GPFS Revisited

Well I am still having issues with GPFS. It turned out the mmbackup wont work with the filesystem size either and a chat with IBM support was not encouraging. Here is what one of our System Admins found out:

The problem was eventually resolved by IBM GPFS developers. It turns out, they never thought their filesystem would be used in this configuration (i.e. 100,000,000 + inodes on a 200GB filesystem). During the time the filesystem was down, we tried multiple times to copy the data off to a different disk. Due to the sheer number of files on the filesystem, every attempt failed. For instance, I found the following commands would have taken weeks to complete:

# cd $src;find . -depth -print | cpio -pamd $dest
# cd $src; tar cf - . | (cd $dest; tar xf -)


Even with the snapshot, I dont think TSM is going to be able to solve this one. This will probably need to be done at the EMC level, where a bit level copy can be made.

So GPFS is not all it was thought to be. So pass it along and make sure you avoid GPFS for application that will produce large numbers of files.

Looking For Contributors

If you work in a large TSM environment and have experienced issues or tasks that you think passing along would help others please let me know. I am looking for contributors to keep this blog current and make it more open. Send me an e-mail with contribution ideas (you'll need more than 1) and what type of environment you work in.

CDP For Unix?

Has anyone heard of when (if ever) Continuous Data Protection for Files will be available on the Unix platform? I could really use this feature with my GPFS system. Since the application creates hundreds of meta data files daily and is proprietary (hence no TDP support) I am getting killed by the backup timeframe since the each volume has in excess of 4 million files already and incrementals take close to 48hrs. to finish. Anyone heard anything at symposiums or seminars?

A Sit Down With ISC Admin Developers

The wonderful people with GUI/Web development at IBM were kind enough to have a conference call with me to discuss issues with the TSM admin interface. They have read my posts and thought I might have a good idea as to why so many people dislike the interface. The truth is, a central console to manage your TSM servers is a better paradigm than a web server per TSM instance process. What was agreed across the board is the problem with WebSphere. They made it sound like they are trying to move to a TSM Express model, which TSM Express uses a WebSphere Lite, and is lot more responsive. The current WebSphere is just too big and slow for the needs of TSM admins.

The other problem with the ISC that is very apparent is the inability to view selected content in the viewable area. See when you select certain commands the results of selecting that command many times will not appear on the visible screen, but below requiring scrolling. Let's face it not everyone managing TSM is an expert so they might miss it and think something is wrong with the interface. The good (and some consider bad) of the old interface was that a selection or command was instantly seen in the results frame. It wasn't perfect but it was more functional than the current model. The other issue, I believe was agreed upon, was the resolution used for the screens. The resolution needs to either be fluid like this blog (meaning resizing itself for various screen resolutions without going beyond the borders of the browser) or set to a specific resolution. The recommended resolution for the web to meet 99% of the users out there is 800x600. I know many of you might think that 800x600 is low, but it can be shown in almost every user’s browser. The ISC however seems built for a resolution of 1280x1024. This doesn't work because many people can't go beyond 1024x768 (like me with my laptop). With items off the screen it makes it harder to use the interface since things like the collapsible section button (minimize) are off the screen. I also suggested that the Libraries for All Servers section be a separate tab that changes color if one of the libraries is having issues. It gets in the way having it there at all times. Sure you can minimize it, but if you select another function it resets itself. I would also like to see the TSM servers easily definable. One of the issues I had is that every admin has to define their own servers to the ISC. So the same server ends up being defined multiple times when a single definition would do. If you can’t login who cares if you can see it? I also think they need to move the servers to an expandable list along the side frame so that you can select the server you want to work with and the context frame switches to that server content.

Another example of how they made the selection more complex is working with client scheduling. Tivoli took away the scheduling link and changed it so that to access client schedules you first select domains, then select the server you want to work with, choose the domain where you think the schedule is, then below the domain list you will see multiple options you can select and one of those will be Client Node Schedules, select that then you can choose the schedules you want to work with. Now after selecting the schedule you want to work with that took 5 selections to get to the schedule you need. And they are under domain so you don’t see a full list of schedules like before. So if you choose the wrong domain…start the process over. I also think they need to add an color coded error window like TSM Manager has. It is so helpful its amazing!

I would like to thank the IBM people who listened to my suggestions. I find it refreshing to see IBM working hard to fix issues to make the users experience more productive. I will admit I really want a central management console (that's why I love TSM Manager) and a streamlined, fast, and simple ISC will definitely win me over. Especially now since my new work place doesn't have TSM Manager! :(

Note: Please leave your own feedback in the comments section so that the TSM ISC Admin Development team can hear what you have to say. I know they would like feedback from more than me.

Unsupported Debian Client Available

Thanks to Harry Redl for the Debian client he has put together. It took him some work but now you can have your Ubuntu and back it up too. According to Harry at adsm.org the package works on Debian Woody, Debian Sarge and Ubuntu 6.06. As Harry stated the package is not supported by IBM and USE AT YOUR OWN RISK! All disclaimers and such apply with this package. No claim is made to its reliability blah blah blah! You hopefully get the idea. Anyway, Harry uses it in his 99% debian shop without a problem...which brings up the following question. Harry, what OS does your TSM server run on?

TSMManager 4.1

Back in June of 2005 I posted a review of TSMManager 4.0. When I was with IBM I had convinced management to purchase the product and we got a sweetheart deal to boot. I was initially drawn to the nice admin console window. As you'll see in the picture below when you open an admin console you are presented with a tabbed window that has a number of different views, one per tab. The first tab is the ALL MESSAGES tab that is basically an active view of the TSM servers activity log. The second tab is the most useful and important tab within TSMManager, in fact it is what makes TSMManager my favorite tool to use. This tab is the ERROR MESSAGES tab and it basically filters the TSM activity log to show only WARNING, ERROR, and SEVERE ERROR messages. Why is this the most important piece to TSMManager? Because, with this one tab 95+% of the time I can deduce a problem within in a minute or two. For example in the provided screen shot you can see we had a number of drive errors, but look closer and you see it's one tape causing the problem. This took me a minute to figure out and I easily resolved it by removing the tape. The next tab I rarely use but it can come in handy. It is the SPECIAL MESSAGES tab and it covers ALL the servers not just the one you have selected to manage at the moment. It really lists the processes and there start and end times. You might use the tab to lookup processes and how they ran, but I haven't found a real use for this tab yet. The next tab is the actual TSM Admin command line. You can issue commands or use the quick commands provided on the side. TSMManager will store the last 20+ commands in a buffer for recall. Overall this window and its tabs are worth the cost of the product in its productivity enhancement.

This is the error tab. Click on it and you'll see how it consolidates all errors into a simple interface. Great for those times you're having server issues.