IBM P7 Strange Behaviour

We have a P7 frame that has 4 LPARs that are used as TSM storage agents from which snapshots of our SAP DB's are mounted for backup. They have always had great performance until one LPAR had a bad HBA that phoned home and was replaced. After it was replaced performance for backups dramatically decreased from 800MB/s to 150MB/s and overall performance of the server would drastically drop. When the DB requiring backup is over 25TB that is a huge hit, and we could not find the root cause.  At first IBM said it was our Hitachi disk that was the problem. We eliminated that right away, so we then replaced the new HBA, checked our fiber, and then checked the GBIC and nothing seemed to fix the situation. During the first week I asked the IBM service technician if we could possibly have a bad drawer or slot and he emphatically said "No! If you did you would have errors all over the place." So we checked firmware, we moved cards within the frame (again), we double checked the fiber, now we were going into the third week. So I kept asking if something could be wrong with the drawer/slots and I kept getting the same answer. The reason I suggested it was due to previous experience. I have seen hardware go bad without totally going "out". So after exhausting everything other than the replacing the slots, IBM finally replaced the slots. Viola! Backup speeds went back to normal and system degradation during the backup disappeared.  So the slots/drawer was the issue. No errors relating to a slot/drawer hardware issue occurred but something caused the slots to degrade performance.  It took almost a month to resolve the issue, I wouldn't say that IBM support was very thorough and at times tried to push off the problem to other vendors (i.e. Hitachi). I can only suggest in the future you trust your instincts and push the CE's to follow down every avenue. My headache is over, but now the RCA begins.

TSM 7

I recently attended a technical briefing from IBM of various storage related topics which included TSM. While they did have an NDA I can say that some of the items we discussed showed promise. I'll be able to discuss more after IBM Pulse this month, but what I can say is that the new Admin Center is pretty slick. It has some nice features and will finally make up for the folly that was the ISC. IBM stressed that they are listening to users and taking their requests and suggestions to try and develop a tool everyone will find useful. That was surprising news seeing as how the majority of people complained about the ISC and it took 7+ years to finally get a replacement. I will say this in defense of the TSM developers, a lot of the ISC push came from above and they were somewhat forced into that fiasco. TSM 7 DB will scale larger and handle more objects and they are really ramping up the capabilities of the client deployment module. More info to come in the next couple weeks.

One item that did come up was the issue of Export and Backup Set tapes being unencrypted from TSM due to the key issue. What I suggested was that they allow TSM servers to backup each others keys and also utilize them so Exports and Backup-Sets could be encrypted, but still shared between TSM servers. Hope they find some way to add that capability.

We did have a Protectier review and it has a lot of promise. I know I have been a Data Domain fanboy for some time. While I didn't see anything that integrated Protectier DeDupe with TSM directly it did show some nice growth capabilities. I'm excited to see how well it works, but I'm fighting a study that shows tape still is the more cost effective backup solution.

I'll post more once PULSE is complete (mid-June) so stay tuned!

Cleaning tape cycles CLI

#IBM 3494#

mtlib -l -q L | grep 3592

#IBM 3584, A.K.A. IBM TS3500#

/opt/java6/bin/java -jar TS3500CLI.jar -a -viewCleaningCartridges -u -p | awk -F',' '{total=total+$9;}END{print total}'

Insert it into your own morning TSM report script! ;-)

Data Domain vs. Protectier

Where I am currently employed we are looking to replace our 3592 based library with a deduplication solution. Currently the higher ups are leaning towards IBM ProtecTIER without having thoroughly investigated any other solutions. Having previously used Data Domain solutions at my previous employ I was somewhat concerned that the ProtecTIER solution would be a bad fit for our environment. I have had some run ins with people who have used IBM's ProtecTIER solution and when compared to those who have used Data Domain (including myself) you immediately see the difference in how they talk about the two products. So I was hoping to find a good write-up showing in-depth details comparing the two solutions and it took a blogger like me to provide a great comparison.  If you would like a good overview of how Data Domain and ProtecTIER stack up against one another in technology and performance check out the following link. It's very informative and solidifies why I would prefer using Data Domain.

Deduplication: Data Domain Vs. ProtecTIER Performance

One item that was not covered was the NFS capabilities of both. While I used VTL functionality with Data Domain, I was a HUGE NFS proponent. You can save a lot of money over a TSM TDP + LAN-Free solution using NFS with 10Gb Ethernet for your DB backups  (since IBM's licensing costs are still questionable). When I was first exploring ProtecTIER they did not yet have NFS capabilities, so I'd like to see a NFS performance comparison between the two products.

Tivoli Storage Manager Next Administrative Interface / Beta Programme

It'll be called: Operations Center (a bit sounds like admincenter ;-) I hope they didn't only change the design!

Upgrading A TSM 5.5 Library Manager to 6.x

I just helped (sort of) perform an upgrade of two TSM library managers from TSM 5.5 to TSM 6.2. First off I'd like to say that the process involved was really not worth the time it took. Our library manager had a 1GB DB and contained no client data. When the library controller contains no client data you can easily move from 5.x to 6.x without all the headaches of a DB upgrade through the extract and insert process (which took 1 hour to complete once we started the insert). Here are the basic steps to easily upgrade a TSM library manager:

• Backup the TSM library manager DB
• Backup Volhist and Devconfig
• Copy all define statements from devconfig into a TSM macro
• Uninstall TSM 5.5
• Install TSM 6.x
• Follow the steps to create a new TSM 6.x server
• Start the TSM 6.x server
• Run the macro to redefine all the servers, devclasses, libraries, drives, and paths
• Check-in the tapes to the library
• Run audit library from each of the library clients

It might seem like a lot, but once you've got the TSM 6.x server up and running, defining the other items is easy and will take a lot less time than running the upgrade process.

NOTE: This only works if you do not perform ANY backups (Client or NAS) to the library manager.