I was having performance issues with a couple TSM 6.2 servers and could not find anything that pointed to the issue. I'm not one to call support unless I'm totally stumped and cannot find help through the web, but this time I finally relented and made the call. The issue was problems with backups failing repeatedly and when researched we were getting internal server errors along with DB table errors. IBM support asked for some DB2 log files and within 30 or so minutes had identified the problem.
TSM has a server option I have never used or heard of that had somehow been set that adversely affected all backups. Somehow the option DBMEMPERCENT was set in the dsmserv.opt file. This option tells TSM what percentage of the overall server's memory it can allocate for use. The default is AUTO and would have been fine, but somehow DBMEMPERCENT was set to 10 in the dsmserv.opt. Which means out of 16GB of RAM I was only using 1.6GB?!?!? How'd that happen? I didn't set it, none of my coworkers remember setting it, so where did it come from? IBM support stated the default was AUTO so the option was manually set. Since I had never used this option and its 6.x specific, I never would have looked for it. Good thing I called support.
Monday, April 30, 2012
Friday, April 27, 2012
db2adutl Error
I recently had an issue with a client and storage agent upgrade that resulted in problems with the db2adutl utility being unable to return any data. Here's the errors:
I pretty much knew what caused the error, the problem was how to fix it. The cause was due to an upgrade of the TSM client on the DB2 server that (after further investigation) could not support the more current TSM Storage Agent. An OS patch would have to be applied, however, that could not be done without an outage. Our only option was to roll the client back to a supported TSM client / storage agent level. The problem was that while attempting to figure out a better solution than rolling back the client the DB2 database had run a backup. When the client API was rolled back it could not "interpret" the new API's backup causing the db2adutl errors.
Support suggested renaming the node or the file space (file space is better since you don't have to stop and start db2 to reset the password as you would with the new node name). I didn't want to have to do either. The backups taken since the rollback were good, but db2adutl couldn't return the list of backups as long as the objects done with the newer API were still present. Luckily I have been dealing with Oracle admins long enough to have a solid grasp on manually deleting objects on the TSM server. When Oracle DBA's neglect their RMAN duties, I pulled out my trusty delete object command and I was able to remove the backup objects from the period of time that the new API had been used. Once completed db2adutl was able to immediately see it's backups and return a list of what was available.
Retrieving
Error: Begin query image failed with TSM return code 136
Error: Get next image failed with TSM return code 2041
I pretty much knew what caused the error, the problem was how to fix it. The cause was due to an upgrade of the TSM client on the DB2 server that (after further investigation) could not support the more current TSM Storage Agent. An OS patch would have to be applied, however, that could not be done without an outage. Our only option was to roll the client back to a supported TSM client / storage agent level. The problem was that while attempting to figure out a better solution than rolling back the client the DB2 database had run a backup. When the client API was rolled back it could not "interpret" the new API's backup causing the db2adutl errors.
Support suggested renaming the node or the file space (file space is better since you don't have to stop and start db2 to reset the password as you would with the new node name). I didn't want to have to do either. The backups taken since the rollback were good, but db2adutl couldn't return the list of backups as long as the objects done with the newer API were still present. Luckily I have been dealing with Oracle admins long enough to have a solid grasp on manually deleting objects on the TSM server. When Oracle DBA's neglect their RMAN duties, I pulled out my trusty delete object command and I was able to remove the backup objects from the period of time that the new API had been used. Once completed db2adutl was able to immediately see it's backups and return a list of what was available.
Thursday, April 26, 2012
TSM Power Admin
I was just made aware of TSM Power Admin by a fellow adsm.org contributor and must say I like some of the features available. I hope to be able to test it soon, but just the ability to run commands against all the servers from the command line (without setting up a server group) is a nice touch. If I do test Power Admin I'll post a review like I did for TSMManager years ago. (Wow it's been that long?!)
Monday, April 23, 2012
TSM 6.1 & 6.2 DB2 Issue
I had a TSM server crash mutliple times over the course a week and after working with Tivoli support and sending them the core files, it was determined that the following error was the cause. Interesting, in that I never thought about the connections from TSM to the DB2 DB. So to summarize, the current connection from TSM to DB2 is not a TCP based but IPC and AIX has a limitation of 1024 IPC connections to DB2 otherwise the application in question (TSM in this case) can crash. The following link has directions on how to convert TSM to DB2 connections to TCP to eliminate this issue.
Subscribe to:
Posts (Atom)