Error Upgrading to 8.1.8.0

I performed an upgrade today on a server going from 8.1.0.0 to 8.1.8.0 and after the upgrade we encountered the following errors in the actlog as the system was starting.

08/14/2019 11:59:59      ANR9999D_2835836899 InstGetDB2stmtPass1(instr.c:550)                          Thread<404>: Error in fetching DB2 statement stats,                          rc=235908/14/2019 11:59:59      ANR9999D Thread<404> issued message 9999 from:08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001000365b4 StdPutText08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010003784c OutDiagToCons08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010000cf08 outDiagfExt08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010019d0d0                          IPRA.$InstGetDB2stmtPass108/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001001a15e0 instrumentStart08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001012d7b98                          AdmInstrumentBegin08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008da68c AdmCommandLocal08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008d7af0 admCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013fbe68 RunCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013f7a48                          ServerMonTwentyMinThread08/14/2019 11:59:59      ANR9999D Thread<404>  0x0000000100010648 StartThread

Although the server started, I wanted to make sure this would not cause issues with the instance. I searched the error ANR9999D_2835836899 and found the following IBM APAR notice. It appears that after the upgrade you may have to manually upgrade the DB2 version using the DB2 update tool. While it appears to be a minor issue, I would recommend you check your instances after upgrade to verify whether you need to run the manual steps to resolve the errors. I've included them here for your convenience.

NOTE: I've added a couple commands (in bold) that we have had to add to the process to make it work.

Local fix

Even though the db2-level might be updated to the latest version 11.1.4.4, run the below set of commands
Stop TSM server and issue the following commands as the instance owner.

• db2 start

• db2 force application all

• db2 terminate

• ipclean

login as root user and run

• /opt/tivoli/tsm/db2/instance/db2iupdt -d <instance name>

login as the instance owner ID and issue the command:

• db2updv111 -d TSMDB1

This issue is supposedly fixed in version 8.1.9.0 and higher.

TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

TSM 6.2.3 and Lower DB Reorg Issue

One of our TSM servers started to experience large numbers of "ANR0530W - internal server error detected messages."  With further investigation we identified that these were related to ANR0162W which are DB deadlock or timeout problems. These errors were causing our DB TDP backups to fail and I eventually called support. I provided our db2diag.log file and a dump of our actlog for the last 24 hrs. and they found the issue to be the DB2 reorg process was locking records and tables and creating the deadlock situation. The problem was compounded in that with TSM versions 6.2.3 or lower the DB2 reorg process cannot be schedule so it can kick off during backups or processes that can create these deadlocks. So to resolve the issue I was told I needed to upgrade our TSM server to 6.2.4 or higher. With TSM 6.2.4 and higher you can schedule the reorg process using the REORGBEGINTIME and REORGDURATION parameters to schedule the reorg within a window. You can see the details of the APAR here.

DBMEMPERCENT...Where'd That Come From?

I was having performance issues with a couple TSM 6.2 servers and could not find anything that pointed to the issue. I'm not one to call support unless I'm totally stumped and cannot find help through the web, but this time I finally relented and made the call. The issue was problems with backups failing repeatedly and when researched we were getting internal server errors along with DB table errors. IBM support asked for some DB2 log files and within 30 or so minutes had identified the problem.

TSM has a server option I have never used or heard of that had somehow been set that adversely affected all backups. Somehow the option DBMEMPERCENT was set in the dsmserv.opt file. This option tells TSM what percentage of the overall server's memory it can allocate for use. The default is AUTO and would have been fine, but somehow DBMEMPERCENT was set to 10 in the dsmserv.opt. Which means out of 16GB of RAM I was only using 1.6GB?!?!? How'd that happen? I didn't set it, none of my coworkers remember setting it, so where did it come from? IBM support stated the default was AUTO so the option was manually set. Since I had never used this option and its 6.x specific, I never would have looked for it. Good thing I called support.

db2adutl Error

I recently had an issue with a client and storage agent upgrade that resulted in problems with the db2adutl utility being unable to return any data. Here's the errors:

Retrieving

Error: Begin query image failed with TSM return code 136

Error: Get next image failed with TSM return code 2041

I pretty much knew what caused the error, the problem was how to fix it. The cause was due to an upgrade of the TSM client on the DB2 server that (after further investigation) could not support the more current TSM Storage Agent. An OS patch would have to be applied, however, that could not be done without an outage. Our only option was to roll the client back to a supported TSM client / storage agent level. The problem was that while attempting to figure out a better solution than rolling back the client the DB2 database had run a backup.  When the client API was rolled back it could not "interpret" the new API's backup causing the db2adutl errors.

Support suggested renaming the node or the file space (file space is better since you don't have to stop and start db2 to reset the password as you would with the new node name). I didn't want to have to do either. The backups taken since the rollback were good, but db2adutl couldn't return the list of backups as long as the objects done with the newer API were still present. Luckily I have been dealing with Oracle admins long enough to have a solid grasp on manually deleting objects on the TSM server. When Oracle DBA's neglect their RMAN duties, I pulled out my trusty delete object command and I was able to remove the backup objects from the period of time that the new API had been used. Once completed db2adutl was able to immediately see it's backups and return a list of what was available.

TSM Server 5.5, 6.1, & 6.2 Bug

I figured I best share with you all a bug I have experienced with TSM Server version 6.2.2 that has resulted in the TSM instance core dumping. The problem stems from TSM being unable to mount a scratch tape with, in our case, a TSM storage agent. When the mount fails it seems TSM lists an internal server error then core dumps a few seconds later.  The fix is in 6.2.3 and higher. It also affects 5.5 and 6.1 versions, check this technote for further details.

Warning: This update has a bug also concerning Data Domain that causes errors if the defined VTL has more than 4400 slots. So when we upgraded the server to fix the LAN-Free bug we inadvertently encountered this bug.

Update: Here is the APAR listing for the Data Domain VTL slot bug in TSM 6.2.x

Memory Efficient Backup

I have come across a few servers that have memory issues when backing up large drives / filesystems and using MEMORYEFFICIENTBACKUP was not resolving the backup failures. The error code I would see was

ANS1030E The operating system refused a TSM request for memory allocation.

The problem was one drive that has millions of small files all under one directory. It's one of those poorly devised custom programs, and I've seen it a hundred times.  I could journal the drive, but we seemed to get better results with an alternative method.  A coworker reminded me of the INCLUDE.FS option using DISKCACHE and specifying the cache location.

INCLUDE.FS G: MEMORYEFFICIENTBACKUP=DISKCACHEMETHOD DISKCACHELOCATION=L:\TSM_cache

With the include.fs TSM will only use the memoryefficientbackup diskcache method for that particular drive, allowing the rest of the backup to perform normally. So if you have a drive or filesystem with millions of files this is an alternative to journaling.

TSM 6.2 GUID Error

So when I and a co-worker were finishing up a TSM 6.2 install we ran into a problem running the tsmapipw command to set the $$_TSMDBMGR_$$ password. After investigating the error and search the web we realized the GUID was the problem and not our IP as was suggested in the following APAR.  The errors we received were:

ANR2987W Session ended because of machine GUID or local host IP
 address mismatch.
ANR0430W Session XXXX for node $$_TSMDBMGR_$$ (AIX) refused
 - node name is locked.

We tried various things and nothing seemed to correct the GUID issue. The TSM server started without any GUID related error (in fact it stated the GUID initialized successfully). So upon further searching we found this APAR that gives the solution.  What they state is that by adding the following line to the dsmserv.opt and restarting the server fixes the problem.

DBMTRUSTEDGUIDIGNORE YES

Once we added the line and restarted the TSM server the tsmapipw command ran successfully.

Weird Query Results From 6.1.4

So we have a couple servers we upgraded from 5.3 to 6.1.4.2 and since the upgrade the following select statement does not work. I posted it on ADSM.org and others report their 6.x servers run it without issue.  Here is the statement and my results:

select server_name, node_name, PLATFORM_NAME || ' ' || cast(CLIENT_OS_LEVEL as char(20)) as OS_Version, cast(client_version as char) || '.' || cast(client_release as char) || '.' || cast(client_level as char) || '.' || cast(client_sublevel as char(2)) as TSM_Version from nodes, status


here is the return code I get:


ANR2034E SELECT: No match found using this criteria.
ANS8001I Return code 11.

I run this through a while loop and append the data to an comma-delimited output file so I can pull it into a spreadsheet and organize clients by version. I can then more easily prioritize my client upgrades.  I really need this to work so I can sort and identify the clients location by server_name. Any ideas?

We found this bug

We found this bug in Hungary. Click for details!

Who Did What?

This weekend I had an issue with a backup failing on a client using LAN-Free. When checking the TSM actlog the error I kept seeing was:

ANR9999D (Session: 265500, Origin: OD0BG-UAX001-STA) mmsshr.c(3874): ThreadId<13> Unable to obtain model type for '/dev/rmt23', rc = 46(SESSION: 265500)

So I had a good idea that the client and TSM Storage Agent were having issues communicating. I knew this because I had seen this error before when the TSM client and TSM Storage Agent were not at the same release levels. Checking the TSM client I found it at 5.4.0.0 and the Storage Agent at 5.3.4.0. Turns out an admin upgraded the client without knowledge of the Storage Agent, causing the backup failures. This is an example of what can happen when someone tries elbow their way into another groups area without the knowledge to do it right.