TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

TSM Client Scheduler Issue

I just recently had an issue with a handful of TSM clients that would not run their backups. The clients all backup to a TSM 5.5.2 server and were all running Windows 2008. The clients use TSM version 6.2.3. The five clients had all been missing their backups for days and what makes the situation more interesting is that there are other Windows 2008 servers with this version of TSM installed and they are all running their schedules without issue.

When reviewing the TSM Schedule log the scheduler listed that it had received the schedule info and was waiting for the TSM server to initiate the schedule. The TSM server never made an attempt to contact the clients in question and never showed any errors other than ANR2578W stating the client missed its schedule. There were no errors in the error log and not much to go by from the TSM server activity log. Even though the TSM client backs up over the public network I switched to polling mode to see if client based initiation of the backup would work. It didn't! The TSM client scheduler would receive the schedule upon polling the TSM server but would never execute it. So now what? I added the TCPCLIENTADDRESS and TCPCLIENTPORT and switched back to SCHEDMODE PROMPTED, still the scheduler would not run backups.

Now I was getting frustrated. I removed the scheduler service and redefined it using dsmcutil and voila, the schedule ran...ONCE! After the initial schedule ran the previous problem returned. Schedules were not running and the TSM server would not show any errors saying it could not contact the client. It just would not run the schedule. Well that left me no choice but to call support. IBM support's response was to make sure the TCPCLIENTADDRESS and TCPCLIENTPORT were defined in the dsm.opt and also to define the client HLADDRESS and LLADDRESS on the TSM server? Define the HL and LL addess? TSM gets that when the client connects doesn't it? Yes and No! It appears that without the optional setting the TSM server can have issues contacting some clients. Why? No idea, but adding the HL and LL address did the trick and the backups have been running without issue since.

How many of you define the HL and LLADDRESS when registering nodes? I've never suspected it was needed until now.