TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

TKLM - Things To Know Part 2

Resolving TKLM Memory Issue

TKLM has a known issue with the Java memory heap size. This memory issue results in TKLM becoming slow to respond or stops issuing keys. You can search for an Out Of Memory condition by reviewing the TKLM /tklm/tip/profiles/TIPProfile/logs/server1/SystemOut.log and looking for the following error:

 java.lang.OutOfMemoryError

If this error is present the short term solution is to restart the primary and replica TKLM instances to resolve the out of memory state. The long term solution is to change the TKLM memory settings in two files used to determine the processes memory allotment.

·         Restart the TKLM primary and replica which will flush the memory in use and allow TKLM to issue keys as before. 

Note: This is a short term solution and does not resolve the problem as it will occur again after a period of time.

·         The permanent solution is to reduce the TKLM audit level to low and change the wsadmin process’s Java memory heap size. This needs to be done in two locations and can be done by following the steps provided:

1.     Backup the /tklm filesystem before you edit the files.

sudo dsmc /tklm

2.     Reduce the TKLM audit level to low by using the TKLM web GUI and navigating to

1)     TKLM > Configuration > Audit

2)     Select Low and click OK

Confirm by looking into this file: /tklm/tip/products/tklm/config/TKLMgrConfig.properties

 Verify that Audit.event.type and Audit.event.outcome variables state the following:

Audit.event.types = runtime, authorization, authorization_terminate, resource_management, key_management

 Audit.event.outcome = failure

3.     Edit wsadmin script and server.xml manually.

1)     You will find the two files that require editing, server.xml and wsadmin.sh, in the following directories:

/tklm/tip/profiles/TIPProfile/config/cells/TIPCell/nodes/TIPNode/servers/server1/server.xml

/tklm/tip/bin/wsadmin.sh

4.      modify the wsadmin -Xmx setting.

Example:

1) Locate and modify the below entry

default value:

PERF_JVM_OPTION(S)="-Xms256m -Xmx256m -Xj9 -Xquickstart"

set max value:

PERF_JVM_OPTION(S)="-Xms256m -Xmx1280m -Xj9 -Xquickstart"

Note: The maximum heap size for wsadmin is 1280Mb

2) Save the changes

5.     Now modify the server.xml file by setting the genericJvmArguments variable to “-Xmx2048m”

1)     Locate and modify the below entry

genericJvmArguments="-Xmx2048m"

2)     Save the changes

6.     As root stop TKLM

1)    /tklm/tip/bin/stopServer.sh server1

7.     As root start TKLM

1)    /tklm/tip/bin/startServer.sh server1