TSM Topics Feed

Wednesday, October 25, 2017

Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

I just wanted to let everyone know that there is a compatibility issue with the Spectrum Protect 8.1.2 client with the 8.1.0 server. If you upgrade the client to SP 8.1.2 and your server is at 8.1.1 or 8.1.0 your server will be unable to start. The error you will get is

Spectrum Protect ANR0162W Supplemental database diagnostic information:  -1:08001:-30082 ([IBM][CLI Driver] SQL30082N  Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID").

The issue has to do with the GSKit8.gskcrypt64.ppc.rte and GSKit8.gskssl64.ppc.rte in the 8.1.2.0 client. When we rolled back to the 8.1.0 client the SP server started without an issue.

Monday, August 07, 2017

TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

Thursday, June 01, 2017

TSM SQL TDP Better Backup Script

On a call today I had an admin complain that the TSM SQL TDP was issuing failed backup errors when the script encountered a database that was offline. So I asked if he had searched Google for an alternate script and he had not. I did a quick query and found this nice batch file that appears to do exactly what is needed. The reader on the site commented that the script did not work on his SQL cluster. Does anyone have a similar script that would work on a cluster? Or could you modify this script to work in a cluster environment? The link to the script is provided below.

http://www.applepie.se/tivoli-storage-manager-and-sql-backup


Tuesday, May 09, 2017

Spectrum Protect 8.1 New DB Functions Supported - UPDATED!

UPDATE (5/9/2017):

I posted on StackOverflow (a great developer/scripting/DB) listserv asking how I could get similar output with the functions available in versions prior to DB2 11 and a user was able to help me out with a script that could return the same data as the one below although it's a little more complicated.  So as you know the SUBSTR function works like this:


Select SUBSTR(FILESPACE_NAME, <START POSITION>,<LENGTH>)

The issue with the earlier DB2 versions is I could only get the LOCATION function to play nice with my script. I was previously only able to figure out a select that got close:
Select -                                                                        
 SUBSTR(FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, - 
 LOCATE('\', FILESPACE_NAME, -                                                  
 LOCATE('\', FILESPACE_NAME)+1)+1)-1)+1) -                                      
 as FSNAME -                                                                    
from filespaces -                                                               
 where node_name='TEST-AP-DAG'
Which output this:

FSNAME                                                           
-----------------------------------------------------------------
TEST-MB-08\6bc391ef-a370-49a1-8f05-b1bed9e5ad55                  
Mailbox Database 1143943276\55ac1670-efc9-4301-ac5c-beb5cd2d77cb 
TEST-MB-05\11dedbd4-1757-45c8-8991-f0f7134ef210                  

I could not figure out a way to get the length. So a kind reader at StackOverflow helped me understand how to get the length by subtracting the 4th '\' location from the 3rd '\' -1. This gives the correct length and allows for the length size to vary rather than be fixed.
Select -
 SUBSTR(FILESPACE_NAME, -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME)+1)+1)+1, -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME)+1)+1)+1)- -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME)+1)+1)-1) -
 as FSNAME -
from filespaces -
 where node_name='TEST-AP-DAG'
Here it is color coded:
(Select SUBSTR(FILESPACE_NAME, <START POSITION>,<LENGTH>)
Select -
 SUBSTR(FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, LOCATE('\', 
 FILESPACE_NAME, LOCATE('\', FILESPACE_NAME)+1)+1)+1, -
 LOCATE('\',FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, LOCATE('\', 
 FILESPACE_NAME,  LOCATE('\', FILESPACE_NAME)+1)+1)+1)-  -
 LOCATE('\',FILESPACE_NAME, LOCATE('\', FILESPACE_NAME,  -
 LOCATE('\', FILESPACE_NAME)+1)+1)-1) -
 as FSNAME -
from FILESPACES -
 where - 
  NODE_NAME='TEST-AP-DAG'
So the final script looks like this
Select -
 VARCHAR(NODE_NAME,12) as NODE, -
 VARCHAR(SUBSTR(FILESPACE_NAME, -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME)+1)+1)+1, -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME)+1)+1)+1)- -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME)+1)+1)-1),30) -
  as Exch_DB,
 date(backup_end) AS LAST_GOOD_BACKUP -
from filespaces -
 where - 
  node_name='TEST-AP-DAG' -
 order by Exch_DB, backup_end asc
The results are exactly like the script that only works with TSM 8.1.


NODE              EXCH_DB                          LAST_GOOD_BACKUP
-------------     ------------------------------- -----------------
TEST-AP-DAG       \TEST-MB-01                            2017-05-08
TEST-AP-DAG       \TEST-MB-02                            2017-05-08
TEST-AP-DAG       \TEST-MB-03                            2017-05-08
TEST-AP-DAG       \TEST-MB-04                            2017-05-08
TEST-AP-DAG       \TEST-MB-05                            2017-05-08
TEST-AP-DAG       \TEST-MB-TEMP                          2017-05-08
TEST-AP-DAG       \MISC-MB-TEMP                          2017-05-08
TEST-AP-DAG       \Mailbox Database 1128394465           2017-05-08
TEST-AP-DAG       \Mailbox Database 1397248650           2017-05-08



(4/20/17):
I upgraded some of my 6.3.4 servers to TSM 8.1 recently and have found the newer DB2 version (v11.1) offers some added functions for SQL queries. I have been trying to build a report to track an Exchange DAG servers backups. The problem is that the easiest way to track the backups of DB's is to view the last completed backup for each of nodes file spaces. That's the easy part. The hard part is that the file space name is huge.

TEST-AP-DAG\Microsoft Exchange Writer\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}\TEST-MB-05\0e41e645-9acf-4b80-bc85-d606e04fe4d8

TEST-AP-DAG\Microsoft Exchange Writer\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}\Mailbox Database 1223544393\2bc06db2-1966-4fd7-9545-f667102b0b7d

So how to extract the DB name, in this case TEST-MB-05 and Mailbox Database 1397248650 from the name when the length of the DB name changes? After some investigation and trial and error I found that the LOCATE_IN_STRING function works in v8.1. Whether it works in 7.x (DB2 v10.5) I can't say since I don't have a 7.x server to run it against. If any of you out there try this and it works let me know. So here's the script. I think you can read it and see what I am doing. If not let me know in the comments and I'll explain...

tsm: TSMSERV>q script DAG-EXCH-RPT f=raw

select -
 VARCHAR(NODE_NAME,12) as NODE, -
 varchar(substring(filespace_name, LOCATE_IN_STRING(filespace_name, '\', 1, 3), -
 LOCATE_IN_STRING(filespace_name, '\', 1, 4) -  LOCATE_IN_STRING(filespace_name, '\', 1, 3)),30) as Exch_DB, -
 date(backup_end) AS LAST_GOOD_BACKUP -
from FILESPACES -
 where -
  node_name='TEST-AP-DAG' -
 order by Exch_DB, backup_end asc


Here is sample output:

NODE              EXCH_DB                          LAST_GOOD_BACKUP
-------------     ------------------------------- -----------------
TEST-AP-DAG       \TEST-MB-01                            2017-04-18
TEST-AP-DAG       \TEST-MB-02                            2017-04-18
TEST-AP-DAG       \TEST-MB-03                            2017-04-18
TEST-AP-DAG       \TEST-MB-04                            2017-04-18
TEST-AP-DAG       \TEST-MB-05                            2017-04-18
TEST-AP-DAG       \TEST-MB-TEMP                          2017-04-18
TEST-AP-DAG       \MISC-MB-TEMP                          2017-04-18
TEST-AP-DAG       \Mailbox Database 1128394465           2017-04-18
TEST-AP-DAG       \Mailbox Database 1397248650           2017-04-18

If another function would work better feel free to correct the script and leave it for everyone in the comments. Now to go and find more functions I can use to manipulate my data!

Friday, May 05, 2017

TSM Menu - Bash Shell Scripting

For years I have maintained a shell script on our jump server that was a menu for accessing all the TSM servers in our environment. It was a nice interface and great for noobies. It allowed for you to set your password and ID variables before the menu loaded so anyone could run the menu under their ID, then presented them with a nice menu to pick which TSM server they wished to access. It was nice and easy. The only problem was when servers were added or decommissioned. When that occurred I would then have to go out and revise the menu. This was a pain and soon to be more painful as we plan to add a lot more TSM servers in the near future. I remember seeing someone post that they had created a menu system based off of the dsm.sys file so I decided to write my own.

Before:

 ################################################################## 
 #                                                                # 
 #                        Infocrossing                            # 
 #                    UNIX ADMINISTRATION                         # 
 #                         TSM  MENU                              # 
 #                                                                # 
 ################################################################## 
                            Omaha                                   
     0)   ouax000                         1)  ouax001               
     2)   ouax002                         3)  ouax003               
     4)   OTSM01                          5)  OTSM02                
 ------------------------------------------------------------------ 
                           Phoenix                                  
     6)   puax001                         7)  puax002               
     8)   puax008                         9)  PTSM01                
     10)  PTSM02                         11)  PTSM20                
 ------------------------------------------------------------------ 
                           Singapore                                
     20)  SIN01ZZ                        99)  STSM01                
 ------------------------------------------------------------------ 
                            Germany                                 
     21)  DTSM01                         22)  DTSM02                
     23)  DTSM03                         24)  DTSM20                
 ------------------------------------------------------------------ 
                            Kings Mtn                               
     25)  CTSM01                         26)  CTSM20                
     27)  CTSM30                         28)  CTSM40                
 ------------------------------------------------------------------ 
                                                                    
                     X/x)  Cross Server                             
                                                                    
                     S/s)  Cmd Prompt                               
                                                                    
                   DD/dd)  Data Domain Admin                        
                                                                    
                     Q/q)  Return to Main Menu                      
                                                                    
                                                                    
     Selection:                                                    

 After (Notice servers listed here that were not in above menu):

 ################################################################
 #                                                              #
 #                         Infocrossing                         #
 #                       Recovery Services                      #
 #                           TSM  MENU                          #
 #                                                              #
 ################################################################
                                                                 
 1) C1IF            12) LABTSM            23) PTSM01             
 2) C1KDC2          13) LUAX001           24) PTSM02             
 3) CTSM01          14) LUAX002           25) PTSM20             
 4) CTSM20          15) LTSM01            26) PTSM25-Arch        
 5) CTSM30          16) NTSM01            27) SIN01ZZ            
 6) CTSM40          17) OUAX003           28) STSM01             
 7) CTSM46-Arch     18) OUAX000           29) Cross Server       
 8) DTSM01          19) OUAX001           30) Refresh Menu       
 9) DTSM02          20) OTSM01            31) Exit               
10) DTSM03          21) OTSM02                                   
11) DTSM20          22) PESTSM01                                   
                                                                 
 Please enter your choice:     

Ok, so the Before is more organized by region but as new servers come in I wanted the menu to update dynamically. So this new menu is built around the BASH shell's select menu function. The script pulls all the TSM server names from the dsm.sys and populates an array that the select statement reads and generates this menu. Make sure the variables the script declares work for you and your dsm.sys. My dsm.sys looks like this if I grep servername:

SERVERNAME         DTSM01          * Germany      LIB-MAN
SERVERNAME         DTSM02          * Germany             
SERVERNAME         DTSM03          * Germany             
SERVERNAME         DTSM20          * Germany      LIB-MAN
SERVERNAME         CTSM01          * Kings_Mnt    LIB-MAN
SERVERNAME         C1KDC2          * Kings_Mnt           
SERVERNAME         C1IF            * Kings_Mnt           
SERVERNAME         CTSM20          * Kings_Mnt           
SERVERNAME         CTSM30          * Kings_Mnt           
SERVERNAME         CTSM40          * Kings_Mnt           
SERVERNAME         CTSM46-Arch     * Kings_Mnt           
SERVERNAME         LUAX001         * Leonia       LIB-MAN
SERVERNAME         LUAX002         * Leonia              
SERVERNAME         LTSM01          * Leonia              
SERVERNAME         NTSM01          * Norcross     LIB-MAN
SERVERNAME         OUAX000         * Omaha        LIB-MAN
SERVERNAME         OUAX001         * Omaha               
SERVERNAME         OUAX003         * Omaha               
SErvername         OTSM01          * Omaha        LIB-MAN
SErvername         OTSM02          * Omaha               
SERVERNAME         PESTSM01        * PEST         LIB-MAN
SERVERNAME         PTSM01          * Tempe        LIB-MAN
SERVERNAME         PTSM02          * Tempe               
SERVERNAME         PTSM20          * Tempe               
SERVERNAME         PTSM25-Arch     * Tempe               
SERVERNAME         STSM01          * Singapore           
SERVERNAME         SIN01ZZ         * Singapore           

SERVERNAME         LABTSM                                

With the dsm.sys file formated this way I can use grep and awk to get specific data. This allows the script to build my menus according to my needs. Especially with the Cross Server sub menu that allows me to run a command across multiple servers at once. In the sub-menu it groups by distinct location which I placed in the servername line. Leave me a comment if you have any comments, questions, or suggestions.

See the script below:

#!/bin/bash
#set -x

### Set Color Variables ###

NORMAL=`echo "\033[0m"`
WHITE=`echo "\033[37m"` #Blue
YELLOW=`echo "\033[33m"` #yellow
GREEN=`echo "\033[32m"`
RED=`echo "\033[31m"`
PURPLE=`echo "\033[35m"`

##### Declare Variables & Arrays ############

prompt="
 Please enter your choice: "

TSM_SYS="/opt/tivoli/tsm/client/ba/bin/dsm.sys"
all_servers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $2}' | sort`)
xservers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $4}' | sort -u`)
Germany=(`cat $TSM_SYS | grep -i servername | grep -i germany |awk -v OFS='\t' '{print $2}'`)
Omaha=(`cat $TSM_SYS | grep -i servername | grep -i omaha |awk -v OFS='\t' '{print $2}'`)
Norcross=(`cat $TSM_SYS | grep -i servername | grep -i norcross |awk -v OFS='\t' '{print $2}'`)
Tempe=(`cat $TSM_SYS | grep -i servername | grep -i tempe |awk -v OFS='\t' '{print $2}'`)
PEST=(`cat $TSM_SYS | grep -i servername | grep -i pest |awk -v OFS='\t' '{print $2}'`)
Leonia=(`cat $TSM_SYS | grep -i servername | grep -i leonia |awk -v OFS='\t' '{print $2}'`)
Kings_Mnt=(`cat $TSM_SYS | grep -i servername | grep -i kings_mnt |awk -v OFS='\t' '{print $2}'| sort`)
Singapore=(`cat $TSM_SYS | grep -i servername | grep -i singapore |awk -v OFS='\t' '{print $2}'`)
lib_man=(`cat $TSM_SYS | grep -i servername | grep -i lib-man | awk -v OFS='\t' '{print $2}'`)

export COLUMNS=70
########################

######################
function clearlogin
######################
{
echo "Please wait while I clear all login information."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
unset TSMID
unset TSMPA
unset DDID
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "Logins have been cleared! "|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 1
}
######################
#  end clearlogin    #
######################

###############
#END MAIN-MENU#
###############

#########################
function tsmadmin
#########################
{
clear
if [ -z "${TSMID}" ]; then
echo ""
echo -e "$YELLOW Enter your TSM Admin ID: $GREEN \c"
read TSMID
echo ""
echo -e "$YELLOW Enter your TSM Admin Password: \c "
#stty -echo
read -s TSMPA
#stty $stty_orig
fi
clear
  tsm-menu
exit
}
####################
#  end tsmadmin    #
####################

#####################
function tsm-menu
#####################
{
all_done=0
export COLUMNS=70
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE #                       $YELLOW  Infocrossing                        $WHITE #"
echo -e "$WHITE #                       Recovery Services                      #"
echo -e "$WHITE #                          $RED TSM  MENU                         $WHITE #"
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#all_servers[@]} + 4))"`
PS3="$prompt "
 select host in "${all_servers[@]}" "Cross Server" "Reset Login" "Refresh Menu" "Exit"; do
   if [ "$host" = "Exit" ]; then
      all_done=1; clear; exit
   elif [ "$host" = "Refresh Menu" ]; then
      all_done=1; tsm-menu
   elif [ "$host" = "Reset Login" ]; then
      all_done=1; clearlogin; tsmadmin
   elif [ "$host" = "Cross Server" ]; then
      all_done=1; xmenu
   elif [ "$REPLY" -gt "$length" ]; then
      echo ""
      echo "You chose option $REPLY which is invalid. Try again! "
      echo ""
   else
       clear
       echo -e "$GREEN "
       dsmadmc -id=$TSMID -pa=$TSMPA -servern="$host"
       all_done=1; tsm-menu
   fi
 done
done
}
##########################
# end function tsm-menu  #
##########################

#####################
function xmenu
#####################
{
all_done=0
export COLUMNS=50
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE #                       $YELLOW  Infocrossing                        $WHITE #"
echo -e "$WHITE #                       Recovery Services                      #"
echo -e "$WHITE #            $RED     TSM Cross Server Command MENU               $WHITE # "
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#xservers[@]} + 4))"`
PS3="$prompt "
 select host in "${xservers[@]}" "All Servers" "Library Managers" "Refresh Menu" "Exit"; do
   if [ "$host" = "Exit" ]; then
      all_done=1; tsm-menu
   elif [ "$host" = "Refresh Menu" ]; then
      all_done=1; xmenu
   elif [ "$host" = "Library Managers" ]; then
       clear
       echo -e "$GREEN "
       xserver $TSMID $TSMPA lib_man
       all_done=1; xmenu
   elif [ "$host" = "All Servers" ]; then
       clear
       echo -e "$GREEN "
       xserver $TSMID $TSMPA all_servers
       all_done=1; xmenu
   elif [ "$REPLY" -gt "$length" ]; then
       echo ""
       echo "You chose option $REPLY which is invalid. Try again! "
       echo ""
   else
       clear
       echo -e "$GREEN "
       xserver $TSMID $TSMPA "$host"
       all_done=1; xmenu
   fi
 done
done
}
##########################
# end function xmenu     #
##########################


###################
function xserver
###################
{
  DC=$3
  echo -e " "
  echo -e "     $YELLOW Enter the command to process: $NORMAL \c"
  read CMD
  echo ""
    echo -e "     $YELLOW Do you want to save the data to a commadelimited csv file? (Yes or No [N])  $NORMAL \c"
    read ANSWER
    case $ANSWER
     in
      y|Y|YES|Yes|yes)
       echo ""
       echo -e "     $YELLOW Enter the file name to save the data: $NORMAL \c"
       read FILE
       echo ""
       touch $FILE
       cat /dev/null > $FILE
        eval varAlias=\${$DC[@]}
        for Server in ${varAlias[@]}
        do
         echo ""
          echo -e "Processing TSM server $Server "
          echo ""
          dsmadmc -id=$1 -pa=$2 -dataonly=yes -commadelimited -servern=$Server $CMD | grep -v ANR | grep -v ANS >> $FILE
         echo ""
        done
       xmenu
      ;;
      n|N|*)
       echo ""
        eval varAlias=\${$DC[@]}
        for Server in ${varAlias[@]}
        do
         echo ""
          echo -e "Processing TSM server $Server "
          echo ""
          dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
         echo ""
        done
        sleep 2
       xmenu
      ;;
      *)
       echo ""
        eval varAlias=\${$DC[@]}
        for Server in ${varAlias[@]}
        do
         echo ""
          echo -e "Processing TSM server $Server "
          echo ""
          dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
         echo ""
        done
        sleep 2
       xmenu
      ;;
    esac
}
########################
# end xserver function #
########################
clear
echo -e "$GREEN"
echo "Please Wait Loading Menu....."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 2

tsmadmin


                                   

Wednesday, February 22, 2017

Learn From My Mistakes

We recently did a DR test and ran into a serious issue with the recovery of the TSM instance. The DR test was to restore the TSM DB from data center A (DC-A) to data center B. The DC-A instance had been created the previous year so we only needed to remove the DB (DSMSERV REMOVEDB TSMDB1) and restore the DB. I followed the steps and restored the DB and after restoring the DB 99% the restore failed. The restore failure error pointed us to the size available being too small for the required. We were off by a couple GB in our DB directory sizes so we had to add another directory to the list. I tried to rerun the format of the space and quickly received and error that the DB was present. I then ran the DSMSERV REMOVEDB TSMDB1 command again then tried to rerun the format and received the same results. We tried multiple commands

db2 uncatalog db TSMDB1

db2idrop serverA

/opt/tivoli/tsm/db2/bin/db2greg -delinstrec service=db2,version=9.7.0.9,instancename=serverA,instancepath=/drtsmserver/serverA/sqllib

This last command allows you to manually drop the DB from the global registry. When we tried this a couple times the instance would be removed from the global registry and when we would try to rerun the restore we would get the error that the DB was present and the instance would reappear in the global registry. (Notice the instance path....that is going to come play into the final solution.)

So at this point we went round and round with various commands to drop/remove/delete the instance from DB2 and the commands would fail saying that the instance was not present and then an attempt to restore the DB would fail saying the DB was present. We went round and round and round until I finally restored the DB under a new ID and instance directory which was successful.....BUUUUUUTTTTTT.....there in lies another issue. Once the DB restore was successfully restored it failed to allow us to bring up the TSM instance because the new ID was not the original owner of the DB instance. Immediately I realized my mistake as this had happened to me years ago and I totally forgot this DB2 requirement. You see DB2 does not allow an ID other than those that have been granted permission to bring the DB up. I get it, it's a security thing, but the TSM instances almost all Admins create only has one active ID allowed ownership of the DB2 instance. You can add as many as you like but it's not something IBM or Tivoli openly suggest or recommend. So if the ID you typically use wont allow the restore or for some reason can't be used and you want to use a new ID for the DB your S.O.L. If it's after the fact, IBM cannot provide a way for you to assign an additional owner of the DB2 TSM instance. IBM's recommendation was go back to DC-A and grant the new ID rights to the instance, run a TSM DB backup, copy/replicate the instance to DC-B, then restore the DB and it would work under the new ID.

By this point in the DR process we were so behind in the recovery of the TSM server we scrapped our portion of the DR test (we were not a significant portion of test, which was to test Recovery Point capabilities and TSM was a fall back). I continued to work with IBM on the restore under the original ID and the looping issue and after a couple attempts, the IBM level 2 DB2 support person identified that the creation of the TSM instance had created DB files in

/drtsmserver/serverA/serverA/NODE0000

and a simple rename of that directory should the the DB present/not present issue. We had repeatedly cleared out the DB and Log file systems but had forgotten about the DB folder and files created in the instance directory. TSM had created that subdirectory with DB related files. We had renamed the /drtsmserver/serverA/sqllib directory more than once, but we had forgotten about the other instance directory as IBM states in this technote.  When we renamed the second serverA directory and tried the restore and it ran to completion successfully. HEAD SLAP! This was one of the times where even with another set of eyes we were not seeing the bigger picture. IBM level one support was probably looking at the wrong issue and level two was stumped because we moved away from the initial issue we called about and were now asking them how bring up the TSM DB that we restored under the alternate ID.

OK at this point I was exhausted and frustrated with myself. So what did I come away with in this situation....


  1. Triple verify your DB file space size is adequate. (DOH!)
  2. From the instance home directory copy the critical files to an alternate location.  
    • This is in case the restore fails and clears them or updates them incorrectly (trust me this happened more than once)
  3. If the restore fails or a rerun of the restore is required rename both <inst home>/<inst name>/<inst name> and <inst home>/<inst name>/sqllib
  4. IMHO I would also create another ID or two on the production server and grant them access to the TSM DB instance just in case you had an issue with the primary ID. 
    • This is not required but cant hurt.