I just wanted to let everyone know that there is a compatibility issue with the Spectrum Protect 8.1.2 client with the 8.1.0 server. If you upgrade the client to SP 8.1.2 and your server is at 8.1.1 or 8.1.0 your server will be unable to start. The error you will get is
Spectrum Protect ANR0162W Supplemental database diagnostic information: -1:08001:-30082 ([IBM][CLI Driver] SQL30082N Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID").
The issue has to do with the GSKit8.gskcrypt64.ppc.rte and GSKit8.gskssl64.ppc.rte in the 8.1.2.0 client. When we rolled back to the 8.1.0 client the SP server started without an issue.
Wednesday, October 25, 2017
Monday, August 07, 2017
TSM/Spectrum Protect 7.x Client Issue
I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:
08/02/17 03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed. Return code = 12.
08/02/17 13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17 03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17 03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.
This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue. I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:
memoryefficientbackup DISKCACHEM
diskcachelocation /tmp
08/02/17 03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed. Return code = 12.
08/02/17 13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17 03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17 03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.
This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue. I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:
memoryefficientbackup DISKCACHEM
diskcachelocation /tmp
Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.
Thursday, June 01, 2017
TSM SQL TDP Better Backup Script
On a call today I had an admin complain that the TSM SQL TDP was issuing failed backup errors when the script encountered a database that was offline. So I asked if he had searched Google for an alternate script and he had not. I did a quick query and found this nice batch file that appears to do exactly what is needed. The reader on the site commented that the script did not work on his SQL cluster. Does anyone have a similar script that would work on a cluster? Or could you modify this script to work in a cluster environment? The link to the script is provided below.
http://www.applepie.se/tivoli-storage-manager-and-sql-backup
http://www.applepie.se/tivoli-storage-manager-and-sql-backup
Friday, May 05, 2017
TSM Menu - Bash Shell Scripting
For years I have maintained a shell script on our jump server that was a menu for accessing all the TSM servers in our environment. It was a nice interface and great for noobies. It allowed for you to set your password and ID variables before the menu loaded so anyone could run the menu under their ID, then presented them with a nice menu to pick which TSM server they wished to access. It was nice and easy. The only problem was when servers were added or decommissioned. When that occurred I would then have to go out and revise the menu. This was a pain and soon to be more painful as we plan to add a lot more TSM servers in the near future. I remember seeing someone post that they had created a menu system based off of the dsm.sys file so I decided to write my own.
Before:
##################################################################
# #
# Infocrossing #
# UNIX ADMINISTRATION #
# TSM MENU #
# #
##################################################################
Omaha
0) ouax000 1) ouax001
2) ouax002 3) ouax003
4) OTSM01 5) OTSM02
------------------------------------------------------------------
Phoenix
6) puax001 7) puax002
8) puax008 9) PTSM01
10) PTSM02 11) PTSM20
------------------------------------------------------------------
Singapore
20) SIN01ZZ 99) STSM01
------------------------------------------------------------------
Germany
21) DTSM01 22) DTSM02
23) DTSM03 24) DTSM20
------------------------------------------------------------------
Kings Mtn
25) CTSM01 26) CTSM20
27) CTSM30 28) CTSM40
------------------------------------------------------------------
X/x) Cross Server
S/s) Cmd Prompt
DD/dd) Data Domain Admin
Q/q) Return to Main Menu
Selection:
After (Notice servers listed here that were not in above menu):
################################################################
# #
# Infocrossing #
# Recovery Services #
# TSM MENU #
# #
################################################################
1) C1IF 12) LABTSM 23) PTSM01
2) C1KDC2 13) LUAX001 24) PTSM02
3) CTSM01 14) LUAX002 25) PTSM20
4) CTSM20 15) LTSM01 26) PTSM25-Arch
5) CTSM30 16) NTSM01 27) SIN01ZZ
6) CTSM40 17) OUAX003 28) STSM01
7) CTSM46-Arch 18) OUAX000 29) Cross Server
8) DTSM01 19) OUAX001 30) Refresh Menu
9) DTSM02 20) OTSM01 31) Exit
10) DTSM03 21) OTSM02
11) DTSM20 22) PESTSM01
Please enter your choice:
Ok, so the Before is more organized by region but as new servers come in I wanted the menu to update dynamically. So this new menu is built around the BASH shell's select menu function. The script pulls all the TSM server names from the dsm.sys and populates an array that the select statement reads and generates this menu. Make sure the variables the script declares work for you and your dsm.sys. My dsm.sys looks like this if I grep servername:
SERVERNAME DTSM01 * Germany LIB-MAN
SERVERNAME DTSM02 * Germany
SERVERNAME DTSM03 * Germany
SERVERNAME DTSM20 * Germany LIB-MAN
SERVERNAME CTSM01 * Kings_Mnt LIB-MAN
SERVERNAME C1KDC2 * Kings_Mnt
SERVERNAME C1IF * Kings_Mnt
SERVERNAME CTSM20 * Kings_Mnt
SERVERNAME CTSM30 * Kings_Mnt
SERVERNAME CTSM40 * Kings_Mnt
SERVERNAME CTSM46-Arch * Kings_Mnt
SERVERNAME LUAX001 * Leonia LIB-MAN
SERVERNAME LUAX002 * Leonia
SERVERNAME LTSM01 * Leonia
SERVERNAME NTSM01 * Norcross LIB-MAN
SERVERNAME OUAX000 * Omaha LIB-MAN
SERVERNAME OUAX001 * Omaha
SERVERNAME OUAX003 * Omaha
SErvername OTSM01 * Omaha LIB-MAN
SErvername OTSM02 * Omaha
SERVERNAME PESTSM01 * PEST LIB-MAN
SERVERNAME PTSM01 * Tempe LIB-MAN
SERVERNAME PTSM02 * Tempe
SERVERNAME PTSM20 * Tempe
SERVERNAME PTSM25-Arch * Tempe
SERVERNAME STSM01 * Singapore
SERVERNAME SIN01ZZ * Singapore
SERVERNAME LABTSM
With the dsm.sys file formated this way I can use grep and awk to get specific data. This allows the script to build my menus according to my needs. Especially with the Cross Server sub menu that allows me to run a command across multiple servers at once. In the sub-menu it groups by distinct location which I placed in the servername line. Leave me a comment if you have any comments, questions, or suggestions.
See the script below:
#!/bin/bash
#set -x
### Set Color Variables ###
NORMAL=`echo "\033[0m"`
WHITE=`echo "\033[37m"` #Blue
YELLOW=`echo "\033[33m"` #yellow
GREEN=`echo "\033[32m"`
RED=`echo "\033[31m"`
PURPLE=`echo "\033[35m"`
##### Declare Variables & Arrays ############
prompt="
Please enter your choice: "
TSM_SYS="/opt/tivoli/tsm/client/ba/bin/dsm.sys"
all_servers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $2}' | sort`)
xservers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $4}' | sort -u`)
Germany=(`cat $TSM_SYS | grep -i servername | grep -i germany |awk -v OFS='\t' '{print $2}'`)
Omaha=(`cat $TSM_SYS | grep -i servername | grep -i omaha |awk -v OFS='\t' '{print $2}'`)
Norcross=(`cat $TSM_SYS | grep -i servername | grep -i norcross |awk -v OFS='\t' '{print $2}'`)
Tempe=(`cat $TSM_SYS | grep -i servername | grep -i tempe |awk -v OFS='\t' '{print $2}'`)
PEST=(`cat $TSM_SYS | grep -i servername | grep -i pest |awk -v OFS='\t' '{print $2}'`)
Leonia=(`cat $TSM_SYS | grep -i servername | grep -i leonia |awk -v OFS='\t' '{print $2}'`)
Kings_Mnt=(`cat $TSM_SYS | grep -i servername | grep -i kings_mnt |awk -v OFS='\t' '{print $2}'| sort`)
Singapore=(`cat $TSM_SYS | grep -i servername | grep -i singapore |awk -v OFS='\t' '{print $2}'`)
lib_man=(`cat $TSM_SYS | grep -i servername | grep -i lib-man | awk -v OFS='\t' '{print $2}'`)
export COLUMNS=70
########################
######################
function clearlogin
######################
{
echo "Please wait while I clear all login information."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
unset TSMID
unset TSMPA
unset DDID
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "Logins have been cleared! "|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 1
}
######################
# end clearlogin #
######################
###############
#END MAIN-MENU#
###############
#########################
function tsmadmin
#########################
{
clear
if [ -z "${TSMID}" ]; then
echo ""
echo -e "$YELLOW Enter your TSM Admin ID: $GREEN \c"
read TSMID
echo ""
echo -e "$YELLOW Enter your TSM Admin Password: \c "
#stty -echo
read -s TSMPA
#stty $stty_orig
fi
clear
tsm-menu
exit
}
####################
# end tsmadmin #
####################
#####################
function tsm-menu
#####################
{
all_done=0
export COLUMNS=70
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE # $YELLOW Infocrossing $WHITE #"
echo -e "$WHITE # Recovery Services #"
echo -e "$WHITE # $RED TSM MENU $WHITE #"
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#all_servers[@]} + 4))"`
PS3="$prompt "
select host in "${all_servers[@]}" "Cross Server" "Reset Login" "Refresh Menu" "Exit"; do
if [ "$host" = "Exit" ]; then
all_done=1; clear; exit
elif [ "$host" = "Refresh Menu" ]; then
all_done=1; tsm-menu
elif [ "$host" = "Reset Login" ]; then
all_done=1; clearlogin; tsmadmin
elif [ "$host" = "Cross Server" ]; then
all_done=1; xmenu
elif [ "$REPLY" -gt "$length" ]; then
echo ""
echo "You chose option $REPLY which is invalid. Try again! "
echo ""
else
clear
echo -e "$GREEN "
dsmadmc -id=$TSMID -pa=$TSMPA -servern="$host"
all_done=1; tsm-menu
fi
done
done
}
##########################
# end function tsm-menu #
##########################
#####################
function xmenu
#####################
{
all_done=0
export COLUMNS=50
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE # $YELLOW Infocrossing $WHITE #"
echo -e "$WHITE # Recovery Services #"
echo -e "$WHITE # $RED TSM Cross Server Command MENU $WHITE # "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#xservers[@]} + 4))"`
PS3="$prompt "
select host in "${xservers[@]}" "All Servers" "Library Managers" "Refresh Menu" "Exit"; do
if [ "$host" = "Exit" ]; then
all_done=1; tsm-menu
elif [ "$host" = "Refresh Menu" ]; then
all_done=1; xmenu
elif [ "$host" = "Library Managers" ]; then
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA lib_man
all_done=1; xmenu
elif [ "$host" = "All Servers" ]; then
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA all_servers
all_done=1; xmenu
elif [ "$REPLY" -gt "$length" ]; then
echo ""
echo "You chose option $REPLY which is invalid. Try again! "
echo ""
else
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA "$host"
all_done=1; xmenu
fi
done
done
}
##########################
# end function xmenu #
##########################
###################
function xserver
###################
{
DC=$3
echo -e " "
echo -e " $YELLOW Enter the command to process: $NORMAL \c"
read CMD
echo ""
echo -e " $YELLOW Do you want to save the data to a commadelimited csv file? (Yes or No [N]) $NORMAL \c"
read ANSWER
case $ANSWER
in
y|Y|YES|Yes|yes)
echo ""
echo -e " $YELLOW Enter the file name to save the data: $NORMAL \c"
read FILE
echo ""
touch $FILE
cat /dev/null > $FILE
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -commadelimited -servern=$Server $CMD | grep -v ANR | grep -v ANS >> $FILE
echo ""
done
xmenu
;;
n|N|*)
echo ""
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
echo ""
done
sleep 2
xmenu
;;
*)
echo ""
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
echo ""
done
sleep 2
xmenu
;;
esac
}
########################
# end xserver function #
########################
clear
echo -e "$GREEN"
echo "Please Wait Loading Menu....."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 2
tsmadmin
Before:
##################################################################
# #
# Infocrossing #
# UNIX ADMINISTRATION #
# TSM MENU #
# #
##################################################################
Omaha
0) ouax000 1) ouax001
2) ouax002 3) ouax003
4) OTSM01 5) OTSM02
------------------------------------------------------------------
Phoenix
6) puax001 7) puax002
8) puax008 9) PTSM01
10) PTSM02 11) PTSM20
------------------------------------------------------------------
Singapore
20) SIN01ZZ 99) STSM01
------------------------------------------------------------------
Germany
21) DTSM01 22) DTSM02
23) DTSM03 24) DTSM20
------------------------------------------------------------------
Kings Mtn
25) CTSM01 26) CTSM20
27) CTSM30 28) CTSM40
------------------------------------------------------------------
X/x) Cross Server
S/s) Cmd Prompt
DD/dd) Data Domain Admin
Q/q) Return to Main Menu
Selection:
After (Notice servers listed here that were not in above menu):
################################################################
# #
# Infocrossing #
# Recovery Services #
# TSM MENU #
# #
################################################################
1) C1IF 12) LABTSM 23) PTSM01
2) C1KDC2 13) LUAX001 24) PTSM02
3) CTSM01 14) LUAX002 25) PTSM20
4) CTSM20 15) LTSM01 26) PTSM25-Arch
5) CTSM30 16) NTSM01 27) SIN01ZZ
6) CTSM40 17) OUAX003 28) STSM01
7) CTSM46-Arch 18) OUAX000 29) Cross Server
8) DTSM01 19) OUAX001 30) Refresh Menu
9) DTSM02 20) OTSM01 31) Exit
10) DTSM03 21) OTSM02
11) DTSM20 22) PESTSM01
Please enter your choice:
Ok, so the Before is more organized by region but as new servers come in I wanted the menu to update dynamically. So this new menu is built around the BASH shell's select menu function. The script pulls all the TSM server names from the dsm.sys and populates an array that the select statement reads and generates this menu. Make sure the variables the script declares work for you and your dsm.sys. My dsm.sys looks like this if I grep servername:
SERVERNAME DTSM01 * Germany LIB-MAN
SERVERNAME DTSM02 * Germany
SERVERNAME DTSM03 * Germany
SERVERNAME DTSM20 * Germany LIB-MAN
SERVERNAME CTSM01 * Kings_Mnt LIB-MAN
SERVERNAME C1KDC2 * Kings_Mnt
SERVERNAME C1IF * Kings_Mnt
SERVERNAME CTSM20 * Kings_Mnt
SERVERNAME CTSM30 * Kings_Mnt
SERVERNAME CTSM40 * Kings_Mnt
SERVERNAME CTSM46-Arch * Kings_Mnt
SERVERNAME LUAX001 * Leonia LIB-MAN
SERVERNAME LUAX002 * Leonia
SERVERNAME LTSM01 * Leonia
SERVERNAME NTSM01 * Norcross LIB-MAN
SERVERNAME OUAX000 * Omaha LIB-MAN
SERVERNAME OUAX001 * Omaha
SERVERNAME OUAX003 * Omaha
SErvername OTSM01 * Omaha LIB-MAN
SErvername OTSM02 * Omaha
SERVERNAME PESTSM01 * PEST LIB-MAN
SERVERNAME PTSM01 * Tempe LIB-MAN
SERVERNAME PTSM02 * Tempe
SERVERNAME PTSM20 * Tempe
SERVERNAME PTSM25-Arch * Tempe
SERVERNAME STSM01 * Singapore
SERVERNAME SIN01ZZ * Singapore
SERVERNAME LABTSM
With the dsm.sys file formated this way I can use grep and awk to get specific data. This allows the script to build my menus according to my needs. Especially with the Cross Server sub menu that allows me to run a command across multiple servers at once. In the sub-menu it groups by distinct location which I placed in the servername line. Leave me a comment if you have any comments, questions, or suggestions.
See the script below:
#!/bin/bash
#set -x
### Set Color Variables ###
NORMAL=`echo "\033[0m"`
WHITE=`echo "\033[37m"` #Blue
YELLOW=`echo "\033[33m"` #yellow
GREEN=`echo "\033[32m"`
RED=`echo "\033[31m"`
PURPLE=`echo "\033[35m"`
##### Declare Variables & Arrays ############
prompt="
Please enter your choice: "
TSM_SYS="/opt/tivoli/tsm/client/ba/bin/dsm.sys"
all_servers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $2}' | sort`)
xservers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $4}' | sort -u`)
Germany=(`cat $TSM_SYS | grep -i servername | grep -i germany |awk -v OFS='\t' '{print $2}'`)
Omaha=(`cat $TSM_SYS | grep -i servername | grep -i omaha |awk -v OFS='\t' '{print $2}'`)
Norcross=(`cat $TSM_SYS | grep -i servername | grep -i norcross |awk -v OFS='\t' '{print $2}'`)
Tempe=(`cat $TSM_SYS | grep -i servername | grep -i tempe |awk -v OFS='\t' '{print $2}'`)
PEST=(`cat $TSM_SYS | grep -i servername | grep -i pest |awk -v OFS='\t' '{print $2}'`)
Leonia=(`cat $TSM_SYS | grep -i servername | grep -i leonia |awk -v OFS='\t' '{print $2}'`)
Kings_Mnt=(`cat $TSM_SYS | grep -i servername | grep -i kings_mnt |awk -v OFS='\t' '{print $2}'| sort`)
Singapore=(`cat $TSM_SYS | grep -i servername | grep -i singapore |awk -v OFS='\t' '{print $2}'`)
lib_man=(`cat $TSM_SYS | grep -i servername | grep -i lib-man | awk -v OFS='\t' '{print $2}'`)
export COLUMNS=70
########################
######################
function clearlogin
######################
{
echo "Please wait while I clear all login information."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
unset TSMID
unset TSMPA
unset DDID
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "Logins have been cleared! "|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 1
}
######################
# end clearlogin #
######################
###############
#END MAIN-MENU#
###############
#########################
function tsmadmin
#########################
{
clear
if [ -z "${TSMID}" ]; then
echo ""
echo -e "$YELLOW Enter your TSM Admin ID: $GREEN \c"
read TSMID
echo ""
echo -e "$YELLOW Enter your TSM Admin Password: \c "
#stty -echo
read -s TSMPA
#stty $stty_orig
fi
clear
tsm-menu
exit
}
####################
# end tsmadmin #
####################
#####################
function tsm-menu
#####################
{
all_done=0
export COLUMNS=70
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE # $YELLOW Infocrossing $WHITE #"
echo -e "$WHITE # Recovery Services #"
echo -e "$WHITE # $RED TSM MENU $WHITE #"
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#all_servers[@]} + 4))"`
PS3="$prompt "
select host in "${all_servers[@]}" "Cross Server" "Reset Login" "Refresh Menu" "Exit"; do
if [ "$host" = "Exit" ]; then
all_done=1; clear; exit
elif [ "$host" = "Refresh Menu" ]; then
all_done=1; tsm-menu
elif [ "$host" = "Reset Login" ]; then
all_done=1; clearlogin; tsmadmin
elif [ "$host" = "Cross Server" ]; then
all_done=1; xmenu
elif [ "$REPLY" -gt "$length" ]; then
echo ""
echo "You chose option $REPLY which is invalid. Try again! "
echo ""
else
clear
echo -e "$GREEN "
dsmadmc -id=$TSMID -pa=$TSMPA -servern="$host"
all_done=1; tsm-menu
fi
done
done
}
##########################
# end function tsm-menu #
##########################
#####################
function xmenu
#####################
{
all_done=0
export COLUMNS=50
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE # $YELLOW Infocrossing $WHITE #"
echo -e "$WHITE # Recovery Services #"
echo -e "$WHITE # $RED TSM Cross Server Command MENU $WHITE # "
echo -e "$WHITE # #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#xservers[@]} + 4))"`
PS3="$prompt "
select host in "${xservers[@]}" "All Servers" "Library Managers" "Refresh Menu" "Exit"; do
if [ "$host" = "Exit" ]; then
all_done=1; tsm-menu
elif [ "$host" = "Refresh Menu" ]; then
all_done=1; xmenu
elif [ "$host" = "Library Managers" ]; then
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA lib_man
all_done=1; xmenu
elif [ "$host" = "All Servers" ]; then
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA all_servers
all_done=1; xmenu
elif [ "$REPLY" -gt "$length" ]; then
echo ""
echo "You chose option $REPLY which is invalid. Try again! "
echo ""
else
clear
echo -e "$GREEN "
xserver $TSMID $TSMPA "$host"
all_done=1; xmenu
fi
done
done
}
##########################
# end function xmenu #
##########################
###################
function xserver
###################
{
DC=$3
echo -e " "
echo -e " $YELLOW Enter the command to process: $NORMAL \c"
read CMD
echo ""
echo -e " $YELLOW Do you want to save the data to a commadelimited csv file? (Yes or No [N]) $NORMAL \c"
read ANSWER
case $ANSWER
in
y|Y|YES|Yes|yes)
echo ""
echo -e " $YELLOW Enter the file name to save the data: $NORMAL \c"
read FILE
echo ""
touch $FILE
cat /dev/null > $FILE
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -commadelimited -servern=$Server $CMD | grep -v ANR | grep -v ANS >> $FILE
echo ""
done
xmenu
;;
n|N|*)
echo ""
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
echo ""
done
sleep 2
xmenu
;;
*)
echo ""
eval varAlias=\${$DC[@]}
for Server in ${varAlias[@]}
do
echo ""
echo -e "Processing TSM server $Server "
echo ""
dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
echo ""
done
sleep 2
xmenu
;;
esac
}
########################
# end xserver function #
########################
clear
echo -e "$GREEN"
echo "Please Wait Loading Menu....."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 2
tsmadmin
Wednesday, February 22, 2017
Learn From My Mistakes
We recently did a DR test and ran into a serious issue with the recovery of the TSM instance. The DR test was to restore the TSM DB from data center A (DC-A) to data center B. The DC-A instance had been created the previous year so we only needed to remove the DB (DSMSERV REMOVEDB TSMDB1) and restore the DB. I followed the steps and restored the DB and after restoring the DB 99% the restore failed. The restore failure error pointed us to the size available being too small for the required. We were off by a couple GB in our DB directory sizes so we had to add another directory to the list. I tried to rerun the format of the space and quickly received and error that the DB was present. I then ran the DSMSERV REMOVEDB TSMDB1 command again then tried to rerun the format and received the same results. We tried multiple commands
db2 uncatalog db TSMDB1
db2idrop serverA
/opt/tivoli/tsm/db2/bin/db2greg -delinstrec service=db2,version=9.7.0.9,instancename=serverA,instancepath=/drtsmserver/serverA/sqllib
This last command allows you to manually drop the DB from the global registry. When we tried this a couple times the instance would be removed from the global registry and when we would try to rerun the restore we would get the error that the DB was present and the instance would reappear in the global registry. (Notice the instance path....that is going to come play into the final solution.)
So at this point we went round and round with various commands to drop/remove/delete the instance from DB2 and the commands would fail saying that the instance was not present and then an attempt to restore the DB would fail saying the DB was present. We went round and round and round until I finally restored the DB under a new ID and instance directory which was successful.....BUUUUUUTTTTTT.....there in lies another issue. Once the DB restore was successfully restored it failed to allow us to bring up the TSM instance because the new ID was not the original owner of the DB instance. Immediately I realized my mistake as this had happened to me years ago and I totally forgot this DB2 requirement. You see DB2 does not allow an ID other than those that have been granted permission to bring the DB up. I get it, it's a security thing, but the TSM instances almost all Admins create only has one active ID allowed ownership of the DB2 instance. You can add as many as you like but it's not something IBM or Tivoli openly suggest or recommend. So if the ID you typically use wont allow the restore or for some reason can't be used and you want to use a new ID for the DB your S.O.L. If it's after the fact, IBM cannot provide a way for you to assign an additional owner of the DB2 TSM instance. IBM's recommendation was go back to DC-A and grant the new ID rights to the instance, run a TSM DB backup, copy/replicate the instance to DC-B, then restore the DB and it would work under the new ID.
By this point in the DR process we were so behind in the recovery of the TSM server we scrapped our portion of the DR test (we were not a significant portion of test, which was to test Recovery Point capabilities and TSM was a fall back). I continued to work with IBM on the restore under the original ID and the looping issue and after a couple attempts, the IBM level 2 DB2 support person identified that the creation of the TSM instance had created DB files in
/drtsmserver/serverA/serverA/NODE0000
and a simple rename of that directory should the the DB present/not present issue. We had repeatedly cleared out the DB and Log file systems but had forgotten about the DB folder and files created in the instance directory. TSM had created that subdirectory with DB related files. We had renamed the /drtsmserver/serverA/sqllib directory more than once, but we had forgotten about the other instance directory as IBM states in this technote. When we renamed the second serverA directory and tried the restore and it ran to completion successfully. HEAD SLAP! This was one of the times where even with another set of eyes we were not seeing the bigger picture. IBM level one support was probably looking at the wrong issue and level two was stumped because we moved away from the initial issue we called about and were now asking them how bring up the TSM DB that we restored under the alternate ID.
OK at this point I was exhausted and frustrated with myself. So what did I come away with in this situation....
db2 uncatalog db TSMDB1
db2idrop serverA
/opt/tivoli/tsm/db2/bin/db2greg -delinstrec service=db2,version=9.7.0.9,instancename=serverA,instancepath=/drtsmserver/serverA/sqllib
This last command allows you to manually drop the DB from the global registry. When we tried this a couple times the instance would be removed from the global registry and when we would try to rerun the restore we would get the error that the DB was present and the instance would reappear in the global registry. (Notice the instance path....that is going to come play into the final solution.)
So at this point we went round and round with various commands to drop/remove/delete the instance from DB2 and the commands would fail saying that the instance was not present and then an attempt to restore the DB would fail saying the DB was present. We went round and round and round until I finally restored the DB under a new ID and instance directory which was successful.....BUUUUUUTTTTTT.....there in lies another issue. Once the DB restore was successfully restored it failed to allow us to bring up the TSM instance because the new ID was not the original owner of the DB instance. Immediately I realized my mistake as this had happened to me years ago and I totally forgot this DB2 requirement. You see DB2 does not allow an ID other than those that have been granted permission to bring the DB up. I get it, it's a security thing, but the TSM instances almost all Admins create only has one active ID allowed ownership of the DB2 instance. You can add as many as you like but it's not something IBM or Tivoli openly suggest or recommend. So if the ID you typically use wont allow the restore or for some reason can't be used and you want to use a new ID for the DB your S.O.L. If it's after the fact, IBM cannot provide a way for you to assign an additional owner of the DB2 TSM instance. IBM's recommendation was go back to DC-A and grant the new ID rights to the instance, run a TSM DB backup, copy/replicate the instance to DC-B, then restore the DB and it would work under the new ID.
By this point in the DR process we were so behind in the recovery of the TSM server we scrapped our portion of the DR test (we were not a significant portion of test, which was to test Recovery Point capabilities and TSM was a fall back). I continued to work with IBM on the restore under the original ID and the looping issue and after a couple attempts, the IBM level 2 DB2 support person identified that the creation of the TSM instance had created DB files in
/drtsmserver/serverA/serverA/NODE0000
and a simple rename of that directory should the the DB present/not present issue. We had repeatedly cleared out the DB and Log file systems but had forgotten about the DB folder and files created in the instance directory. TSM had created that subdirectory with DB related files. We had renamed the /drtsmserver/serverA/sqllib directory more than once, but we had forgotten about the other instance directory as IBM states in this technote. When we renamed the second serverA directory and tried the restore and it ran to completion successfully. HEAD SLAP! This was one of the times where even with another set of eyes we were not seeing the bigger picture. IBM level one support was probably looking at the wrong issue and level two was stumped because we moved away from the initial issue we called about and were now asking them how bring up the TSM DB that we restored under the alternate ID.
OK at this point I was exhausted and frustrated with myself. So what did I come away with in this situation....
- Triple verify your DB file space size is adequate. (DOH!)
- From the instance home directory copy the critical files to an alternate location.
- This is in case the restore fails and clears them or updates them incorrectly (trust me this happened more than once)
- If the restore fails or a rerun of the restore is required rename both <inst home>/<inst name>/<inst name> and <inst home>/<inst name>/sqllib
- IMHO I would also create another ID or two on the production server and grant them access to the TSM DB instance just in case you had an issue with the primary ID.
- This is not required but cant hurt.
Thursday, August 25, 2016
Why Are You Not Using Google & YouTube?
I had an OS Admin contact me through LinkedIn and G-Mail asking for help on trying to find his archived data. He didn't have much experience with TSM and was looking for information on how to find the long term backups (i.e. Archives). I asked him if he even tried to search Google? Google and YouTube are great resources for all your needs. For example if you want to learn more about the new Operations Center you can see a plethora of videos by searching YouTube. You can also use google to find all sorts of related documents and pages when it concerns APARs and errors. If you haven't done your due diligence you make yourself look dumb. Shoot, you can even search Google from my website and get my posts and outside relevant web pages!
So here is a list of YouTube videos you can reference:
Backup and Archive
Server Administration
Setup TSM Deduplication
Tivoli Data Protection Agents
TDP for Virtual Environments
So here is a list of YouTube videos you can reference:
Backup and Archive
Server Administration
Setup TSM Deduplication
Tivoli Data Protection Agents
TDP for Virtual Environments
Subscribe to:
Posts (Atom)