Monday, December 04, 2017

SystemState Restore Issue with Windows 8.x Client

3/7/2017 Update
I have had a number of people request the test flag and one response said it was kicking up errors when used with the CAD service/process. Please note this flag is only needed in the event of a system state restore and you would not (I hope) be using the CAD/Scheduler to restore your system state. So the testflag should only be activated in the dsm.opt in the event you are doing a manual restore of the system state and DO NOT leave it in the dsm.opt as it will affect daily use.


1/24/2018 Update
After waiting for IBM to post an update to this issue I decided I would let you know that IBM has an undocumented flag for this issue that allows the client to restore the System State online rather than work with the PE. Our Windows admins spoke with Microsoft and they state emphatically that they support online system state restores, so is appears IBM does not starting with the 8.1.x client. Here is the flag to allow the SP client to perform online system state restores.

ONLINE_SYSTEMSTATE_RESTORE_ALLOWED

12/4/2018
I wanted to make everyone aware that starting with the 8.1 Windows Client, IBM has deprecated the System State restore process so that any restore attempt will require the use of a WinPE environment. With the 8.1.x client you can no longer do a complete restore of a server within the normal windows environment and will have to make sure you have a compatible WinPE for every Windows Hardware type or environment. This, in my opinion, is a huge break of the software. Although IBM stated that Microsoft does not support online system state restores, Microsoft disagrees. I say this knowing that if you are on a 7.x or earlier version of the client IBM will support you with the system state restore, but as of 8.1 you will be directed to build a Windows PE environment and follow a series of stepto create the necessary disc. The support person I spoke with treated this process as if it should be second nature to any Windows Admin.

We recently were migrating a clients server and in the process the disks were corrupted. The client installed on the new machine was 8.1.2 and when the Windows Admins tried to restore the system state they experienced the following error.

 


After speaking with support we were directed to the following document by Andrew Raibeck that discussed creating the WinPE disk image with the TSM client code that would allow the offline restore of the System State. We attempted to build the ISO but when the Windows Admin tried to run the dsmc client we experienced the following error:



Who do we call when we experience an error like this in a WinPE environment? IBM or Microsoft? So you have two choices, have a Windows PE available for each hardware type or try using the testflag but understand IBM may support its use. If you were fully aware of this and took precautions what were they? My thinking is that many admins have no idea this is going to bite them and its better you know now than find out the hard way.

 

Wednesday, November 22, 2017

Spectrum Protect 8.1 New DB Functions Supported - UPDATE to the UPDATE!

UPDATE (11/22/2017):

I have found another interesting select option I was previously unaware of that coincides with the previous LOCATE option. We have an interesting host naming convention where I work and we use the hostname for the NODE NAME when defining the servers to TSM/SP. It is as follows:

<servername>.<datacenterID>.<OS_type>.<customer>.<domain.net>
Example:   CQSDBS01.US01.ULX.BS.US32118.NET
The issue I have is with trying to seperate out the data is a report by customer. Of course each customer has their own domains but I wanted a report that broke out the report by the customer ID. I wanted something like this:
Customer      Occupancy GB
----------    ------------
UDC2                 38144
903                  33358
CFG                  30321
ULT01                 1942
MAQ                    639
MSM                   1557
MX                       9
MX3                      0
ELO                   6741
GUS                      4
BFG                  11234
VS                     166
PMTX                    80
POS                  14148
SAM                   8031
SUX                  18562
The problem was not with breaking out the customer ID because the LOCATE select worked perfectly for extracting the Customer IDs with their various lengths. The issue was how to sum each customers data without experiencing an error requiring I use GROUP BY which caused more headaches because I could not figure out how to group by the NODE NAME from which I was getting the customer ID. So I googled GROUP BY in DB2 with SUBSTR since I was using a SUBSTR to extract the customer IDs. As turns out the GROUP BY is more flexible than I originally thought. Check this out:

Select -                           
 VARCHAR(SUBSTR(NODE_NAME, -        
 LOCATE('.',NODE_NAME, -           
 LOCATE('.', NODE_NAME, -          
 LOCATE('.', NODE_NAME)+1)+1)+1, - 
 LOCATE('.',NODE_NAME, -           
 LOCATE('.',NODE_NAME, -           
 LOCATE('.', NODE_NAME, -          
 LOCATE('.', NODE_NAME)+1)+1)+1)- -
 LOCATE('.',NODE_NAME, -           
 LOCATE('.', NODE_NAME, -          
 LOCATE('.', NODE_NAME)+1)+1)-1),8) -
 as CUSTOMER, -                    
 sum(TOTAL_MB)/1024 AS TOTAL_GB -   
from auditocc -                    
 where node_name like '%.%.%.%.%' - 
group by VARCHAR(SUBSTR(NODE_NAME, -
 LOCATE('.',NODE_NAME, -           
 LOCATE('.', NODE_NAME, -          
 LOCATE('.', NODE_NAME)+1)+1)+1, - 
 LOCATE('.',NODE_NAME, -           
 LOCATE('.',NODE_NAME, -           
 LOCATE('.', NODE_NAME, -          
 LOCATE('.', NODE_NAME)+1)+1)+1)- -
 LOCATE('.',NODE_NAME, -           
 LOCATE('.', NODE_NAME, -          
 LOCATE('.', NODE_NAME)+1)+1)-1),8) 

In DB2 I could actually group by the SUBSTR of the value I was extracting, even with it being complex with the LOCATE options in use. MIND BLOWN! 

mind blown wat animated GIF

UPDATE (5/9/2017):

I posted on StackOverflow (a great developer/scripting/DB) listserv asking how I could get similar output with the functions available in versions prior to DB2 11 and a user was able to help me out with a script that could return the same data as the one below although it's a little more complicated.  So as you know the SUBSTR function works like this:


Select SUBSTR(FILESPACE_NAME, <START POSITION>,<LENGTH>)

The issue with the earlier DB2 versions is I could only get the LOCATION function to play nice with my script. I was previously only able to figure out a select that got close:
Select -                                                                        
 SUBSTR(FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, - 
 LOCATE('\', FILESPACE_NAME, -                                                  
 LOCATE('\', FILESPACE_NAME)+1)+1)-1)+1) -                                      
 as FSNAME -                                                                    
from filespaces -                                                               
 where node_name='TEST-AP-DAG'
Which output this:

FSNAME                                                           
-----------------------------------------------------------------
TEST-MB-08\6bc391ef-a370-49a1-8f05-b1bed9e5ad55                  
Mailbox Database 1143943276\55ac1670-efc9-4301-ac5c-beb5cd2d77cb 
TEST-MB-05\11dedbd4-1757-45c8-8991-f0f7134ef210                  

I could not figure out a way to get the length. So a kind reader at StackOverflow helped me understand how to get the length by subtracting the 4th '\' location from the 3rd '\' -1. This gives the correct length and allows for the length size to vary rather than be fixed.
Select -
 SUBSTR(FILESPACE_NAME, -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME)+1)+1)+1, -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME)+1)+1)+1)- -
 LOCATE('\',FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME, -
 LOCATE('\', FILESPACE_NAME)+1)+1)-1) -
 as FSNAME -
from filespaces -
 where node_name='TEST-AP-DAG'
Here it is color coded:
(Select SUBSTR(FILESPACE_NAME, <START POSITION>,<LENGTH>)
Select -
 SUBSTR(FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, LOCATE('\', 
 FILESPACE_NAME, LOCATE('\', FILESPACE_NAME)+1)+1)+1, -
 LOCATE('\',FILESPACE_NAME, LOCATE('\',FILESPACE_NAME, LOCATE('\', 
 FILESPACE_NAME,  LOCATE('\', FILESPACE_NAME)+1)+1)+1)-  -
 LOCATE('\',FILESPACE_NAME, LOCATE('\', FILESPACE_NAME,  -
 LOCATE('\', FILESPACE_NAME)+1)+1)-1) -
 as FSNAME -
from FILESPACES -
 where - 
  NODE_NAME='TEST-AP-DAG'
So the final script looks like this
Select -
 VARCHAR(NODE_NAME,12) as NODE, -
 VARCHAR(SUBSTR(FILESPACE_NAME, -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME)+1)+1)+1, -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME)+1)+1)+1)- -
  LOCATE('\',FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME, -
  LOCATE('\', FILESPACE_NAME)+1)+1)-1),30) -
  as Exch_DB,
 date(backup_end) AS LAST_GOOD_BACKUP -
from filespaces -
 where - 
  node_name='TEST-AP-DAG' -
 order by Exch_DB, backup_end asc
The results are exactly like the script that only works with TSM 8.1.


NODE              EXCH_DB                          LAST_GOOD_BACKUP
-------------     ------------------------------- -----------------
TEST-AP-DAG       \TEST-MB-01                            2017-05-08
TEST-AP-DAG       \TEST-MB-02                            2017-05-08
TEST-AP-DAG       \TEST-MB-03                            2017-05-08
TEST-AP-DAG       \TEST-MB-04                            2017-05-08
TEST-AP-DAG       \TEST-MB-05                            2017-05-08
TEST-AP-DAG       \TEST-MB-TEMP                          2017-05-08
TEST-AP-DAG       \MISC-MB-TEMP                          2017-05-08
TEST-AP-DAG       \Mailbox Database 1128394465           2017-05-08
TEST-AP-DAG       \Mailbox Database 1397248650           2017-05-08



(4/20/17):
I upgraded some of my 6.3.4 servers to TSM 8.1 recently and have found the newer DB2 version (v11.1) offers some added functions for SQL queries. I have been trying to build a report to track an Exchange DAG servers backups. The problem is that the easiest way to track the backups of DB's is to view the last completed backup for each of nodes file spaces. That's the easy part. The hard part is that the file space name is huge.

TEST-AP-DAG\Microsoft Exchange Writer\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}\TEST-MB-05\0e41e645-9acf-4b80-bc85-d606e04fe4d8

TEST-AP-DAG\Microsoft Exchange Writer\{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}\Mailbox Database 1223544393\2bc06db2-1966-4fd7-9545-f667102b0b7d

So how to extract the DB name, in this case TEST-MB-05 and Mailbox Database 1397248650 from the name when the length of the DB name changes? After some investigation and trial and error I found that the LOCATE_IN_STRING function works in v8.1. Whether it works in 7.x (DB2 v10.5) I can't say since I don't have a 7.x server to run it against. If any of you out there try this and it works let me know. So here's the script. I think you can read it and see what I am doing. If not let me know in the comments and I'll explain...

tsm: TSMSERV>q script DAG-EXCH-RPT f=raw

select -
 VARCHAR(NODE_NAME,12) as NODE, -
 varchar(substring(filespace_name, LOCATE_IN_STRING(filespace_name, '\', 1, 3), -
 LOCATE_IN_STRING(filespace_name, '\', 1, 4) -  LOCATE_IN_STRING(filespace_name, '\', 1, 3)),30) as Exch_DB, -
 date(backup_end) AS LAST_GOOD_BACKUP -
from FILESPACES -
 where -
  node_name='TEST-AP-DAG' -
 order by Exch_DB, backup_end asc


Here is sample output:

NODE              EXCH_DB                          LAST_GOOD_BACKUP
-------------     ------------------------------- -----------------
TEST-AP-DAG       \TEST-MB-01                            2017-04-18
TEST-AP-DAG       \TEST-MB-02                            2017-04-18
TEST-AP-DAG       \TEST-MB-03                            2017-04-18
TEST-AP-DAG       \TEST-MB-04                            2017-04-18
TEST-AP-DAG       \TEST-MB-05                            2017-04-18
TEST-AP-DAG       \TEST-MB-TEMP                          2017-04-18
TEST-AP-DAG       \MISC-MB-TEMP                          2017-04-18
TEST-AP-DAG       \Mailbox Database 1128394465           2017-04-18
TEST-AP-DAG       \Mailbox Database 1397248650           2017-04-18

If another function would work better feel free to correct the script and leave it for everyone in the comments. Now to go and find more functions I can use to manipulate my data!

Wednesday, October 25, 2017

Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

I just wanted to let everyone know that there is a compatibility issue with the Spectrum Protect 8.1.2 client with the 8.1.0 server. If you upgrade the client to SP 8.1.2 and your server is at 8.1.1 or 8.1.0 your server will be unable to start. The error you will get is

Spectrum Protect ANR0162W Supplemental database diagnostic information:  -1:08001:-30082 ([IBM][CLI Driver] SQL30082N  Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID").

The issue has to do with the GSKit8.gskcrypt64.ppc.rte and GSKit8.gskssl64.ppc.rte in the 8.1.2.0 client. When we rolled back to the 8.1.0 client the SP server started without an issue.

Monday, August 07, 2017

TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

Thursday, June 01, 2017

TSM SQL TDP Better Backup Script

On a call today I had an admin complain that the TSM SQL TDP was issuing failed backup errors when the script encountered a database that was offline. So I asked if he had searched Google for an alternate script and he had not. I did a quick query and found this nice batch file that appears to do exactly what is needed. The reader on the site commented that the script did not work on his SQL cluster. Does anyone have a similar script that would work on a cluster? Or could you modify this script to work in a cluster environment? The link to the script is provided below.

http://www.applepie.se/tivoli-storage-manager-and-sql-backup


Friday, May 05, 2017

TSM Menu - Bash Shell Scripting

For years I have maintained a shell script on our jump server that was a menu for accessing all the TSM servers in our environment. It was a nice interface and great for noobies. It allowed for you to set your password and ID variables before the menu loaded so anyone could run the menu under their ID, then presented them with a nice menu to pick which TSM server they wished to access. It was nice and easy. The only problem was when servers were added or decommissioned. When that occurred I would then have to go out and revise the menu. This was a pain and soon to be more painful as we plan to add a lot more TSM servers in the near future. I remember seeing someone post that they had created a menu system based off of the dsm.sys file so I decided to write my own.

Before:

 ################################################################## 
 #                                                                # 
 #                        Infocrossing                            # 
 #                    UNIX ADMINISTRATION                         # 
 #                         TSM  MENU                              # 
 #                                                                # 
 ################################################################## 
                            Omaha                                   
     0)   ouax000                         1)  ouax001               
     2)   ouax002                         3)  ouax003               
     4)   OTSM01                          5)  OTSM02                
 ------------------------------------------------------------------ 
                           Phoenix                                  
     6)   puax001                         7)  puax002               
     8)   puax008                         9)  PTSM01                
     10)  PTSM02                         11)  PTSM20                
 ------------------------------------------------------------------ 
                           Singapore                                
     20)  SIN01ZZ                        99)  STSM01                
 ------------------------------------------------------------------ 
                            Germany                                 
     21)  DTSM01                         22)  DTSM02                
     23)  DTSM03                         24)  DTSM20                
 ------------------------------------------------------------------ 
                            Kings Mtn                               
     25)  CTSM01                         26)  CTSM20                
     27)  CTSM30                         28)  CTSM40                
 ------------------------------------------------------------------ 
                                                                    
                     X/x)  Cross Server                             
                                                                    
                     S/s)  Cmd Prompt                               
                                                                    
                   DD/dd)  Data Domain Admin                        
                                                                    
                     Q/q)  Return to Main Menu                      
                                                                    
                                                                    
     Selection:                                                    

 After (Notice servers listed here that were not in above menu):

 ################################################################
 #                                                              #
 #                         Infocrossing                         #
 #                       Recovery Services                      #
 #                           TSM  MENU                          #
 #                                                              #
 ################################################################
                                                                 
 1) C1IF            12) LABTSM            23) PTSM01             
 2) C1KDC2          13) LUAX001           24) PTSM02             
 3) CTSM01          14) LUAX002           25) PTSM20             
 4) CTSM20          15) LTSM01            26) PTSM25-Arch        
 5) CTSM30          16) NTSM01            27) SIN01ZZ            
 6) CTSM40          17) OUAX003           28) STSM01             
 7) CTSM46-Arch     18) OUAX000           29) Cross Server       
 8) DTSM01          19) OUAX001           30) Refresh Menu       
 9) DTSM02          20) OTSM01            31) Exit               
10) DTSM03          21) OTSM02                                   
11) DTSM20          22) PESTSM01                                   
                                                                 
 Please enter your choice:     

Ok, so the Before is more organized by region but as new servers come in I wanted the menu to update dynamically. So this new menu is built around the BASH shell's select menu function. The script pulls all the TSM server names from the dsm.sys and populates an array that the select statement reads and generates this menu. Make sure the variables the script declares work for you and your dsm.sys. My dsm.sys looks like this if I grep servername:

SERVERNAME         DTSM01          * Germany      LIB-MAN
SERVERNAME         DTSM02          * Germany             
SERVERNAME         DTSM03          * Germany             
SERVERNAME         DTSM20          * Germany      LIB-MAN
SERVERNAME         CTSM01          * Kings_Mnt    LIB-MAN
SERVERNAME         C1KDC2          * Kings_Mnt           
SERVERNAME         C1IF            * Kings_Mnt           
SERVERNAME         CTSM20          * Kings_Mnt           
SERVERNAME         CTSM30          * Kings_Mnt           
SERVERNAME         CTSM40          * Kings_Mnt           
SERVERNAME         CTSM46-Arch     * Kings_Mnt           
SERVERNAME         LUAX001         * Leonia       LIB-MAN
SERVERNAME         LUAX002         * Leonia              
SERVERNAME         LTSM01          * Leonia              
SERVERNAME         NTSM01          * Norcross     LIB-MAN
SERVERNAME         OUAX000         * Omaha        LIB-MAN
SERVERNAME         OUAX001         * Omaha               
SERVERNAME         OUAX003         * Omaha               
SErvername         OTSM01          * Omaha        LIB-MAN
SErvername         OTSM02          * Omaha               
SERVERNAME         PESTSM01        * PEST         LIB-MAN
SERVERNAME         PTSM01          * Tempe        LIB-MAN
SERVERNAME         PTSM02          * Tempe               
SERVERNAME         PTSM20          * Tempe               
SERVERNAME         PTSM25-Arch     * Tempe               
SERVERNAME         STSM01          * Singapore           
SERVERNAME         SIN01ZZ         * Singapore           

SERVERNAME         LABTSM                                

With the dsm.sys file formated this way I can use grep and awk to get specific data. This allows the script to build my menus according to my needs. Especially with the Cross Server sub menu that allows me to run a command across multiple servers at once. In the sub-menu it groups by distinct location which I placed in the servername line. Leave me a comment if you have any comments, questions, or suggestions.

See the script below:

#!/bin/bash
#set -x

### Set Color Variables ###

NORMAL=`echo "\033[0m"`
WHITE=`echo "\033[37m"` #Blue
YELLOW=`echo "\033[33m"` #yellow
GREEN=`echo "\033[32m"`
RED=`echo "\033[31m"`
PURPLE=`echo "\033[35m"`

##### Declare Variables & Arrays ############

prompt="
 Please enter your choice: "

TSM_SYS="/opt/tivoli/tsm/client/ba/bin/dsm.sys"
all_servers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $2}' | sort`)
xservers=(`cat $TSM_SYS | grep -i servername | awk -v OFS='\t' '{print $4}' | sort -u`)
Germany=(`cat $TSM_SYS | grep -i servername | grep -i germany |awk -v OFS='\t' '{print $2}'`)
Omaha=(`cat $TSM_SYS | grep -i servername | grep -i omaha |awk -v OFS='\t' '{print $2}'`)
Norcross=(`cat $TSM_SYS | grep -i servername | grep -i norcross |awk -v OFS='\t' '{print $2}'`)
Tempe=(`cat $TSM_SYS | grep -i servername | grep -i tempe |awk -v OFS='\t' '{print $2}'`)
PEST=(`cat $TSM_SYS | grep -i servername | grep -i pest |awk -v OFS='\t' '{print $2}'`)
Leonia=(`cat $TSM_SYS | grep -i servername | grep -i leonia |awk -v OFS='\t' '{print $2}'`)
Kings_Mnt=(`cat $TSM_SYS | grep -i servername | grep -i kings_mnt |awk -v OFS='\t' '{print $2}'| sort`)
Singapore=(`cat $TSM_SYS | grep -i servername | grep -i singapore |awk -v OFS='\t' '{print $2}'`)
lib_man=(`cat $TSM_SYS | grep -i servername | grep -i lib-man | awk -v OFS='\t' '{print $2}'`)

export COLUMNS=70
########################

######################
function clearlogin
######################
{
echo "Please wait while I clear all login information."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
unset TSMID
unset TSMPA
unset DDID
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .16;done;echo;done;
echo "Logins have been cleared! "|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 1
}
######################
#  end clearlogin    #
######################

###############
#END MAIN-MENU#
###############

#########################
function tsmadmin
#########################
{
clear
if [ -z "${TSMID}" ]; then
echo ""
echo -e "$YELLOW Enter your TSM Admin ID: $GREEN \c"
read TSMID
echo ""
echo -e "$YELLOW Enter your TSM Admin Password: \c "
#stty -echo
read -s TSMPA
#stty $stty_orig
fi
clear
  tsm-menu
exit
}
####################
#  end tsmadmin    #
####################

#####################
function tsm-menu
#####################
{
all_done=0
export COLUMNS=70
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE #                       $YELLOW  Infocrossing                        $WHITE #"
echo -e "$WHITE #                       Recovery Services                      #"
echo -e "$WHITE #                          $RED TSM  MENU                         $WHITE #"
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#all_servers[@]} + 4))"`
PS3="$prompt "
 select host in "${all_servers[@]}" "Cross Server" "Reset Login" "Refresh Menu" "Exit"; do
   if [ "$host" = "Exit" ]; then
      all_done=1; clear; exit
   elif [ "$host" = "Refresh Menu" ]; then
      all_done=1; tsm-menu
   elif [ "$host" = "Reset Login" ]; then
      all_done=1; clearlogin; tsmadmin
   elif [ "$host" = "Cross Server" ]; then
      all_done=1; xmenu
   elif [ "$REPLY" -gt "$length" ]; then
      echo ""
      echo "You chose option $REPLY which is invalid. Try again! "
      echo ""
   else
       clear
       echo -e "$GREEN "
       dsmadmc -id=$TSMID -pa=$TSMPA -servern="$host"
       all_done=1; tsm-menu
   fi
 done
done
}
##########################
# end function tsm-menu  #
##########################

#####################
function xmenu
#####################
{
all_done=0
export COLUMNS=50
while ((!all_done )); do
clear
echo -e "$WHITE ################################################################$NORMAL "
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE #                       $YELLOW  Infocrossing                        $WHITE #"
echo -e "$WHITE #                       Recovery Services                      #"
echo -e "$WHITE #            $RED     TSM Cross Server Command MENU               $WHITE # "
echo -e "$WHITE #                                                              #$NORMAL "
echo -e "$WHITE ################################################################$NORMAL "
echo -e " "
length=`echo "$((${#xservers[@]} + 4))"`
PS3="$prompt "
 select host in "${xservers[@]}" "All Servers" "Library Managers" "Refresh Menu" "Exit"; do
   if [ "$host" = "Exit" ]; then
      all_done=1; tsm-menu
   elif [ "$host" = "Refresh Menu" ]; then
      all_done=1; xmenu
   elif [ "$host" = "Library Managers" ]; then
       clear
       echo -e "$GREEN "
       xserver $TSMID $TSMPA lib_man
       all_done=1; xmenu
   elif [ "$host" = "All Servers" ]; then
       clear
       echo -e "$GREEN "
       xserver $TSMID $TSMPA all_servers
       all_done=1; xmenu
   elif [ "$REPLY" -gt "$length" ]; then
       echo ""
       echo "You chose option $REPLY which is invalid. Try again! "
       echo ""
   else
       clear
       echo -e "$GREEN "
       xserver $TSMID $TSMPA "$host"
       all_done=1; xmenu
   fi
 done
done
}
##########################
# end function xmenu     #
##########################


###################
function xserver
###################
{
  DC=$3
  echo -e " "
  echo -e "     $YELLOW Enter the command to process: $NORMAL \c"
  read CMD
  echo ""
    echo -e "     $YELLOW Do you want to save the data to a commadelimited csv file? (Yes or No [N])  $NORMAL \c"
    read ANSWER
    case $ANSWER
     in
      y|Y|YES|Yes|yes)
       echo ""
       echo -e "     $YELLOW Enter the file name to save the data: $NORMAL \c"
       read FILE
       echo ""
       touch $FILE
       cat /dev/null > $FILE
        eval varAlias=\${$DC[@]}
        for Server in ${varAlias[@]}
        do
         echo ""
          echo -e "Processing TSM server $Server "
          echo ""
          dsmadmc -id=$1 -pa=$2 -dataonly=yes -commadelimited -servern=$Server $CMD | grep -v ANR | grep -v ANS >> $FILE
         echo ""
        done
       xmenu
      ;;
      n|N|*)
       echo ""
        eval varAlias=\${$DC[@]}
        for Server in ${varAlias[@]}
        do
         echo ""
          echo -e "Processing TSM server $Server "
          echo ""
          dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
         echo ""
        done
        sleep 2
       xmenu
      ;;
      *)
       echo ""
        eval varAlias=\${$DC[@]}
        for Server in ${varAlias[@]}
        do
         echo ""
          echo -e "Processing TSM server $Server "
          echo ""
          dsmadmc -id=$1 -pa=$2 -dataonly=yes -servern=$Server $CMD
         echo ""
        done
        sleep 2
       xmenu
      ;;
    esac
}
########################
# end xserver function #
########################
clear
echo -e "$GREEN"
echo "Please Wait Loading Menu....."|while read x;do for((i=0;i<${#x};i++));do echo -n "${x:$i:1}";sleep .03;done;echo;done;
sleep 2

tsmadmin