Monday, December 19, 2011

DB2 Doesn't Make A Difference

I've been working with some IBM reps/consultants lately, and I find it kind of funny how they talk about TSM. We were discussing the issue with some queries to the TSM DB being so hard to process that many times they don't return any data, when the IBM rep said "With DB2 that wont happen." I laughed and said, "DB2 didn't help that much." For example try something like this and see how long it takes to get a response.

select  cast(sum(b.file_size/1073741824) as decimal(18,2)) AS GB_SIZE from backups a, contents b where a.node_name in ('DEV01_ORA','DEV02_ORA','DEV03_ORA','PRD01_ORA','PROD02_ORA') and a.backup_date < '2011-11-01 00:00:00' and a.object_id=b.object_id

I'm running this query to determine the amount of space I would free up if I deleted old oracle backup objects that they DBA's never reconciled through RMAN. I ran it over 30 minutes ago.....still waiting! The problem is the schema has not changed enough in the TSM table structure to make some select statements run any better than in pre-DB2 days. Anyone else seen this? 

(Yes! I know if I used a specific NODE_NAME then TSM would probably return some data, but handles queries 1000x times more complex than these in the non-TSM world)

Tuesday, December 06, 2011

TSM 6.x Client Deployment

Has anyone used TSM's client deployment process/function?  How well does it work? Is it worth the effort? I have a lot of servers we need to install TSM too and would like to utilize it if it will work.

Wednesday, November 16, 2011

Tivoli Storage Manager Reporting and Monitoring v6.3

This is a query from the TSM v6.3 agent:

select node_name, count(distinct volume_name) from volumeusage a, stgpools b where (a.stgpool_name=b.stgpool_name) and devclass in (select DEVCLASS_NAME from devclasses where devtype in ('3570','3590','3592','4MM','8MM','DLT','DTF','ECARTRIDGE','GENERICTAPE','LTO','QIC')) group by node_name

Could you run it on a TSM v5.x.x.x productive system for me!

Do you get any result in 10 minutes?

Tuesday, October 18, 2011

Simple TSM 6.2 Server Restore

I just completed a DR test and we had to restore one of our TSM servers from a Data Domain replicated copy. This was our first time restoring a TSM server from a replicated DD copy and after importing the replicated volumes and defining our initiators we set about restoring the TSM database. Our AIX server had been restored from an image (SysBack) and we had a current volhist and devconfig file so we began our restore. If you think that the restore from a Data Domain is not relevant to your environment because you use tape, think again. The Data Domain mimics an STK library with IBM drives and so we had to follow the same directions as anyone using tape backup.

To restore the TSM 6.x DB from tape you must have your volhist and devconfig files. You will need to modify the devconfig so that the only lines are those defining the devclass, server name, and server password; all other lines should be deleted. Then you need lines defining a manual library, a tape drive, and a line defining a path to the drive (which for us was an LTO3 drive).


Note: Do not define an element address or serial with the drive, TSM will detect these when you run the DSMSERV RESTORE DB command.

When running the DSMSERV RESTORE DB command TSM will start up and query the devconfig file to retrieve the information on the devclass, drive, library type, server name, and password. Once TSM has successfully queried the tape drive it will query the volhist file for the most current DB backup volume depending on whether you are restoring to the most current date or to a specific point in time. When TSM has identified the volume to use it will prompt you to mount the tape. When I saw the mount I went into the Data Domain web based GUI and moved the DB backup volume from its "virtual slot" to the drive that is /dev/rmt1. Once the tape was mounted, TSM was able to recognize the tape had been loaded and began restoring the DB. If more than one tape is required to complete the restore TSM will prompt you for each tape. With the library web GUI available you can move the tapes as needed and accomplish the restore. Once the restore completes you can bring TSM back up and audit/fix anything that could be out of sync. With the switch to DB2 I was expecting a little more work to get TSM back up and running, but surprisingly it was quite simple.

Now if you don't have a SysBack of your TSM server the rebuild can take a lot longer and requires you to recreate some of the DB2 dependent files. I might have to do a BRM restore without an image in the near future and if I do I'll post a step by step process for everyone.  If anyone has already done this and would like to post the process on TSMAdmin let me know.

Sunday, October 02, 2011

TSM Symposium 2011, Dresden

Here's a picture:
Harry REDL, _flex, Norbert POTT

Thursday, September 15, 2011

Double The Trouble

So the TSMDBMGR stanza was accidentally removed from a TSM 6.x server and the log space filled because DB backups could not be run. So upon bringing TSM back up (which is a nightmare with DB2) TSM support let my coworker know she had to take two DB backups to clear the logs. OK, so I have heard this since 6.x came out but it turns out the reason is due to a bug in TSM that they have not cleared up. So IBM's solution for now is that two DB backups clear the logs. You think that would be fixed by now.....go figure!

Tuesday, August 30, 2011

Solaris Driver Issue

I have a Solaris (version 10) server that I am trying to configure LTO drives on for a Storage Agent. My problem is that I ran the procedure to assign the drives to the IBMtape driver but doc state a reboot is needed. Has anyone done a successful modunload/modload of the IBMtape driver thereby avoiding a reboot? The server is a production box and the DB takes 45 minutes to take down and come back up so reboots are usually scheduled and the next one is weeks off. Any ideas or help is appreciated.

Thursday, August 25, 2011

Coming Soon

Many moons ago I put together some videos running through a full DR of the TSM system running on a Windows platform. I deleted them about a year ago but plan to re-create them as a lot of people have been asking about them. Within the next month I'm doing a DR for one of our clients and will record the process for you to see. I know the way I've setup TSM will no doubt be different from yours, but I'll try and cut the videos in a way that's useful to all.

More to follow


Thursday, August 04, 2011

Archiving The Actlog

I just had a friend e-mail me and state he was being contacted by his previous employer complaining he made changes to their TSM retention times causing them to lose data. The TSM admin who took over is blaming him stating he changed the retention and now the data is gone. My friend does not remember making any retention changes and the problem is that any change to the copygroup updates the "Last Update by (administrator)" and the "Last Update Date/Time" so it's not substantial evidence of who did what. The only way to verify what had really occurred would be to either keep the actlog for an extremely long period of time, or dump it to a text file that you zip and archive. (Even then it's a text file and could be tampered with) In the case of my friend, he left the company last November and anyone could have altered the copygroup since then.

How many of you archive your TSM Activity Log, and how long do you keep it for? Obviously it good for security and tracking purposes, but who manages it and can you reliably keep it in a read-only state? Of course this is also a case where a bi-monthly audit of retention settings would have helped.

Thursday, July 28, 2011

Windows Admins & VSS

I've been running into a number of Windows Admins lately who are not familiar with the Microsoft Volume Shadow Copy Service. I guess you have to be an admin who works with backups to have any familiarity with it??? Is this common?  If you have admins who seem to lack knowledge of VSS I would recommend the following MS TechNet library reference. Lately I've been dealing with the object enumeration and missing system writer issues, if anyone has found a solution for the object enumeration issue I'm all ears!!!

Sunday, July 24, 2011

TSM v6.x

I'm curious to find out your opinion about the TSM v6 published a few years ago.

Do you use v6 at all, are you afraid to use it? what about deduplication or simple you are just satisfied with v5?

And of course what about the ITM and ISC/AC/TIP whatever this is called?

I'm really interested in your view!


/* I opened the same topic at: */

Saturday, July 23, 2011

This is to let you know v1beta RC2 has been published.

In brief, this program will make your daily TSM work easier by using the entire terminal size with colors and by redefining the existing commands or by extending the original ones.

In addition, the program can handle historical/archive data as well!!!

See details at:

Will you try it and send us your experiences or suggestions?

You can also contact us via several other ways:


TSM Symposium 2011

Don't forget: TSM Symposium 2011:
( 27th-30th September 2011 at the Hilton Hotel, Dresden )

Thursday, June 23, 2011

What Problem? It's Working As Designed!

When upgrading from TSM 5.x to 6.x you have probably found that your select queries don't return in table format but in list format. You ask yourself, "WTH is going on here?" First off don't complain to IBM, it's "working as designed." You see with the change of the TSM DB to DB2 many of table column widths have changed. To have your select statements return in table format IBM states that you should use the CAST function on certain (if not all) columns. Otherwise, the best alternative is to save the results in commadelimited format and open the data in Excel. Here's the APAR??? that states it's not a problem. To quote Lady Gaga, TSM 6.x was "Born This Way!"

NOTE: Is it me or does IBM's last example leave one scratching their head wondering how using cast helped???

Tuesday, June 21, 2011

Memory Efficient Backup

I have come across a few servers that have memory issues when backing up large drives / filesystems and using MEMORYEFFICIENTBACKUP was not resolving the backup failures. The error code I would see was

ANS1030E The operating system refused a TSM request for memory allocation.

The problem was one drive that has millions of small files all under one directory. It's one of those poorly devised custom programs, and I've seen it a hundred times.  I could journal the drive, but we seemed to get better results with an alternative method.  A coworker reminded me of the INCLUDE.FS option using DISKCACHE and specifying the cache location.


With the include.fs TSM will only use the memoryefficientbackup diskcache method for that particular drive, allowing the rest of the backup to perform normally. So if you have a drive or filesystem with millions of files this is an alternative to journaling.

Tuesday, June 14, 2011

Drive Matching

While working with a VTL I realized how much I hated matching drive definitions on a LAN-Free client or remote library client and decided to do something about it. So after searching the web and finding a helpful script to pull rmt and serial numbers from the tape drives discovered on the LAN-Free or Lib-Client I worked it into an AIX shell script that will match the drives rmt with the correct drive definition on the library manager. The key to this working is to have the library manager listed in the lib-clients dsm.sys and to define the following TSM script on the library manager.

-=-=-=-=-=-=-TSM Server Script-=-=-=-=-=-=-=-

def script drive_match desc="This script is used by a shell script to match drives to rmts to help create DEFINE PATH statements"
upd script drive_match "select 'define path $1 ' || drive_name || ' srct=server destt=drive libr=' || library_name || ' device=/dev/$2 -"
upd script drive_match " online=yes' from drives where DRIVE_SERIAL='$3' "

-=-=-=-=-=-=-UNIX Shell Script-=-=-=-=-=-=-=-

ID=`cat /home/tsmadmin/VARS/ADSMID`
PA=`cat /home/tsmadmin/VARS/ADSMPA`
touch ./drv_match.mac
cat /dev/null > ./drv_match.mac
echo "This script will build the DEFINE PATH statements for a server"
echo ""
echo "Enter the Library Manager Name: \c"
read SERV
echo ""
echo "What is the TSM server name these paths are going to be generated for: \c"
echo ""
lsdev |grep fscsi | while read FSCS rest
   TAPES=$(lsdev -p $FSCS | grep rmt | grep -v ALT |cut -f1 -d" ")
     for TAPE in $TAPES
         SERIAL=$(lscfg -vl $TAPE | grep Serial | awk -F"." '{printf "%d", $NF}')
         FCS=$(echo $FSCS | sed 's/fscsi/fcs/')
         dsmadmc -id=$ID -pa=$PA -dataonly=yes -commadelimited -servern=$SERV run drive_match $TSMSRV $TAPE $SERIAL | grep -v ANR >> ./drv_match.mac
cat ./drv_match.mac

-=-=-=-=-End Script-=-=-=-=-=-

You'll notice I save the output as a macro so it can be reviewed before executed. If you'd like to run the macro automatically then just add a dsmadmc line at the end.

The following is some sample output:

define path tsm8 VTL3_00332 srct=server destt=drive libr=VTL3 device=/dev/rmt88 online=yes
define path tsm8 VTL3_00336 srct=server destt=drive libr=VTL3 device=/dev/rmt89 online=yes
define path tsm8 VTL3_00340 srct=server destt=drive libr=VTL3 device=/dev/rmt90 online=yes
define path tsm8 VTL4_00444 srct=server destt=drive libr=VTL4 device=/dev/rmt91 online=yes

Friday, June 10, 2011

TSM Server V6 AIX Install/Upgrade Gotchas

So after setting up numerous TSM 6.1 and 6.2 servers here are a few of the things that have been little gotchas. They were never upgrade stoppers but they did cause some headaches as we determined what was causing the errors.

  • Check the ATAPE version (recommend 11.x)
  • Check your xlC C++ runtime level (recommend or greater)
  • With AIX you must have IOCP set to available or else you'll have to update the OS setting and reboot the server.
  • Make sure the user id that the TSM 6.x instance is running under has ulimit set to unlimited. Real pain when you go to create disk volumes and you forgot to set the ulimit. It was my absentminded moment!
  • Don't forget that the tsmdbmgr.log file ownership needs to be the new ID not root.
  • Also when using RAW disk volumes for TSM diskpools chown the device file (example: /dev/rtsmdata01) to the new user id or TSM will say it's unavailable.
  • With a recent upgrade we could not get the TSM DB backup to execute without an error. It turned out the TSM client's dsmtca file ownership had been accidentally changed to the TSM server's user ID and it MUST BE OWNED BY ROOT for the backup of the TSM DB to execute successfully.
If you have anything you've experienced you consider a "GOTCHA" , whether it be UNIX or Windows, then leave a comment and help others who might run across the same issue(s).

    Tuesday, May 31, 2011

    Where, how and why are tapes still used for backup?

    Not strictly a TSM only topic, but this fascinating thread has been raging on over at LinkedIn for the last few days and I thought it would be worth sharing here. The provocative question asked was:
    "Is tape still used for backup? Why is tape still being used these days when disk and cloud are available?"

    Cue an avalanche of fascinating pro- and anti-tape pitches and opinions. Wading through the comments may take a while (I pitched-in with a couple of comments too) but it's a useful read as, frankly, these are the questions that clients/customers/decision-makers - and indeed ourselves - should be asking to ensure that we continue to use the right technology for the right purpose.

    Of course, TSM itself is well-positioned for in many ways given its long-running support of disk, both in random access and more recently in sequential file (virtual tapes!).

    David Mc
    London, UK

    Thursday, May 26, 2011

    Get Device Info From AIX

    I needed to gather the serial numbers by device (rmt) in AIX for our VTL, and the grep command in AIX is not as functional as my Linux box at home, so the -A and -B options are not available. Those two flags allow you to grab X lines before and after the grep'ed expression which is really handy when running lscfg -vp. Here's a grep command to gather just the rmt(s) with their serial and WWN. If you need to change the number of lines just change the a (after) and b (before) value, and s is what is "grep'ed".

    lscfg -vp | awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=1 a=5 s="rmt"

    Wednesday, May 18, 2011

    Getting TSM Script Info

    So you want to query a script and see what it executes but don't like the standard Q SCRIPT results? Don't forget about the FORMAT= options available. Most people know FORMAT=DETAIL but with the Q SCRIPT command there are two additional formats. FORMAT=LINE and FORMAT=RAW. I like RAW but LINE is decent if you want to see the script by sequence.

    UPDATE: As reader Erwann reminded me (I had long forgotten), there is also a F=MACRO option which will display the script in a form that can be run as a macro to define the script to another TSM server. If you issue Help Q SCRIPT it does not list the MACRO option, but it is valid and does work. So you could dump all your scripts to macro format and send it to an outfile. Then you could define them to an alternate TSM server (Q SCRIPT * OUTFILE=SCRIPTS.MAC F=Macro).

    tsm: TSMLIBM>q script LIBR_INFO f=macro
    DEFINE SCRIPT  LIBR_INFO DESC="Library path information"
    UPDATE SCRIPT  LIBR_INFO "q libr $1"
    UPDATE SCRIPT  LIBR_INFO "q libr $1 f=d"
    UPDATE SCRIPT  LIBR_INFO "q path $1 srct=serv destt=libr f=d"
    UPDATE SCRIPT  LIBR_INFO "q path * srct=serv destt=dr libr=$1 f=d"
    UPDATE SCRIPT  LIBR_INFO "select library_name,library_type,library_serial from libraries where library_name ='$1'"
    UPDATE SCRIPT  LIBR_INFO "select source_name,destination_name,library_name,device serial from paths where library_name='$1'"
    UPDATE SCRIPT  LIBR_INFO "select drive_name,drive_serial,element,library_name from drives where library_name='$1'"

    Here are the examples:

    tsm: TSMLIBM>q script LIBR_INFO f=line   

    Name           Line       Command                                                     
    ----------     ------     ------------------------------------------------------------
    LIBR_INFO      5          q libr $1                                                   
                   10         q libr $1 f=d                                               
                   15         q path $1 srct=serv destt=libr f=d                          
                   20         q path * srct=serv destt=dr libr=$1 f=d                     
                   25         select library_name,library_type,library_serial from        
                               libraries where library_name ='$1'                         
                   30         select source_name,destination_name,library_name,device     
                               serial from paths where library_name='$1'                  
                   35         select drive_name,drive_serial,element,library_name from    
                               drives where library_name='$1'             

    tsm: TSMLIBM>q script LIBR_INFO f=raw
    q libr $1
    q libr $1 f=d
    q path $1 srct=serv destt=libr f=d
    q path * srct=serv destt=dr libr=$1 f=d
    select library_name,library_type,library_serial from libraries where library_name ='$1'
    select source_name,destination_name,library_name,device serial from paths where library_name='$1'
    select drive_name,drive_serial,element,library_name from drives where library_name='$1'

    Friday, April 29, 2011

    Delete Old Filespaces

    Been looking to do some cleanup and I wanted an easy way to do it. So...I wrote this script. I can take the list returned and either make it a macro or an internal TSM script. Sure I could build a shell script that processed a list with the TSM server_name, the node_name, and FSID but I thought I'd show how you can create a command out of SQL returned data.

    select 'del filespace ' || node_name || ' ' || cast(FILESPACE_ID as char(3)) || ' type=backup nametype=fsid wait=yes' AS COMMAND from filespaces where cast((current_timestamp-backup_end)days as decimal)>90

    Here is an example of the results from the script:

    del filespace NODE1 37  type=backup nametype=fsid wait=yes
    del filespace NODE1 9   type=backup nametype=fsid wait=yes
    del filespace NODE1 36  type=backup nametype=fsid wait=yes
    del filespace NEO_1 32  type=backup nametype=fsid wait=yes
    del filespace NEO_1 31  type=backup nametype=fsid wait=yes
    del filespace NEO_1 23  type=backup nametype=fsid wait=yes
    del filespace NEO_1 29  type=backup nametype=fsid wait=yes

    Tuesday, April 26, 2011

    TSM 6.2 GUID Error

    So when I and a co-worker were finishing up a TSM 6.2 install we ran into a problem running the tsmapipw command to set the $$_TSMDBMGR_$$ password. After investigating the error and search the web we realized the GUID was the problem and not our IP as was suggested in the following APAR.  The errors we received were:

    ANR2987W Session ended because of machine GUID or local host IP
     address mismatch.
    ANR0430W Session XXXX for node $$_TSMDBMGR_$$ (AIX) refused
     - node name is locked.

    We tried various things and nothing seemed to correct the GUID issue. The TSM server started without any GUID related error (in fact it stated the GUID initialized successfully). So upon further searching we found this APAR that gives the solution.  What they state is that by adding the following line to the dsmserv.opt and restarting the server fixes the problem.


    Once we added the line and restarted the TSM server the tsmapipw command ran successfully.

    Thursday, February 03, 2011

    Send CSV File As Attachment

    Have a report that is too big to be column output and don't like the list look of TSM 6.1? Sure you could reformat you comma delimited file for output but I'd rather send it as an attachment in their mail and let them open it in Excel. So if you have access to a Linux/UNIX server this is a simple solution I used recently and it worked great.

    uuencode /tmp/tsm/stg_backup_rpt.csv /tmp/tsm/stg_backup_rpt.csv | mailx -s "TSM Servers Backup STGPool Status" ","

    There are other ways with other mail handlers (i.e. mail, mailx, sendmail, etc.) but this one was easy and worked.

    Tuesday, January 11, 2011

    Select X Days Back

    As you all know I have been writing and rewriting scripts for my 6.x servers and have been experiencing interesting results when I use the timestampdiff option when determining how far back a script should look. So when running my summary report query I was getting inaccurate results when using the following script:

    example 1:
    select schedule_name, entity, start_time, end_time, timestampdiff(16, char(end_time-start_time)) || ' ' || timestampdiff(8, char(end_time-start_time)) || ':' || timestampdiff(8, char(end_time-start_time)) || ':' || substr(cast((end_time-start_time) as char(22)),13,2) AS ELAPSED_TIME, affected from summary where activity='STGPOOL BACKUP' and timestampdiff(8, char(current_timestamp-start_time))<=24

    This script would not return all the STGPOOL BACKUP jobs from the summary table. Not sure why, but I have a couple ideas. So, I found that the following is easier to use and returned the correct results.

    example 2:
    select schedule_name, entity, start_time, end_time, timestampdiff(16, char(end_time-start_time)) || ' ' || timestampdiff(8, char(end_time-start_time)) || ':' || timestampdiff(8, char(end_time-start_time)) || ':' || substr(cast((end_time-start_time) as char(22)),13,2) AS ELAPSED_TIME, affected from summary where activity='STGPOOL BACKUP' and start_time>(CURRENT TIMESTAMP - 10 DAYS)

    So whenever you would like to query X days back I suggest example 2 for ease of use.