TSMAdmin

TSM Backup Issue

2012-01-23T10:49:00.000-08:00

Anyone had an issue where their backups were extremely slow and their Interrupts were huge? I've got 400GB DB's taking 40hrs to backup over a 4 port Ether-channel connection. No errors in my AIX errpt and the network guys are telling me they don't think it's them. Any suggestions on what to look at are appreciated. Below is an example when I run entstat.

ETHERNET STATISTICS (en8) :

Device Type: IEEE 802.3ad Link Aggregation

Hardware Address: 00:14:5e:e7:26:41

Elapsed Time: 9 days 19 hours 20 minutes 35 seconds

Transmit Statistics: Receive Statistics:

-------------------- -------------------

Packets: 5470416553 Packets: 24510516113

Bytes: 440661650021 Bytes: 32245892708954

Interrupts: 0 Interrupts: 6027433898

Transmit Errors: 0 Receive Errors: 691

Packets Dropped: 0 Packets Dropped: 0

Bad Packets: 0

Max Packets on S/W Transmit Queue: 298

S/W Transmit Queue Overflow: 0

Current S/W+H/W Transmit Queue Length: 355

Broadcast Packets: 8786 Broadcast Packets: -1346793420

Multicast Packets: 225928 Multicast Packets: 136913

No Carrier Sense: 0 CRC Errors: 0

DMA Underrun: 0 DMA Overrun: 691

Lost CTS Errors: 0 Alignment Errors: 0

Max Collision Errors: 0 No Resource Errors: 0

Late Collision Errors: 0 Receive Collision Errors: 0

Deferred: 141004 Packet Too Short Errors: 0

SQE Test: 0 Packet Too Long Errors: 0

Timeout Errors: 0 Packets Discarded by Adapter: 0

Single Collision Count: 0 Receiver Start Count: 0

Multiple Collision Count: 0

Current HW Transmit Queue Length: 355

General Statistics:

-------------------

No mbuf Errors: 0

Adapter Reset Count: 0

Adapter Data Rate: 1701737521

Driver Flags: Up Broadcast Running

Simplex 64BitSupport ChecksumOffload

PrivateSegment LargeSend DataRateSet

TSM Device Handling in Windows

2012-01-19T12:13:00.000-08:00

I have to say that TSM on Windows is good for small to medium size solutions and I'm not ANTI-Windows. I just cringe when dealing with devices in Windows. I hate its driver handling and most of all I hate how Windows presents library and tape drives. So I was working with a TSM server where the tape library would not initialize. It was an older SCSI library, not Fiber. I tried restarting the library, the TSM server, reloading drivers, and updating the drivers and nothing worked.

Duh! <Head Slap!!!>

That's because on the initial reboot that caused the library to stop communicating the device ID's changed. So the library went from LB1.0.0.2 to LB1.0.0.3. Nobody touched the SCSI card or library but the device definition changed! Seriously? All the drives changed to mtX.X.X.3 also. Now I don't use Windows all that much but luckily I remembered the TSMDLST program that is installed with the TSM server. It's under C:\Porgram Files\tivoli\tsm\console and will pull the information from Windows for you in a readable format. So next time your library goes offline make sure you use it to compare the device definitions and serials with what is defined in TSM. It will save you a lot of time and headache. You can find more information on issues like this here.

TSM Client Scheduler Issue

2012-01-18T08:19:00.000-08:00

I just recently had an issue with a handful of TSM clients that would not run their backups. The clients all backup to a TSM 5.5.2 server and were all running Windows 2008. The clients use TSM version 6.2.3. The five clients had all been missing their backups for days and what makes the situation more interesting is that there are other Windows 2008 servers with this version of TSM installed and they are all running their schedules without issue.

When reviewing the TSM Schedule log the scheduler listed that it had received the schedule info and was waiting for the TSM server to initiate the schedule. The TSM server never made an attempt to contact the clients in question and never showed any errors other than ANR2578W stating the client missed its schedule. There were no errors in the error log and not much to go by from the TSM server activity log. Even though the TSM client backs up over the public network I switched to polling mode to see if client based initiation of the backup would work. It didn't! The TSM client scheduler would receive the schedule upon polling the TSM server but would never execute it. So now what? I added the TCPCLIENTADDRESS and TCPCLIENTPORT and switched back to SCHEDMODE PROMPTED, still the scheduler would not run backups.

Now I was getting frustrated. I removed the scheduler service and redefined it using dsmcutil and voila, the schedule ran...ONCE! After the initial schedule ran the previous problem returned. Schedules were not running and the TSM server would not show any errors saying it could not contact the client. It just would not run the schedule. Well that left me no choice but to call support. IBM support's response was to make sure the TCPCLIENTADDRESS and TCPCLIENTPORT were defined in the dsm.opt and also to define the client HLADDRESS and LLADDRESS on the TSM server? Define the HL and LL addess? TSM gets that when the client connects doesn't it? Yes and No! It appears that without the optional setting the TSM server can have issues contacting some clients. Why? No idea, but adding the HL and LL address did the trick and the backups have been running without issue since.

How many of you define the HL and LLADDRESS when registering nodes? I've never suspected it was needed until now.

DB2 Doesn't Make A Difference

2011-12-19T10:00:00.000-08:00

I've been working with some IBM reps/consultants lately, and I find it kind of funny how they talk about TSM. We were discussing the issue with some queries to the TSM DB being so hard to process that many times they don't return any data, when the IBM rep said "With DB2 that wont happen." I laughed and said, "DB2 didn't help that much." For example try something like this and see how long it takes to get a response.

select cast(sum(b.file_size/1073741824) as decimal(18,2)) AS GB_SIZE from backups a, contents b where a.node_name in ('DEV01_ORA','DEV02_ORA','DEV03_ORA','PRD01_ORA','PROD02_ORA') and a.backup_date < '2011-11-01 00:00:00' and a.object_id=b.object_id

I'm running this query to determine the amount of space I would free up if I deleted old oracle backup objects that they DBA's never reconciled through RMAN. I ran it over 30 minutes ago.....still waiting! The problem is the schema has not changed enough in the TSM table structure to make some select statements run any better than in pre-DB2 days. Anyone else seen this?

(Yes! I know if I used a specific NODE_NAME then TSM would probably return some data, but handles queries 1000x times more complex than these in the non-TSM world)

TSM 6.x Client Deployment

2011-12-06T20:51:00.001-08:00

Has anyone used TSM's client deployment process/function? How well does it work? Is it worth the effort? I have a lot of servers we need to install TSM too and would like to utilize it if it will work.

Tivoli Storage Manager Reporting and Monitoring v6.3

2011-11-16T05:13:00.001-08:00

This is a query from the TSM v6.3 agent:

select node_name, count(distinct volume_name) from volumeusage a, stgpools b where (a.stgpool_name=b.stgpool_name) and devclass in (select DEVCLASS_NAME from devclasses where devtype in ('3570','3590','3592','4MM','8MM','DLT','DTF','ECARTRIDGE','GENERICTAPE','LTO','QIC')) group by node_name

Could you run it on a TSM v5.x.x.x productive system for me!

Do you get any result in 10 minutes?

Simple TSM 6.2 Server Restore

2011-10-18T09:57:00.000-07:00

I just completed a DR test and we had to restore one of our TSM servers from a Data Domain replicated copy. This was our first time restoring a TSM server from a replicated DD copy and after importing the replicated volumes and defining our initiators we set about restoring the TSM database. Our AIX server had been restored from an image (SysBack) and we had a current volhist and devconfig file so we began our restore. If you think that the restore from a Data Domain is not relevant to your environment because you use tape, think again. The Data Domain mimics an STK library with IBM drives and so we had to follow the same directions as anyone using tape backup.

To restore the TSM 6.x DB from tape you must have your volhist and devconfig files. You will need to modify the devconfig so that the only lines are those defining the devclass, server name, and server password; all other lines should be deleted. Then you need lines defining a manual library, a tape drive, and a line defining a path to the drive (which for us was an LTO3 drive).

DEFINE LIBRARY MANLIB LIBTYPE=MANUAL
DEFINE DRIVE MANLIB DRIVE1 ONLINE=YES
DEFINE PATH TSMSERV1 DRIVE1 SRCT=SERVER DESTT=DRIVE LIBR=MANLIB DEVICE=/dev/rmt1 ONLINE=YES

Note: Do not define an element address or serial with the drive, TSM will detect these when you run the DSMSERV RESTORE DB command.

When running the DSMSERV RESTORE DB command TSM will start up and query the devconfig file to retrieve the information on the devclass, drive, library type, server name, and password. Once TSM has successfully queried the tape drive it will query the volhist file for the most current DB backup volume depending on whether you are restoring to the most current date or to a specific point in time. When TSM has identified the volume to use it will prompt you to mount the tape. When I saw the mount I went into the Data Domain web based GUI and moved the DB backup volume from its "virtual slot" to the drive that is /dev/rmt1. Once the tape was mounted, TSM was able to recognize the tape had been loaded and began restoring the DB. If more than one tape is required to complete the restore TSM will prompt you for each tape. With the library web GUI available you can move the tapes as needed and accomplish the restore. Once the restore completes you can bring TSM back up and audit/fix anything that could be out of sync. With the switch to DB2 I was expecting a little more work to get TSM back up and running, but surprisingly it was quite simple.

Now if you don't have a SysBack of your TSM server the rebuild can take a lot longer and requires you to recreate some of the DB2 dependent files. I might have to do a BRM restore without an image in the near future and if I do I'll post a step by step process for everyone. If anyone has already done this and would like to post the process on TSMAdmin let me know.

TSM Symposium 2011, Dresden

2011-10-02T11:15:00.000-07:00

Here's a picture:
Harry REDL, _flex, Norbert POTT

Double The Trouble

2011-09-15T09:27:00.000-07:00

So the TSMDBMGR stanza was accidentally removed from a TSM 6.x server and the log space filled because DB backups could not be run. So upon bringing TSM back up (which is a nightmare with DB2) TSM support let my coworker know she had to take two DB backups to clear the logs. OK, so I have heard this since 6.x came out but it turns out the reason is due to a bug in TSM that they have not cleared up. So IBM's solution for now is that two DB backups clear the logs. You think that would be fixed by now.....go figure!

Solaris Driver Issue

2011-08-30T10:34:00.000-07:00

I have a Solaris (version 10) server that I am trying to configure LTO drives on for a Storage Agent. My problem is that I ran the procedure to assign the drives to the IBMtape driver but doc state a reboot is needed. Has anyone done a successful modunload/modload of the IBMtape driver thereby avoiding a reboot? The server is a production box and the DB takes 45 minutes to take down and come back up so reboots are usually scheduled and the next one is weeks off. Any ideas or help is appreciated.

Free Generator of Barcode Tape Labels for Ultrium LTO Backup Libraries

2011-08-26T06:04:00.000-07:00

If you haven't heard about it: Free Generator of Barcode Tape Labels for Ultrium LTO Backup Libraries

Coming Soon

2011-08-25T01:33:00.000-07:00

Many moons ago I put together some videos running through a full DR of the TSM system running on a Windows platform. I deleted them about a year ago but plan to re-create them as a lot of people have been asking about them. Within the next month I'm doing a DR for one of our clients and will record the process for you to see. I know the way I've setup TSM will no doubt be different from yours, but I'll try and cut the videos in a way that's useful to all.

More to follow

Cheers

Archiving The Actlog

2011-08-04T10:33:00.000-07:00

I just had a friend e-mail me and state he was being contacted by his previous employer complaining he made changes to their TSM retention times causing them to lose data. The TSM admin who took over is blaming him stating he changed the retention and now the data is gone. My friend does not remember making any retention changes and the problem is that any change to the copygroup updates the "Last Update by (administrator)" and the "Last Update Date/Time" so it's not substantial evidence of who did what. The only way to verify what had really occurred would be to either keep the actlog for an extremely long period of time, or dump it to a text file that you zip and archive. (Even then it's a text file and could be tampered with) In the case of my friend, he left the company last November and anyone could have altered the copygroup since then.

How many of you archive your TSM Activity Log, and how long do you keep it for? Obviously it good for security and tracking purposes, but who manages it and can you reliably keep it in a read-only state? Of course this is also a case where a bi-monthly audit of retention settings would have helped.

Funny TSM v6.2.2.0 install message ;-)

2011-08-02T09:07:00.000-07:00

Windows Admins & VSS

2011-07-28T14:32:00.000-07:00

I've been running into a number of Windows Admins lately who are not familiar with the Microsoft Volume Shadow Copy Service. I guess you have to be an admin who works with backups to have any familiarity with it??? Is this common? If you have admins who seem to lack knowledge of VSS I would recommend the following MS TechNet library reference. Lately I've been dealing with the object enumeration and missing system writer issues, if anyone has found a solution for the object enumeration issue I'm all ears!!!

TSM v6.x

2011-07-24T12:47:00.000-07:00

I'm curious to find out your opinion about the TSM v6 published a few years ago.

Do you use v6 at all, are you afraid to use it? what about deduplication or simple you are just satisfied with v5?

And of course what about the ITM and ISC/AC/TIP whatever this is called?

I'm really interested in your view!

Cheers.

/* I opened the same topic at: adsm.org. */

tsmadm.pl

2011-07-23T07:51:00.001-07:00

This is to let you know tsmadm.pl v1beta RC2 has been published.

In brief, this program will make your daily TSM work easier by using the entire terminal size with colors and by redefining the existing commands or by extending the original ones.

In addition, the program can handle historical/archive data as well!!!

See details at: tsmadm.pl

Will you try it and send us your experiences or suggestions?

You can also contact us via several other ways:

SIGNUP FOR NEWSLETTER | USE OUR FORUM | FOLLOW US ON TWITTER | JOIN OUR FACEBOOK FANPAGE | SUPPORT

TSM Symposium 2011

2011-07-23T07:44:00.000-07:00

Don't forget: TSM Symposium 2011: http://tsm2011.uni-koeln.de/
( 27th-30th September 2011 at the Hilton Hotel, Dresden )

What Problem? It's Working As Designed!

2011-06-23T09:49:00.000-07:00

When upgrading from TSM 5.x to 6.x you have probably found that your select queries don't return in table format but in list format. You ask yourself, "WTH is going on here?" First off don't complain to IBM, it's "working as designed." You see with the change of the TSM DB to DB2 many of table column widths have changed. To have your select statements return in table format IBM states that you should use the CAST function on certain (if not all) columns. Otherwise, the best alternative is to save the results in commadelimited format and open the data in Excel. Here's the APAR??? that states it's not a problem. To quote Lady Gaga, TSM 6.x was "Born This Way!"

NOTE: Is it me or does IBM's last example leave one scratching their head wondering how using cast helped???

Memory Efficient Backup

2011-06-21T11:47:00.000-07:00

I have come across a few servers that have memory issues when backing up large drives / filesystems and using MEMORYEFFICIENTBACKUP was not resolving the backup failures. The error code I would see was

ANS1030E The operating system refused a TSM request for memory allocation.

The problem was one drive that has millions of small files all under one directory. It's one of those poorly devised custom programs, and I've seen it a hundred times. I could journal the drive, but we seemed to get better results with an alternative method. A coworker reminded me of the INCLUDE.FS option using DISKCACHE and specifying the cache location.

INCLUDE.FS G: MEMORYEFFICIENTBACKUP=DISKCACHEMETHOD DISKCACHELOCATION=L:\TSM_cache

With the include.fs TSM will only use the memoryefficientbackup diskcache method for that particular drive, allowing the rest of the backup to perform normally. So if you have a drive or filesystem with millions of files this is an alternative to journaling.

Drive Matching

2011-06-14T10:08:00.000-07:00

While working with a VTL I realized how much I hated matching drive definitions on a LAN-Free client or remote library client and decided to do something about it. So after searching the web and finding a helpful script to pull rmt and serial numbers from the tape drives discovered on the LAN-Free or Lib-Client I worked it into an AIX shell script that will match the drives rmt with the correct drive definition on the library manager. The key to this working is to have the library manager listed in the lib-clients dsm.sys and to define the following TSM script on the library manager.

-=-=-=-=-=-=-TSM Server Script-=-=-=-=-=-=-=-

def script drive_match desc="This script is used by a shell script to match drives to rmts to help create DEFINE PATH statements"
upd script drive_match "select 'define path $1 ' || drive_name || ' srct=server destt=drive libr=' || library_name || ' device=/dev/$2 -"
upd script drive_match " online=yes' from drives where DRIVE_SERIAL='$3' "

-=-=-=-=-=-=-UNIX Shell Script-=-=-=-=-=-=-=-

#!/bin/ksh
ID=`cat /home/tsmadmin/VARS/ADSMID`
PA=`cat /home/tsmadmin/VARS/ADSMPA`
touch ./drv_match.mac
cat /dev/null > ./drv_match.mac
clear
echo "This script will build the DEFINE PATH statements for a server"
echo ""
echo "Enter the Library Manager Name: \c"
read SERV
echo ""
echo "What is the TSM server name these paths are going to be generated for: \c"
read TSMSRV
echo ""
lsdev |grep fscsi | while read FSCS rest
do
TAPES=$(lsdev -p $FSCS | grep rmt | grep -v ALT |cut -f1 -d" ")
for TAPE in $TAPES
do
SERIAL=$(lscfg -vl $TAPE | grep Serial | awk -F"." '{printf "%d", $NF}')
FCS=$(echo $FSCS | sed 's/fscsi/fcs/')
dsmadmc -id=$ID -pa=$PA -dataonly=yes -commadelimited -servern=$SERV run drive_match $TSMSRV $TAPE $SERIAL | grep -v ANR >> ./drv_match.mac
done
done
cat ./drv_match.mac

-=-=-=-=-End Script-=-=-=-=-=-

You'll notice I save the output as a macro so it can be reviewed before executed. If you'd like to run the macro automatically then just add a dsmadmc line at the end.

The following is some sample output:

define path tsm8 VTL3_00332 srct=server destt=drive libr=VTL3 device=/dev/rmt88 online=yes
define path tsm8 VTL3_00336 srct=server destt=drive libr=VTL3 device=/dev/rmt89 online=yes
define path tsm8 VTL3_00340 srct=server destt=drive libr=VTL3 device=/dev/rmt90 online=yes
define path tsm8 VTL4_00444 srct=server destt=drive libr=VTL4 device=/dev/rmt91 online=yes

TSM Server V6 AIX Install/Upgrade Gotchas

2011-06-10T10:09:00.000-07:00

So after setting up numerous TSM 6.1 and 6.2 servers here are a few of the things that have been little gotchas. They were never upgrade stoppers but they did cause some headaches as we determined what was causing the errors.

Check the ATAPE version (recommend 11.x)
Check your xlC C++ runtime level (recommend 9.0.0.8 or greater)
With AIX you must have IOCP set to available or else you'll have to update the OS setting and reboot the server.
Make sure the user id that the TSM 6.x instance is running under has ulimit set to unlimited. Real pain when you go to create disk volumes and you forgot to set the ulimit. It was my absentminded moment!
Don't forget that the tsmdbmgr.log file ownership needs to be the new ID not root.
Also when using RAW disk volumes for TSM diskpools chown the device file (example: /dev/rtsmdata01) to the new user id or TSM will say it's unavailable.
With a recent upgrade we could not get the TSM DB backup to execute without an error. It turned out the TSM client's dsmtca file ownership had been accidentally changed to the TSM server's user ID and it MUST BE OWNED BY ROOT for the backup of the TSM DB to execute successfully.

If you have anything you've experienced you consider a "GOTCHA" , whether it be UNIX or Windows, then leave a comment and help others who might run across the same issue(s).

Where, how and why are tapes still used for backup?

2011-05-31T04:30:00.000-07:00

Not strictly a TSM only topic, but this fascinating thread has been raging on over at LinkedIn for the last few days and I thought it would be worth sharing here. The provocative question asked was:

"Is tape still used for backup? Why is tape still being used these days when disk and cloud are available?"

Link: http://linkd.in/lo6VnM

Cue an avalanche of fascinating pro- and anti-tape pitches and opinions. Wading through the comments may take a while (I pitched-in with a couple of comments too) but it's a useful read as, frankly, these are the questions that clients/customers/decision-makers - and indeed ourselves - should be asking to ensure that we continue to use the right technology for the right purpose.

Of course, TSM itself is well-positioned for in many ways given its long-running support of disk, both in random access and more recently in sequential file (virtual tapes!).

David Mc

London, UK

Get Device Info From AIX

2011-05-26T09:45:00.000-07:00

I needed to gather the serial numbers by device (rmt) in AIX for our VTL, and the grep command in AIX is not as functional as my Linux box at home, so the -A and -B options are not available. Those two flags allow you to grab X lines before and after the grep'ed expression which is really handy when running lscfg -vp. Here's a grep command to gather just the rmt(s) with their serial and WWN. If you need to change the number of lines just change the a (after) and b (before) value, and s is what is "grep'ed".

lscfg -vp | awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=1 a=5 s="rmt"

Getting TSM Script Info

2011-05-18T09:31:00.000-07:00

So you want to query a script and see what it executes but don't like the standard Q SCRIPT results? Don't forget about the FORMAT= options available. Most people know FORMAT=DETAIL but with the Q SCRIPT command there are two additional formats. FORMAT=LINE and FORMAT=RAW. I like RAW but LINE is decent if you want to see the script by sequence.

UPDATE: As reader Erwann reminded me (I had long forgotten), there is also a F=MACRO option which will display the script in a form that can be run as a macro to define the script to another TSM server. If you issue Help Q SCRIPT it does not list the MACRO option, but it is valid and does work. So you could dump all your scripts to macro format and send it to an outfile. Then you could define them to an alternate TSM server (Q SCRIPT * OUTFILE=SCRIPTS.MAC F=Macro).

tsm: TSMLIBM>q script LIBR_INFO f=macro
DEFINE SCRIPT LIBR_INFO DESC="Library path information"
UPDATE SCRIPT LIBR_INFO "q libr $1"
UPDATE SCRIPT LIBR_INFO "q libr $1 f=d"
UPDATE SCRIPT LIBR_INFO "q path $1 srct=serv destt=libr f=d"
UPDATE SCRIPT LIBR_INFO "q path * srct=serv destt=dr libr=$1 f=d"
UPDATE SCRIPT LIBR_INFO "select library_name,library_type,library_serial from libraries where library_name ='$1'"
UPDATE SCRIPT LIBR_INFO "select source_name,destination_name,library_name,device serial from paths where library_name='$1'"
UPDATE SCRIPT LIBR_INFO "select drive_name,drive_serial,element,library_name from drives where library_name='$1'"

Here are the examples:

tsm: TSMLIBM>q script LIBR_INFO f=line

Name Line Command
Number
---------- ------ ------------------------------------------------------------
LIBR_INFO 5 q libr $1
10 q libr $1 f=d
15 q path $1 srct=serv destt=libr f=d
20 q path * srct=serv destt=dr libr=$1 f=d
25 select library_name,library_type,library_serial from
libraries where library_name ='$1'
30 select source_name,destination_name,library_name,device
serial from paths where library_name='$1'
35 select drive_name,drive_serial,element,library_name from
drives where library_name='$1'

tsm: TSMLIBM>q script LIBR_INFO f=raw
q libr $1
q libr $1 f=d
q path $1 srct=serv destt=libr f=d
q path * srct=serv destt=dr libr=$1 f=d
select library_name,library_type,library_serial from libraries where library_name ='$1'
select source_name,destination_name,library_name,device serial from paths where library_name='$1'
select drive_name,drive_serial,element,library_name from drives where library_name='$1'

Delete Old Filespaces

2011-04-29T08:52:00.000-07:00

Been looking to do some cleanup and I wanted an easy way to do it. So...I wrote this script. I can take the list returned and either make it a macro or an internal TSM script. Sure I could build a shell script that processed a list with the TSM server_name, the node_name, and FSID but I thought I'd show how you can create a command out of SQL returned data.

select 'del filespace ' || node_name || ' ' || cast(FILESPACE_ID as char(3)) || ' type=backup nametype=fsid wait=yes' AS COMMAND from filespaces where cast((current_timestamp-backup_end)days as decimal)>90

Here is an example of the results from the script:

COMMAND
------------------------------------------------------------
del filespace NODE1 37 type=backup nametype=fsid wait=yes
del filespace NODE1 9 type=backup nametype=fsid wait=yes
del filespace NODE1 36 type=backup nametype=fsid wait=yes
del filespace NEO_1 32 type=backup nametype=fsid wait=yes
del filespace NEO_1 31 type=backup nametype=fsid wait=yes
del filespace NEO_1 23 type=backup nametype=fsid wait=yes
del filespace NEO_1 29 type=backup nametype=fsid wait=yes

TSM 6.2 GUID Error

2011-04-26T09:58:00.000-07:00

So when I and a co-worker were finishing up a TSM 6.2 install we ran into a problem running the tsmapipw command to set the $$_TSMDBMGR_$$ password. After investigating the error and search the web we realized the GUID was the problem and not our IP as was suggested in the following APAR. The errors we received were:

ANR2987W Session ended because of machine GUID or local host IP
address mismatch.
ANR0430W Session XXXX for node $$_TSMDBMGR_$$ (AIX) refused
- node name is locked.

We tried various things and nothing seemed to correct the GUID issue. The TSM server started without any GUID related error (in fact it stated the GUID initialized successfully). So upon further searching we found this APAR that gives the solution. What they state is that by adding the following line to the dsmserv.opt and restarting the server fixes the problem.

DBMTRUSTEDGUIDIGNORE YES

Once we added the line and restarted the TSM server the tsmapipw command ran successfully.

Send CSV File As Attachment

2011-02-03T21:11:00.000-08:00

Have a report that is too big to be column output and don't like the list look of TSM 6.1? Sure you could reformat you comma delimited file for output but I'd rather send it as an attachment in their mail and let them open it in Excel. So if you have access to a Linux/UNIX server this is a simple solution I used recently and it worked great.

uuencode /tmp/tsm/stg_backup_rpt.csv /tmp/tsm/stg_backup_rpt.csv | mailx -s "TSM Servers Backup STGPool Status" "test1@mymail.com,test2@mymail.com"

There are other ways with other mail handlers (i.e. mail, mailx, sendmail, etc.) but this one was easy and worked.

Select X Days Back

2011-01-11T12:01:00.000-08:00

As you all know I have been writing and rewriting scripts for my 6.x servers and have been experiencing interesting results when I use the timestampdiff option when determining how far back a script should look. So when running my summary report query I was getting inaccurate results when using the following script:

example 1:
select schedule_name, entity, start_time, end_time, timestampdiff(16, char(end_time-start_time)) || ' ' || timestampdiff(8, char(end_time-start_time)) || ':' || timestampdiff(8, char(end_time-start_time)) || ':' || substr(cast((end_time-start_time) as char(22)),13,2) AS ELAPSED_TIME, affected from summary where activity='STGPOOL BACKUP' and timestampdiff(8, char(current_timestamp-start_time))<=24

This script would not return all the STGPOOL BACKUP jobs from the summary table. Not sure why, but I have a couple ideas. So, I found that the following is easier to use and returned the correct results.

example 2:
select schedule_name, entity, start_time, end_time, timestampdiff(16, char(end_time-start_time)) || ' ' || timestampdiff(8, char(end_time-start_time)) || ':' || timestampdiff(8, char(end_time-start_time)) || ':' || substr(cast((end_time-start_time) as char(22)),13,2) AS ELAPSED_TIME, affected from summary where activity='STGPOOL BACKUP' and start_time>(CURRENT TIMESTAMP - 10 DAYS)

So whenever you would like to query X days back I suggest example 2 for ease of use.

TSM 6.x Time Calculation

2010-12-31T09:20:00.000-08:00

I use the summary table frequently and like to calculate the elapsed time by subtracting the END_TIME from the START_TIME. In TSM 5.x you could easily do this because the TSM DB was setup for returning a day/time when calculating those two fields, but TSM 6.x's DB2 database does not work the same way. When calculating the difference in 6.x the result looks like this:

   ENTITY: STG_V0653 -> STG_T10K_C2
  START_TIME: 2010-12-28 08:00:11.000000
   END_TIME: 2010-12-28 22:39:29.000000
ELAPSED_TIME: 143918.000000
   AFFECTED: 258
GB BACKED UP: 8814.00

You'll notice that the value given is actually in HHMMSS without the colon seperators. So my select formats the output to a more time based solution. You might ask why I didn't use timestampdiff, but the problem with the timstampdiff format is that it does not return in a time format like previous (5.x) versions. The timestampdiff would return the format calculation of either days, hours, minutes, or seconds only. So I actually worked with a friend who has a little more experience with DB2 and we came up with the following.

select schedule_name, entity, start_time, end_time, substr(cast((end_time-start_time) as char(22)),7,2) || ' ' || substr(cast((end_time-start_time) as char(22)),9,2) || ':' || substr(cast((end_time-start_time) as char(22)),11,2) || ':' || substr(cast((end_time-start_time) as char(22)),13,2) AS ELAPSED_TIME, affected from summary where activity='STGPOOL BACKUP' and timestampdiff(8, char(current_timestamp-start_time))<=24

Results:

SCHEDULE_NAME: AS_BACK_V655

ENTITY: STG_655 -> STG_10_C2

START_TIME: 2010-12-28 08:00:26.000000

END_TIME: 2010-12-29 07:40:46.000000

ELAPSED_TIME: 00 23:40:20

AFFECTED: 632

So now it shows the elapsed time as Days, Hours:Minutes:Seconds

You could eliminate the Day by removing the first calculation, but I have some storage pool backup processes that can run VERY long so I included it.

UPDATE: Weird Query Results From 6.1.4

2010-12-17T16:23:00.000-08:00

So the problem has been identified and listed in an APAR. In the document it states that SELECT commands on multiple tables not working as expected on a 6.x server. The table list is quite large and it looks like the problem is resolved in 6.1.5 and 6.2.2. Which means I have no fix unless I upgrade the servers.

Weird Query Results From 6.1.4

2010-12-15T21:42:00.000-08:00

So we have a couple servers we upgraded from 5.3 to 6.1.4.2 and since the upgrade the following select statement does not work. I posted it on ADSM.org and others report their 6.x servers run it without issue. Here is the statement and my results:

select server_name, node_name, PLATFORM_NAME || ' ' || cast(CLIENT_OS_LEVEL as char(20)) as OS_Version, cast(client_version as char) || '.' || cast(client_release as char) || '.' || cast(client_level as char) || '.' || cast(client_sublevel as char(2)) as TSM_Version from nodes, status

here is the return code I get:

ANR2034E SELECT: No match found using this criteria.
ANS8001I Return code 11.

I run this through a while loop and append the data to an comma-delimited output file so I can pull it into a spreadsheet and organize clients by version. I can then more easily prioritize my client upgrades. I really need this to work so I can sort and identify the clients location by server_name. Any ideas?

Update of SQL For TSM 6.1

2010-11-29T10:43:00.000-08:00

So the previous select is not compatible with TSM 6.1 so I was searching through the IBM Redbooks for TSM and found changes to make TSM conform to SQL standards. There are changes to the TSM supported time functions. Here is the link, it's under Appendix B.

def script event-check desc="Events - Exceptions"
upd script event-check "/* ---------------------------------------------*/"
upd script event-check "/* Script Name: event-check */"
upd script event-check "/* ---------------------------------------------*/"
upd script event-check ' select -'
upd script event-check ' schedule_name, -'
upd script event-check ' cast(SUBSTR(CHAR(actual_start),12,8) as char(8)) AS START, - '
upd script event-check ' node_name, -'
upd script event-check ' cast(status as char(10)) as "STATUS", -'
upd script event-check ' case -'
upd script event-check " when result=0 then ' 0-Succ' -"
upd script event-check " when result=4 then ' 4-SkFi' -"
upd script event-check " when result=8 then ' 8-Warn' -"
upd script event-check " when result=12 then '12-Errs' -"
upd script event-check " else cast(result as char(7)) -"
upd script event-check ' end -'
upd script event-check ' as "RESULT" -'
upd script event-check ' from events -'
upd script event-check ' where timestampdiff(8, char(current_timestamp-scheduled_start))<=24 -'
upd script event-check ' and result<>0 and node_name is not NULL'

Previously Unknown TSM SQL Feature

2010-11-26T11:21:00.000-08:00

So while doing some script maintenance on a TSM server I came across this select statement and was shocked to see the CASE option in a select. I've used it in shell scripts but didn't think TSM allowed for it. Well it does and it works. Go figure! (Runs on a TSM 5.3.5.2 server).

/* ---------------------------------------------*/
/* Script Name: eventx */
/* ---------------------------------------------*/
select -
schedule_name as "Schedule Name ", -
cast(substring(cast(scheduled_start as character(26)) from 12 for 8) as char(8)) as "Sched", -
node_name, -
cast(status as char(10)) as "Status", -
case -
when result=0 then ' 0-Succ' -
when result=4 then ' 4-SkFi' -
when result=8 then ' 8-Warn' -
when result=12 then '12-Errs' -
else cast(result as char(7)) -
end -
as "Result" -
from events -
where current_timestamp-scheduled_start<interval '24' hours -
and result<>0

I saw this and a whole new world opened up before my eyes! The possibilities!

TSM 6.1...Finally!!!

2010-11-03T07:54:00.000-07:00

So I have landed on my feet as a contractor and where I am currently working they are migrating to 6.1.4 TSM from 5.x versions. In their testing they found a 36GB DB took 14 hours for the complete migration of the DB. This time includes completing storage pool migration, stopping all processes, and bringing down the original TSM server clean. So I am not looking forward to the large 100+GB DB's. I also find it interesting how many companies have not migrated to version 6. The only reason they are going to 6.1.x is that it has been tested and 6.2 has not (their testing process is long and tedious). So what has everyone else seen as their average migration time from 5.x to 6.x?

TSM Admin Contract - Cheshire, UK

2010-09-21T08:31:00.000-07:00

I just received this job posting for Cheshire, England. If you're qualified and interested give Matthew a call.

The successful TSM Administrator will be working for a large company based just outside Chester, UK.

The key points for the role are:

Recent commercial experience of using TSM v6.2
Install TSM Server / TSM Client on AIX
Present / configure tape drives for use with TSM
Some of the work will involve deployment for the TSM project
General TSM support & maintenance, resolving PRMs with IBM support

The contract attracts very competitive rates and will last at least 4 months.

You will need to be eligible to work in the UK and hold a HMRC Tier 1 Visa.

Many thanks,

Matt

Matthew Burrows RECRUITER
4020 LAKESIDE, SOLIHULL PARKWAY
BIRMINGHAM BUSINESS PARK, SOLIHULL, BIRMINGHAM, B37 7YN
T +44 (0)121 717 2828 F +44 (0)845 643 9702
E MBURROWS@TEKSYSTEMS.CO.UK

Multiple Telecommuting Positions Open

2010-09-03T07:30:00.000-07:00

Artech is trying to staff for a number of telecommuting TSM Admin positions. If you are a high level TSM admin and looking for work contact Arjun at Artech.

Arjun Kumar Dhir Sr. Executive Staffing
Artech Information Systems LLC 240 Cedar Knolls Road, Suite 100 \| Cedar Knolls, NJ 07927
Office: 973.967.3427 \| Fax: 973.998.2599
Email: Arjun_Dheer@artechinfo.com \| Website: www.artechinfo.com

Good TSM Oracle Backup Blog

2010-09-02T08:34:00.000-07:00

I came across this website and it has a lot of TSM Oracle backup related information. If you perform Oracle TDP backups then check out this site.

May I Have One Free Brain Dump...And Another?

2010-07-25T20:43:00.000-07:00

OK, so I was contacted by a person from India (I only state that so you'll know why there are so many grammar/typos in the questions) who proceeded to ask me 18 of the following questions of which I was reluctant to respond. I did, however, respond with the following note letting them know what I thought of such e-mails:

(Keep reading past the first set of questions...it continues.)

I would like to point out that the questions you asked are all easily answered by reading the documentation, accessing ADSM.org, or simply by searching the web using Google. Having previously helped in the interviewing and hiring process, I would stress that your inability to utilize the resources already available would make me hesitant to consider you for employment. You need to be a self motivated, problem solving, and resourceful person to excel in IT. The one question I always asked when hiring was, "If you have a problem and you can't resolve it, what would be your steps in finding a solution?" The answer to this one question usually determined if they were good candidates. In the future try to use the resources available to find the answers, and note that just typing HELP on the admin command line can usually provide a majority of the answers you require.

How to configure the library in unix level? Depends...if you mean discover the hardware and such it is dependent on the UNIX version. I always used AIX for my TSM servers so cfgmgr was used to discover the hardware and define it to the OS.
How to make a communication b/w TSM and library? Most communication happens over the control point. For example in a 3584 you will select one of the drives to be the control point and the HBA it is zoned in the SAN to will be the HBA the library controller (robot) shows up under. All communication happens across this connection. If you use ACSLS then its a whole different process.
Different type of H/W components available in library? Again this depends on the library type/manufacturer. You could have LTO, 3590, SDLT, VTL options along with dual robots.
In which way you will access the library? There are various tools to access the library but I tend to use the web interface more than I use the AIX library/tape tools (i.e. tapeutil)
What will happen when the volume will move to different storage slot inside the library.How to rectify the same? TSM should track the slots and where every tape is. If for some reason the inventory is inaccurate you can run both a library level audit and a TSM level audit. Lookup the AUDIT LIBRARY command in the TSM Admin Reference.
If you are running out of scratch tapes then what are the steps you will do to overcome that? Of course there is the standard RECLAIM STG and MOVE DATA commands but a lot depends on how good my processes are at freeing up tapes. It could just come down to deleting old DB backups or I might go so far as investigating whether the Oracle DBA's have been keeping their RMAN catalogue sync'd with TSM. If they haven't I have a very good chance of freeing up a lot of tapes when I manually delete their data from TSM.
How to checkout the tapes from library? If using DRM then MOVE DRMEDIA if not then the CHECKOUT LIBVOLUME command. See the TSM Admin Reference for more details.
How to manual clean the drive? You should not need to manually clean a drive but if you decide to then it depends on if you initiate it from the library's web interface or from TSM. From TSM it's a simple CLEAN DRIVE
Which will act as a interface b/w library & TSM? I don't quite understand what you mean? In the case of TSM you usually don't need the library's web interface much, most of the handling can be done through TSM.
When the volume is throwing error while mounting then what you will do as a TSM admin? I usually check to see if the tape has a history of mount errors, if so I check it out of the library, if not then it could be the drive or accessor are in need of service. If mount errors continue on the drive I will call in a service technician to look at the drive.
What is the use of Audit Library? Makes sure that the library physical inventory matches TSM's inventory. If they get out of sync you could have mount failures and errors due to TSM thinking it is grabbing a different tape then it should have found in that slot.
If the stgpool reusedelay parameter is 2 then what will happen when volume becomes empty? After two days the volume is available to be used as scratch again. The reusedelay parameter is good to use if there is a possibility that you might need data from beyond the normal expired time. So technically if you keep 5 days worth of TSM DBbackups then the oldest TSM DB could request tapes that have since expired and been sent back onsite to be used as scratch so you should set the resuse delay parameter to the same setting or same+1 day as the number of TSM DB backups you keep.
Uses of Reclamation: Think of reclamation as a Defrag for tape. Reclamation can tape two or more tapes and condense them down to one tape if used correctly. Again lookup RECLAMATION in the TSM Admin Reference.
Depends on library type/config and the number of scratch I need to load. For example I currently am working with a 3584 that has a 10 tape I/O door and needed to load 50 scratch. So I paused the library, opened the storage frame's door and manual inserted all 50 into the open slots (I have a number of columns not in use). I then closed the library and allowed the library's audit to complete before I ran the CHECKIN LIBVOLUME SEARCH=YES CHECKL=BARCODE STATUS=SCRATCH otherwise the standard checkin process through the I/O door works fine. With ACSLS and 3494 libraries the tapes are automatically entered into the library and the CHECKIN LIBVOLUME SEARCH=YES CHECKLABEL=NO STATUS=SCRATCH command is used. Notice it is a little different from a SCSI based library like the 3584.
Use of reusedelay & maxscratch? Maxscratch will determine how many scratch a sequential TSM storage pool can use. The reuse delay parameter will determine the number of days a tape that shows as empty in TSM will be in "limbo" before TSM will allow it to be reassigned back to a scratch status.
Why we need multiple management class in a single domain? Multiple management classes in a single domain allow for you to reference different retention rules "on the fly" when running a backup or archive by using the INCLUDE . So say I had a management class called 1_YEAR that keeps UNLIMITED versions for 1 year. To use it on specific data types I would put an include in a clients DSM.OPT or in a client option set.

INCLUDE "C:\USERS\...\MY DOCUMENTS\WORD\*.DOC" 1_YEAR

Diff b/w session & process? A session is a TSM client/admin/alternate TSM server connection where a process is an internal task (i.e. MIGRATION) running on the TSM server.
Different types of client established session? There are standard client sessions, datamover sessions (NAS), storage agent sessions (LAN-Free), TDP sessions (Exchange, SQL, Oracle, etc), and TSM Server sessions (If the server is a library client accessing the library manager or if using the virtual volume feature).

So after answering these questions, with the added commentary on what I thought about such requests, you would think the person would get a clue and start researching the answers to any further questions...NOPE! I just received the following e-mail from the same person:

Pls clear me.....

Difference between other application and TSM?
About features of TSM.
Make clear me on Roll forward Mode & Storage pool Hierarchy?
What is meant by copy storage pool?
What is need of SAN in TSM environment?
Where you will mention the destination of backup?
If client want 3 inactive version for 60 days what are the parameter we will mention?
Explain about scheduling and its types
Explain some types of tapes with its capacity.
What are the types of copy group
Where will you allocate the no.of tapes?
How will you check the node belongs to particular domain?
Where will you define the server for the particular tape?
How will you know the particular volume which has data?
Advantages of Linux?
How to configuring the TSM?
How to see the Database time of yesterday missed file?
About device class and types?
What is meant by Admin Process?
How to see the node started confirmation from server side?
In Migration, why we r giving LOWMIG?
Retension period for node?
Diff between foreground and background?
Tell me the device name for library in TSM on AIX?
Difference between clientopt and clopset?
Difference between label and checkinlibvolume?
Backupset where it will be used?
If recovery log 100% what will happen?
Suppose TSM server crashed ...what we will do?
What is client option set?
About Restore process and Data Shredding?

Seriously?!? I would never hire this person...I can understand asking a question or two but 31? I just answered 18 out of shear boredom...are some people really this lazy?

TSM to DPM transition

2010-07-19T21:49:00.000-07:00

I am currently working a contract transitioning a companies old TSM environment to Microsoft DPM and am discovering once again how Microsoft operates. Just like many of their software products when they were first released, Microsoft's product lacks certain features that others have had for years. As they revise the product they slowly reach a close parity, but it's never quite equal to older more mature products. The clincher is that it's a Microsoft tool and Windows shops don't know any better. As I learn more about DPM I'm convinced it's not going to supplant TSM or Netbackup by features, but shear lack of backup expertise from the administrators. Their MS rep will come in and tout the product with the limited information he has, show them a decent bundle price, sell them DPM, and not be heard of until the next licensing cycle. (Does The Music Man ring a bell?) The reporting capabilities alone are awful! You can't easily create your own and DPM runs on MS SQL. Go figure! And what's with Microsoft pushing their PowerShell?? You can script reports but have to know how to script within MS PowerShell. I thought the whole purpose of Windows was to eliminate the command line? (Not that I mind, I just find it ironic) So as I learn more I'll post what I find, but so far I'm shocked they thought DPM was a better tool than TSM...what they really needed was a good TSM admin not new software.

2010-06-21T13:42:00.000-07:00

I was contacted today for a position with J.P. Morgan Chase and thought I would pass it on since it does not allow remote/telecommuting and would require me to move to either Houston, TX or Columbus, OH. If you have senior level TSM skills and are either willing to move or live within these metro areas send Melissa your resume. All I ask is that you tell her you saw the post on TSMExpert.org. (And no, I don't get a referral fee!)

Hi Chad; thanks again for your time today. As discussed, I recruit for our core infrastructure tech roles (storage, server) at JPMorgan Chase, and would love to be in touch with any talented individuals for current or future needs our managers may have.

The role I called you about today is for a TSM expert who would be willing to work the weekends. We offer great benefits+ competitive salary, including OT pay + 10% shift differential due to the nature and hours of the roles. Individuals can be based in Columbus or Houston, and we are offering a $10K net lump sum to help with relocation for those who live outside a reasonable commute distance of the two sites.

Also, due to a couple of recent team promotions, I will be posting two more roles that require skills in both NBU and intermediate TSM (with command line environment knowledge).

I appreciate any referrals you may be able to offer!

Best regards, and thanks again!

Melissa

Melissa Greenwood | JPMorgan Chase Corporate Recruiting | 614-213-6670 | Melissa.J.Greenwood@jpmchase.com

TSM Console Manager

2010-06-19T13:52:00.000-07:00

While catching up with my e-mail I came across Spirit Software's TSM Console Manager. I had been asked to review it and did not get the e-mail until today. So I thought I would post the video describing the product and giving you and idea of how it works. Oh, and the cost isn't bad either...it's FREE!

Updated E-mail

2010-06-19T11:02:00.000-07:00

I apologize to anyone who has sent me an e-mail to tsmexpert@hotmail.com since I really have not checked it consistently and thought I had switched everything over to my gmail account. If you have a question or would like to submit an article I have updated my e-mail link or you can contact me at chadsmal@gmail.com (yes that's one l).

On The Bench

2010-06-11T17:32:00.000-07:00

Well I am back in the job market today. When searching TSM positions I notice there are quite a few positions available out there. I wonder how long it took those of you out there to find new positions? I'm pretty optimistic. At least for the moment.

I was let go due to cost cutting demanded from the higher ups, so I'm thankful I can use Infocrossing as a solid reference. If anyone has suggestions or is hiring feel free to send me an e-mail.

When Data Isn't There, But Is Really There!

2010-05-30T19:43:00.000-07:00

OK so the title seems odd, but I had this occur recently when our UNIX Admins decided to convert a subdirectory to a file system. So they created the file system and wiped out the subdirectory expecting us to restore the data. When you would query for the backups nothing would show as having been backed up. (Let me clarify that the Admins did the change and then a week later alerted us and asked for the restore.) So the project manager asked for the restore but did not clarify the directory had gone from /opt/ar/ to being file system /opt/ar/. So any

dsmc res /opt/ar/* /opt/ar/ -pick -inactive -subdir=yes

returned no data. What I had to do was

dsmc res /opt/* /opt/ar/ -pick -inactive -subdir=yes -pitd=

and then the data would show and I could select it manually for restore. If there is one thing I have learned with TSM is always double, triple, and quadruple check for "missing" data using various methods, chances are it is probably there. Remember the user is clueless....don't get me started on symbolic links and how users don't always know the actual location of their data.

Tivoli Storage Manager v6.2 install and see the Exception Stack Trace for more information... [sorry for the long entry]

2010-05-14T05:13:00.000-07:00

Am I too old for that? or this is the real future? Is it realy the way it has to be? I don't think so... :-(

--- CUT --- Welcome Tivoli Storage Manager 6.2 Licensed Materials - Property of IBM Corp. (c) IBM Corporation and other(s) 1994, 2010. All rights reserved. US Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. It is highly recommended that you stop all other programs before continuing with this installation. PRESS TO CONTINUE: =============================================================================== International Program License Agreement Part 1 - General Terms BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, CLICKING ON AN "ACCEPT" BUTTON, OR OTHERWISE USING THE PROGRAM, LICENSEE AGREES TO THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON BEHALF OF LICENSEE, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND LICENSEE TO THESE TERMS. IF YOU DO NOT AGREE TO THESE TERMS, - DO NOT DOWNLOAD, INSTALL, COPY, ACCESS, CLICK ON AN "ACCEPT" BUTTON, OR USE THE PROGRAM; AND - PROMPTLY RETURN THE UNUSED MEDIA, DOCUMENTATION, AND PROOF OF ENTITLEMENT TO THE PARTY FROM WHOM IT WAS OBTAINED FOR A REFUND OF THE AMOUNT PAID. IF THE PROGRAM WAS DOWNLOADED, DESTROY ALL COPIES OF THE PROGRAM. Press Enter to continue viewing the license agreement, or enter "1" to accept the agreement, "2" to decline it, "3" to print it, "4" to read non-IBM terms, or "99" to go back to the previous screen.: 1 =============================================================================== Component Selection ------------------- Select the components to install. 1- Tivoli Storage Manager Server 2- Tivoli Storage Manager License 3- Tivoli Storage Manager Devices 4- Tivoli Storage Manager Storage Agent ENTER A COMMA-SEPARATED LIST OF NUMBERS REPRESENTING THE DESIRED CHOICES: 1,2 =============================================================================== Deployment Engine Initialization -------------------------------- Please Wait ... Step 1 of 15............................................................... Step 2 of 15................................................................................. Step 3 of 15... Step 4 of 15... Step 5 of 15.... Step 6 of 15..... Step 7 of 15.................................................................... ..... Step 8 of 15..................... Step 9 of 15.... Step 10 of 15........ Step 11 of 15............... Step 12 of 15............. Step 13 of 15... Step 14 of 15... Step 15 of 15............................................................ Completed. =============================================================================== Pre-Installation Summary ------------------------ Review the Following Summary Before You Continue: Product Name: Tivoli Storage Manager Install Folder: /opt/tivoli/tsm Components: TSM Server,DB2 9.7,TSM Client API 32 bit,TSM Client API 64 bit,TSM License Disk Space Information (for Installation Target): Required: 1,815,384,781 bytes Available: 478,806,016 bytes PRESS TO CONTINUE: =============================================================================== Not Enough Disk Space --------------------- Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: Warning! This installation requires 1,731.29 MB of free disk space, but there are only 456.62 MB available at: /opt/tivoli/tsm Please free at least 1,274.66 MB to proceed with the installation. PRESS TO RECALCULATE AVAILABLE DISK SPACE, OR TYPE 'QUIT' TO EXIT THE INSTALLER: There is now enough disk space to proceed with the installation. PRESS TO BEGIN INSTALLATION: =============================================================================== Installing... ------------- [==================|==================|==================|==================] [------------------|------------------|------------------|------------------] =============================================================================== Installation Complete --------------------- The product did not install. See /var/tivoli/tsm/log.txt for details. PRESS TO EXIT THE INSTALLER: --- CUT --- total 800 drwxrwxr-x 2 root system 256 May 14 14:09 . drwxrwxr-x 3 root system 256 May 14 12:34 .. -rw-rw-r-- 1 root system 320432 May 14 14:09 log.txt -rw-rw-r-- 1 root system 84750 May 14 14:01 logs.zip -rw-rw-r-- 1 root system 0 May 14 13:55 rxa_message.log -rw-rw-r-- 1 root system 0 May 14 13:55 rxa_trace.log and the of that SMALL log: . . . Fri May 14 14:01:50.310 CEST 2010 : FINE : PlanStepEventOccurred(Failed target 'step_00001_TSM_Server_1_1') (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred) Fri May 14 14:01:50.311 CEST 2010 : FINE : COIEventType.ERROR (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred) Fri May 14 14:01:50.313 CEST 2010 : FINE : MachinePlanEventOccurred(An exception was received during the processing of MachinePlan: localhost. See the Exception Stack Trace for more information.) (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred) Fri May 14 14:01:50.313 CEST 2010 : FINE : COIEventType.ERROR. Last step started was: TSM_Server_1_1 (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred) Fri May 14 14:01:50.314 CEST 2010 : FINE : DeploymentPlanEventOccurred(Failed Deployment Plan 'install') (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred) Fri May 14 14:01:50.315 CEST 2010 : FINE : COIEventType.ERROR (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred) Fri May 14 14:01:50.926 CEST 2010 : FINE : Exception via callback from DeploymentPlanProcessor.process(INSTALL): (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install) Fri May 14 14:01:50.927 CEST 2010 : FINE : COIEventType.ERROR reported (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install) Fri May 14 14:01:50.928 CEST 2010 : STDERR : com.ibm.ac.coi.ext.ia.COIWrapperException: COIEventType.ERROR reported Fri May 14 14:01:50.929 CEST 2010 : STDERR : at com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred(COIWrapperPluginImpl.java:121) Fri May 14 14:01:50.930 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListener(COIRuntimeImpl.java:226) Fri May 14 14:01:50.931 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListeners(COIRuntimeImpl.java:215) Fri May 14 14:01:50.932 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.utils.ParallelPlanListener.eventOccurred(ParallelPlanListener.java:105) Fri May 14 14:01:50.934 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListener(COIRuntimeImpl.java:229) Fri May 14 14:01:50.935 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListeners(COIRuntimeImpl.java:219) Fri May 14 14:01:50.936 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.MachinePlanImpl.process(MachinePlanImpl.java:254) Fri May 14 14:01:50.937 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.utils.MachinePlanRunner.run(MachinePlanRunner.java:64) Fri May 14 14:01:50.938 CEST 2010 : STDERR : at java.lang.Thread.run(Thread.java:736) Fri May 14 14:01:50.939 CEST 2010 : STDERR : Caused by: Fri May 14 14:01:50.940 CEST 2010 : STDERR : com.ibm.ac.coi.api.exception.DeploymentPlanExecutionException: One or more Machine Plans within the DeploymentPlan failed. Failure count = 1 Fri May 14 14:01:50.941 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.utils.ParallelPlanListener.eventOccurred(ParallelPlanListener.java:92) Fri May 14 14:01:50.942 CEST 2010 : STDERR : ... 5 more Fri May 14 14:01:50.943 CEST 2010 : SEVERE : Raw Exception in DeploymentPlanProcessor.process(INSTALL): COIEventType.ERROR reported (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install) Fri May 14 14:01:50.944 CEST 2010 : FINER : THROW (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install) . . .

and bla, bla, bla, bla...

Sorry, but nice week-end!!! ;-)

TSM Upcoming Events and Live Technical Training

2010-05-05T09:29:00.000-07:00

FAQ page

TSM

My Notifications

19 May 2010 - TSM and Electronic Vaulting for Disaster Recovery
27 May 2010 - Implementation and Use of TSM Client/Server Data De-duplication with TSM 6.2
08 June 2010 - Encryption and Security with TSM Version 6
13 June 2010 - TSM V6.2 Automatic Client Deployment

NetbackupExpert?

2010-04-14T10:55:00.000-07:00

Once again licensing time has arrived and Symantec is giving away Netbackup. IBM's pricing model makes it hard to justify staying with TSM when Symantec is pricing their product sooooo low. Sure I can pull out graphs, reports, and pretty little charts to try and help management understand the difference, but let's face it, it all comes down to the bottom line. IBM doesn't like to budge on pricing and Symantec has nothing to lose. Anyone have some Netbackup training materials I can borrow?

Obstacles Upgrading to 6.1

2010-03-18T22:44:00.003-07:00

I have a server with 8 instances of TSM running on it. The first instance is the library controller and the other seven are production instances. So the problem has been how to best reconfigure everything for TSM 6.1 since IBM says it's best not to run multiple instances on the same server unless they are in their own LPAR. So what I have been trying to figure out is how best to migrate to 6.1 with the least impact. I figure I will need to merge the 7 production instances into 3 or 4, but then what would be the best procedure for dividing up the hardware (i.e. NICs and HBA's) for the other LPARs. Any suggestions?

Tivoli Storage Manager v6.2 is coming in March

2010-02-25T12:39:00.001-08:00

Tivoli Storage Manager v6.2

Finally! MySQL backups with TSM

2010-02-23T12:04:00.001-08:00

Computer Technology Review has an announcement from Zmanda stating they are releasing a TSM backup agent (Zmanda Recovery Manager for MySQL). The ZRM-TSM agent is available immediately. You can read the full article here. Those of you who have been asking for a TDP for MySQL, the wait is over.

Update: Weird Volhist Records (Part Two)

2009-12-23T09:37:00.000-08:00

IBM did get back to me about my library manager/library client volume history devclass issue and I found it interesting how development handled it. Initially IBM support said the following:

Our server development has confirmed with our design, it is possible that the remote volhist entry can be different. Development believe this is their design. However, the existing document did not document this clearly. They agreed to open a Doc. APAR to properly document this for command query volhist.

In other words they felt that no action was needed other than to document that this possible occurrence within the TSM Admin and Reference guides. So I asked it be escalated since it is definitely a flaw and finally heard back from support and was told the following:

They agree what you observed or reported in this case is incorrect even though it does not cause any function lose. They have agreed to take the APAR IC65048 as a defect ( instead of Document ). However, this APAR would take a big code change to "fix" this issue and after first evaluate, they will not be able to deliver a fix in the service stream. They request to open a DCR ( design change record ) so development can make this change on a release boundary so that there is sufficient testing for a code change.
So it looks like IBM will eventually fix this. Thanks to IBM support for helping me get development to at least go beyond the Doc. APAR.

Data Loss Top 10

2009-12-07T13:03:00.000-08:00

Ontrack, which handles data recovery, compiled a list of the strangest reasons data was lost. You can read their list here. I remember when I was teaching TSM some students had some of the craziest stories for data loss (many were disgruntled employee stories).

One story I heard from a student occurred when a manager was stupid enough to fire an employee, but require him to work the rest of the week if he wanted his pay. So the employee worked the rest of the week and on his way out (unescorted) he went into the server room and proceeded to urinate into the mainframe. Supposedly he shorted out the mainframe and caused all kind of havoc. If you have a story you'd like to share feel free to leave it in the comments section.

Update: TSM 2009 Symposium

2009-10-29T14:20:00.000-07:00

I received a response to my query on the cost of the 2009 TSM Symposium presentations USB stick. The following will provide you with the needed purchase information:

The USB-stick with all symposium presentations and materials in PDF-format is
available for 50 EUR plus shipping costs (Germany 3 EUR, European Union 5 EUR,
US 8 EUR).

Payment is accepted only by Paypal. Please transfer the respective amount by
using the Paypal service (http://www.paypal.com/) to
nc-kallecl(at)netcologne.de. Add your post address as optional message for the
payment or mail it to tsm2009@uni-koeln.de.

Pay Pal will make the necessary conversion from dollars to euros and expect to pay around $86 with current conversion rates. If you'd like to check the conversion rates try Google's currency conversion tool.

TSM 2009 Symposium

2009-10-21T09:42:00.000-07:00

I went looking for the presentations given at the 2009 TSM Symposium and discovered that the presentations will be provided on a USB stick for around 50 Euro plus shipping. Previously the presentations had been posted for the community to access, but it looks like we wont have that opportunity going forward. I didn't see anything for U.S. cost and will post an update when I find out if/how we can purchase the symposium presentations. I would recommend you talk your company into purchasing it since I have always found the presentations to be very informative, and with 6.1 out it might help you with any planned migration/upgrade.

Tivoli User Community

2009-10-08T10:00:00.000-07:00

I had the opportunity this week to attend the Arizona Tivoli User Group and gain some valuable information on TSM Fastback, TSM 6.1, and Tivoli Storage Virtualization Management. They covered these products in the morning session and had a hands on demonstration of TSM 6.1 Admin interface and reporting features in the afternoon. What was even better was they offered a free certification exam so I got caught up on my TSM certification. Not every group will be able to afford free certifications, but it is definitely worth your time to check out your local or regional group for contacts and information. To find the Tivoli User Group in your are check out the Global Tivoli User Community website. If there is no group within your area you can still join the global group and attend webinars and participate in online activities.

Update: Weird Volhist Records

2009-10-01T10:26:00.000-07:00

Upon further investigation I found that my other shared library environment, that has multiple NAS devices defined to the library manager, is also showing the NAS device class for all LTO-3 volumes used by the library clients. If anyone has a NAS attached to a shared library environment please compare your volhist devclass to those of your library clients and tell me if you see the same situation. CRAZY THING IS, it doesn't seem to affect the library clients from using their tapes or acquiring scratch. What my concern is, is that I am slowing losing use of tapes that should be marked back as scratch, but due to the volhistory settings is unable to be released. I am wondering if this is a bug in TSM that could be affecting others.

Weird Volhist Records

2009-09-30T10:32:00.000-07:00

I am reviewing my Volume History file on my ACSLS library controller and it shows volumes as REMOTE and owned by the library clients but the weird thing is it shows the DEVCLASS as a device class that is not even defined on the library clients. Why would TSM do this? So I have a device class called IBM-LTO-3 on all my TSM instances (8 total including the library manager) and on the library manager there is a device class called NAS-DEV-CL. I show tons of tape volumes using the NAS-DEV-CL devclass on the library manager's volume history but not on the other instances (since they don't even have the NAS-DEV-CL devclass defined). Why or how would this occur? I tried an audit library on one of my library clients but it is not completing (I've had issues with audit library commands from library clients before). Any ideas?

Here is an example of what I see:

tsm: PD-703-S-AITSM-1>select * from volhistory where volume_name='L40202'

DATE_TIME: 2009-08-19 03:33:07.000000
UNIQUE: 0
TYPE: REMOTE
BACKUP_SERIES:
BACKUP_OPERATION:
VOLUME_SEQ:
DEVCLASS: NAS-DEV-CL
VOLUME_NAME: L40202
LOCATION: PD703-UAX007
COMMAND:

When I query the volume on the library client I see:

Volume Name: L40202
Storage Pool Name: NP-STD-TAPE
Device Class Name: IBM-LTO-3
Estimated Capacity: 1.6 T
Scaled Capacity Applied:
Pct Util: 0.4
Volume Status: Filling
Access: Read-Only
Pct. Reclaimable Space: 26.1
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 14
Write Pass Number: 1
Approx. Date Last Written: 08/26/09 11:01:12
Approx. Date Last Read: 09/22/09 11:19:00
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
Last Update Date/Time: 09/14/09 13:14:21
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: None

So the question is how did this volume history record on the library manager end up with the devclass that doesn't exist on the library client? It doesn't seem to affect the volumes or processing daily tasks, but I am worried its not freeing up the tapes to return to a scratch status when they are reclaimed.

What do you think?

2009-09-24T05:06:00.000-07:00

Smarter Tivoli Storage Manager Reclamation Configuration Tool

This is what we need?

Personal Backups

2009-09-17T09:31:00.001-07:00

A friend of mine lost a bunch of family pictures and video when his hard drives died (he lost both at the same time). As backup people we are all well aware that the current home computer user does not have an easy and safe way to backup all their data easily. Oh, sure they can buy a USB drive but those can also be destroyed in a fire, flood, or even stolen. There are some online options but wanted to ask those out there what their experiences are with them and any suggestions. I currently have 200+GB I need protected so what are the safest options...and no I wont be buying a tape drive. ;-)

Scalar i2000 Frustrations

2009-08-20T13:39:00.000-07:00

So for any Scalar i2000 users out there, why is it at every D.R. test I have to reboot the library to get the robot to discover correctly? Is there an inherent issue with the internal switch? I have done 3 D.R. tests with i2000's and each time I have to get the D.R. site people to reboot the darn thing before I can detect the robot. What's up with that? How do those of you who use them daily like them? I get a bad taste in my mouth when I hear I have to use one. At least it had IBM drives in it.

We found this bug

2009-08-11T09:17:00.000-07:00

We found this bug in Hungary. Click for details!

TSM 5.5 to 6.1 Video

2009-08-06T10:15:00.000-07:00

If you don't already subscribe to IBM's TSM Information Update & Storage Newsletter then you might not be aware of the following video IBM has posted to their website. They have provided a video tutorial upgrading TSM 5.5 to 6.1. Check it out here.

TSM Server Scripts Sleep Option

2009-07-23T21:59:00.001-07:00

I have been frustrated with the lack of a sleep command with TSM scripts for a long time and just today a co-worker sent me this link which I think sums up how we've had to get around the it within TSM server scripts. The WAIT=YES option is only good for commands that allow it, and is not exactly what many of us need in certain situations. Since this document's modified date shows as 7-23-2009, I will take it that there is no respite from the issue with TSM 6.1. Basically IBM recommends you break the script in two and where you need the sleep you have the script schedule a onetime schedule of the secondary script for X number of minutes in the future. It does work. I've use this process on some scripts, but you'd think they would have added a sleep option by now. So I assume when it's absolutely needed many of you resort to a regular shell script to execute tasks, any alternate processes than that? I'd like to hear about it.

TSM's VTL DeDupe Reclaim Problem

2009-07-17T09:58:00.000-07:00

So a friend of mine sent me this link to an article from Scott Waterhouse's "The Backup Blog" that discusses a known issue with TSM not reclaiming VTL tapes when dedupe is in use. I would recommend you check it out if you are using dedupe in your environment since it looks like it affects TSM no matter what dedupe product is in use when you use it with a Virtual Tape Library. The problem is solved with release 5.5.2 or higher.

Still Looking For TSM Admins In Boulder, CO

2009-07-07T11:03:00.000-07:00

Hello,
My name is Arjun and I'm a recruiter at Artech.

Artech has an urgent contract for one of our direct clients:

Job Title: TSM Administrator
Location: BOULDER, CO
Job Description:
Required Skill: TSM support

If you are qualified, available, interested, planning to make a change, or know of a friend who might have the required qualifications and interest, please call me ASAP at (973) 993-9383 Ext.3319, even if we have spoken recently about a different position. If you do respond via e-mail please include a daytime phone number so I can reach you. In considering candidates, time is of the essence, so please respond ASAP.

Artech is a global IT Consulting company with over 30 Fortune 500 customers. You may visit our website at http://www.artechinfo.com/ to learn more about us.

Thank you.
Sincerely yours,
Arjun Dheer
(973) 993-9383 Ext.3319
arjun_dheer@artechinfo.com

Rosetta Stone for UNIX

2009-06-15T13:36:00.001-07:00

Know AIX? Ever wonder what that simple command in AIX is on Solaris or HP/UX? Need to know what the command for tcp/ip management is in Linux compared to OS/X? Now you can! Check out this helpful website Rosetta Stone for UNIX. My boss passed this along a while ago and I thought I had passed it along to the community, but I cannot remember if I did. So, here is a great website for UNIX admins and those learning UNIX to help you transition from one vendor to another.

TSM Checkout and ACSLS Question

2009-06-12T11:07:00.001-07:00

At Infocrossing we have a tape operations group and they handle the physical opening of the cap and insert returned tapes and removing tapes for offsite. Because of this I didn't realize (until I had to handle the checkout myself) that TSM is checking out my tapes by filling the cap and pausing, waiting for the operations people to take the tapes out, then resuming the eject. OK, normal process except I have two 40 tape cap doors. When an eject has more than 40 tapes I figured that TSM would use the next cap door when I leave a specific cap identifier out of the checkout command, but it does not use the alternate cap. Both caps are set to a priority higher than zero so I find it frustrating TSM only uses one cap door. ACSLS itself allows you to use 0,0,* and it will checkout to all available caps above priority zero, but TSM does not accept the *. So what am I missing here? Is it possible with TSM to use both cap doors and not have the checkout pause for the one cap to be emptied? I didn't see anything suggesting REMOVE=BULK is any different from REMOVE=YES. I searched Google but didn't fine what I was looking for, and didn't find anything on ADSM.org so any ideas? Here's an example of my command:

CHECKOUT LIBVOLUME LIBRARY1 REMOVE=YES VOLLIST=FILE:/tsm/logs/chkout.list

Whether I specify a cap door or not it only uses one of the two.

Technical Flash

2009-06-05T07:25:00.001-07:00

This was in the TSM Info e-mail sent out yesterday. The last one is worth looking into since a bad System State backup can ruin your whole day.

News & Technical Flashes

Title: TSM V61 Server may crash due to exhausted system resources

Title: Backupsets Generated with non-IBM LTO Generation 4 Drives Can Exhibit Overwritten Labels

Title: Tivoli Storage Manager 6.1 Installation Failure on AIX 5.3 due to AIX prerequisite APAR check

Title: Incorrect Windows system state backups and restores with TSM 5.5 and 6.1 Windows clients

Windows 2003 Volume Shadow Copy Service (VSS) Hotfixes

2009-06-04T21:00:00.000-07:00

If you are need of VSS hotfixes checkout this page from IBM. I plan to add it to the link list...might start a list of links to patches and updates, but that could be difficult to keep current. I know this is an older reference but I needed the fixes recently and it seems the VSS issues don't go away, they just keep coming back with each patch of Windows.

Need a job? Willing to move to Boulder, CO?

2009-06-02T23:11:00.001-07:00

So with the state of the economy and many people out of work it's good to know IBM is hiring. IBM is looking for solid TSM candidates for various accounts, the only caveat is that the positions require you work from the Boulder, CO IBM facility. So if you are willing to relocate yourself, consider contacting a friendly IT recruiter from Artech or CDI and let the hiring begin!

TSMExpert Changes

2009-05-30T22:01:00.001-07:00

So I have recently made numerous changes to the site and hope you all like it. I was finally able to get the inline comment piece working and made changes to the categories list and template. So what I have realized is the following:

Blogger is somewhat restrictive when your site outgrows normal blogging

WordPress.com is inadequate in its customization to use as a alternative

I need to consider a host that offers WordPress for more functionality
(GoDaddy! probably)

I need to learn XHTML and CSS

This took to much of my spare time
(I have 4 kids, I don't have spare time!)

I hope you like the changes and if you have any suggestions or comments I will add those shortly. Also the contributors list will be back up when I figure how they changed the widget.

Calculating Active Data

2009-05-22T10:32:00.000-07:00

I was recently asked to calculate the amount of active data in TSM storage for file system backups, not TDP's and had some interesting results. If you search TSM active data in Google the first result will be this IBM support doc that explains how to calculate active data to help size an Active Data Storage Pool. IBM recommends that you use the EXPORT NODE command with PREVIEW=YES to determine the amount of active data. This, in theory, should work well but for TSM to process this request it has to analyze the backups table and who knows what else to gather the data. I have 10 instances I needed to gather the information from, they all vary in TSM DB size, and the amount of managed data stored. My smallest DB is a new instance that is 25GB and my largest is 155GB and size did not matter when it came to how fast the information was calculated. The TSM instance with the largest DB completed the taks in over two days (YES TWO DAYS!). Two TSM instances were still running the EXPORT NODE query after THREE DAYS and they had moderate to large sized DB's.

So what caused this problem? It all comes down to the number of files TSM has to inspect. The two instances that never completed the query have large numbers of Windows nodes and have the most registered nodes overall. These two instances seemed to be crawling through the process and where they should have calculated into the ten to twenty TB, as the next largest instance did after just over two days, the problem two were still in the 7 to 6 TB range and increasing slowly. My only explanation (and this is a guess) is that due to the fact that Windows servers tend to have hundred of thousands if not millions of files which TSM gets bogged down trying to inspect them all. I didn't notice a performance impact but IBM claims that it is a resource intensive task and should be run during non-peak hours. How can you do that when it runs 24hrs or more?

Finally after three days, and no end in sight, I canceled the processes and now have to figure out some other way to calculate the amount of active data they have stored. I could calculate (i.e. guesstimate) by summing the amount of space used per file space.

Example:

select cast(sum(capacity*(pct_util/100)) AS decimal (18,2)) As Total_Used_Space from filespaces where node_name in (select node_name from nodes where domain_name like '%STD%')

The problem is that this will not be accurate and will probably cause me to oversize any Active Data Pool I create. Now that's not a horrible thing (more space is always better than to little) but the whole process seems to time consuming for something that on the surface should be fairly easy to calculate. This where I hope the new DB2 database can help, but until someone has it up and running and can try this process we can't say if there is a reasonable solution to find the total active data in TSM within larger instances.

Addendum

Are wondering why I used the '%STD%' filter in the select statement above? The team here at Infocrossing separates the file system backups and application (TDP) backups with different domain names using a standardized naming process. This is great because it also allowed us to run the EXPORT NODE command using wild cards for the specific domains to include in the export query.

EXPORT NODE * DOMAINS=*STD* FILEDATA=ALLACTIVE PREVIEW=YES

I highly recommend you follow a similar process when creating domains and even schedules to make it easier to process groups of nodes. So you could create a WIN-STD-DOM domain or like we do for our TDP's WIN-APP-DOM. These are examples but they can make life easier.

New Look!

2009-05-21T23:12:00.000-07:00

I am trying a new look for the site. I made the change to resolve some problems with commenting and items that broke over the course of the last year. I could have used the old template but thought I would try something new. Let me know if you like it. If not I can always go back to the clean and simple template I was using. If anyone has a suggestion for look and feel leave a comment. I love new ideas.

TSM 6.1 Upgrade - Need To Know!

2009-05-14T09:43:00.000-07:00

So In researching the TSM 5.5 to 6.1 upgrade I have come across a number of issues that should have been compiled into a complete list to keep admins informed. So here it goes.

Things to know about TSM 6.1:

Although stated that the 6.1 DB should be the same size as the 5.5 DB the TSM community is claiming as much as 4x the space is required
It does not support RAW volumes for the TSM DB and Log
It has added archive logging to the existing log process (i.e. more disk space required to run)
It cannot generate or define backupsets at this time
It does not support NAS or Backupset TOC's at this time
It will not allow a DB upgrade if TOC's or Backupsets exist on the current TSM server
NQR (No Query Restore) is now slower due to how 6.1 processes the data before sending it to the client

I have been hearing of upgrades taking an extremely long time so be aware that if doing an upgrade the TSM source instance has to be down to allow the upgrade when doing it across the LAN or on the same server. Even with the media method your source instance has to be down to perform the DB dump since 6.1 cannot use a standard TSM DB backup.

TSM 6.1 Upgrade - FYI

2009-05-05T09:41:00.000-07:00

For those of you looking to upgrade your current TSM instances to 6.1 take note of this issue with upgrading the DB.

At this time a database containing backup sets or tables of contents (TOCs) cannot be upgraded to V6.1. The database upgrade utilities check for defined backup sets and existing TOCs. If either exists, the upgrade stops and a message is issued saying that the upgrade is not possible at the time. In addition, any operation on a V6.1 server that tries to create or load a TOC fails.

When support is restored by a future V6.1 fix pack, the database upgrade and all backup set and TOC operations will be fully enabled.

I haven't heard if this will be fixed in the first patch of 6.1 but keep it in mind when considering upgrading your system rather than starting from scratch.

TSM 6.1 Debian Client Request

2009-04-24T08:17:00.000-07:00

Well TSM 6.1 has officially been out for about a month now and I have already received a request for a Debian client. Maybe Harry can help out in that area. Anyone want to try their hand at putting one together?

ACSLS Checkout Script Update

2009-04-22T06:58:00.000-07:00

I Modified the ACSLS checkout script due to an Admin with BYU pointing out it doesn't work when the library clients reside on seperate servers. It was changed due to how TSM processes the CMDFILE= option in my MOVE DRM macro, plus if your TSM server names vary the while loop is not as easy to setup. So here is the new script and macro. I switched it to use an array for the servernames for easier processing when names don't follow a similar convention. Also you might need to modify the grep if your volumes don't start with L0 thru 4. If your volser is a different range just modify the grep so it will work for you.

<---------------------MOVE DRM MACRO------------------------>

move drm * so=dbb wherestate=mo tostate=vault remove=no WAIT=YES
move drm * so=dbb wherestate=vaultr tostate=onsiter

<------------------------Cut Below-------------------------->

#!/bin/ksh

cd /usr/tivoli/tsm/client/ba/bin

ADSMID=`cat /usr/local/scripts/ADSMID`
ADSMPASS=`cat /usr/local/scripts/ADSMPASS`

OFFSITE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/offsite.txt
DRVOLS=/usr/tivoli/tsm/client/ba/bin/OFFSITE/Vol_List
RETRIEVE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/retrieve.txt

set -A SERV TSM-1 TSM-2 TSM-3 TSM-4

cp $OFFSITE "$OFFSITE.BAK"

mv /usr/tivoli/tsm/client/ba/bin/OFFSITE/Vol_List /usr/tivoli/tsm/client/ba/bin/OFFSITE/Vol_List.bak

printf "Use this list to determine tapes that are to go offsite report any discrepancies to the Recovery Services Team.\n\n" > $OFFSITE
printf " \n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE
printf " The following tapes will checkout from the library\n\n" >> $OFFSITE
printf " and should be sent offsite\n\n" >> $OFFSITE
printf " Current as of: `date`\n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE

printf "Use this list to determine tapes that are to come back onsite from Iron Mountain for reuse. Report any discrepancies to the Recovery Service Team.\n\n" > $RETRIEVE
printf " \n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE
printf " Tapes to be brought back onsite from Iron Mountain\n" >> $RETRIEVE
printf " and placed back into TSM library for scratch.\n\n" >> $RETRIEVE
printf " Current as of: `date`\n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE

i=0
while [ $i -lt 4 ]
do
dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=${SERV[$i]} -dataonly=yes "select volume_name from drmedia where state='MOUNTABLE' and volume_name not in (select volume_name from drives where volume_name is not NULL) " grep L[0-4] >> $DRVOLS

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=${SERV[$i]} -dataonly=yes "select volume_name from drmedia where state='VAULTRETRIEVE' " grep L[0-4] >> $RETRIEVE

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=${SERV[$i]} 'macro /usr/tivoli/tsm/client/ba/bin/move_drmedia.mac'

sleep 120

i=$(( $i + 1 ))
done

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=

CHECKIN LIBVOL search=yes stat=private checklabel=barcode vollist=FILE:$DRVOLS

sleep 180

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=

CHECKOUT LIBVOL checkl=no remove=bulk vollist=FILE:$DRVOLS

cat $DRVOLS sort >> $OFFSITE

mail -s "Daily Tape Rotation - Tapes to be sent to Iron Mountain" tape_rpt < $OFFSITE mail -s "Daily Tape Return" tape_rpt < $RETRIEVE

TSM Admin Needed In Bahrain!

2009-04-17T11:35:00.000-07:00

I could use some help here in Bahrain.
Need some certified Tivoli people ASAP.

Please answer all the questions below as best you can upfront as it will save us both time and get you submitted quicker to the MGR/Client.

Sometimes the client will cover expenses and sometimes they won't, so give me a rate for both below.

THIS HELPS ME SPEED UP THE PROCESS!!!

CANDIDATE:
POSITION:
YOUR CURRENT LOCATION:
CONTACT #'s :
EMAIL:
RATE ALL INCLUSIVE NO EXPENSES COVERED (Corp to Corp or 1099) :
RATE + EXPENSES:
CITIZENSHIP:
IF YOUR AN H-1 PROVIDE/(EMAIL & PHONE NUMBERS) OF THE ACTUAL COMPANY WHO CURRENTLY HOLDS YOUR H-1, AND WHO I SHOULD ASK FOR ABOUT YOUR SERVICES?
Does the person have any Pending offers? If so, when do you need an answer by from me?
How many years does this person have left on his H-1 in the US before it expires?
AVAILABILITY:

Please tell me how many years you have with EACH Skill listed in the job as best you can and rank yourself with each skill from 1 to 5--meaning advanced/expert as the top end.

SUMMARY OF WHY YOU FIT THIS ROLE , BE SPECIFIC:

Forrest Murphy
Sr Information Technology/Engineering Recruiter
ERP-Oracle, SAP, Peoplesoft, JDE
972-607-9897 Direct Dial
214-207-2478 Cell
972-496-2398 Fax
850-205-2185 Fax for Corp
509-277-2478 Email Fax
fmurphy@csifl.com
IM Address: fwnynj@yahoo.com

Consulting Solutions International
3512 Maclay Blvd, South
Tallahassee, FL 32312
www.csifl.com

New TSM Management Tool

2009-04-14T13:48:00.000-07:00

I have been informed of a new TSM management tool called TSM Studio. It looks pretty slick, but I won't lie, I haven't had a chance to use it yet. If you are looking for a TSM tool check it out. I plan on posting a quick overview of all the TSM tools in the near future. If you have experience with any TSM tool (i.e. TSMManager, TSM Explorer, TSM Studio, AutoVault, Servergraph, Aptare, Bocada, or EMC's DPA) and are willing to write up a review of the product, let me know and we'll publish it here for others to use when deciding on a tool for their own environment.

Don't Miss - TSM V6.1 DB Upgrade Webcast

2009-04-14T11:18:00.000-07:00

If you are looking to move to TSM V6.1 in the near future don't miss this webcast on April 28th at 11:00am EST. This is part one in what I believe is a two part webcast (could be more than two). You can either click on the title of this post or find the sign up here.

TSM 6.1 Docs Online

2009-03-27T13:39:00.000-07:00

I found this link listed on ADSM.org and thought I would pass it along. Here is a link that will take you to the TSM 6.1 documentation. Supposedly the upgrade doc alone is over 200 pagers. Not sure if any of you are "dying" to try 6.1, but I am planning on testing it soon.

IBM Considers Buying Sun!

2009-03-20T11:45:00.001-07:00

Did I.Q.'s just drop substantially around the world? Go sell crazy somewhere else! Why would IBM even consider buying Sun? Definitely not for their Server technology, it's gotta be for their software portfolio. Most people are not aware that IBM's makes more money from their software division than hardware. (At least they used to...still think they do). But still, I gotta think Sun would be better served by a buyout from HP or Cisco...maybe Dell, but Dell seems uninterested from what I have read.

Question About Active Data Pools

2009-03-12T16:47:00.000-07:00

Geoffrey Huntley recently asked me what situation would an Active Data pool be useful? To be honest, I couldn't think of a good situation. Since I don't use an Active Data Pool I thought I would throw the question out to our fellow readers to get some feedback as to how you might be using ADPs. My biggest reason for not using them was the PIT restore issue. You can read my full gripe here. Basically if TSM wont utilize it when doing a PIT then what's the benefit? I'd say more than half, if not 75%, my restores are from older dates.

Active Data Pool - What's The Point?

2009-03-12T08:58:00.000-07:00

To go along with the ADP story I have moved this older post up for easy reference.

With the release of TSM 5.4 Tivoli has added the ability to create an active data storage pool to allow for faster restore times. We have been looking into using them at work when I stumbled upon this interesting factoid in the description of the active data pools limitations:

The server will not attempt to retrieve client files from an active-data pool during a point-in-time restore. Point-in-time restores require both active and inactive file versions. Active-data pools contain only active file versions. For optimal efficiency during point-in-time restores and to avoid switching between active-data pools and primary or copy storage pools, the server retrieves both active and inactive versions from the same storage pool and volumes.

So my question is why don't they allow the client to restore all the active data first then restore the inactive, or why didn't they implement a multi-session restore process when they added active data pools to the product, thereby speeding up the restore process? With the amount of P-I-T restores I do, this issue makes the whole active data pool useless not just for me but I'm sure for many of you.

TSM 6.1 Technical Overview Presentation

2009-02-26T09:45:00.000-08:00

IBM gave a Technical Overview of TSM 6.1 today and I thought I would post the links to the presentation for everyone. To go to the IBM presentation page click here.

To replay the presentation you can click here.

Who Did What?

2009-02-17T10:13:00.000-08:00

This weekend I had an issue with a backup failing on a client using LAN-Free. When checking the TSM actlog the error I kept seeing was:

ANR9999D (Session: 265500, Origin: OD0BG-UAX001-STA) mmsshr.c(3874): ThreadId<13> Unable to obtain model type for '/dev/rmt23', rc = 46(SESSION: 265500)

So I had a good idea that the client and TSM Storage Agent were having issues communicating. I knew this because I had seen this error before when the TSM client and TSM Storage Agent were not at the same release levels. Checking the TSM client I found it at 5.4.0.0 and the Storage Agent at 5.3.4.0. Turns out an admin upgraded the client without knowledge of the Storage Agent, causing the backup failures. This is an example of what can happen when someone tries elbow their way into another groups area without the knowledge to do it right.

ART: Restore Testing software for TSM

2009-01-14T09:35:00.000-08:00

We have a new product that does something unusual.

ART (Automated Restore Testing for TSM) test-restores a random sample of files. And it does this for every node at your site, automatically, on a schedule.

We don't think anyone has ever done this kind of comprehensive restore testing before.

ART has uncovered problems at every customer site it has tested. The problems are usually operational, and often easy to fix.

If you'd like to give the free trial a whirl (it's full-featured but limited to 20% of your nodes), go to www.tsmworks.com/download. We appreciate feedback from TSM experts out there.
Thanks!

TSM & File System Support

2009-01-07T07:15:00.001-08:00

First off I hope everyone had a good holiday season. Now that we can focus on work again I wanted to discuss a topic of file system types. I just had an incident where a Solaris server had ZFS used for some newer file systems. The admins had added them without consulting us, and we didn't catch it because TSM didn't even attempt to back them up. Our client level was 5.4.1.0 and ZFS support was added with the 5.4.1.2 update. Once I updated the client the file systems were backed up successfully and show the correct format. We did see one file system was returning a type of UNKNOWN and that should have alerted us, but we were not receiving errors or failures on the backup of the server in question.

So here is the question, how do you keep something like this from happening in the future as new, more bleeding edge file system types are added? Obviously you need to inform your Unix Admins to work with you whenever they add a newer file system type, but if they don't alert you, and TSM doesn't report failures, how would you know? It's bound to happen as the Linux community adds newer, more robust file system types. Other than stay as current as possible with my TSM client levels (which wont always be the fix) what would you suggest?

LAN-Free Unknown Feature

2008-11-24T12:44:00.000-08:00

My boss brought this article to my attention so I thought I would pass it on. It discusses an unknown or little discussed capability of LAN-Free to act as a "pass-thru" server for other clients. In other words a normal client can connect to TSM through the LAN-Free agent that resides on another server. Basically the Storage Agent becomes a dummy TSM server. This can be helpful when you have backups that need to go to tape directly but the network connection between the client and the TSM server is hurt by inadequate bandwidth between switches, or firewall issues. You can check the article out here. I remember seeing this discussed on ADSM.org before, but have never attempted to try it.

Texas Government Pulls Plug on IBM

2008-11-18T09:33:00.000-08:00

Well if you get a chance check out this article on ZDnet. It looks like IBM is losing the Texas government account due to backup failures and lost data. Having previously worked for IBM what I would like to know is what is not being said. When IBM takes over an account on an outsourcing deal they have to assume the horrible practices already in place then do their best to convert to a better software/hardware solution and processes. For example when I was with IBM we had a remote site that was using Arcserve for their backups and had two weeks worth of tapes. A couple months into the contract they lost a server and needed to restore the data and guess what...they couldn't. The process in place was for them to rotate two weeks worth of tapes continually. The tapes were four to five years old and had never been tested. They found out after the fact that the tapes had gone bad some time in the past and they were pretty much out of luck. Is this IBM's fault? They took some of the blame even though IBM was only a couple months into the contract and the smaller remote sites were secondary in the process change timeline.

The Monring News did report the following:

In a Nov. 3 letter to the governor’s office, IBM acknowledges the company overreached by assuming responsibility for existing technological conditions that are inadequate, inconsistent and not sustainable.

AIX Image Backup Performance Issue

2008-10-25T10:36:00.000-07:00

I am running 3 simultaneous image backup jobs on an AIX server and the HBA only shows 20% utilized and my throughput sucks. I am using -imagetype=static and the FS's in questions are EMC clones split off and remounted to a new server. When I have done image backups before they were wicked fast...now not so much. Any ideas what I might have missed? I have an ACSLS STK 8500 with LTO-3 drives. Even over gig-ether I've seen better performance. We did check the CPU and memory usage and they are fine. Lots of I/O wait, not sure why I would see that with statc image backups.

TSM facebook Group

2008-10-22T09:35:00.000-07:00

If you're on facebook and would like to network with other TSM admins check out the new TSM group. It's currently looking to grow and can allow us all to network for information, jobs, tips, and just getting to know each other. Thanks to Henrik Wils for creating it!

EMC Backup Advisor

2008-10-16T09:46:00.000-07:00

I am currently working a project to install and setup EMC Backup Advisor in our environment. If anyone has experience with the product I'd like to hear your thoughts and opinions. So far it seems like it can monitor almost anything. Hopefully it doesn't become such a huge/complex tool that we'll spend half our time trying to manage it.

Layout Issue

2008-09-23T09:22:00.000-07:00

For anyone who liked the Table Of Contents feature, I have had to temporarily remove it because the javascript was not playing nice with IE or Firefox (at least not on my system). The only browser that handled it correctly was Google Chrome. I hope to have it back soon, I liked the feature.

The BACKUP VM command is not supported on this client (ANS1184W)

2008-09-17T19:35:00.000-07:00

Just a heads up, so others don't run into this - TSM Integration w/VCB is not supported on x64.
As per this posting to ADSM-L by Andy many months ago:

At the time 5.5 VCB support was being developed, the 64-bit VMware VCB framework was not available (though I think one has since been introduced). So TSM support is currently 32-bit Windows 2003.

Good news is that 'backup vm' works on the x64 platform when using the x32 client.

x64 binaries - v5.5.1.0 - C:\progra~1\tivoli\tsm\baclient
x32 binaries - v5.5.1.0 - C:\progra~1\tivoli\tsm\baclient32
C:\progra~1\tivoli\tsm\baclient32.dsmc.exe backup vm -vmlist="fileserver[fileserver_fs]"

Storage Agent Device Issue

2008-09-17T01:19:00.000-07:00

So one of my co-workers was tasked with figuring out why a drive was shown in TSM as being assigned to a storage agent when the storage agent was not actually using it (we had even restarted the agent and the library controller still showed the agent as owner of the drive). As it turns out there is an issue with down level TSM device drivers being used between an AIX TSM server and Windows Storage agents. You can find the APAR here. Here is a brief rundown of the problem.

After a review with the Windows and AIX Device Driver development team, it was determined that one of the Windows STA's placed a persistent registration on a drive, failed the persistent reservation, and then failed to release the registration during the recovery path. The registration causes a problem for AIX but not Windows due to the differences in SCSI 2 and SCSI 3 used by Windows and AIX. This hasbeen fixed and the recommendation was to upgrade to the current Windows Device Driver - 6.1.9.3. If you are not at 6.1.9.3 or higher, this is most likely your causing your problem.

You can download the latest driver from here:
ftp://ftp.software.ibm.com/storage/devdrvr/Windows/

Here is the tsm apar I mentioned:
http://www-01.ibm.com/support/entdocview.wss?uid=swg1IC57753&NotUpdateReferer=

Defaults should never be trusted

2008-09-01T16:14:00.000-07:00

I've performed a few audits recently and this one keeps popping up, again and again!

Please be aware that the defaults provided when defining a new storage pool will turn on collocation by GROUP. If this default is specified and no collocation groups are defined then the TSM server will collocate data by NODE!

Pleasant little surprise eh?

COLlocate
Specifies whether the server attempts to keep data belonging to a single client node, group of client nodes, or client file space stored on as few volumes as possible. This parameter is optional. The default value is GROUP.

If you specify COLLOCATE=GROUP but do not define any collocation groups or if you specify COLLOCATE=GROUP but do not add nodes to a collocation group, data is collocated by node.

It never hurts to double check everything, here's some queries I use all the time to sanity check collocation: (replace the stgpool_name as needed)

select stgpool_name, access, collocate, maxscratch, reusedelay from stgpools
select count(distinct node_name) as "Number of Nodes", volume_name from volumeusage where stgpool_name='%_TAPE' group by volume_name order by "Number of Nodes" desc
select count(distinct volume_name) as "Number of Tapes", node_name from volumeusage where stgpool_name='%_TAPE' group by node_name, stgpool_name order by "Number of Tapes" desc
select count(distinct volume_name) as "Number of Tapes", node_name, stgpool_name from volumeusage group by node_name, stgpool_name order by "Number of Tapes" desc

Mislabeled Tapes Headache

2008-08-23T10:50:00.000-07:00

I am periodically running into a situation where TSM will check in our tapes as private or relabel them to private, causing me headaches until I get the tapes status updated. So how do you identify the tapes that are mislabeled? Well if you have setup a library controller instance (which I recommend even if you will only have one instance for clients) then this script should work for you. What you will find when tapes are mislabeled is that they will show the library controller as their owner, will have a NULL value for LAST USE, and will of course be in private status. So a simple select statement can identify them for use.

select VOLUME_NAME from LIBVOLUMES where volume_name not in (select volume_name from volumes) and status='Private' and last_use is NULL and owner=''

If your library uses different types of media (LTO-2 and LTO-3) and you have different volume label series for them you can add another filter to further refine the results. For example my LTO-3 tape labels all start with L3 so I use:

select VOLUME_NAME from LIBVOLUMES where volume_name not in (select volume_name from volumes) and status='Private' and last_use is NULL and owner='' and volume_name like 'L3'

Once TSM has generated a list you can then run it through a while loop to reset their status or do further refining of the list if you like. If you are proficient with scripts you can automate the whole process and make life a lot easier.

Rebinding Archives

2008-07-17T09:23:00.000-07:00

I would like to thank Nate Tade for allowing me to link to his article on how to extend (rebind) the retention for archives due to expire. I haven't had to extend archives so this is a good "How To". You can find the article by clicking the title of this post or going here.

Arjun Kumar Dhir Sr. Executive Staffing
Artech Information Systems LLC 240 Cedar Knolls Road, Suite 100 \| Cedar Knolls, NJ 07927
Office: 973.967.3427 \| Fax: 973.998.2599
Email: Arjun_Dheer@artechinfo.com \| Website: www.artechinfo.com