Win2K & Win2003 - Data Currently Unavailable On Server

This past weekend we performed a DR test on quit a few servers and discovered a situation that was a wake up call for anyone still using Windows 2000 or Windows 2003. The issue is that when we ran a restore the IBM Spectrum Protect 8.1 server issued a warning on all the files for the particular node with the following message type…

ANE4035W (Session: 79, Node: ODC-WINCS-1-DR)  File '\\od-wincs-1\e$\834\c-oh\files-20170823-123000' currently unavailable on server.

The tape was in a readonly state and accessible by the library so it was not a tape unavailable problem. I then querried the file and ran some tests suggested by IBM in the following link

http://www-01.ibm.com/support/docview.wss?uid=swg21249032

The data is present and on tape but TSM will not restore it. (See the queries below) IBM would not help us because the OS is unsupported as is the TSM client. We could not restore the System State, System Services, or any of the Data to the DR replica. What we could do is access the data from a supported client level on a supported OS level (i.e. TSM Client 6.4 from a Windows 2008 server). We were unable to recover the Win2K/Win2003 servers during the DR test and unfortunately I know many companies out there still have some of these servers in their environment. It's time to upgrade! Your data is at risk!

tsm: OMATSM02>select * from backups where node_name='ODC-WINCS-1-DR' and FILESPACE_ID=1 and hl_name=upper('\834\c-oh\') and LL_NAME=upper('files-20170823-123000')

      NODE_NAME: ODC-WINCS-1-DR
FILESPACE_NAME: \\od-wincs-1\e$
   FILESPACE_ID: 1
          STATE: ACTIVE_VERSION
           TYPE: DIR
        HL_NAME: \834\C-OH\
        LL_NAME: FILES-20170823-123000
      OBJECT_ID: 11926208445
    BACKUP_DATE: 2018-08-10 03:04:37.000000
DEACTIVATE_DATE:
          OWNER:
     CLASS_NAME: WIN-DR-MC
    ACTUAL_SIZE:

tsm: OMATSM02>show bfo 11926208445

Bitfile Object: 11926208445
  Active
**Sub-bitfile 11926208445 is stored in the following aggregate(s)
Super-bitfile: 11926199674, Offset: 26157087, Length 735, Deduped: F


tsm: OMATSM02>show bfo 11926199674

Bitfile Object: 11926199674
**Super-bitfile 11926199674 contains following aggregated bitfiles,
Bitfile Id, offset, length, active state or owner, link bfid
11926199674   0             8732          Active
11926199676   8732          1527          Active
.
.
.
11926208946   26573407      5510          Active
11926208947   26578917      49168         Active

**Sub-bitfile 11926199674 is stored in the following aggregate(s)
  Super-bitfile: 11926199674, Offset: 0, Length 8732, Deduped: F

**Archival Bitfile Entry
  Bitfile Type: PRIMARY  Storage Format: 22
  Bitfile Size: 26631381  Number of Segments: 1, flags: 0
  Storage Pool ID: 33  Volume ID: 22903  Volume Name: L42803


tsm: OMATSM02>q vol L42803 f=d

                   Volume Name: L42803
             Storage Pool Name: WIN-DR-TAPE
             Device Class Name: IBM-LTO-4
            Estimated Capacity: 4.2 T
       Scaled Capacity Applied:
                      Pct Util: 100.0
                 Volume Status: Full
                        Access: Read-Only
        Pct. Reclaimable Space: 0.0
               Scratch Volume?: Yes
               In Error State?: No
      Number of Writable Sides: 1
       Number of Times Mounted: 227
             Write Pass Number: 1
     Approx. Date Last Written: 08/12/18   16:44:25
        Approx. Date Last Read: 08/25/18   22:06:34
           Date Became Pending:
        Number of Write Errors: 0
         Number of Read Errors: 0
               Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
         Last Update Date/Time: 08/25/18   20:29:37
          Begin Reclaim Period:
            End Reclaim Period:
  Drive Encryption Key Manager: IBM Spectrum Protect
       Logical Block Protected: No



tsm: OMATSM02>q libvol dr-libr L42803

Library Name     Volume Name     Status               Owner          Last Use      Home        Device
                                                                                   Element     Type
------------     -----------     ----------------     ----------     ---------     -------     ------
DR-LIBR          L42803          Private                             Data          1,089

Learn From My Mistakes

We recently did a DR test and ran into a serious issue with the recovery of the TSM instance. The DR test was to restore the TSM DB from data center A (DC-A) to data center B. The DC-A instance had been created the previous year so we only needed to remove the DB (DSMSERV REMOVEDB TSMDB1) and restore the DB. I followed the steps and restored the DB and after restoring the DB 99% the restore failed. The restore failure error pointed us to the size available being too small for the required. We were off by a couple GB in our DB directory sizes so we had to add another directory to the list. I tried to rerun the format of the space and quickly received and error that the DB was present. I then ran the DSMSERV REMOVEDB TSMDB1 command again then tried to rerun the format and received the same results. We tried multiple commands

db2 uncatalog db TSMDB1

db2idrop serverA

/opt/tivoli/tsm/db2/bin/db2greg -delinstrec service=db2,version=9.7.0.9,instancename=serverA,instancepath=/drtsmserver/serverA/sqllib

This last command allows you to manually drop the DB from the global registry. When we tried this a couple times the instance would be removed from the global registry and when we would try to rerun the restore we would get the error that the DB was present and the instance would reappear in the global registry. (Notice the instance path....that is going to come play into the final solution.)

So at this point we went round and round with various commands to drop/remove/delete the instance from DB2 and the commands would fail saying that the instance was not present and then an attempt to restore the DB would fail saying the DB was present. We went round and round and round until I finally restored the DB under a new ID and instance directory which was successful.....BUUUUUUTTTTTT.....there in lies another issue. Once the DB restore was successfully restored it failed to allow us to bring up the TSM instance because the new ID was not the original owner of the DB instance. Immediately I realized my mistake as this had happened to me years ago and I totally forgot this DB2 requirement. You see DB2 does not allow an ID other than those that have been granted permission to bring the DB up. I get it, it's a security thing, but the TSM instances almost all Admins create only has one active ID allowed ownership of the DB2 instance. You can add as many as you like but it's not something IBM or Tivoli openly suggest or recommend. So if the ID you typically use wont allow the restore or for some reason can't be used and you want to use a new ID for the DB your S.O.L. If it's after the fact, IBM cannot provide a way for you to assign an additional owner of the DB2 TSM instance. IBM's recommendation was go back to DC-A and grant the new ID rights to the instance, run a TSM DB backup, copy/replicate the instance to DC-B, then restore the DB and it would work under the new ID.

By this point in the DR process we were so behind in the recovery of the TSM server we scrapped our portion of the DR test (we were not a significant portion of test, which was to test Recovery Point capabilities and TSM was a fall back). I continued to work with IBM on the restore under the original ID and the looping issue and after a couple attempts, the IBM level 2 DB2 support person identified that the creation of the TSM instance had created DB files in

/drtsmserver/serverA/serverA/NODE0000

and a simple rename of that directory should the the DB present/not present issue. We had repeatedly cleared out the DB and Log file systems but had forgotten about the DB folder and files created in the instance directory. TSM had created that subdirectory with DB related files. We had renamed the /drtsmserver/serverA/sqllib directory more than once, but we had forgotten about the other instance directory as IBM states in this technote.  When we renamed the second serverA directory and tried the restore and it ran to completion successfully. HEAD SLAP! This was one of the times where even with another set of eyes we were not seeing the bigger picture. IBM level one support was probably looking at the wrong issue and level two was stumped because we moved away from the initial issue we called about and were now asking them how bring up the TSM DB that we restored under the alternate ID.

OK at this point I was exhausted and frustrated with myself. So what did I come away with in this situation....

1. Triple verify your DB file space size is adequate. (DOH!)
2. From the instance home directory copy the critical files to an alternate location.  
• This is in case the restore fails and clears them or updates them incorrectly (trust me this happened more than once)
3. If the restore fails or a rerun of the restore is required rename both <inst home>/<inst name>/<inst name> and <inst home>/<inst name>/sqllib
4. IMHO I would also create another ID or two on the production server and grant them access to the TSM DB instance just in case you had an issue with the primary ID. 
• This is not required but cant hurt.

Restoring TSM Without A Volhist

Someone in the comments to an old post just asked for directions/instructions on restoring TSM without a volume history or devconfig file. Well, I got some bad news and some not so bad but not fun news. We will start with the not so bad news. If you don't have a devconfig, don't panic! You can recreate the devconfig. That's fairly simple, just a pain. TSM has to have a devconfig file to initialize its devices so if the devconfig is not present you'll have to create one. Typically you do this when you rebuild a TSM instance. For example at a DR site you install TSM on the DR server, define the dsmserv.opt, and then you define base devices on the new install. Once that has been done you can bring down TSM and attempt a restore using the newly defined device(s). 

Now for the bad news. Without the volhist, if you don't know what volume(s) were used for DBBAckup your kind of screwed. The old DSMSERV DISPLAY DBBACKUPVOLUME command has been removed/deleted and IBM now says the following

DSMSERV DISPLAY DBBACKUPVOLUME - Information about volumes used for database backup is available from the volume history file. The volume history file is now required to restore the database. 

You can find a list of TSM Server deleted commands, utilities, and options at the following link.