New Version of TSMExplorer

There is a new version of TSMExplorer available and it has added support for Spectrum Protect Plus. See can download a trial version of 5.2.3 here.  TSMExplorer has a load of features and provides a great central management console when you have multiple TSM/Spectrum Protect Servers. With the added support of managing Spectrum Protect Plus it's even easier to manage your environment.

Here is a video of an older version of TSMExplorer so you can get a feel for the product.




Here is a video of the client agent and how it works with the TSMExplorer console to allow access to the client options files and logs for easier management.

Finally - A Universal Backup Agent!

I was contacted by a representative of Spictera about their SPFS tool for Spectrum Protect. This product allows you to use a local mount on your Linux server as a place to save your backup or application dump that is a direct connection to the IBM Spectrum Protect backend storage. It uses Spectrum Protect API calls and allows you to implement versioning and retention policies just as you would normal file system backups or archives. This product can solve your issues with proprietary applications that IBM does not, and probably never will, support. Now you can backup your MongoDB or SAP HANA a lot easier by using the native backup process built into your application. Currently it only supports the following OS's.

Supported operating systems are:

• CentOS 6 for x86_64
• CentOS 7 for x86_64
• CentOS 8 for x86_64
• Debian 8 for x86_64
• Debian 9 for x86_64
• Red Hat 6 for x86_64, ppc64 be, s390x (zLinux)
• Red Hat 7 for x86_64, ppc64 le, s390x (zLinux)
• Red Hat 8 for x86_64, ppc64 le, s390x (zLinux)
• SuSE 11 for x86_64, ppc64 be, s390x (zLinux)
• SuSE 12 for x86_64, ppc64 le, s390x (zLinux)
• SuSE 15 for x86_64, ppc64 le, s390x (zLinux)
• Ubuntu 14.04 for x86_64, ppc64 le, s390x (zLinux)
• Ubuntu 18.04 for x86_64, ppc64 le, s390x (zLinux)

I would assume if it support CentOS x86_64 you should have no trouble with it on Red Hat. I'll let Spictera confirm that though. I've provided a presentation on the product for your review.

HBA Info Script

I thought I would pass this script along for anyone needing a way to out HBA info on Linux in a nicely formatted table.

#!/bin/bash
echo "FC_HOST     |   PortID   |    WWN             | State      | Speed    "
echo "------------+------------+--------------------+------------+----------"
for hba in `ls -d /sys/class/fc_host/host*`;do
  FC_HOST=`basename $hba`
  PortID=`cat $hba/port_id`
  wwpn=`cat $hba/port_name`
  state=`cat $hba/port_state`
  speed=`cat $hba/speed`
  hba=`cat $hba/symbolic_name`
  echo "$FC_HOST| $PortID | $wwpn | $state| $speed" | awk 'BEGIN{OFS=FS="|"}{for(i=1;i<=NF;i++){$i=sprintf("%-12s",$i)};print}'
done #|sort -k3n,6
echo "------------+------------+--------------------+------------+----------"

The output looks like this

FC_HOST     |   PortID   |    WWN             | State      | Speed    

------------+------------+--------------------+------------+----------

host0       | 0xffffffff | 0x20003890a540876c | Linkdown   | unknown  

host11      | 0xffffffff | 0x20003890a540876d | Linkdown   | unknown  

host12      | 0x640000   | 0x100000109b1bbe00 | Online     | 32 Gbit  

host13      | 0xffffffff | 0x20003890a5803461 | Linkdown   | 40 Gbit  

host14      | 0xffffffff | 0x20003890a5803462 | Linkdown   | 40 Gbit  

host15      | 0x640000   | 0x100000109b1bbe01 | Online     | 32 Gbit  

host16      | 0x03eac0   | 0x100000109b1b9a77 | Online     | 16 Gbit  

host17      | 0x04eac0   | 0x100000109b1b9a78 | Online     | 16 Gbit  

------------+------------+--------------------+------------+----------

If you would like more detail I added a couple more columns to show make, model, firmware, and driver version. This is a fixed width column layout and the minimum column width can be adjusted for your needs.

#!/bin/bash
echo "FC_HOST         |   PortID       |    WWN             | State          | Speed          | Make           | Model          | Firmware       | Driver"
echo "----------------+----------------+--------------------+----------------+----------------+----------------+----------------+----------------+---------------"
for hba in `ls -d /sys/class/fc_host/host*`;do
  FC_HOST=`basename $hba`
  PortID=`cat $hba/port_id`
  wwpn=`cat $hba/port_name`
  state=`cat $hba/port_state`
  speed=`cat $hba/speed`
  hba=`cat $hba/symbolic_name`
  hba_make=`echo $hba | cut -f1 -d ' ' `
  hba_model=`echo $hba | cut -f2 -d ' ' `
  hba_fw=`echo $hba | cut -f3 -d ' ' `
  hba_driver=`echo $hba | cut -f4 -d ' ' `
  echo "$FC_HOST| $PortID | $wwpn | $state| $speed | $hba_make | $hba_model | $hba_fw | $hba_driver " | awk 'BEGIN{OFS=FS="|"}{for(i=1;i<=NF;i++){$i=sprintf("%-16s",$i)};print}'
done #|sort -k3n,6

echo "----------------+----------------+--------------------+----------------+----------------+----------------+----------------+----------------+---------------"

The output looks similar to this:

End of the Road?

For those of you who come here periodically and are wondering why content has been lacking, I wanted to let you all know that it is due to my current employment situation. Currently my employer is in the process of replacing Spectrum Protect throughout the company with Commvault. Once the conversion has completed I will be released and so am currently looking for new employment. Since the 8.1.8 upgrade of our servers I have been working steadily on reporting and prepping for the conversion. The initial thought was that I would stay employed and transition to learning Commvault, but that turned out to not be the case. I have enjoyed managing ADSM/TSM/Spectrum Protect environments for the last 22 years but have seen a drop off of its use as IBM has retreated from marketing their products and moved more towards other areas of the IT field. As I have looked for jobs in the Spectrum Protect realm I have noticed that they are rarer than ever, especially with the current situation the world finds itself. I plan to leave the site and up and will post anything you the readers feel will benefit the Spectrum Protect community. Feel free to send me your article for review and if I feel it fits the nature of this website it will be posted. Unfortunately, unless things change I see myself transitioning into a new role with a cloud emphasis and posts here will end unless you all take over the mantle and provide other admins with advice and examples from your own experiences. I have enjoyed posting and helping the community since founding this website and wish you all success going forward.

Error Upgrading to 8.1.8.0

I performed an upgrade today on a server going from 8.1.0.0 to 8.1.8.0 and after the upgrade we encountered the following errors in the actlog as the system was starting.

08/14/2019 11:59:59      ANR9999D_2835836899 InstGetDB2stmtPass1(instr.c:550)                          Thread<404>: Error in fetching DB2 statement stats,                          rc=235908/14/2019 11:59:59      ANR9999D Thread<404> issued message 9999 from:08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001000365b4 StdPutText08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010003784c OutDiagToCons08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010000cf08 outDiagfExt08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010019d0d0                          IPRA.$InstGetDB2stmtPass108/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001001a15e0 instrumentStart08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001012d7b98                          AdmInstrumentBegin08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008da68c AdmCommandLocal08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008d7af0 admCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013fbe68 RunCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013f7a48                          ServerMonTwentyMinThread08/14/2019 11:59:59      ANR9999D Thread<404>  0x0000000100010648 StartThread

Although the server started, I wanted to make sure this would not cause issues with the instance. I searched the error ANR9999D_2835836899 and found the following IBM APAR notice. It appears that after the upgrade you may have to manually upgrade the DB2 version using the DB2 update tool. While it appears to be a minor issue, I would recommend you check your instances after upgrade to verify whether you need to run the manual steps to resolve the errors. I've included them here for your convenience.

NOTE: I've added a couple commands (in bold) that we have had to add to the process to make it work.

Local fix

Even though the db2-level might be updated to the latest version 11.1.4.4, run the below set of commands
Stop TSM server and issue the following commands as the instance owner.

• db2 start

• db2 force application all

• db2 terminate

• ipclean

login as root user and run

• /opt/tivoli/tsm/db2/instance/db2iupdt -d <instance name>

login as the instance owner ID and issue the command:

• db2updv111 -d TSMDB1

This issue is supposedly fixed in version 8.1.9.0 and higher.

Audit Occupancy Issue

2019 Update
IBM has resolved this issue with version 8.1.6 and higher. As of 8.1.6 the standard AUDIT LICENSE will update all clients whether they use standard or directory storage pools.

1/24/2018 Update 
IBM Support contacted me and stated that the developers said the issue was actually a defect (APAR IT23153) and that you can fix the issue by logging on the SP server as the instance owner and running the following commands:

• db2 connect to tsmdb1
• db2 set schema tsmdb1
• db2 "update tsmdb1.sd_pool_clusters set physoccupancy=1 where physoccupancy=0"
• login as admin to Spectrum Protect instance
• run AUDIT LICense command
• run Q AUDITOCCupancy command

You should now see the AUDITOCC table is populated.


1/23/2018 
 We setup a new Spectrum Protect instance with a Directory Container Storage Pool and when we went to gather storage data for billing purposes we discovered that the AUDITOCC table in the database showed no storage data. The first troubleshooting step was to check to see if AUDITSTORAGE option was set to YES which it was. Then we reran the AUDIT LICENSE command to have Spectrum Protect audit the storage and after completing there was still no data in the AUDITOCC table. I contacted IBM and asked for support to tell me why I had no data in my AUDITOCC table and was met with level 1 support unable to provide an answer (which was probably due to them not knowing my servers full configuration). When the PMR was passed up to the chain to level 2 support and they reviewed the log details I sent in, they responded that DIRECTORY CONTAINER storage pools do not provide occupancy data to the AUDITOCC table. To gather the required data that the AUDITOCC table normally would provide you have to run the GENERATE DEDUPSTATS command.

GENERATE DEDUPSTATS <STORAGE POOL> <NODE NAME>

Please note that there is nothing in the Spectrum Protect manuals that will inform you that DIRECTORY CONTAINER storage pools do not provide AUDITOCC table data. IBM Level 2 support has opened a PMR requesting that the documentation be updated for the QUERY AUDITOCC command. I am not sure if you can use a * with the node name and have GENERATE DEDUPSTATS but will be trying it today and will let you all know the results in the comments below.

IBM Spectrum Protect 8.1.6 Tier to Cloud by State

A coworker recently did a presentation on Amazon S3 with Netbackup and I went searching for the capabilities available with TSM/Spectrum Protect. It appears as of 8.1.6 they have added some additional features that greatly expand the usage of cloud based storage. Check out this informative demo from IBM.

Q STATUS UPDATE!

So it turns out that TSM is calculating Front-End usage automatically when you install Ops Center or have SET STATUSMONITOR ON. I had installed Operations Center when I installed the latest SP and wasn't familiar witht he front-end licensing model until now. So lucky me it show's the data our management wants without me having to go to each application server to manually gather the data. Note that it is supposed to show the total in MB but from the command line it appears as if SP didn't calculate correctly. If you go into Ops Center and check licensing then it shows that the 

Front-End Capacity (MB): 148.21 

According to the Front-End licensing model this instance ACTIVE DATA is acutally 148TB.  If you you want to know the overall ACTIVE DATA the SP instance is protecting then look at the Front-End numbers. When you do a Q STATUS, if it's not reporting in TB than it probably is expriencing the bug that needs correcting by IBM, but that value is probably correct but in TB not MB.  Ops Center will also tell you how many nodes are not reporting and you can select that and view the list to see if any should be reporting.

Q STATUS Question

The company to which I am employed was recently acquired by another company that is a heavy Netbackup user. They bill not based on overall occupancy but on the "Front End Usage" of their servers. I don't know how that works for accurate billing but it's what they do and now we are being asked to provide "Front End Capacity" numbers so they can compare. The problem is ADSM/TSM/SP has always had issues with reporting "active only" data. Just recently I was working with an select IBM provided to pull the active for file systems but they also stated we had to go to the individual application servers to pull DB data. Not fun. So when I recently looked at all the detail of the Q STATUS I noticed the following near the bottom.

                 SUR Occupancy (TB): 703                

            SUR Occupancy Date/Time: 2019-01-07, 10:23:18

            Front-End Capacity (MB): 148.21             

             Front-End Client Count: 408                   

            Front-End Capacity Date: 2019-01-07, 10:23:18

                   Product Offering: IBM Spectrum Protect

According to IBM these last few statements are:

SUR Occupancy (TB)

If you have an IBM Spectrum Protect Suite (SUR) license, this field specifies the SUR occupancy on the server. The SUR occupancy is the amount of space that is used to store data that is managed by the IBM Spectrum Protect products that are included in the SUR bundle.

SUR Occupancy Date/Time

Specifies the date and time when SUR occupancy data was last collected.

Front-End Capacity (MB)

Specifies the amount of primary data that is reported as being backed up by clients. Clients include applications, virtual machines, and systems. This value is used for the front-end licensing model.

Front-End Client Count

Specifies the number of clients that reported capacity usage based on the front-end licensing model.

Front-End Capacity Date

Specifies the date and time when front-end capacity data was last collected.

Product Offering

Specifies a product offering.

So what exactly is the SUR license and does the EE license cover that? How is the data generated because my server reports front-end data at 148MB but if it is actually TB then that would fit more closely with overall capacity as front-end would be about 21% of overall occupancy. IBM obviously has some function to gather front-end numbers so why do they also have us going down the rabbit hold of gathering the data application data from the clients?

Active Data Report

So me an a coworker have been hounding IBM for a way to generate an active data report for management. For various reasons they want to be able to see how much data a server would require for rebuild/restore and also to guage any growth. As many of you might be aware this has been something the TSM/Spectrum Protect community has been requesting for some time. So IBM created a perl script that when analyzed issues the following SQL select:

(I've added continuation dashes to make the select more readable)

select -
(sum(bk.bfsize )/1048576) as front_end_size_mega_byte, -
count(bk.bfsize ) as number_of_objects -
from -
 backups b, backup_objects bk -
where -
 b.state='ACTIVE_VERSION' -
 and -
 b.object_id=bk.objid -
 and -
 b.filespace_id in -
   ( select f.filespace_id -
     from filespaces f -
     where -
      b.node_name=f.node_name -
     and -
      f.filespace_id=b.filespace_id -
     and -
      f.filespace_type not like 'API:%' -
     and -
      f.filespace_type not like 'TDP%' -
   ) -
   and -
    b.node_name in -
   ( select node_name -
     from nodes -
     where repl_mode not in('RECEIVE','SYNCRECEIVE') -
    )

You will notice, however, that the select excludes node replicas and any API or TDP data. This is purely for file system backups. Supposedly they have something for TDP/API backups but I have not worked with it yet. I will post it as soon as I have a chance to review it and make it readable.

NOTE: This command can take a somewhat considerable amount of time to run. I have seen it take upwards of 10+ minutes to complete when run during our non-backup window times. Be patient!

Win2K & Win2003 - Data Currently Unavailable On Server

This past weekend we performed a DR test on quit a few servers and discovered a situation that was a wake up call for anyone still using Windows 2000 or Windows 2003. The issue is that when we ran a restore the IBM Spectrum Protect 8.1 server issued a warning on all the files for the particular node with the following message type…

ANE4035W (Session: 79, Node: ODC-WINCS-1-DR)  File '\\od-wincs-1\e$\834\c-oh\files-20170823-123000' currently unavailable on server.

The tape was in a readonly state and accessible by the library so it was not a tape unavailable problem. I then querried the file and ran some tests suggested by IBM in the following link

http://www-01.ibm.com/support/docview.wss?uid=swg21249032

The data is present and on tape but TSM will not restore it. (See the queries below) IBM would not help us because the OS is unsupported as is the TSM client. We could not restore the System State, System Services, or any of the Data to the DR replica. What we could do is access the data from a supported client level on a supported OS level (i.e. TSM Client 6.4 from a Windows 2008 server). We were unable to recover the Win2K/Win2003 servers during the DR test and unfortunately I know many companies out there still have some of these servers in their environment. It's time to upgrade! Your data is at risk!

tsm: OMATSM02>select * from backups where node_name='ODC-WINCS-1-DR' and FILESPACE_ID=1 and hl_name=upper('\834\c-oh\') and LL_NAME=upper('files-20170823-123000')

      NODE_NAME: ODC-WINCS-1-DR
FILESPACE_NAME: \\od-wincs-1\e$
   FILESPACE_ID: 1
          STATE: ACTIVE_VERSION
           TYPE: DIR
        HL_NAME: \834\C-OH\
        LL_NAME: FILES-20170823-123000
      OBJECT_ID: 11926208445
    BACKUP_DATE: 2018-08-10 03:04:37.000000
DEACTIVATE_DATE:
          OWNER:
     CLASS_NAME: WIN-DR-MC
    ACTUAL_SIZE:

tsm: OMATSM02>show bfo 11926208445

Bitfile Object: 11926208445
  Active
**Sub-bitfile 11926208445 is stored in the following aggregate(s)
Super-bitfile: 11926199674, Offset: 26157087, Length 735, Deduped: F


tsm: OMATSM02>show bfo 11926199674

Bitfile Object: 11926199674
**Super-bitfile 11926199674 contains following aggregated bitfiles,
Bitfile Id, offset, length, active state or owner, link bfid
11926199674   0             8732          Active
11926199676   8732          1527          Active
.
.
.
11926208946   26573407      5510          Active
11926208947   26578917      49168         Active

**Sub-bitfile 11926199674 is stored in the following aggregate(s)
  Super-bitfile: 11926199674, Offset: 0, Length 8732, Deduped: F

**Archival Bitfile Entry
  Bitfile Type: PRIMARY  Storage Format: 22
  Bitfile Size: 26631381  Number of Segments: 1, flags: 0
  Storage Pool ID: 33  Volume ID: 22903  Volume Name: L42803


tsm: OMATSM02>q vol L42803 f=d

                   Volume Name: L42803
             Storage Pool Name: WIN-DR-TAPE
             Device Class Name: IBM-LTO-4
            Estimated Capacity: 4.2 T
       Scaled Capacity Applied:
                      Pct Util: 100.0
                 Volume Status: Full
                        Access: Read-Only
        Pct. Reclaimable Space: 0.0
               Scratch Volume?: Yes
               In Error State?: No
      Number of Writable Sides: 1
       Number of Times Mounted: 227
             Write Pass Number: 1
     Approx. Date Last Written: 08/12/18   16:44:25
        Approx. Date Last Read: 08/25/18   22:06:34
           Date Became Pending:
        Number of Write Errors: 0
         Number of Read Errors: 0
               Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
         Last Update Date/Time: 08/25/18   20:29:37
          Begin Reclaim Period:
            End Reclaim Period:
  Drive Encryption Key Manager: IBM Spectrum Protect
       Logical Block Protected: No



tsm: OMATSM02>q libvol dr-libr L42803

Library Name     Volume Name     Status               Owner          Last Use      Home        Device
                                                                                   Element     Type
------------     -----------     ----------------     ----------     ---------     -------     ------
DR-LIBR          L42803          Private                             Data          1,089