Wednesday, August 14, 2019

Error Upgrading to

I performed an upgrade today on a server going from to and after the upgrade we encountered the following errors in the actlog as the system was starting.

08/14/2019 11:59:59      ANR9999D_2835836899 InstGetDB2stmtPass1(instr.c:550)                          Thread<404>: Error in fetching DB2 statement stats,                          rc=235908/14/2019 11:59:59      ANR9999D Thread<404> issued message 9999 from:08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001000365b4 StdPutText08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010003784c OutDiagToCons08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010000cf08 outDiagfExt08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010019d0d0                          IPRA.$InstGetDB2stmtPass108/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001001a15e0 instrumentStart08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001012d7b98                          AdmInstrumentBegin08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008da68c AdmCommandLocal08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008d7af0 admCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013fbe68 RunCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013f7a48                          ServerMonTwentyMinThread08/14/2019 11:59:59      ANR9999D Thread<404>  0x0000000100010648 StartThread

Although the server started, I wanted to make sure this would not cause issues with the instance. I searched the error ANR9999D_2835836899 and found the following IBM APAR notice. It appears that after the upgrade you may have to manually upgrade the DB2 version using the DB2 update tool. While it appears to be a minor issue, I would recommend you check your instances after upgrade to verify whether you need to run the manual steps to resolve the errors. I've included them here for your convenience.

NOTE: I've added a couple commands (in bold) that we have had to add to the process to make it work.

Local fix

Even though the db2-level might be updated to the latest version, run the below set of commands
Stop TSM server and issue the following commands as the instance owner.
  • db2 start
  • db2 force application all
  • db2 terminate
  • ipclean

login as root user and run
  • /opt/tivoli/tsm/db2/instance/db2iupdt -d <instance name>

login as the instance owner ID and issue the command:

  • db2updv111 -d TSMDB1

This issue is supposedly fixed in version and higher.

Tuesday, June 18, 2019

Audit Occupancy Issue

2019 Update
IBM has resolved this issue with version 8.1.6 and higher. As of 8.1.6 the standard AUDIT LICENSE will update all clients whether they use standard or directory storage pools.

1/24/2018 Update 
IBM Support contacted me and stated that the developers said the issue was actually a defect (APAR IT23153) and that you can fix the issue by logging on the SP server as the instance owner and running the following commands:

  • db2 connect to tsmdb1
  • db2 set schema tsmdb1
  • db2 "update tsmdb1.sd_pool_clusters set physoccupancy=1 where physoccupancy=0"
  • login as admin to Spectrum Protect instance
  • run AUDIT LICense command
  • run Q AUDITOCCupancy command

You should now see the AUDITOCC table is populated.

 We setup a new Spectrum Protect instance with a Directory Container Storage Pool and when we went to gather storage data for billing purposes we discovered that the AUDITOCC table in the database showed no storage data. The first troubleshooting step was to check to see if AUDITSTORAGE option was set to YES which it was. Then we reran the AUDIT LICENSE command to have Spectrum Protect audit the storage and after completing there was still no data in the AUDITOCC table. I contacted IBM and asked for support to tell me why I had no data in my AUDITOCC table and was met with level 1 support unable to provide an answer (which was probably due to them not knowing my servers full configuration). When the PMR was passed up to the chain to level 2 support and they reviewed the log details I sent in, they responded that DIRECTORY CONTAINER storage pools do not provide occupancy data to the AUDITOCC table. To gather the required data that the AUDITOCC table normally would provide you have to run the GENERATE DEDUPSTATS command.


Please note that there is nothing in the Spectrum Protect manuals that will inform you that DIRECTORY CONTAINER storage pools do not provide AUDITOCC table data. IBM Level 2 support has opened a PMR requesting that the documentation be updated for the QUERY AUDITOCC command. I am not sure if you can use a * with the node name and have GENERATE DEDUPSTATS but will be trying it today and will let you all know the results in the comments below.

Wednesday, February 20, 2019

IBM Spectrum Protect 8.1.6 Tier to Cloud by State

A coworker recently did a presentation on Amazon S3 with Netbackup and I went searching for the capabilities available with TSM/Spectrum Protect. It appears as of 8.1.6 they have added some additional features that greatly expand the usage of cloud based storage. Check out this informative demo from IBM.

Wednesday, January 16, 2019


So it turns out that TSM is calculating Front-End usage automatically when you install Ops Center or have SET STATUSMONITOR ON. I had installed Operations Center when I installed the latest SP and wasn't familiar witht he front-end licensing model until now. So lucky me it show's the data our management wants without me having to go to each application server to manually gather the data. Note that it is supposed to show the total in MB but from the command line it appears as if SP didn't calculate correctly. If you go into Ops Center and check licensing then it shows that the 

Front-End Capacity (MB): 148.21 

According to the Front-End licensing model this instance ACTIVE DATA is acutally 148TB.  If you you want to know the overall ACTIVE DATA the SP instance is protecting then look at the Front-End numbers. When you do a Q STATUS, if it's not reporting in TB than it probably is expriencing the bug that needs correcting by IBM, but that value is probably correct but in TB not MB.  Ops Center will also tell you how many nodes are not reporting and you can select that and view the list to see if any should be reporting.

Monday, January 14, 2019

Q STATUS Question

The company to which I am employed was recently acquired by another company that is a heavy Netbackup user. They bill not based on overall occupancy but on the "Front End Usage" of their servers. I don't know how that works for accurate billing but it's what they do and now we are being asked to provide "Front End Capacity" numbers so they can compare. The problem is ADSM/TSM/SP has always had issues with reporting "active only" data. Just recently I was working with an select IBM provided to pull the active for file systems but they also stated we had to go to the individual application servers to pull DB data. Not fun. So when I recently looked at all the detail of the Q STATUS I noticed the following near the bottom.

                 SUR Occupancy (TB): 703                
            SUR Occupancy Date/Time: 2019-01-07, 10:23:18
            Front-End Capacity (MB): 148.21             
             Front-End Client Count: 408                   
            Front-End Capacity Date: 2019-01-07, 10:23:18
                   Product Offering: IBM Spectrum Protect

According to IBM these last few statements are:

SUR Occupancy (TB)
If you have an IBM Spectrum Protect Suite (SUR) license, this field specifies the SUR occupancy on the server. The SUR occupancy is the amount of space that is used to store data that is managed by the IBM Spectrum Protect products that are included in the SUR bundle.
SUR Occupancy Date/Time
Specifies the date and time when SUR occupancy data was last collected.
Front-End Capacity (MB)
Specifies the amount of primary data that is reported as being backed up by clients. Clients include applications, virtual machines, and systems. This value is used for the front-end licensing model.
Front-End Client Count
Specifies the number of clients that reported capacity usage based on the front-end licensing model.
Front-End Capacity Date
Specifies the date and time when front-end capacity data was last collected.
Product Offering
Specifies a product offering.

So what exactly is the SUR license and does the EE license cover that? How is the data generated because my server reports front-end data at 148MB but if it is actually TB then that would fit more closely with overall capacity as front-end would be about 21% of overall occupancy. IBM obviously has some function to gather front-end numbers so why do they also have us going down the rabbit hold of gathering the data application data from the clients?