New Version of TSMExplorer

There is a new version of TSMExplorer available and it has added support for Spectrum Protect Plus. See can download a trial version of 5.2.3 here.  TSMExplorer has a load of features and provides a great central management console when you have multiple TSM/Spectrum Protect Servers. With the added support of managing Spectrum Protect Plus it's even easier to manage your environment.

Here is a video of an older version of TSMExplorer so you can get a feel for the product.




Here is a video of the client agent and how it works with the TSMExplorer console to allow access to the client options files and logs for easier management.

Audit Occupancy Issue

2019 Update
IBM has resolved this issue with version 8.1.6 and higher. As of 8.1.6 the standard AUDIT LICENSE will update all clients whether they use standard or directory storage pools.

1/24/2018 Update 
IBM Support contacted me and stated that the developers said the issue was actually a defect (APAR IT23153) and that you can fix the issue by logging on the SP server as the instance owner and running the following commands:

• db2 connect to tsmdb1
• db2 set schema tsmdb1
• db2 "update tsmdb1.sd_pool_clusters set physoccupancy=1 where physoccupancy=0"
• login as admin to Spectrum Protect instance
• run AUDIT LICense command
• run Q AUDITOCCupancy command

You should now see the AUDITOCC table is populated.


1/23/2018 
 We setup a new Spectrum Protect instance with a Directory Container Storage Pool and when we went to gather storage data for billing purposes we discovered that the AUDITOCC table in the database showed no storage data. The first troubleshooting step was to check to see if AUDITSTORAGE option was set to YES which it was. Then we reran the AUDIT LICENSE command to have Spectrum Protect audit the storage and after completing there was still no data in the AUDITOCC table. I contacted IBM and asked for support to tell me why I had no data in my AUDITOCC table and was met with level 1 support unable to provide an answer (which was probably due to them not knowing my servers full configuration). When the PMR was passed up to the chain to level 2 support and they reviewed the log details I sent in, they responded that DIRECTORY CONTAINER storage pools do not provide occupancy data to the AUDITOCC table. To gather the required data that the AUDITOCC table normally would provide you have to run the GENERATE DEDUPSTATS command.

GENERATE DEDUPSTATS <STORAGE POOL> <NODE NAME>

Please note that there is nothing in the Spectrum Protect manuals that will inform you that DIRECTORY CONTAINER storage pools do not provide AUDITOCC table data. IBM Level 2 support has opened a PMR requesting that the documentation be updated for the QUERY AUDITOCC command. I am not sure if you can use a * with the node name and have GENERATE DEDUPSTATS but will be trying it today and will let you all know the results in the comments below.

Q STATUS UPDATE!

So it turns out that TSM is calculating Front-End usage automatically when you install Ops Center or have SET STATUSMONITOR ON. I had installed Operations Center when I installed the latest SP and wasn't familiar witht he front-end licensing model until now. So lucky me it show's the data our management wants without me having to go to each application server to manually gather the data. Note that it is supposed to show the total in MB but from the command line it appears as if SP didn't calculate correctly. If you go into Ops Center and check licensing then it shows that the 

Front-End Capacity (MB): 148.21 

According to the Front-End licensing model this instance ACTIVE DATA is acutally 148TB.  If you you want to know the overall ACTIVE DATA the SP instance is protecting then look at the Front-End numbers. When you do a Q STATUS, if it's not reporting in TB than it probably is expriencing the bug that needs correcting by IBM, but that value is probably correct but in TB not MB.  Ops Center will also tell you how many nodes are not reporting and you can select that and view the list to see if any should be reporting.

Q STATUS Question

The company to which I am employed was recently acquired by another company that is a heavy Netbackup user. They bill not based on overall occupancy but on the "Front End Usage" of their servers. I don't know how that works for accurate billing but it's what they do and now we are being asked to provide "Front End Capacity" numbers so they can compare. The problem is ADSM/TSM/SP has always had issues with reporting "active only" data. Just recently I was working with an select IBM provided to pull the active for file systems but they also stated we had to go to the individual application servers to pull DB data. Not fun. So when I recently looked at all the detail of the Q STATUS I noticed the following near the bottom.

                 SUR Occupancy (TB): 703                

            SUR Occupancy Date/Time: 2019-01-07, 10:23:18

            Front-End Capacity (MB): 148.21             

             Front-End Client Count: 408                   

            Front-End Capacity Date: 2019-01-07, 10:23:18

                   Product Offering: IBM Spectrum Protect

According to IBM these last few statements are:

SUR Occupancy (TB)

If you have an IBM Spectrum Protect Suite (SUR) license, this field specifies the SUR occupancy on the server. The SUR occupancy is the amount of space that is used to store data that is managed by the IBM Spectrum Protect products that are included in the SUR bundle.

SUR Occupancy Date/Time

Specifies the date and time when SUR occupancy data was last collected.

Front-End Capacity (MB)

Specifies the amount of primary data that is reported as being backed up by clients. Clients include applications, virtual machines, and systems. This value is used for the front-end licensing model.

Front-End Client Count

Specifies the number of clients that reported capacity usage based on the front-end licensing model.

Front-End Capacity Date

Specifies the date and time when front-end capacity data was last collected.

Product Offering

Specifies a product offering.

So what exactly is the SUR license and does the EE license cover that? How is the data generated because my server reports front-end data at 148MB but if it is actually TB then that would fit more closely with overall capacity as front-end would be about 21% of overall occupancy. IBM obviously has some function to gather front-end numbers so why do they also have us going down the rabbit hold of gathering the data application data from the clients?

Spectrum Protect Server 8.1.x Paging Space Issue

I thought I would pass this along...I have two Spectrum Protect 8.1.4 instances on the same AIX 7 LPAR that since their upgrade to this version has had frequent exhausting of paging space crashes. We reviewed and checked processes and figured it was an AIX issue. Upon deeper investigation I noticed there were some previous APARs for DB2 exhausting paging space due to a 64k block size. Investigating that further led me to the following APAR for Spectrum Protect 

IT23731: SPECTRUM PROTECT SERVER ON AIX MAY RUN INTO PAGING AND OUT OF MEMORY CONDITION

So to check I ran the following shell command to list the top 10 processes using memory/paging

# svmon -Pt 10 -O summary=basic,unit=MB

Unit: MB

----------------------------------------------------------------

     Pid Command          Inuse      Pin     Pgsp  Virtual

 4391072 db2sysc       74805.10     51.4 15266.27 72775.38

 7274744 db2sysc       71323.35     50.2 15579.61 71796.98

10486106 db2vend       71164.58     47.6 15084.41 71589.28

 4915692 dsmserv       15230.22     49.5  6629.03 20345.98

 2490670 dsmserv        1931.09     49.6   299.12  2197.74

 5701694 db2sysc         632.43     47.6     42.1   732.73

 8126714 db2sysc         569.93     47.6     29.1   665.60

 6553630 db2vend         561.76     47.6     37.0   660.47

 6422602 db2sysc         559.20     47.6     37.2   657.31

 4849682 db2sysc         559.20     47.6     37.2   657.31

So the db2sysc and dsm2vend along with the dsmserv are using up large amounts of paging.

Here is my using in topas

PAGING SPACE
Size,MB   64512
% Used     60
% Free     40

So what's the solution? Restart the TSM instance(s) until you can upgrade to 8.1.5 

DB2 Select Example In TSM/SP

While trying to create a specific report that a manager requested I realized I needed something more powerful. Well now that TSM/Spectrum Protect runs on DB2 we have all the power we need. Here is an example of generating a report with an Inner and Outer Join.

====================================
Inner and Outer Join Example
====================================

Part 1 - Will return the columns from two tables into a third temporary table

Select -
 varchar(o.node_name,45) ENTITY, -
 cast(s.SUM_AFFECTED as dec(12,0)) AFFECTED,
 cast(s.SUM_BYTES/1073741824 as decimal(12,2)) GB_BACKED_UP, -
 cast(o.SUM_NUM_FILES as dec(15,0)) TOTAL_FILES_STORED, -
 cast(o.SUM_LOGICAL_MB/1024 as dec(12,2)) GB_TSM_OCCUPANCY -
 from -


Part 2 - Gather two columns from occupancy

select node_name, -
    sum(NUM_FILES) SUM_NUM_FILES, -
    sum(LOGICAL_MB) SUM_LOGICAL_MB -
  from occupancy -
    group by node_name


Part 3 - Gather three columns from summary for BACKUP and ARCHIVES for last 30 days

select varchar(entity,45) ENTITY, -
    sum(bytes/1073741824) SUM_BYTES, -
    sum(affected) SUM_AFFECTED -
  from summary  -
   where -
    end_time>=(current_timestamp - 30 days) -
    and activity in ('BACKUP','ARCHIVE') -
   group by entity


select -
 varchar(o.node_name,45) ENTITY, -
 cast(s.SUM_AFFECTED as dec(12,0)) AFFECTED, -
 cast(s.SUM_BYTES as decimal(12,2)) GB_BACKED_UP, -
 cast(o.SUM_NUM_FILES as dec(15,0)) TOTAL_FILES_STORED, -
 cast(o.SUM_LOGICAL_MB/1024 as dec(12,2)) GB_TSM_OCCUPANCY -
 from -
 (select node_name, -
    sum(NUM_FILES) SUM_NUM_FILES, -
    sum(LOGICAL_MB) SUM_LOGICAL_MB -
  from occupancy -
    group by node_name) o, -
 (select entity, -
    sum(bytes/1073741824) SUM_BYTES, -
    sum(affected) SUM_AFFECTED -
  from summary  -
   where -
    end_time>=(current_timestamp - 30 days) -
    and activity in ('BACKUP','ARCHIVE') -
   group by entity) s -
 where ENTITY in (select node_name from nodes where domain_name<>'MN-STD-POL-DOM') -
  and o.node_name=s.entity -
  order by ENTITY

The generated report shows the total amount of files backed up, GB backed up for the last 30 days while also showing the total files stored in TSM and the nodes current total occupancy in GB.

ENTITY              AFFECTED        GB_BACKED_UP      TOTAL_FILES_STORED      GB_TSM_OCCUPANCY
----------   ---------------     ---------------     -------------------     -----------------
DORPORA05                920            47513.00                    8354              73263.60
IXPFLDNAS              60792               27.00                15443947               9434.27
ILDNASJ                12680                5.00                 1972680                715.60
ILDNASK                    8                0.00                   83004                197.48
ILDNASL                10739                4.00                 1649434                574.51
ILDNASM                45448               25.00                 1221367               1722.03
ILDNASM                 4386                7.00                 1122826                557.64
ILDNASN                    5                0.00                   99769                 46.52
ILDNASO                 1688                0.00                  459683                138.16
ILDNASP                 9369               11.00                 1402332               1623.92
ILDNASR                28787               29.00                  462479                457.15
ILDNASS                    5                0.00                   36771                 17.51
ILDNAST                 6329               21.00                  982728               1038.08
ILDNASU                 6848               19.00                 1548369                671.68
ILDNASV                18197               48.00                  999905               1333.92
ILDNASW                    5                0.00                  245366                116.89
RSD1W030S              59979                7.00                 5207060                547.23
RSD1W031S               3444                2.00                 5070450                473.15
RSD1W241S               6879               47.00                  517358               1782.92
RSD1W242S               6478               51.00                  995006               2109.45
RSD1W266S               4377                7.00                  492810                184.69
RSD1W267S               5700                3.00                  328391                100.95
RSD1W320C              40881               12.00                 2692541                576.50
RSD1W341C               8194               55.00                  456567               1120.11
RSD1W342C              17003                7.00                  625487                281.84
RSD1W343C               4628                2.00                  298699                 79.63
RSD1W504SV              1941               12.00                   42444                551.35
RSD1W505SV              1079              105.00                   38524               4033.06
RSD1W5067S               683               64.00                   22389               1343.82
RSD1W506SV               489                0.00                   14349                  4.53
RSD1W508SV              2662               31.00                   79340                699.12
RSD1W509SV              4017               74.00                  139073               1617.35
RSD1W510SV               698              544.00                   18086               9161.23
RSD1W511SV              1365              295.00                   65303               6588.06

Spectrum Protect Server 8.1.4 DB Issue

Recently we ran into a situation where our Spectrum Protect 8.1.4 server started to crash. We reviewed the db2diag.log file and identified the error as ANR0131E Server DB space exhausted. Upon review of our DB file systems the OS showed 7GB available in the 5 file systems assigned for DB space.

/dev/tsm20log 111616.00   9784.93   92%      205     1% /tsmserver/tsm20log

/dev/tsm20arch 102400.00  66821.52   35%      100     1% /tsmserver/tsm20arch

/dev/tsm20fail 101376.00 100964.16    1%       21     1% /tsmserver/tsm20fail

/dev/tsm20dblv01 101376.00   7622.45   93%      104     1% /tsmserver/tsm20dblv01

/dev/tsm20dblv02 101376.00   7622.48   93%      104     1% /tsmserver/tsm20dblv02

/dev/tsm20dblv03 101376.00   7622.45   93%      104     1% /tsmserver/tsm20dblv03

/dev/tsm20dblv04 101376.00   7622.46   93%      105     1% /tsmserver/tsm20dblv04

/dev/tsm20dblv05 101376.00   7622.45   93%      105     1% /tsmserver/tsm20dblv05

The server had been crashing from the previous night and we had at first thought it was due to the active log filling and not writing out to the archive log fast enough. What was weird is we could bring TSM up and it would run for an hour or two then crash again. When we identified it was not the active log but the DB space (earlier the next morning) we were confused. Why would we get a space notification when the DB has 35GB of space still available. My co-worker created an additional file system of the same size as the other DB file systems and ran the dsmserv extend dbspace command and the file systems almost immediately jumped to 99% utilized. So it appears Spectrum Protect needed to write 35+GB of DB data and could not due to space available? We also noticed that our offshore people had added tsm20dblv05 a few months back but had used a different disk array (slow SATA) and had not informed us, which we think might have contributed to the performance issues we had been seeing on this server recently and maybe was related to the DB space issue??? Not sure, but be aware that this can occur. I will need to research how DB2 allocates space and in what chunk size it does so.

SystemState Restore Issue with Windows 8.x Client

3/7/2017 Update
I have had a number of people request the test flag and one response said it was kicking up errors when used with the CAD service/process. Please note this flag is only needed in the event of a system state restore and you would not (I hope) be using the CAD/Scheduler to restore your system state. So the testflag should only be activated in the dsm.opt in the event you are doing a manual restore of the system state and DO NOT leave it in the dsm.opt as it will affect daily use.


1/24/2018 Update
After waiting for IBM to post an update to this issue I decided I would let you know that IBM has an undocumented flag for this issue that allows the client to restore the System State online rather than work with the PE. Our Windows admins spoke with Microsoft and they state emphatically that they support online system state restores, so is appears IBM does not starting with the 8.1.x client. Here is the flag to allow the SP client to perform online system state restores.

ONLINE_SYSTEMSTATE_RESTORE_ALLOWED

12/4/2018
I wanted to make everyone aware that starting with the 8.1 Windows Client, IBM has deprecated the System State restore process so that any restore attempt will require the use of a WinPE environment. With the 8.1.x client you can no longer do a complete restore of a server within the normal windows environment and will have to make sure you have a compatible WinPE for every Windows Hardware type or environment. This, in my opinion, is a huge break of the software. Although IBM stated that Microsoft does not support online system state restores, Microsoft disagrees. I say this knowing that if you are on a 7.x or earlier version of the client IBM will support you with the system state restore, but as of 8.1 you will be directed to build a Windows PE environment and follow a series of stepto create the necessary disc. The support person I spoke with treated this process as if it should be second nature to any Windows Admin.

We recently were migrating a clients server and in the process the disks were corrupted. The client installed on the new machine was 8.1.2 and when the Windows Admins tried to restore the system state they experienced the following error.

 

After speaking with support we were directed to the following document by Andrew Raibeck that discussed creating the WinPE disk image with the TSM client code that would allow the offline restore of the System State. We attempted to build the ISO but when the Windows Admin tried to run the dsmc client we experienced the following error:

Who do we call when we experience an error like this in a WinPE environment? IBM or Microsoft? So you have two choices, have a Windows PE available for each hardware type or try using the testflag but understand IBM may support its use. If you were fully aware of this and took precautions what were they? My thinking is that many admins have no idea this is going to bite them and its better you know now than find out the hard way.
 

Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

I just wanted to let everyone know that there is a compatibility issue with the Spectrum Protect 8.1.2 client with the 8.1.0 server. If you upgrade the client to SP 8.1.2 and your server is at 8.1.1 or 8.1.0 your server will be unable to start. The error you will get is

Spectrum Protect ANR0162W Supplemental database diagnostic information:  -1:08001:-30082 ([IBM][CLI Driver] SQL30082N  Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID").

The issue has to do with the GSKit8.gskcrypt64.ppc.rte and GSKit8.gskssl64.ppc.rte in the 8.1.2.0 client. When we rolled back to the 8.1.0 client the SP server started without an issue.

TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.