Active Data Report

So me an a coworker have been hounding IBM for a way to generate an active data report for management. For various reasons they want to be able to see how much data a server would require for rebuild/restore and also to guage any growth. As many of you might be aware this has been something the TSM/Spectrum Protect community has been requesting for some time. So IBM created a perl script that when analyzed issues the following SQL select:

(I've added continuation dashes to make the select more readable)

select -
(sum(bk.bfsize )/1048576) as front_end_size_mega_byte, -
count(bk.bfsize ) as number_of_objects -
from -
 backups b, backup_objects bk -
where -
 b.state='ACTIVE_VERSION' -
 and -
 b.object_id=bk.objid -
 and -
 b.filespace_id in -
   ( select f.filespace_id -
     from filespaces f -
     where -
      b.node_name=f.node_name -
     and -
      f.filespace_id=b.filespace_id -
     and -
      f.filespace_type not like 'API:%' -
     and -
      f.filespace_type not like 'TDP%' -
   ) -
   and -
    b.node_name in -
   ( select node_name -
     from nodes -
     where repl_mode not in('RECEIVE','SYNCRECEIVE') -
    )

You will notice, however, that the select excludes node replicas and any API or TDP data. This is purely for file system backups. Supposedly they have something for TDP/API backups but I have not worked with it yet. I will post it as soon as I have a chance to review it and make it readable.

NOTE: This command can take a somewhat considerable amount of time to run. I have seen it take upwards of 10+ minutes to complete when run during our non-backup window times. Be patient!

Win2K & Win2003 - Data Currently Unavailable On Server

This past weekend we performed a DR test on quit a few servers and discovered a situation that was a wake up call for anyone still using Windows 2000 or Windows 2003. The issue is that when we ran a restore the IBM Spectrum Protect 8.1 server issued a warning on all the files for the particular node with the following message type…

ANE4035W (Session: 79, Node: ODC-WINCS-1-DR)  File '\\od-wincs-1\e$\834\c-oh\files-20170823-123000' currently unavailable on server.

The tape was in a readonly state and accessible by the library so it was not a tape unavailable problem. I then querried the file and ran some tests suggested by IBM in the following link

http://www-01.ibm.com/support/docview.wss?uid=swg21249032

The data is present and on tape but TSM will not restore it. (See the queries below) IBM would not help us because the OS is unsupported as is the TSM client. We could not restore the System State, System Services, or any of the Data to the DR replica. What we could do is access the data from a supported client level on a supported OS level (i.e. TSM Client 6.4 from a Windows 2008 server). We were unable to recover the Win2K/Win2003 servers during the DR test and unfortunately I know many companies out there still have some of these servers in their environment. It's time to upgrade! Your data is at risk!

tsm: OMATSM02>select * from backups where node_name='ODC-WINCS-1-DR' and FILESPACE_ID=1 and hl_name=upper('\834\c-oh\') and LL_NAME=upper('files-20170823-123000')

      NODE_NAME: ODC-WINCS-1-DR
FILESPACE_NAME: \\od-wincs-1\e$
   FILESPACE_ID: 1
          STATE: ACTIVE_VERSION
           TYPE: DIR
        HL_NAME: \834\C-OH\
        LL_NAME: FILES-20170823-123000
      OBJECT_ID: 11926208445
    BACKUP_DATE: 2018-08-10 03:04:37.000000
DEACTIVATE_DATE:
          OWNER:
     CLASS_NAME: WIN-DR-MC
    ACTUAL_SIZE:

tsm: OMATSM02>show bfo 11926208445

Bitfile Object: 11926208445
  Active
**Sub-bitfile 11926208445 is stored in the following aggregate(s)
Super-bitfile: 11926199674, Offset: 26157087, Length 735, Deduped: F


tsm: OMATSM02>show bfo 11926199674

Bitfile Object: 11926199674
**Super-bitfile 11926199674 contains following aggregated bitfiles,
Bitfile Id, offset, length, active state or owner, link bfid
11926199674   0             8732          Active
11926199676   8732          1527          Active
.
.
.
11926208946   26573407      5510          Active
11926208947   26578917      49168         Active

**Sub-bitfile 11926199674 is stored in the following aggregate(s)
  Super-bitfile: 11926199674, Offset: 0, Length 8732, Deduped: F

**Archival Bitfile Entry
  Bitfile Type: PRIMARY  Storage Format: 22
  Bitfile Size: 26631381  Number of Segments: 1, flags: 0
  Storage Pool ID: 33  Volume ID: 22903  Volume Name: L42803


tsm: OMATSM02>q vol L42803 f=d

                   Volume Name: L42803
             Storage Pool Name: WIN-DR-TAPE
             Device Class Name: IBM-LTO-4
            Estimated Capacity: 4.2 T
       Scaled Capacity Applied:
                      Pct Util: 100.0
                 Volume Status: Full
                        Access: Read-Only
        Pct. Reclaimable Space: 0.0
               Scratch Volume?: Yes
               In Error State?: No
      Number of Writable Sides: 1
       Number of Times Mounted: 227
             Write Pass Number: 1
     Approx. Date Last Written: 08/12/18   16:44:25
        Approx. Date Last Read: 08/25/18   22:06:34
           Date Became Pending:
        Number of Write Errors: 0
         Number of Read Errors: 0
               Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
         Last Update Date/Time: 08/25/18   20:29:37
          Begin Reclaim Period:
            End Reclaim Period:
  Drive Encryption Key Manager: IBM Spectrum Protect
       Logical Block Protected: No



tsm: OMATSM02>q libvol dr-libr L42803

Library Name     Volume Name     Status               Owner          Last Use      Home        Device
                                                                                   Element     Type
------------     -----------     ----------------     ----------     ---------     -------     ------
DR-LIBR          L42803          Private                             Data          1,089

Spectrum Protect Server 8.1.x Paging Space Issue

I thought I would pass this along...I have two Spectrum Protect 8.1.4 instances on the same AIX 7 LPAR that since their upgrade to this version has had frequent exhausting of paging space crashes. We reviewed and checked processes and figured it was an AIX issue. Upon deeper investigation I noticed there were some previous APARs for DB2 exhausting paging space due to a 64k block size. Investigating that further led me to the following APAR for Spectrum Protect 

IT23731: SPECTRUM PROTECT SERVER ON AIX MAY RUN INTO PAGING AND OUT OF MEMORY CONDITION

So to check I ran the following shell command to list the top 10 processes using memory/paging

# svmon -Pt 10 -O summary=basic,unit=MB

Unit: MB

----------------------------------------------------------------

     Pid Command          Inuse      Pin     Pgsp  Virtual

 4391072 db2sysc       74805.10     51.4 15266.27 72775.38

 7274744 db2sysc       71323.35     50.2 15579.61 71796.98

10486106 db2vend       71164.58     47.6 15084.41 71589.28

 4915692 dsmserv       15230.22     49.5  6629.03 20345.98

 2490670 dsmserv        1931.09     49.6   299.12  2197.74

 5701694 db2sysc         632.43     47.6     42.1   732.73

 8126714 db2sysc         569.93     47.6     29.1   665.60

 6553630 db2vend         561.76     47.6     37.0   660.47

 6422602 db2sysc         559.20     47.6     37.2   657.31

 4849682 db2sysc         559.20     47.6     37.2   657.31

So the db2sysc and dsm2vend along with the dsmserv are using up large amounts of paging.

Here is my using in topas

PAGING SPACE
Size,MB   64512
% Used     60
% Free     40

So what's the solution? Restart the TSM instance(s) until you can upgrade to 8.1.5 

DB2 Select Example In TSM/SP

While trying to create a specific report that a manager requested I realized I needed something more powerful. Well now that TSM/Spectrum Protect runs on DB2 we have all the power we need. Here is an example of generating a report with an Inner and Outer Join.

====================================
Inner and Outer Join Example
====================================

Part 1 - Will return the columns from two tables into a third temporary table

Select -
 varchar(o.node_name,45) ENTITY, -
 cast(s.SUM_AFFECTED as dec(12,0)) AFFECTED,
 cast(s.SUM_BYTES/1073741824 as decimal(12,2)) GB_BACKED_UP, -
 cast(o.SUM_NUM_FILES as dec(15,0)) TOTAL_FILES_STORED, -
 cast(o.SUM_LOGICAL_MB/1024 as dec(12,2)) GB_TSM_OCCUPANCY -
 from -


Part 2 - Gather two columns from occupancy

select node_name, -
    sum(NUM_FILES) SUM_NUM_FILES, -
    sum(LOGICAL_MB) SUM_LOGICAL_MB -
  from occupancy -
    group by node_name


Part 3 - Gather three columns from summary for BACKUP and ARCHIVES for last 30 days

select varchar(entity,45) ENTITY, -
    sum(bytes/1073741824) SUM_BYTES, -
    sum(affected) SUM_AFFECTED -
  from summary  -
   where -
    end_time>=(current_timestamp - 30 days) -
    and activity in ('BACKUP','ARCHIVE') -
   group by entity


select -
 varchar(o.node_name,45) ENTITY, -
 cast(s.SUM_AFFECTED as dec(12,0)) AFFECTED, -
 cast(s.SUM_BYTES as decimal(12,2)) GB_BACKED_UP, -
 cast(o.SUM_NUM_FILES as dec(15,0)) TOTAL_FILES_STORED, -
 cast(o.SUM_LOGICAL_MB/1024 as dec(12,2)) GB_TSM_OCCUPANCY -
 from -
 (select node_name, -
    sum(NUM_FILES) SUM_NUM_FILES, -
    sum(LOGICAL_MB) SUM_LOGICAL_MB -
  from occupancy -
    group by node_name) o, -
 (select entity, -
    sum(bytes/1073741824) SUM_BYTES, -
    sum(affected) SUM_AFFECTED -
  from summary  -
   where -
    end_time>=(current_timestamp - 30 days) -
    and activity in ('BACKUP','ARCHIVE') -
   group by entity) s -
 where ENTITY in (select node_name from nodes where domain_name<>'MN-STD-POL-DOM') -
  and o.node_name=s.entity -
  order by ENTITY

The generated report shows the total amount of files backed up, GB backed up for the last 30 days while also showing the total files stored in TSM and the nodes current total occupancy in GB.

ENTITY              AFFECTED        GB_BACKED_UP      TOTAL_FILES_STORED      GB_TSM_OCCUPANCY
----------   ---------------     ---------------     -------------------     -----------------
DORPORA05                920            47513.00                    8354              73263.60
IXPFLDNAS              60792               27.00                15443947               9434.27
ILDNASJ                12680                5.00                 1972680                715.60
ILDNASK                    8                0.00                   83004                197.48
ILDNASL                10739                4.00                 1649434                574.51
ILDNASM                45448               25.00                 1221367               1722.03
ILDNASM                 4386                7.00                 1122826                557.64
ILDNASN                    5                0.00                   99769                 46.52
ILDNASO                 1688                0.00                  459683                138.16
ILDNASP                 9369               11.00                 1402332               1623.92
ILDNASR                28787               29.00                  462479                457.15
ILDNASS                    5                0.00                   36771                 17.51
ILDNAST                 6329               21.00                  982728               1038.08
ILDNASU                 6848               19.00                 1548369                671.68
ILDNASV                18197               48.00                  999905               1333.92
ILDNASW                    5                0.00                  245366                116.89
RSD1W030S              59979                7.00                 5207060                547.23
RSD1W031S               3444                2.00                 5070450                473.15
RSD1W241S               6879               47.00                  517358               1782.92
RSD1W242S               6478               51.00                  995006               2109.45
RSD1W266S               4377                7.00                  492810                184.69
RSD1W267S               5700                3.00                  328391                100.95
RSD1W320C              40881               12.00                 2692541                576.50
RSD1W341C               8194               55.00                  456567               1120.11
RSD1W342C              17003                7.00                  625487                281.84
RSD1W343C               4628                2.00                  298699                 79.63
RSD1W504SV              1941               12.00                   42444                551.35
RSD1W505SV              1079              105.00                   38524               4033.06
RSD1W5067S               683               64.00                   22389               1343.82
RSD1W506SV               489                0.00                   14349                  4.53
RSD1W508SV              2662               31.00                   79340                699.12
RSD1W509SV              4017               74.00                  139073               1617.35
RSD1W510SV               698              544.00                   18086               9161.23
RSD1W511SV              1365              295.00                   65303               6588.06

Personal Back Ups!

I have been doing this backup/archive work for almost 20 years and I can't go a week without some acquaintance asking me about backing up their personal data. It used to be large portable USB drives were the best option but as home internet connections have increased there throughput to the point of some offering giga-bit speeds, it makes more sense to use a cloud based backup provider. This article is based off of my experience and I only offer my opinion here so take it for what it is, but I recommend everyone look into a good backup solution or else you could loose all the important memories you have accumulated as life has become more digital. 

So what do I use to protect my memories? I used to use Dropbox, then when Google Drive became available I used it, and then I started using Microsoft SkyDrive now known as OneDrive. I began to exclusively use OneDrive several years ago when I was able to get more than the paltry 5 - 10GB that the others allowed. At one point I freely acquired 100GB by using Bing and gathering Bing points. I am cheap and didn't want to pay for my cloud space. I had portable hard drives with my data but knew it wasn't the solution. So when my free OneDrive space was due to expire I was offered an Office365 family plan with 1TB of space for up to 5 people in the family. With my wife and I each having 128GB iphone that my wife has actually filled with video and pictures during trips, I knew it was the best solution for us. Not only did I get 1TB for each of us, it could cover our personal home computer and our daughter phone also. Oh, and did I mention it comes with access to download Office to our home and daughters laptop?  All this is good but it would be useless if the OneDrive app was terrible. Truthfully, I find it quite easy to use. Microsoft has continued to fix bugs and add features it has become a great tool I use almost daily. 

I write this article because it's important you think about how you will protect memories. Sure you can upload to Google Photo's or put all your items on Facebook, but each has it's own drawbacks. Google Photo's does not keep the photo at its original resolution, and Facebook is not designed to be a backup solution. I have lost personal photos and documents from when I was younger due to losing them during moves or becoming damaged by mishandling. I will never get them back, had I scanned/digitzed them I would have been able to keep them safe. I also understand that nothing is 100% protected or safe, but by putting my files in the cloud they are more safe than on a portable drive or sitting in a drawer. 

Spectrum Protect Server 8.1.4 DB Issue

Recently we ran into a situation where our Spectrum Protect 8.1.4 server started to crash. We reviewed the db2diag.log file and identified the error as ANR0131E Server DB space exhausted. Upon review of our DB file systems the OS showed 7GB available in the 5 file systems assigned for DB space.

/dev/tsm20log 111616.00   9784.93   92%      205     1% /tsmserver/tsm20log

/dev/tsm20arch 102400.00  66821.52   35%      100     1% /tsmserver/tsm20arch

/dev/tsm20fail 101376.00 100964.16    1%       21     1% /tsmserver/tsm20fail

/dev/tsm20dblv01 101376.00   7622.45   93%      104     1% /tsmserver/tsm20dblv01

/dev/tsm20dblv02 101376.00   7622.48   93%      104     1% /tsmserver/tsm20dblv02

/dev/tsm20dblv03 101376.00   7622.45   93%      104     1% /tsmserver/tsm20dblv03

/dev/tsm20dblv04 101376.00   7622.46   93%      105     1% /tsmserver/tsm20dblv04

/dev/tsm20dblv05 101376.00   7622.45   93%      105     1% /tsmserver/tsm20dblv05

The server had been crashing from the previous night and we had at first thought it was due to the active log filling and not writing out to the archive log fast enough. What was weird is we could bring TSM up and it would run for an hour or two then crash again. When we identified it was not the active log but the DB space (earlier the next morning) we were confused. Why would we get a space notification when the DB has 35GB of space still available. My co-worker created an additional file system of the same size as the other DB file systems and ran the dsmserv extend dbspace command and the file systems almost immediately jumped to 99% utilized. So it appears Spectrum Protect needed to write 35+GB of DB data and could not due to space available? We also noticed that our offshore people had added tsm20dblv05 a few months back but had used a different disk array (slow SATA) and had not informed us, which we think might have contributed to the performance issues we had been seeing on this server recently and maybe was related to the DB space issue??? Not sure, but be aware that this can occur. I will need to research how DB2 allocates space and in what chunk size it does so.