Tuesday, May 26, 2020

Finally - A Universal Backup Agent!


I was contacted by a representative of Spictera about their SPFS tool for Spectrum Protect. This product allows you to use a local mount on your Linux/Unix server as a place to save your backup or application dumb that is a direct connection to the IBM Spectrum Protect backend storage. It used Spectrum Protect API calls and allows you to implement versioning and retention policies just as you would normal file system backups or archives. This product can solve your issues with proprietary applications that IBM does not and probably never will support. Now you can backup your MongoDB or SAP HANA a lot easier by using the native backup process built into your application. Currently it only supports the following OS's.


Supported operating systems are:
  • CentOS 6 for x86_64
  • CentOS 7 for x86_64
  • CentOS 8 for x86_64
  • Debian 8 for x86_64
  • Debian 9 for x86_64
  • Red Hat 6 for x86_64, ppc64 be, s390x (zLinux)
  • Red Hat 7 for x86_64, ppc64 le, s390x (zLinux)
  • Red Hat 8 for x86_64, ppc64 le, s390x (zLinux)
  • SuSE 11 for x86_64, ppc64 be, s390x (zLinux)
  • SuSE 12 for x86_64, ppc64 le, s390x (zLinux)
  • SuSE 15 for x86_64, ppc64 le, s390x (zLinux)
  • Ubuntu 14.04 for x86_64, ppc64 le, s390x (zLinux)
  • Ubuntu 18.04 for x86_64, ppc64 le, s390x (zLinux)
I would assume if it support CentOS x86_64 you should have no trouble with it on Red Hat. I'll let Spictera confirm that though. I've provided a presentation on the product for your review.


Wednesday, May 13, 2020

HBA Info Script

I thought I would pass this script along for anyone needing a way to out HBA info on Linux in a nicely formatted table.

#!/bin/bash
echo "FC_HOST     |   PortID   |    WWN             | State      | Speed    "
echo "------------+------------+--------------------+------------+----------"
for hba in `ls -d /sys/class/fc_host/host*`;do
  FC_HOST=`basename $hba`
  PortID=`cat $hba/port_id`
  wwpn=`cat $hba/port_name`
  state=`cat $hba/port_state`
  speed=`cat $hba/speed`
  hba=`cat $hba/symbolic_name`
  echo "$FC_HOST| $PortID | $wwpn | $state| $speed" | awk 'BEGIN{OFS=FS="|"}{for(i=1;i<=NF;i++){$i=sprintf("%-12s",$i)};print}'
done #|sort -k3n,6
echo "------------+------------+--------------------+------------+----------"


The output looks like this


FC_HOST     |   PortID   |    WWN             | State      | Speed    
------------+------------+--------------------+------------+----------
host0       | 0xffffffff | 0x20003890a540876c | Linkdown   | unknown  
host11      | 0xffffffff | 0x20003890a540876d | Linkdown   | unknown  
host12      | 0x640000   | 0x100000109b1bbe00 | Online     | 32 Gbit  
host13      | 0xffffffff | 0x20003890a5803461 | Linkdown   | 40 Gbit  
host14      | 0xffffffff | 0x20003890a5803462 | Linkdown   | 40 Gbit  
host15      | 0x640000   | 0x100000109b1bbe01 | Online     | 32 Gbit  
host16      | 0x03eac0   | 0x100000109b1b9a77 | Online     | 16 Gbit  
host17      | 0x04eac0   | 0x100000109b1b9a78 | Online     | 16 Gbit  
------------+------------+--------------------+------------+----------

If you would like more detail I added a couple more columns to show make, model, firmware, and driver version. This is a fixed width column layout and the minimum column width can be adjusted for your needs.

#!/bin/bash
echo "FC_HOST         |   PortID       |    WWN             | State          | Speed          | Make           | Model          | Firmware       | Driver"
echo "----------------+----------------+--------------------+----------------+----------------+----------------+----------------+----------------+---------------"
for hba in `ls -d /sys/class/fc_host/host*`;do
  FC_HOST=`basename $hba`
  PortID=`cat $hba/port_id`
  wwpn=`cat $hba/port_name`
  state=`cat $hba/port_state`
  speed=`cat $hba/speed`
  hba=`cat $hba/symbolic_name`
  hba_make=`echo $hba | cut -f1 -d ' ' `
  hba_model=`echo $hba | cut -f2 -d ' ' `
  hba_fw=`echo $hba | cut -f3 -d ' ' `
  hba_driver=`echo $hba | cut -f4 -d ' ' `
  echo "$FC_HOST| $PortID | $wwpn | $state| $speed | $hba_make | $hba_model | $hba_fw | $hba_driver " | awk 'BEGIN{OFS=FS="|"}{for(i=1;i<=NF;i++){$i=sprintf("%-16s",$i)};print}'
done #|sort -k3n,6

echo "----------------+----------------+--------------------+----------------+----------------+----------------+----------------+----------------+---------------"

The output looks similar to this:



Tuesday, April 21, 2020

End of the Road?

For those of you who come here periodically and are wondering why content has been lacking, I wanted to let you all know that it is due to my current employment situation. Currently my employer is in the process of replacing Spectrum Protect throughout the company with Commvault. Once the conversion has completed I will be released and so am currently looking for new employment. Since the 8.1.8 upgrade of our servers I have been working steadily on reporting and prepping for the conversion. The initial thought was that I would stay employed and transition to learning Commvault, but that turned out to not be the case. I have enjoyed managing ADSM/TSM/Spectrum Protect environments for the last 22 years but have seen a drop off of its use as IBM has retreated from marketing their products and moved more towards other areas of the IT field. As I have looked for jobs in the Spectrum Protect realm I have noticed that they are rarer than ever, especially with the current situation the world finds itself. I plan to leave the site and up and will post anything you the readers feel will benefit the Spectrum Protect community. Feel free to send me your article for review and if I feel it fits the nature of this website it will be posted. Unfortunately, unless things change I see myself transitioning into a new role with a cloud emphasis and posts here will end unless you all take over the mantle and provide other admins with advice and examples from your own experiences. I have enjoyed posting and helping the community since founding this website and wish you all success going forward.

Wednesday, August 14, 2019

Error Upgrading to 8.1.8.0

I performed an upgrade today on a server going from 8.1.0.0 to 8.1.8.0 and after the upgrade we encountered the following errors in the actlog as the system was starting.

08/14/2019 11:59:59      ANR9999D_2835836899 InstGetDB2stmtPass1(instr.c:550)                          Thread<404>: Error in fetching DB2 statement stats,                          rc=235908/14/2019 11:59:59      ANR9999D Thread<404> issued message 9999 from:08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001000365b4 StdPutText08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010003784c OutDiagToCons08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010000cf08 outDiagfExt08/14/2019 11:59:59      ANR9999D Thread<404>  0x000000010019d0d0                          IPRA.$InstGetDB2stmtPass108/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001001a15e0 instrumentStart08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001012d7b98                          AdmInstrumentBegin08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008da68c AdmCommandLocal08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001008d7af0 admCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013fbe68 RunCommand08/14/2019 11:59:59      ANR9999D Thread<404>  0x00000001013f7a48                          ServerMonTwentyMinThread08/14/2019 11:59:59      ANR9999D Thread<404>  0x0000000100010648 StartThread

Although the server started, I wanted to make sure this would not cause issues with the instance. I searched the error ANR9999D_2835836899 and found the following IBM APAR notice. It appears that after the upgrade you may have to manually upgrade the DB2 version using the DB2 update tool. While it appears to be a minor issue, I would recommend you check your instances after upgrade to verify whether you need to run the manual steps to resolve the errors. I've included them here for your convenience.

NOTE: I've added a couple commands (in bold) that we have had to add to the process to make it work.

Local fix

Even though the db2-level might be updated to the latest version 11.1.4.4, run the below set of commands
Stop TSM server and issue the following commands as the instance owner.
  • db2 start
  • db2 force application all
  • db2 terminate
  • ipclean

login as root user and run
  • /opt/tivoli/tsm/db2/instance/db2iupdt -d <instance name>

login as the instance owner ID and issue the command:

  • db2updv111 -d TSMDB1

This issue is supposedly fixed in version 8.1.9.0 and higher.

Tuesday, June 18, 2019

Audit Occupancy Issue

2019 Update
IBM has resolved this issue with version 8.1.6 and higher. As of 8.1.6 the standard AUDIT LICENSE will update all clients whether they use standard or directory storage pools.

1/24/2018 Update 
IBM Support contacted me and stated that the developers said the issue was actually a defect (APAR IT23153) and that you can fix the issue by logging on the SP server as the instance owner and running the following commands:

  • db2 connect to tsmdb1
  • db2 set schema tsmdb1
  • db2 "update tsmdb1.sd_pool_clusters set physoccupancy=1 where physoccupancy=0"
  • login as admin to Spectrum Protect instance
  • run AUDIT LICense command
  • run Q AUDITOCCupancy command

You should now see the AUDITOCC table is populated.


1/23/2018 
 We setup a new Spectrum Protect instance with a Directory Container Storage Pool and when we went to gather storage data for billing purposes we discovered that the AUDITOCC table in the database showed no storage data. The first troubleshooting step was to check to see if AUDITSTORAGE option was set to YES which it was. Then we reran the AUDIT LICENSE command to have Spectrum Protect audit the storage and after completing there was still no data in the AUDITOCC table. I contacted IBM and asked for support to tell me why I had no data in my AUDITOCC table and was met with level 1 support unable to provide an answer (which was probably due to them not knowing my servers full configuration). When the PMR was passed up to the chain to level 2 support and they reviewed the log details I sent in, they responded that DIRECTORY CONTAINER storage pools do not provide occupancy data to the AUDITOCC table. To gather the required data that the AUDITOCC table normally would provide you have to run the GENERATE DEDUPSTATS command.

GENERATE DEDUPSTATS <STORAGE POOL> <NODE NAME>

Please note that there is nothing in the Spectrum Protect manuals that will inform you that DIRECTORY CONTAINER storage pools do not provide AUDITOCC table data. IBM Level 2 support has opened a PMR requesting that the documentation be updated for the QUERY AUDITOCC command. I am not sure if you can use a * with the node name and have GENERATE DEDUPSTATS but will be trying it today and will let you all know the results in the comments below.

Wednesday, February 20, 2019

IBM Spectrum Protect 8.1.6 Tier to Cloud by State

A coworker recently did a presentation on Amazon S3 with Netbackup and I went searching for the capabilities available with TSM/Spectrum Protect. It appears as of 8.1.6 they have added some additional features that greatly expand the usage of cloud based storage. Check out this informative demo from IBM.