Finally - A Universal Backup Agent!

I was contacted by a representative of Spictera about their SPFS tool for Spectrum Protect. This product allows you to use a local mount on your Linux server as a place to save your backup or application dump that is a direct connection to the IBM Spectrum Protect backend storage. It uses Spectrum Protect API calls and allows you to implement versioning and retention policies just as you would normal file system backups or archives. This product can solve your issues with proprietary applications that IBM does not, and probably never will, support. Now you can backup your MongoDB or SAP HANA a lot easier by using the native backup process built into your application. Currently it only supports the following OS's.

Supported operating systems are:

• CentOS 6 for x86_64
• CentOS 7 for x86_64
• CentOS 8 for x86_64
• Debian 8 for x86_64
• Debian 9 for x86_64
• Red Hat 6 for x86_64, ppc64 be, s390x (zLinux)
• Red Hat 7 for x86_64, ppc64 le, s390x (zLinux)
• Red Hat 8 for x86_64, ppc64 le, s390x (zLinux)
• SuSE 11 for x86_64, ppc64 be, s390x (zLinux)
• SuSE 12 for x86_64, ppc64 le, s390x (zLinux)
• SuSE 15 for x86_64, ppc64 le, s390x (zLinux)
• Ubuntu 14.04 for x86_64, ppc64 le, s390x (zLinux)
• Ubuntu 18.04 for x86_64, ppc64 le, s390x (zLinux)

I would assume if it support CentOS x86_64 you should have no trouble with it on Red Hat. I'll let Spictera confirm that though. I've provided a presentation on the product for your review.

Archive Report

Where I work we have a process that bi-monthly generates a mksysb then archives it to TSM. Recently an attempt to use an archived mksysb found that sometimes the mksysb process does not create a valid file, but it is still archived to TSM. So the other AIX admins asked me to generate a report that would show the amount of data that was archived and on what date it occurred. Now I would have told them it was impossible if they had asked for data from the backup table, but our archive table is not as large as the backups so I gave it a go.

First problem was determining the best table(s) to use. I could use the summary table, but it doesn't tell me what schedule ran and some of these UNIX servers do have archive schedules other than the mksysb process. The idea I came up with was to query the contents table and join it with the archive table using the object_id field.  Here's an example of the command:

select a.node_name, a.filespace_name, a.object_id, cast((b.file_size/1048576)as integer(9,2))AS SIZE_MB , cast((a.ARCHIVE_DATE)as date) as ARCHIVE from archives a, contents b where a.node_name=b.node_name and a.filespace_name='/mksysb_apitsm' and a.filespace_name=b.filespace_name and a.object_id=b.object_id and a.node_name like 'USA%'

This select takes at least 20 hours to run across 6 TSM servers. I guess that I should be happy it returns at all, but TSM is DB2! It should be a lot faster, so I am wondering if I could clean up the script or add something that would make the index the data faster??? I am considering dropping the "like" and just matching node_name between the two tables. Would putting node_name matching first then matching object_id be faster? Would I be better off running it straight out of DB2? Suggestions appreciated.

Archiving The Actlog

I just had a friend e-mail me and state he was being contacted by his previous employer complaining he made changes to their TSM retention times causing them to lose data. The TSM admin who took over is blaming him stating he changed the retention and now the data is gone. My friend does not remember making any retention changes and the problem is that any change to the copygroup updates the "Last Update by (administrator)" and the "Last Update Date/Time" so it's not substantial evidence of who did what. The only way to verify what had really occurred would be to either keep the actlog for an extremely long period of time, or dump it to a text file that you zip and archive. (Even then it's a text file and could be tampered with) In the case of my friend, he left the company last November and anyone could have altered the copygroup since then.

How many of you archive your TSM Activity Log, and how long do you keep it for? Obviously it good for security and tracking purposes, but who manages it and can you reliably keep it in a read-only state? Of course this is also a case where a bi-monthly audit of retention settings would have helped.

Backupsets & TCopy

OK, I am going to throw a question out into the TSM world and see what you people think. I was asked at work if a Backupset could be copied using the UNIX tcopy command? I thought about it and since Backupsets are independent of the TSM server I would think, in theory, it should. The only issue I could see would be needing to define the copy to TSM, since it would have no record of it. The benefit would be that I would not need to mount the tapes all over again to run the Backupset. The reason I ask is that a customer wanted two copies of a full backup of a server to archive forever. Anyone ever tried it, any theories? They are all welcome!

Archiving - The Great Debate!

I knew when I decided to cover archiving that many of you would have comments on it, so I am expecting your feedback and a good discussion on the matter. The problems I see with archiving are many but most companies fall within one of three categories:

Category 1 - They don’t know what they need archived

• They don’t know where all the data needing archiving is


• They don’t know how long they should keep their data


• They don’t know where to begin to discover data that meets archive requirements


• They don’t want to spend the money on a true archiving solution

Category 2 - They somewhat know what they need archived

• They know some of the data needing archiving


• They don’t have all data identified


• They have an idea on how to identify data needing long term retention


• They are willing to investigate purchasing a true archiving solution

Category 3 - They know exactly what they need archived

• They know all data needing archiving


• They know retention times for all data


• They constantly review systems and apps to identify data that meets archiving requirements


• They are willing to pay for a true archiving solution

The major problem with archiving is how many companies fall within the first two categories and getting them to the point of category 3 is sometimes impossible. It’s amazing to see huge companies scratch their heads and get that perplexed look when you discuss archiving with TSM. Most people come from the old school of taking a weekend or month end full and keeping it forever.  They think this protects them in case they need any data in the future and so the customer follows this pattern and doesn’t see the problems inherent in that scenario. The problems are numerous but the most glaring one is that 99.99% of the data on those weekend or month end tapes will never be needed and you are now paying a huge amount of money for tapes and offsite storage.  Now introduce TSM into the mix and the customer or management are accustomed to the old process and wonder why they now need to identify their data for archiving.

Let’s be honest most companies become overwhelmed when asked to do discovery on specific locations and data types that should be archived, so to make it easier and less work they try to make TSM conform to the old process. Unfortunately as TSM admins we tend to either not argue the case, or when we do dissent we are overruled. So you end up doing backupsets (If you are actually archiving whole machines please see a Psychologist immediately) and relying on the customer to keep his restores to a minimum. The problem is that backupsets sound good, they give management that false sense of security, which gets them off your back, and they are independent of any particular TSM server. The truth is that they stink! Backupsets are the worst archiving process you could use. Sure Tivoli has supposedly updated them to make them more functional in 5.3, but the truth is you still end up using too many resources, waste tape, and pay more for offsite storage due to that increase in tape usage. We wont even talk about the restore times, and what happens when syntax is wrong. So backupsets are the wrong solution for anything but DR needs or portability.

TSM is an adequate archiving tool. It does a good job for small to moderate archiving, but when you have situations where the customer needs to have very descriptive meta data stored with the archives to make retrieval easier you need an enterprise tool like IBM Content Manager, Xerox DocuShare, or one of the many others out there. The problems always seem to come down to cost.  What do you do when the customer or management can’t part with the money to truly protect themselves?  That is where you need to work with them to explicitly identify the data they need archived and that retention requirements are met, that management classes and include statements are used to match data with retention times, that they document the owners of the data for future reference, and that the documents and contact information are reviewed at least once per year. I had a situation where data was being archived and a few years down the road some one asked for data and the person who had been managing the archive had left the company and no one knew what process was in place and what data was being archived. They didn’t know who all the owners of the data were and the previous manager had not done any transition or hand over to other personnel.

You need to do constant review and audit of archiving processes and standards. Too many times requirements, laws, and applications change and you find yourself without the data required. Archiving tends to be like Ron Popeil’s Rotisserie, “Set it and forget it!” This is the breaking point. As a TSM admin even I have fallen into the trap of forgetting about archive jobs and assuming they are working. So I had to make changes to how we handled archive jobs and retention. Typically I recommend reviewing requirements and processes at least twice a year if not quarterly. This will hopefully allow you to identify any issues with new data brought online, changes in requirements, and application changes. Schedules need to be reviewed, shell scripts need to be checked, and archive data should periodically be audited to make sure they are performing correctly. DO NOT RELY ON YOUR SCHEDULE EVENT RECORDS! THEY DON”T GIVE YOU A COMPLETE PICTURE! What if the customer or management decides to change the location he or she stores the data? What do you do when the customer or management wants data archived from a directory weekly but does not want the data to be deleted? What if the customer wants data kept online (in the library) and also sends a copy offsite? These are the issues you will have to deal with as you work with archives. If you were expecting solutions and answers I only have suggestions. There is no one-way to do archiving so you have to find the best process that fits your needs. The key is helping your company or customer understand what is best for them even if they don’t initially like what they hear. When it comes to data the customer is not always right. Of course you can’t make the company or customer do exactly what you’d like but you’ll have to do your best to help them understand how much they stand to lose if they don’t follow the right procedures.