End of the Road?

For those of you who come here periodically and are wondering why content has been lacking, I wanted to let you all know that it is due to my current employment situation. Currently my employer is in the process of replacing Spectrum Protect throughout the company with Commvault. Once the conversion has completed I will be released and so am currently looking for new employment. Since the 8.1.8 upgrade of our servers I have been working steadily on reporting and prepping for the conversion. The initial thought was that I would stay employed and transition to learning Commvault, but that turned out to not be the case. I have enjoyed managing ADSM/TSM/Spectrum Protect environments for the last 22 years but have seen a drop off of its use as IBM has retreated from marketing their products and moved more towards other areas of the IT field. As I have looked for jobs in the Spectrum Protect realm I have noticed that they are rarer than ever, especially with the current situation the world finds itself. I plan to leave the site and up and will post anything you the readers feel will benefit the Spectrum Protect community. Feel free to send me your article for review and if I feel it fits the nature of this website it will be posted. Unfortunately, unless things change I see myself transitioning into a new role with a cloud emphasis and posts here will end unless you all take over the mantle and provide other admins with advice and examples from your own experiences. I have enjoyed posting and helping the community since founding this website and wish you all success going forward.

IBM Spectrum Protect 8.1.6 Tier to Cloud by State

A coworker recently did a presentation on Amazon S3 with Netbackup and I went searching for the capabilities available with TSM/Spectrum Protect. It appears as of 8.1.6 they have added some additional features that greatly expand the usage of cloud based storage. Check out this informative demo from IBM.

Q STATUS Question

The company to which I am employed was recently acquired by another company that is a heavy Netbackup user. They bill not based on overall occupancy but on the "Front End Usage" of their servers. I don't know how that works for accurate billing but it's what they do and now we are being asked to provide "Front End Capacity" numbers so they can compare. The problem is ADSM/TSM/SP has always had issues with reporting "active only" data. Just recently I was working with an select IBM provided to pull the active for file systems but they also stated we had to go to the individual application servers to pull DB data. Not fun. So when I recently looked at all the detail of the Q STATUS I noticed the following near the bottom.

                 SUR Occupancy (TB): 703                

            SUR Occupancy Date/Time: 2019-01-07, 10:23:18

            Front-End Capacity (MB): 148.21             

             Front-End Client Count: 408                   

            Front-End Capacity Date: 2019-01-07, 10:23:18

                   Product Offering: IBM Spectrum Protect

According to IBM these last few statements are:

SUR Occupancy (TB)

If you have an IBM Spectrum Protect Suite (SUR) license, this field specifies the SUR occupancy on the server. The SUR occupancy is the amount of space that is used to store data that is managed by the IBM Spectrum Protect products that are included in the SUR bundle.

SUR Occupancy Date/Time

Specifies the date and time when SUR occupancy data was last collected.

Front-End Capacity (MB)

Specifies the amount of primary data that is reported as being backed up by clients. Clients include applications, virtual machines, and systems. This value is used for the front-end licensing model.

Front-End Client Count

Specifies the number of clients that reported capacity usage based on the front-end licensing model.

Front-End Capacity Date

Specifies the date and time when front-end capacity data was last collected.

Product Offering

Specifies a product offering.

So what exactly is the SUR license and does the EE license cover that? How is the data generated because my server reports front-end data at 148MB but if it is actually TB then that would fit more closely with overall capacity as front-end would be about 21% of overall occupancy. IBM obviously has some function to gather front-end numbers so why do they also have us going down the rabbit hold of gathering the data application data from the clients?

Rumor Mill Update!

So my source at IBM was right except for the name, but she was close, in that IBM is going to be restructuring its storage software group under the Spectrum Storage moniker (Specter was close but Spectrum is a better title). ZDNet states that IBM will be investing $1 Billion in software defined storage which also means TSM is no longer under the Tivoli division....(Please, Please, Please let this be true!) In my opinion Tivoli was a horrible brand name and IBM should have never put ADSM under its umbrella.

Hopefully we will see some huge changes to TSM to make it more competitive. Some cloud connectivity would be a good start as more companies are implementing hybrid data-centers. I'd also like to see the new administration and reporting tool become a little more flexible with user defined reports, but we'll have to wait and see where this restructuring takes TSM or should we now call it Spectrum Storage Manager?

TSM Rumor Mill

So I was talking with an IBM source about IBM's recent layoffs and she stated that many groups were being restructured and TSM was affected. According to my source she said that TSM is possibly going to have a name change. This is no shock for those who remember when TSM was called ADSM under the ADSTAR (ADvanced STorage And Retrieval) division. A partial name drop was something along the lines of Specter blah blah blah!! So take this rumor with some reservations. IBM is going through a major upheaval as they decide what their service path with be in the future, so it's anybodies guess where TSM will end up.

IBM P7 Strange Behaviour

We have a P7 frame that has 4 LPARs that are used as TSM storage agents from which snapshots of our SAP DB's are mounted for backup. They have always had great performance until one LPAR had a bad HBA that phoned home and was replaced. After it was replaced performance for backups dramatically decreased from 800MB/s to 150MB/s and overall performance of the server would drastically drop. When the DB requiring backup is over 25TB that is a huge hit, and we could not find the root cause.  At first IBM said it was our Hitachi disk that was the problem. We eliminated that right away, so we then replaced the new HBA, checked our fiber, and then checked the GBIC and nothing seemed to fix the situation. During the first week I asked the IBM service technician if we could possibly have a bad drawer or slot and he emphatically said "No! If you did you would have errors all over the place." So we checked firmware, we moved cards within the frame (again), we double checked the fiber, now we were going into the third week. So I kept asking if something could be wrong with the drawer/slots and I kept getting the same answer. The reason I suggested it was due to previous experience. I have seen hardware go bad without totally going "out". So after exhausting everything other than the replacing the slots, IBM finally replaced the slots. Viola! Backup speeds went back to normal and system degradation during the backup disappeared.  So the slots/drawer was the issue. No errors relating to a slot/drawer hardware issue occurred but something caused the slots to degrade performance.  It took almost a month to resolve the issue, I wouldn't say that IBM support was very thorough and at times tried to push off the problem to other vendors (i.e. Hitachi). I can only suggest in the future you trust your instincts and push the CE's to follow down every avenue. My headache is over, but now the RCA begins.

Tivoli Storage Manager v6.2 is coming in March

Tivoli Storage Manager v6.2

TSM 5.5 to 6.1 Video

If you don't already subscribe to IBM's TSM Information Update & Storage Newsletter then you might not be aware of the following video IBM has posted to their website. They have provided a video tutorial upgrading TSM 5.5 to 6.1. Check it out here.

Still Looking For TSM Admins In Boulder, CO

Hello,
My name is Arjun and I'm a recruiter at Artech.

Artech has an urgent contract for one of our direct clients:

Job Title: TSM Administrator
Location: BOULDER, CO
Job Description:
Required Skill: TSM support

If you are qualified, available, interested, planning to make a change, or know of a friend who might have the required qualifications and interest, please call me ASAP at (973) 993-9383 Ext.3319, even if we have spoken recently about a different position. If you do respond via e-mail please include a daytime phone number so I can reach you. In considering candidates, time is of the essence, so please respond ASAP.

Artech is a global IT Consulting company with over 30 Fortune 500 customers. You may visit our website at http://www.artechinfo.com/ to learn more about us.

Thank you.
Sincerely yours,
Arjun Dheer
(973) 993-9383 Ext.3319
arjun_dheer@artechinfo.com

GPFS Revisited

Well I am still having issues with GPFS. It turned out the mmbackup wont work with the filesystem size either and a chat with IBM support was not encouraging. Here is what one of our System Admins found out:

The problem was eventually resolved by IBM GPFS developers. It turns out, they never thought their filesystem would be used in this configuration (i.e. 100,000,000 + inodes on a 200GB filesystem). During the time the filesystem was down, we tried multiple times to copy the data off to a different disk. Due to the sheer number of files on the filesystem, every attempt failed. For instance, I found the following commands would have taken weeks to complete:

# cd $src;find . -depth -print | cpio -pamd $dest
# cd $src; tar cf - . | (cd $dest; tar xf -)


Even with the snapshot, I dont think TSM is going to be able to solve this one. This will probably need to be done at the EMC level, where a bit level copy can be made.

So GPFS is not all it was thought to be. So pass it along and make sure you avoid GPFS for application that will produce large numbers of files.

Disk Question

Ok so in the DS4000 Redbook (page 153) they give a TSM example for setting up disk and they describe using a RAID 10 configuration. What!?! I've always heard and taught that TSM should use TSM mirroring since the extra DB copies are used for writing to help with performance and TSM keeps them all sync'd. So why would I create two RAID 10 arrays and let the DS4000 handle the mirroring? Why not create two RAID 1 arrays and mirror thru TSM? This has implications not only on the DS4000 series but on every high-end enterprise disk out there. So which is it IBM?

Leaving IBM

Friday will be my last day with IBM.  It’s been a great 5 years, but I realized that the only way my career would grow was to make a change.  I have learned a lot and am thankful for the opportunity IBM provided.  My new employer will be Infocrossing, and I look forward to the challenges that I will encounter. For one it won’t be a majority IBM hardware user so I will be able to get my hands dirty with ACSLS and STK libraries.  I love IBM equipment so it will be interesting to see what it’s like in the non-IBM world. I will still be doing TSM so this blog is not going anywhere.

Have You Checked Your Drive Firmware Lately?

Well the weekend is over and I thought I would update you on the problem SAP system.  In the last update we had discussed how this one TSM server was consistently having its mounts go into a RESERVED and hanging. The only thing we could do was cycle the TSM server. So after talking to support they stated it was a known problem listed in APAR IC49066 and fixed in the TSM server 5.3.3.2 release. So we upgraded the TSM server, updated ATAPE on the problem server to the same level as on the controller, and turned on SANDISCOVERY. The system came back up and started to mount tapes, albeit slowly. So we let it run and all seemed well for about the first 8 hours then all hell broke loose. TSM started taking longer and longer to mount tapes until it couldn’t mount anything. We began receiving the following errors also:

7/13/2006 9:38:45 AM ANR8779E Unable to open drive /dev/rmt25, error number=16.
7/13/2006 9:38:45 AM ANR8311E An I/O error occurred while accessing drive DR25 (/dev/rmt25) for SETMODE operation, errno = 9.

We reopened the problem ticket and talked to a number of Tivoli support reps and almost had to force them to have us run a trace. The original problem fix began Friday and here we were still trying to fix it well into early Sunday morning. So after getting the trace to Tivoli they looked it over and seemed perplexed at the errors. The errors seemed as if Tivoli was trying to mount tapes for drives that the library client was not pathed for. Also it seemed TSM was polling the library for drives but was unable to get a response so it went into a polling loop until it found a drive. This caused our mount queue to get as high as 100+ tape mounts waiting and the mounts that did complete sometimes took 30-45 minutes to do so. It was NUTS! So Sunday morning a new Tivoli Rep was assigned (John Wang). As we discussed the problem and Tivoli was trying to get a developer to analyze the trace John mentioned how he had seen this type of error before in a call about an LTO-1 library. He stated that it took 3 weeks to determine the problem but that the end result was that the firmware on the drives was down-level and causing the mount issues. So I checked my 3584’s web interface (I feel bad for all those people out there without web interfaces on their libraries) and found the drive firmware level at 57F7. This seemed down-level from what little information we had so I had my oncall person call 1-800-IBM-SERV and place a SEV-1 service call. The CE called me and we discussed the firmware level. When he saw how down-level it was he was surprised and lectured me on making sure we always check with CE’s before we do any changes to the environment. The CE then gathered his needed software and came to the account to update the drives. I brought all the TSM servers down and after 30 minutes the library had dismounted all tapes from the drives. The CE then proceeded to update the firmware, which actually only took 15-20 minutes. I expected longer. So we went from firmware level 57F7 to 64D0. Huge jump! So after the firmware upgrade I audited the library and brought the controller back up. Viola! It started mounting tapes as soon as the library initialized and the response was back to what it should have been. It’s now Tuesday morning and there have been no problems. So before you upgrade TSM be sure to have checked your libraries firmware (both library and drives). It could mean the difference between sink and swim!

Managing RAW Volumes

Well I was recently called by another IBMer and asked how to use RAW volumes. The person called to ask why sometimes DSMFMT will format quite fast on one machine then take forever on another machine. Well one thing you have to understand about DSMFMT is that it's taking that file and making the space within it RAW. If you've ever looked inside an unused TSM volume you'll see it is a text file filled with ADSM over and over (they might have changed the fill but when I was teaching TSM thats what we saw). So why use RAW instead of actual files (other than files being a redundant process)? FAST! EASY! and when speed in DR is key it's the only way to go. It's actually easier than one would think and with a little script you can manage your RAW volumes and hdisks easily. I'll post the script along with a script to querry the serial and WWN of your tape drives. These two scripts come courtesy of Hari Patel my co-worker who is a PERL mad man. (Download tar/zip)