SystemState Restore Issue with Windows 8.x Client

3/7/2017 Update
I have had a number of people request the test flag and one response said it was kicking up errors when used with the CAD service/process. Please note this flag is only needed in the event of a system state restore and you would not (I hope) be using the CAD/Scheduler to restore your system state. So the testflag should only be activated in the dsm.opt in the event you are doing a manual restore of the system state and DO NOT leave it in the dsm.opt as it will affect daily use.


1/24/2018 Update
After waiting for IBM to post an update to this issue I decided I would let you know that IBM has an undocumented flag for this issue that allows the client to restore the System State online rather than work with the PE. Our Windows admins spoke with Microsoft and they state emphatically that they support online system state restores, so is appears IBM does not starting with the 8.1.x client. Here is the flag to allow the SP client to perform online system state restores.

ONLINE_SYSTEMSTATE_RESTORE_ALLOWED

12/4/2018
I wanted to make everyone aware that starting with the 8.1 Windows Client, IBM has deprecated the System State restore process so that any restore attempt will require the use of a WinPE environment. With the 8.1.x client you can no longer do a complete restore of a server within the normal windows environment and will have to make sure you have a compatible WinPE for every Windows Hardware type or environment. This, in my opinion, is a huge break of the software. Although IBM stated that Microsoft does not support online system state restores, Microsoft disagrees. I say this knowing that if you are on a 7.x or earlier version of the client IBM will support you with the system state restore, but as of 8.1 you will be directed to build a Windows PE environment and follow a series of stepto create the necessary disc. The support person I spoke with treated this process as if it should be second nature to any Windows Admin.

We recently were migrating a clients server and in the process the disks were corrupted. The client installed on the new machine was 8.1.2 and when the Windows Admins tried to restore the system state they experienced the following error.

 

After speaking with support we were directed to the following document by Andrew Raibeck that discussed creating the WinPE disk image with the TSM client code that would allow the offline restore of the System State. We attempted to build the ISO but when the Windows Admin tried to run the dsmc client we experienced the following error:

Who do we call when we experience an error like this in a WinPE environment? IBM or Microsoft? So you have two choices, have a Windows PE available for each hardware type or try using the testflag but understand IBM may support its use. If you were fully aware of this and took precautions what were they? My thinking is that many admins have no idea this is going to bite them and its better you know now than find out the hard way.
 

Spectrum Protect 8.1.0 vs. 8.1.2 Client Issue

I just wanted to let everyone know that there is a compatibility issue with the Spectrum Protect 8.1.2 client with the 8.1.0 server. If you upgrade the client to SP 8.1.2 and your server is at 8.1.1 or 8.1.0 your server will be unable to start. The error you will get is

Spectrum Protect ANR0162W Supplemental database diagnostic information:  -1:08001:-30082 ([IBM][CLI Driver] SQL30082N  Security processing failed with reason "24" ("USERNAME AND/OR PASSWORD INVALID").

The issue has to do with the GSKit8.gskcrypt64.ppc.rte and GSKit8.gskssl64.ppc.rte in the 8.1.2.0 client. When we rolled back to the 8.1.0 client the SP server started without an issue.

TSM/Spectrum Protect 7.x Client Issue

I ran across an issue with the TSM/Spectrum Protect client recently that through investigation showed that the issue has not been patched and the supposed resolution works....sometimes. The issue is that when an AIX backup is running you can experience the following error:

08/02/17   03:24:52 ANS1512E Scheduled event 'CS-FS-U-02.00' failed.  Return code = 12.
08/02/17   13:54:36 ANS2820E An interrupt has occurred. The current operation will end and the client will shut down.
08/03/17   03:39:20 calloc() failed: Size 31496 File ../mem/mempool.cpp Line 1090
08/03/17   03:39:20 ANS1999E Incremental processing of '/usr/ibm' stopped.

This is a memory issue and can occur if TSM cannot allocate enough memory during the scheduled backup. The interesting thing is that manual backups run without issue. It is only when a scheduled backup is run that we experience the error. The server in question has only 8GB so we do know that real memory is limited so I followed the troubleshooting tips I saw online and checked the file system for excessive files. Querying the file system determined that was not the issue.  I could not exclude the file system and so I added the MEMORYEFFICIENTBACKUP YES option thinking that would resolve the issue. Unfortunately the addition of the MEMORYEFFICIENTBACKUP YES option did not work. Subsequent investigation showed some people had to downgrade their TSM client version to a 6.4 level to resolve the problem. Unwilling to do so I changed the option so it used the disk cache function. So far using disk cache has worked without issue, but it's concerning that a change to the TSM/SP client has created this memory issue. So the fix was to add the following two options to my dsm.sys:

   memoryefficientbackup DISKCACHEM
   diskcachelocation     /tmp

Please note that my /tmp file system is over 2GB in size and only 1% used so make sure you have a sufficiently sized file system you can use if you use the DISKCACHEMETHOD option.

Why Are You Not Using Google & YouTube?

I had an OS Admin contact me through LinkedIn and G-Mail asking for help on trying to find his archived data. He didn't have much experience with TSM and was looking for information on how to find the long term backups (i.e. Archives). I asked him if he even tried to search Google? Google and YouTube are great resources for all your needs. For example if you want  to learn more about the new Operations Center you can see a plethora of videos by searching YouTube. You can also use google to find all sorts of related documents and pages when it concerns APARs and errors. If you haven't done your due diligence you make yourself look dumb. Shoot, you can even search Google from my website and get my posts and outside relevant web pages!

So here is a list of YouTube videos you can reference:

Backup and Archive

Server Administration

Setup TSM Deduplication

Tivoli Data Protection Agents

TDP for Virtual Environments

Remote Site Backup

Ok here is the scenario, we have a 2.8TB Windows file server at a remote location and we have to back it up over the WAN.  We can't put a TSM server locally due to many issues (one is that it's not worth wasting resources on 2-3 servers). Due to throughput the backup was calculated to possibly take 25+ days to complete. So the idea I came up with was to copy the data onto a 4TB USB  drive and then ship the drive to our Data Center and attach it to another Windows server as the same drive name/label and the drive has at the remote site, setup a dsm.opt with the node name of the remote server and run the backup. The hope is that then when we run the backup remotely it will see all it's data as already backed up and then pickup with incremental backups. Will it work? Anyone tried it? I'm worried there are too many file variables involved and it will still try and backup the files all over again.

Client Lockdown

A directive came down for 75+ windows servers to be "locked down" when it came to accessing their backup data. The TSM client will allow anyone to open the client GUI and restore files with which they have permission. So I considered the various ways to keep anyone from accessing TSM and restoring data; we could set permissions on the GUI and command line to not allow executing unless in the admin group, We could delete the command line and GUI executable, or we could simply set the SESSIONINIT option on the server.  After weighing the options the SESSIONINIT was the easier and most direct way to keep anyone but a TSM admin from restoring data.  Once SESSIONINIT is set the TSM client GUI, command line, and web GUI will not be allowed to initiate a session. All restores will have to be executed through a schedule from the TSM server. Of course you can temporarily turn SESSIONINIT off, but only a TSM admin can do so, making it easier to track who's accessed the data.


(Note: SESSIONINIT does not support the CAD)


The problem was how to update 75+ Windows servers options file and then restart the TSM Scheduler. So you can change any MANAGEDSERVICES options to WEBCLIENT using a client option set, but SESSIONINIT is another problem. As it turns out, if you set SESSIONINIT on the TSM server you have to put the HLAddress and LLAddress in the node definition on the server.  The client dsm.opt must have the TCPCLIENTADDRESS and TCPCLIENTPORT in the dsm.opt. What we didn't know was that we also had to put SESSIONINIT  SERVERONLY in the dsm.opt also. If you set SESSIONINIT on the TSM server and not on the client and the scheduler was defined with /validate:yes then you will get "password" errors and the scheduler will crash. The reason for this is that TSM does not allow client initiated sessions, but the scheduler when started is trying to validate its password. Since the scheduler can't validate the password it fails similar to when the password has not been set when using PASSWORDACCESS GENERATE. 


We had the TCP Client settings, but adding SESSIONINIT to all 75+ servers would have been a chore...unless you know how to use the Windows command prompt and dsmcutil. Here's how I added the SESSIONINIT option to all 75 servers.

Example of how to remotely add a line to the dsm.opt


c:\echo SESSIONINIT SERVERONLY >> \\WINSERVPRD20\C$\progra~1\tivoli\tsm\baclient\dsm.opt

I put the command for all 75 into a batch file and ran it looking for errors (not all our servers had TSM installed in the default location). Then I used the dsmcutil command to stop and start the TSM scheduler remotely.

Example of stopping and starting the TSM Scheduler service

c:\C:\progra~1\tivoli\tsm\baclient\dsmcutil stop /name:"TSM Client Acceptor" /machine:WINSERVPRD20

c:\C:\progra~1\tivoli\tsm\baclient\dsmcutil start /name:"TSM Client Acceptor" /machine:WINSERVPRD20

I needed to restart the scheduler on all 75 servers so once again I created a Windows command line batch file with the following commands in a list and it restarted all 75 quickly and easily from my own desktop. 

 

TSM 6.x Client Deployment

Has anyone used TSM's client deployment process/function?  How well does it work? Is it worth the effort? I have a lot of servers we need to install TSM too and would like to utilize it if it will work.

Memory Efficient Backup

I have come across a few servers that have memory issues when backing up large drives / filesystems and using MEMORYEFFICIENTBACKUP was not resolving the backup failures. The error code I would see was

ANS1030E The operating system refused a TSM request for memory allocation.

The problem was one drive that has millions of small files all under one directory. It's one of those poorly devised custom programs, and I've seen it a hundred times.  I could journal the drive, but we seemed to get better results with an alternative method.  A coworker reminded me of the INCLUDE.FS option using DISKCACHE and specifying the cache location.

INCLUDE.FS G: MEMORYEFFICIENTBACKUP=DISKCACHEMETHOD DISKCACHELOCATION=L:\TSM_cache

With the include.fs TSM will only use the memoryefficientbackup diskcache method for that particular drive, allowing the rest of the backup to perform normally. So if you have a drive or filesystem with millions of files this is an alternative to journaling.

TSM 6.1 Debian Client Request

Well TSM 6.1 has officially been out for about a month now and I have already received a request for a Debian client. Maybe Harry can help out in that area. Anyone want to try their hand at putting one together?

Bare Metal Restore

All the best for 2008
Bare Meal Restore - Put simply

For W2K the TSM dsm.opt DOMAIN ALL-LOCAL is the default, this will include
SYSTEMOBJECT backup for incrementals.

Backup Commands:
dsmc incr

TSM Windows 2000 restore steps:
1. Repartition, format harddrive
2. Install W2K, set computername to original
3. Install appropriate W2K Service Pack
4. Install tsm ba client
5. dsmc restore c:\ -replace=all -subdir=yes
6. dsmc restore systemobject
7. Reboot

Cheers
Ryan Partington

TSM Client Install Help

I found a good set of How-To's and documentation from IBM while looking for client overviews of installing the TSM client on UNIX platforms. It is an IBM Education Assistant covering installation, migration, and upgrading. It's always been part of the Infocenter but I never noticed it. Duh! There is a nice slide show walk-through covering installation of the TSM client on different platforms. Also some PDF's and even a Silent Install how-to. The main page can be found here and the installation help can be found here.

Vulnerabilities Found In TSM Client!!!

(APARs IC52905 and IC53616)

Wow I'm glad I added the news section to the blog I saw this article about a possible exploit with the TSM CAD on client systems. I'd try to explain it but it's best to check the article. It can leave systems open for execution of arbitrary code. Check it out here and here. This exploit goes back as far as the 5.1 client so be aware of the issue even with older clients.

Affected Products:

IBM Tivoli Storage Manager Client version 5.1
IBM Tivoli Storage Manager Client version 5.2
IBM Tivoli Storage Manager Client version 5.3
IBM Tivoli Storage Manager Client version 5.4

Solution:

Apply client update package 5.4.1.2 (UK27738 and UK27739) :
http://www.ibm.com/support/docview.wss?uid=swg24016585

Apply client update package 5.3.5.3 (UK29248 and UK29249) :
http://www.ibm.com/support/docview.wss?uid=swg24016838

Apply client update packages 5.2.5.2 and 5.1.8.1 :
http://www.ibm.com/support/docview.wss?uid=swg24016985
http://www.ibm.com/support/docview.wss?uid=swg24016586

Original Advisory:
http://www-1.ibm.com/support/docview.wss?uid=swg21268775