Spectrum Protect Server 8.1.x Paging Space Issue

I thought I would pass this along...I have two Spectrum Protect 8.1.4 instances on the same AIX 7 LPAR that since their upgrade to this version has had frequent exhausting of paging space crashes. We reviewed and checked processes and figured it was an AIX issue. Upon deeper investigation I noticed there were some previous APARs for DB2 exhausting paging space due to a 64k block size. Investigating that further led me to the following APAR for Spectrum Protect 

IT23731: SPECTRUM PROTECT SERVER ON AIX MAY RUN INTO PAGING AND OUT OF MEMORY CONDITION

So to check I ran the following shell command to list the top 10 processes using memory/paging

# svmon -Pt 10 -O summary=basic,unit=MB

Unit: MB

----------------------------------------------------------------

     Pid Command          Inuse      Pin     Pgsp  Virtual

 4391072 db2sysc       74805.10     51.4 15266.27 72775.38

 7274744 db2sysc       71323.35     50.2 15579.61 71796.98

10486106 db2vend       71164.58     47.6 15084.41 71589.28

 4915692 dsmserv       15230.22     49.5  6629.03 20345.98

 2490670 dsmserv        1931.09     49.6   299.12  2197.74

 5701694 db2sysc         632.43     47.6     42.1   732.73

 8126714 db2sysc         569.93     47.6     29.1   665.60

 6553630 db2vend         561.76     47.6     37.0   660.47

 6422602 db2sysc         559.20     47.6     37.2   657.31

 4849682 db2sysc         559.20     47.6     37.2   657.31

So the db2sysc and dsm2vend along with the dsmserv are using up large amounts of paging.

Here is my using in topas

PAGING SPACE
Size,MB   64512
% Used     60
% Free     40

So what's the solution? Restart the TSM instance(s) until you can upgrade to 8.1.5 

Spectrum Protect Server 8.1.4 DB Issue

Recently we ran into a situation where our Spectrum Protect 8.1.4 server started to crash. We reviewed the db2diag.log file and identified the error as ANR0131E Server DB space exhausted. Upon review of our DB file systems the OS showed 7GB available in the 5 file systems assigned for DB space.

/dev/tsm20log 111616.00   9784.93   92%      205     1% /tsmserver/tsm20log

/dev/tsm20arch 102400.00  66821.52   35%      100     1% /tsmserver/tsm20arch

/dev/tsm20fail 101376.00 100964.16    1%       21     1% /tsmserver/tsm20fail

/dev/tsm20dblv01 101376.00   7622.45   93%      104     1% /tsmserver/tsm20dblv01

/dev/tsm20dblv02 101376.00   7622.48   93%      104     1% /tsmserver/tsm20dblv02

/dev/tsm20dblv03 101376.00   7622.45   93%      104     1% /tsmserver/tsm20dblv03

/dev/tsm20dblv04 101376.00   7622.46   93%      105     1% /tsmserver/tsm20dblv04

/dev/tsm20dblv05 101376.00   7622.45   93%      105     1% /tsmserver/tsm20dblv05

The server had been crashing from the previous night and we had at first thought it was due to the active log filling and not writing out to the archive log fast enough. What was weird is we could bring TSM up and it would run for an hour or two then crash again. When we identified it was not the active log but the DB space (earlier the next morning) we were confused. Why would we get a space notification when the DB has 35GB of space still available. My co-worker created an additional file system of the same size as the other DB file systems and ran the dsmserv extend dbspace command and the file systems almost immediately jumped to 99% utilized. So it appears Spectrum Protect needed to write 35+GB of DB data and could not due to space available? We also noticed that our offshore people had added tsm20dblv05 a few months back but had used a different disk array (slow SATA) and had not informed us, which we think might have contributed to the performance issues we had been seeing on this server recently and maybe was related to the DB space issue??? Not sure, but be aware that this can occur. I will need to research how DB2 allocates space and in what chunk size it does so.

Find WWN's in AIX

So I use the following script to find WWN's in AIX. Does anyone have a better script they'd like to share?

#!/usr/bin/ksh

CSV="," 

for FCSX in `lscfg | grep fcs | awk '{ print $2 }' | sort`
do 
echo ${FCSX}${CSV}`lscfg -vl ${FCSX} | grep "Network Address" | sed -e "s/^.*\.//"`${CSV}`lscfg -l ${FCSX} | awk '{ print $2 }'` 
done

Poor Performance Followup

As a follow-up to the previous poor performance post I thought I'd post what the outcome was. As it turns out we checked performance tuning settings in TSM and AIX and no performance increase was seen. We asked the DB2 admins to review any of their settings and they could not find any tunables that had not already been implemented. We sent in servermon.pl output and although they saw performance that was sub-par, they couldn't designate what was causing it. There are no server/adapter/switch/disk/tape errors so nothing emerged as the culprit for our poor throughput performance.

So we reviewed the backup time of each TSM storage agent server used to backup this 101 TB SAP database. At the time the storage agents that perform the backup consisted of 5 LPARS, 4 of those in a single frame each with their own assigned I/O drawer. The 5th was in a separate 740 frame with its own I/O drawer. The 5th storage agent was completing the backup in a fraction of the time of the other 4 so we concluded we must be overloading the CEC on the 740. We moved one of the four storage agents out of the frame to a secondary frame and the results were awesome. See below:

You'll notice that the backup time didn't change with the update of the tape drives from E06 to E07. Hardware layout matters more than the performance of the tape drives. When a vendor tells you just updating the hardware to newer iterations will increase performance take it will a grain of salt. In our case we did testing of the new tape drives and no performance gains were seen but the go ahead was given to upgrade to the newer hardware and as you'll see we didn't gain anything until we reworked the environment. Our task now is to identify how to increase TSM internal job performance (i.e. migration and storage pool backup) which has not seen significant performance gains from the tape upgrades.

Tivoli Storage Manager v6.2 install and see the Exception Stack Trace for more information... [sorry for the long entry]

Am I too old for that? or this is the real future? Is it realy the way it has to be? I don't think so... :-(

--- CUT ---

Welcome
Tivoli Storage Manager 6.2


Licensed Materials - Property of IBM Corp. (c) IBM Corporation and other(s)
1994, 2010. All rights reserved. US Government Users Restricted Rights --
Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM
Corp.


It is highly recommended that you stop all other programs before continuing
with this installation.


PRESS TO CONTINUE:

===============================================================================


International Program License Agreement

Part 1 - General Terms

BY DOWNLOADING, INSTALLING, COPYING, ACCESSING, CLICKING ON AN
"ACCEPT" BUTTON, OR OTHERWISE USING THE PROGRAM, LICENSEE AGREES TO
THE TERMS OF THIS AGREEMENT. IF YOU ARE ACCEPTING THESE TERMS ON
BEHALF OF LICENSEE, YOU REPRESENT AND WARRANT THAT YOU HAVE FULL
AUTHORITY TO BIND LICENSEE TO THESE TERMS. IF YOU DO NOT AGREE TO
THESE TERMS,

- DO NOT DOWNLOAD, INSTALL, COPY, ACCESS, CLICK ON AN "ACCEPT" BUTTON,
OR USE THE PROGRAM; AND

- PROMPTLY RETURN THE UNUSED MEDIA, DOCUMENTATION, AND PROOF OF
ENTITLEMENT TO THE PARTY FROM WHOM IT WAS OBTAINED FOR A REFUND OF THE
AMOUNT PAID. IF THE PROGRAM WAS DOWNLOADED, DESTROY ALL COPIES OF THE
PROGRAM.


Press Enter to continue viewing the license agreement, or enter "1" to
accept the agreement, "2" to decline it, "3" to print it, "4" to read
non-IBM terms, or "99" to go back to the previous screen.: 1




===============================================================================
Component Selection
-------------------

Select the components to install.


1- Tivoli Storage Manager Server
2- Tivoli Storage Manager License
3- Tivoli Storage Manager Devices
4- Tivoli Storage Manager Storage Agent

ENTER A COMMA-SEPARATED LIST OF NUMBERS REPRESENTING THE DESIRED CHOICES: 1,2



===============================================================================
Deployment Engine Initialization
--------------------------------

Please Wait
...
Step 1 of 15...............................................................
Step 2 of 15.................................................................................
Step 3 of 15...
Step 4 of 15...
Step 5 of 15....
Step 6 of 15.....
Step 7 of 15.................................................................... .....
Step 8 of 15.....................
Step 9 of 15....
Step 10 of 15........
Step 11 of 15...............
Step 12 of 15.............
Step 13 of 15...
Step 14 of 15...
Step 15 of 15............................................................
Completed.


===============================================================================
Pre-Installation Summary
------------------------

Review the Following Summary Before You Continue:

Product Name:
Tivoli Storage Manager

Install Folder:
/opt/tivoli/tsm

Components:
TSM Server,DB2 9.7,TSM Client API 32 bit,TSM Client API 64 bit,TSM License

Disk Space Information (for Installation Target):
Required: 1,815,384,781 bytes
Available: 478,806,016 bytes

PRESS TO CONTINUE:


===============================================================================
Not Enough Disk Space
---------------------


Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:
Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:
Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:
Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:
Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:
Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:
Warning!

This installation requires 1,731.29 MB of free disk space, but there are only
456.62 MB available at:

/opt/tivoli/tsm

Please free at least 1,274.66 MB to proceed with the installation.

PRESS TO RECALCULATE AVAILABLE DISK SPACE,
OR TYPE 'QUIT' TO EXIT THE INSTALLER:

There is now enough disk space to proceed with the installation.

PRESS TO BEGIN INSTALLATION:



===============================================================================
Installing...
-------------

[==================|==================|==================|==================]
[------------------|------------------|------------------|------------------]



===============================================================================
Installation Complete
---------------------

The product did not install.
See /var/tivoli/tsm/log.txt for details.


PRESS TO EXIT THE INSTALLER:

--- CUT ---

total 800
drwxrwxr-x 2 root system 256 May 14 14:09 .
drwxrwxr-x 3 root system 256 May 14 12:34 ..
-rw-rw-r-- 1 root system 320432 May 14 14:09 log.txt
-rw-rw-r-- 1 root system 84750 May 14 14:01 logs.zip
-rw-rw-r-- 1 root system 0 May 14 13:55 rxa_message.log
-rw-rw-r-- 1 root system 0 May 14 13:55 rxa_trace.log

and the of that SMALL log:

.
.
.
Fri May 14 14:01:50.310 CEST 2010 : FINE : PlanStepEventOccurred(Failed target 'step_00001_TSM_Server_1_1') (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred)
Fri May 14 14:01:50.311 CEST 2010 : FINE : COIEventType.ERROR (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred)
Fri May 14 14:01:50.313 CEST 2010 : FINE : MachinePlanEventOccurred(An exception was received during the processing of MachinePlan: localhost. See the Exception Stack Trace for more information.) (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred)
Fri May 14 14:01:50.313 CEST 2010 : FINE : COIEventType.ERROR. Last step started was: TSM_Server_1_1 (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred)
Fri May 14 14:01:50.314 CEST 2010 : FINE : DeploymentPlanEventOccurred(Failed Deployment Plan 'install') (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred)
Fri May 14 14:01:50.315 CEST 2010 : FINE : COIEventType.ERROR (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred)
Fri May 14 14:01:50.926 CEST 2010 : FINE : Exception via callback from DeploymentPlanProcessor.process(INSTALL): (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install)
Fri May 14 14:01:50.927 CEST 2010 : FINE : COIEventType.ERROR reported (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install)
Fri May 14 14:01:50.928 CEST 2010 : STDERR : com.ibm.ac.coi.ext.ia.COIWrapperException: COIEventType.ERROR reported
Fri May 14 14:01:50.929 CEST 2010 : STDERR : at com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl$SimpleListener.eventOccurred(COIWrapperPluginImpl.java:121)
Fri May 14 14:01:50.930 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListener(COIRuntimeImpl.java:226)
Fri May 14 14:01:50.931 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListeners(COIRuntimeImpl.java:215)
Fri May 14 14:01:50.932 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.utils.ParallelPlanListener.eventOccurred(ParallelPlanListener.java:105)
Fri May 14 14:01:50.934 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListener(COIRuntimeImpl.java:229)
Fri May 14 14:01:50.935 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.COIRuntimeImpl.notifyListeners(COIRuntimeImpl.java:219)
Fri May 14 14:01:50.936 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.MachinePlanImpl.process(MachinePlanImpl.java:254)
Fri May 14 14:01:50.937 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.utils.MachinePlanRunner.run(MachinePlanRunner.java:64)
Fri May 14 14:01:50.938 CEST 2010 : STDERR : at java.lang.Thread.run(Thread.java:736)
Fri May 14 14:01:50.939 CEST 2010 : STDERR : Caused by:
Fri May 14 14:01:50.940 CEST 2010 : STDERR : com.ibm.ac.coi.api.exception.DeploymentPlanExecutionException: One or more Machine Plans within the DeploymentPlan failed. Failure count = 1
Fri May 14 14:01:50.941 CEST 2010 : STDERR : at com.ibm.ac.coi.impl.utils.ParallelPlanListener.eventOccurred(ParallelPlanListener.java:92)
Fri May 14 14:01:50.942 CEST 2010 : STDERR : ... 5 more
Fri May 14 14:01:50.943 CEST 2010 : SEVERE : Raw Exception in DeploymentPlanProcessor.process(INSTALL): COIEventType.ERROR reported (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install)
Fri May 14 14:01:50.944 CEST 2010 : FINER : THROW (from com.ibm.ac.coi.ext.ia.COIWrapperPluginImpl.install)
.
.
.


and bla, bla, bla, bla...

Sorry, but nice week-end!!! ;-)

AIX Image Backup Performance Issue

I am running 3 simultaneous image backup jobs on an AIX server and the HBA only shows 20% utilized and my throughput sucks. I am using -imagetype=static and the FS's in questions are EMC clones split off and remounted to a new server. When I have done image backups before they were wicked fast...now not so much. Any ideas what I might have missed? I have an ACSLS STK 8500 with LTO-3 drives. Even over gig-ether I've seen better performance. We did check the CPU and memory usage and they are fine. Lots of I/O wait, not sure why I would see that with statc image backups.

ANR0380W The database buffer pool recovered from an overcommit of 867 pages. Consider increasing your buffer pool size by 3468 kilobytes

This error occurs when the BUFPOOL is incorrectly sized or SELFTUNE is enabled and the server has been recently restarted - Check the following:

• Is SELFTUNEBUFPOOLSIZE YES in use in the dsmserv.opt?
  
  tsm:> query opt
  
• What is the default configured BUFPOOLSIZE?
  
  tsm:> query opt
  
• What is the the current BUFPOOLSIZE and is SELFTUNE enabled?
  
  tsm:> query db f=d
  
  Multiply the buffer pool pages by 4096 and then divide by 1024.
  

On AIX if BUFPOOL is currently set to less than 10% increase the value by editing the dsmserv.opt, otherwise increase BUFPOOL by the amount specificied (3468K) and then restart the server.

TSM AIX Performance Issue With ML05

I found this APAR interesting and thought I would pass it along. There seems to be a bug in the TSM client for AIX when updated to ML05 and Direct I/O is in use on the filesystems. I can't explain the details well enough so read the post here. This bug can cause significant increases in backup time. They give a good example in the APAR description.

Have You Checked Your Drive Firmware Lately?

Well the weekend is over and I thought I would update you on the problem SAP system.  In the last update we had discussed how this one TSM server was consistently having its mounts go into a RESERVED and hanging. The only thing we could do was cycle the TSM server. So after talking to support they stated it was a known problem listed in APAR IC49066 and fixed in the TSM server 5.3.3.2 release. So we upgraded the TSM server, updated ATAPE on the problem server to the same level as on the controller, and turned on SANDISCOVERY. The system came back up and started to mount tapes, albeit slowly. So we let it run and all seemed well for about the first 8 hours then all hell broke loose. TSM started taking longer and longer to mount tapes until it couldn’t mount anything. We began receiving the following errors also:

7/13/2006 9:38:45 AM ANR8779E Unable to open drive /dev/rmt25, error number=16.
7/13/2006 9:38:45 AM ANR8311E An I/O error occurred while accessing drive DR25 (/dev/rmt25) for SETMODE operation, errno = 9.

We reopened the problem ticket and talked to a number of Tivoli support reps and almost had to force them to have us run a trace. The original problem fix began Friday and here we were still trying to fix it well into early Sunday morning. So after getting the trace to Tivoli they looked it over and seemed perplexed at the errors. The errors seemed as if Tivoli was trying to mount tapes for drives that the library client was not pathed for. Also it seemed TSM was polling the library for drives but was unable to get a response so it went into a polling loop until it found a drive. This caused our mount queue to get as high as 100+ tape mounts waiting and the mounts that did complete sometimes took 30-45 minutes to do so. It was NUTS! So Sunday morning a new Tivoli Rep was assigned (John Wang). As we discussed the problem and Tivoli was trying to get a developer to analyze the trace John mentioned how he had seen this type of error before in a call about an LTO-1 library. He stated that it took 3 weeks to determine the problem but that the end result was that the firmware on the drives was down-level and causing the mount issues. So I checked my 3584’s web interface (I feel bad for all those people out there without web interfaces on their libraries) and found the drive firmware level at 57F7. This seemed down-level from what little information we had so I had my oncall person call 1-800-IBM-SERV and place a SEV-1 service call. The CE called me and we discussed the firmware level. When he saw how down-level it was he was surprised and lectured me on making sure we always check with CE’s before we do any changes to the environment. The CE then gathered his needed software and came to the account to update the drives. I brought all the TSM servers down and after 30 minutes the library had dismounted all tapes from the drives. The CE then proceeded to update the firmware, which actually only took 15-20 minutes. I expected longer. So we went from firmware level 57F7 to 64D0. Huge jump! So after the firmware upgrade I audited the library and brought the controller back up. Viola! It started mounting tapes as soon as the library initialized and the response was back to what it should have been. It’s now Tuesday morning and there have been no problems. So before you upgrade TSM be sure to have checked your libraries firmware (both library and drives). It could mean the difference between sink and swim!

Help! Problem With Shared Library Client!

Ok, here is my problem.  I have a large shared library that supports 5 TSM servers. All of the TSM servers, save one, run without any problems. The one with problems is pathed to 32 of the 5 drives (they are designated just for its use) and periodically its mount requests go into a RESERVED state but never mount.  I have tried FORCESYNC'ing both sides to see if it was communication but that didn't work.  I have tried setting paths offline then back online and that didn't work either. The only thing that works is to cycle the TSM server then it comes back up and starts mounting tapes like nothing was wrong.  This happens on average every 24-48 hours.  This is a large DB client TSM instance so the reboots are hurting critical business apps backups. I don't see any glaring errors in the activity log or in the AIX errpt. Any suggestions would be gladly received.

Oracle TDP and HACMP 5.3 Incompatibility Resolved

Well I was have a heck of a time with this problem. It seems the 5.3 HACMP binaries are incompatible with the Oracle 64-bit API so the TDP would crash when attempting to backup a DB on a cluster. I was expecting a fix and was told it would be available by January. So I waited, and watched and never saw an update to the TDP. Frustration mounted until I was able to get ahold of a TDP developer who pointed out that it was the HACMP side that had to do the patching and now all is well. So if you are looking to use HACMP and backup Oracle DB's with the TDP make sure they have the latest patch levels installed or the 64-bit API will not work.

TSM Library Sharing

TSM is truly an Enterprise Backup tool and one of the key selling points is how well it allows you to consolidate and centralize the backup environment. One feature I have used extensively is the library sharing option. This feature allows multiple TSM servers to use the same library without conflict. It can be easier than partitioning the library, allows for dynamic allocation of tape drives, and allows for a single shared scratch pool. The key to using a shared library is creating a separate TSM instance as a dedicated library manager (whether on the same server as another TSM instance or on its own hardware). The library manager instance will not perform any backups and will not be a big instance. From my experience it will have very little impact on the physical server. When creating our library manager instance I was unsure how large to make the DB so I made it 2GB, and it currently is 4.6% utilized. The library it manages is a 9 frame 42 drive 3584. The library supports 3 TSM backup instances (2 on one server and 1 on a separate physical server) with a total of 951 clients. We put the library manager instance on the server with 2 instances, thereby creating a third TSM instance. By creating the third instance dedicated to managing the library you will have increased uptime and when you need to do maintenance or restart the instance the library manager is back up very quickly. This will not be the case if you make a production backup TSM instance the library manager due to a production TSM servers need to run expiration and usually having a larger log size making restart time take that much longer.

The other nice feature of library sharing is the ability to dynamically allocate drives to specific TSM servers. I have seen some administrators take twenty drives and path ten to one TSM instance and ten to another but this defeats the purpose. If you setup the library manager and TSM backup instances correctly then you’ll allow every instance connected to the library manager to see all the drives. This coupled with the device class mount limit setting can allow you to change drive usage on the fly. So say you have two TSM servers TSM-A and TSM-B using a twenty drive 3584 managed by a library manager instance called TSM-C. Each TSM instance sees all twenty drives but you have their device class mount limits set to ten. Now each TSM instance can only use ten drives out of the twenty. Lets say on a specific day TSM-A server is in need of more drives and TSM-B is currently only using three drives. You could update the TSM device class on each to allow TSM-A fifteen drives and TSM-B five drives.

Recently we purchased some newer hardware to replace our 9 frame LTO-1 library and are looking at using two small AIX servers using HACMP as the library controller since the new library will need to be available at all times and will be supporting at least six TSM servers. Unfortunately I have had to partition this new library due to application demands. The problem we are experiencing with this new library is that the logical portioning requires us to use the Virtual I/O option in the 3584. This option allows the library to automatically insert volumes when placed in the I/O door. This would not be a problem if the library did not then require us to assign the volumes to a specific logical library. This is done through the library and not through TSM, which adds another process to our tape management. I would have preferred to not have partitioned the library and allowed a TSM library manager instance to handle allocation of tape drives, but alas I am not able to at this time (still awaiting the HACMP instances).

Got HACMP?

Well our group got stung again this last week. While working a HACMP implementation we were asked to install the Oracle TDP and TSM client. We installed the TSM client and all went well and we then installed the TDP and the DBA’s could not get the password file to generate. Each time they tried the utility would core dump. When trying to run the utility multiple ways it always core dumped. After calling support we were informed that the current 64-bit Oracle TDP (And from what I have found through Google it could be any 64-bit TDP). It seems they rebuilt HACMP version 5.3 and the TDP is not compatible with the new way it was compiled. There is a patch projected to be released as of end of the month, but we cannot wait that long so our options are to use temporary disk and mirror the DB, then break and backup the mirrored copy when we want to backup, or we can roll back to HACMP 5.2. The problem with the latter is that we basically have to rebuild the boxes. So be warned there are issues with the latest and greatest version of HACMP at this time.