Recently we ran into a situation where our Spectrum Protect 8.1.4 server started to crash. We reviewed the db2diag.log file and identified the error as
ANR0131E Server DB space exhausted. Upon review of our DB file systems the OS showed 7GB available in the 5 file systems assigned for DB space.
/dev/tsm20log 111616.00 9784.93
92% 205 1%
/tsmserver/tsm20log
/dev/tsm20arch 102400.00 66821.52
35% 100 1%
/tsmserver/tsm20arch
/dev/tsm20fail 101376.00 100964.16
1% 21 1%
/tsmserver/tsm20fail
/dev/tsm20dblv01 101376.00 7622.45
93% 104 1%
/tsmserver/tsm20dblv01
/dev/tsm20dblv02 101376.00 7622.48
93% 104 1%
/tsmserver/tsm20dblv02
/dev/tsm20dblv03 101376.00 7622.45
93% 104 1%
/tsmserver/tsm20dblv03
/dev/tsm20dblv04 101376.00 7622.46
93% 105 1%
/tsmserver/tsm20dblv04
/dev/tsm20dblv05 101376.00 7622.45
93% 105 1%
/tsmserver/tsm20dblv05
The server had been crashing from the previous night and we had at first thought it was due to the active log filling and not writing out to the archive log fast enough. What was weird is we could bring TSM up and it would run for an hour or two then crash again. When we identified it was not the active log but the DB space (earlier the next morning) we were confused. Why would we get a space notification when the DB has 35GB of space still available. My co-worker created an additional file system of the same size as the other DB file systems and ran the
dsmserv extend dbspace command and the file systems almost immediately jumped to 99% utilized. So it appears Spectrum Protect needed to write 35+GB of DB data and could not due to space available? We also noticed that our offshore people had added tsm20dblv05 a few months back but had used a different disk array (slow SATA) and had not informed us, which we think might have contributed to the performance issues we had been seeing on this server recently and maybe was related to the DB space issue??? Not sure, but be aware that this can occur. I will need to research how DB2 allocates space and in what chunk size it does so.