In case you were thinking of getting me a “Just because it is August” gift…
John made a good point… While telling Blackboard about this is pointless, the community at large ought to be aware of another undocumented workspace issue. I found an 8GB .bak in the /u01/app/nodeA/weblogic81/webctbackup on the active JMS node. Taking out user accessible nodes is okay in my book as with 18-20 of them in our clusters, we can lose one and no client would ever know. Mail, chat, learning context administration and other services in CE/Vista fail without a functional JMS node.
An administrator did a template reassignment with “Force archive before template reassignment” set to true. For some reason the file was placed on the JMS node. It should have been deleted. However, it was not. I caught it in time as another large file was dropped within 10 minutes of me deleting the first. I only caught it time because I was at my desk working (not in meetings, at home, or asleep).
This came within one GB of completely filling up the file system. We do not have huge hard drives on these nodes, just 3 times the size we need except for this. Nor do we allow the nodes accrue a ton of logs or junk.
Maybe this is something Blackboard has resolved this for future versions like Vista 4 or 8. Maybe one day we will have official or unofficial documentation about this kind of stuff.
The answers I anticipate from Blackboard:
- This is functioning as designed. I bet composing the archive requires something from the JMS node, so it must reside there. The JVM is too small as is /var/tmp, so the file system is the best place.
- Use a bigger hard drive.
- Set “Force archive before template reassignment” to false.
Even if Blackboard agrees this is bad, then it might get fixed on Vista 8. Certainly it will not get fixed in the officially supported Vista 3.
If you want to confirm if you have the potential for this problem, then you should have a $NODENAME/weblogic81/webctbackup or a $NODENAME/weblogic92/webctbackup directory. We only have them on all four JMS nodes, but have have seen them on four (out of 76) other nodes. The other 72 nodes lack this directory. While you are at it, make sure you know about the other undocumented work spaces I have mentioned.
So I wanted to open a support ticket. However, in thinking about what I can ask for the company to do arrayed against what they are willing to offer for support, I realized… I am not going to get a resoultion for the ticket.
- It is functioning as designed.
- They are just going to tell us the workarounds we have already implemented.
So, what is the point? Other than distracting employees of the company with something they are never going to solve, I get no benefit. I just get to be the passive-aggressive, CYAer, paper pusher who gets to point at the fact I opened a pointless support ticket to justify my employment.
Yes, the problem could trigger a cascade of events which would result in the failure of services for about 3,000 active users. We stood at the brink twice yesterday and the day before. Because we DBAs are responsive, we saved it. The next time we will do the same.
The company is not going to release another patch for the product unless forced to do so (aka glaring security hole). So even if we could convince them of a bug, then no resolution would be provided in this version. I’ll have to replicate to see if the same problem exists in a newer version they do adequately support. If so, then I would have justification in opening a ticket.
Now… how to I identify an 8GB section archive…