I noticed one the nodes in a development cluster was down. So I started it again. The second start failed, so I ended up looking at logs to figure out why. The error in the WebCTServer.000000000.log said:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280036]Missing the file store file “WLS_DIAGNOSTICS000001.DAT” in the directory “$VISTAHOME/./servers/$NODENAME/data/store/diagnostics”
So I looked to see if the file was there. It wasn’t.
I tried touching a file at the right location and starting it. Another failed start with a new error:
There was an error while reading from the log file.
So I tried copying to WLS_DIAGNOSTICS000002.DAT to WLS_DIAGNOSTICS000001.DAT and starting again. This got me a successful startup. Examination of the WLS files revealed the the 0 and 1 files have updated time stamps while the 2 file hasn’t changed since the first occurance of the error.
That suggests to me Weblogic is unaware of the 2 file and only aware of the 0 and 1 files. Weird.
At least I tricked the software into running again.
Some interesting discussion about these files.
- Apparently I could have just renamed the files. CONFIRMED
- The files capture JDBC diagnostic data. Maybe I need to look at the JDBC pool settings. DONE (See comment below)
- Apparently these files grow and add a new file when it reaches 2GB. Sounds to me like we should purge these files like we do logs. CONFIRMED
- There was a bug in a similar version causing these to be on by default.
Guess that gives me some work for tomorrow.