Interactive Archives

My jaw dropped at the end of this blog post Cloud Hosting and Academic Research.

There is a value in keeping significant old systems around, even if they no longer have active user bases.  A cloud hosting model seems so right to me–it’s scalable and robust. It just makes sense. But the hosting costs are a problem. Even if the total amount of money is small, grants are for specific work and have end dates. I can still be running a 10+ year old UNIX box, but I can’t still be paying hosting fees for a research project whose funding ended years ago, no matter how small that bill is.  Grants end–there’s no provision for “long term hosting.”  Our library can help us archive data, but they are not yet ready to “archive” an interactive system.  I hope companies that provide hosting services will consider donating long-term hosting for research.

Opening up a new area of digital archives by preserving the really cool works of the faculty seems like something I might enjoy.

My mentor in web design and server administration might have been described as a pack rat. He… Well, I guess, we kept around versions of web pages a decade old. Nothing really found deletion. The public just missed it by use of permissions.

When building my portfolio, my mistake was not gathering up the whole files to replicate the sites I designed. I’m no longer doing web design or even programming. So it is okay.

A professor in Geology had a pretty cool Virtual Museum for Fossils. The site moved around a few times, eventually ending up on the main web server also hosting WWW. Of course, HTML, images, and Flash files are easy to archive. Take the files and place them on a web server. Since they are static, it is easy to keep around for a long time. As long as the standards remain honored, they should be good. Developers of web browsers have pressure to go for the new, which potentially abandons the old eventually.

Scripted web sites using Perl, PHP, ASP, or JSP, JavaScript, or AJAX require a working interpreter. Still, some things might not be backwards compatible.

About a year ago my mother ran across 8mm video film. An uncle found a place who converted it to DVD. Will we even be using DVDs in a decade? Maybe the 8mm needs to go on Blueray?

Going back to the scripted web sites, should an archived web site’s code be updated to work on the new version of the interpreter? Maybe. If makers of the interpreters allowed for running in a backwards compatible mode, then all would be good. Even better, to be able to add to a script a variable that tells the interpreter which back version to pretend to use. For administrators, they could have the programmers check non-working scripts by just telling the interpreter to simulate an older version.

links for 2007-09-27

BbWorld Presentation Redux Part II – Monitoring

Much of what I might write in these posts about Vista is knowledge accumulated from the efforts of my coworkers.

This is part two in a series of blog posts on our presentation at BbWorld ’07, on the behalf of the Georgia VIEW project, Maintaining Large Vista Installations (2MB PPT).

Part one covered automation of Blackboard Vista 3 tasks. Next, let’s look at monitoring.

Several scripts we have written are in place to collect data. One of the special scripts connects to Weblogic on each node to capture data from several MBeans. Other scripts watch for problems with hardware, the operating system, database, and even login to Vista. Each server (node or database) has, I think, 30-40 monitors. A portion of items we monitor is in the presentation. Every level of our clusters are watched for issues. The data from these scripts are collected into two applications.

  1. Nagios sends us alerts when values from the monitoring scripts on specific criteria fall outside of our expectations. Green means good; yellow means warning; red means bad. Thankfully none in our group are colorblind. Nagios can also send email and pages for alerts. Finding the sweet spot where we get alerted for a problem but avoid false positives perhaps is the most difficult.
  2. An AJAX application two excellent members of our Systems group created called internallyl Stats creates graphs of the same monitored data. Nagios tells us a node failed a test. Stats tells us when the problem started, how long it lasted, and if others also displayed similar issues.We also can use stats to watch trends. For example, we know two peaks by watching WIO usage rise to a noonish peak slough by ~20% and peak again in the evening fairly consistently over weeks and months.

We also use AWStats to provide web server log summary data. Web server logs show activity of the users: where they go, how much, etc.

In summary, Nagios gives us a heads up there is a problem. Stats allows us to trend performance of nodes and databases. AWStats allows us to trend overall user activity.

Coradiant TrueSight was featured in the vendor area at BbWorld. This product looks promising for determining where users encounter issues. Blackboard is working with them, but I suspect its likely for Vista 4 and CE 6.

We have fantastic data. Unfortunately, interpreting the data proves more complex. Say the load on a server hosting a starts climbing, its the point we get pages and continues to climb. What does one do? Remove it from the cluster? Restart it? Restarting it will simply shift the work to another node in the cluster. Say the same happens with the database. Restarting the database will kick all the users out of Vista. Unfortunately, Blackboard does not provide a playbook on what to do with every support possibility. Also, if you ask three DBAs, then you will likely get three answers.
😀

Its important to balance the underreaction and overreaction. When things go wrong, people want us to fix the problem. Vista is capable of handling many faults and not handling very similar faults. The link example was a failed firewall upgrade. I took a similar tact with another firewall problem earlier this week. I ultimately had to restart the cluster that evening because it didn’t recover.

Part three will discuss the node types.