Diff and Spaces

Amy’s dump of the CE/Vista settings table ended up with a slightly different format than mine. I was able to use sed to rejoin the correct lines. This resulted in two files with spaces on about half of the lines in the file. Ouch.

Thankfully diff has a -b flag to ignore the spaces. Really useful in this case.

Now… To figure out why a pattern I copy and paste from a file is not found with grep.
🙂

Tar Owner

Some of my previous blog posts about a little discovery have save me time in the past.

I made a tar of some system integration scripts we wrote for Vista 3 and brought them over to a Vista 8 test box to work on updating them for this new version. A decision we made earlier to change the userid came to light.

Using gtar -zxvf file.tar.gz gave me the wrong owner. Thankfully the people who made tar long ago solved this problem with –owner. Look for it in the man pages. Using gtar -zxvf file.tar.gz --owner newid gave me the files I needed with the right owner.

Also, thankfully, the old owner account was still on the box so I go into it and remove the files before extracting with the new id. Hopefully my Systems people (who read this blog) will not kill that account or our access to it until after this project is done?
🙂

First Edition AWK

When I first started in IT at Valdosta State, my boss had The AWK Programming Language. My initial dabbling with awk was based on the guidance of this book.

  1. I am impressed Amazon has the book in stock.
  2. I am impressed the sale price is over $70 (3 pages / $1 while typical is >10 pages / $1).

Admittedly, the authors are the ones who created awk. It is a first edition.

For you non-geeks: Sorry for taxing your brain cells. awk is perhaps my favorite program to use next to grep.

On the Fourth through Sixth Loops of Ready 2 Wear

I really have to stop listening to the same song played over and over. It may affect my thinking….

We had another node crash due to the Sun JVM issue. Our start script failed to make a file in /var so the node did not become fully operational as expected. While waiting for those with permission to delete some stuff to free up space, I went looking for what I could delete myself. Naturally /var/tmp seemed a likely place. I found 1,171 files named Axis#####axis. (Replace the #s with well… numbers.) They used up only 42MB. Most were small. Looking across all our machines there are thousands of these dating back to February of this year.

I love the Unix file command. It will tell you what kind of files are there. So I used file | sort -k 2 to sort by the type. Almost all of the files were either plain text or JPEG or GIFs. One file, called a “c program file” turned out to be a JavaScript (based on the C syntax). I downloaded a JPEG file locally, renamed it to have the .jpg extension, and opened it in an image viewer. It opened correctly. Seems its a graphic of a table.

It would seem our Blackboard Vista 3 has been collecting these files for months. They do not take up very much space. There are not nearly enough files to represent a download of content by all users. Our /var would fill up hourly in that case.

Axis is an Apache SOAP project. Vista’s exposed APIs use Axis, I believe. So, the running hypothesis is several of our campuses are using a product which is contacting the APIs to upload content. Its spread out enough that all four clusters are affected. Its something that started about February.

Suspect #1 Respondus – Chosen because we know it hits the APIs to upload content. Discounted because the content is lecture materials. Respondus works with assessments (aka quizzes, tests, exams).

Suspect #2 Impatica – Chosen because the JavaScript file references PPT. Impatica compacts PowerPoint (aka PPT) files and allows them to play without needing a PPT player. Their support pages teach users how to use the Campus Edition 4 user interface to upload content into a course. O-kay….

Suspects #n Softchalk, Diploma, Microsoft .Learn, etc. – I haven’t really investigated any of these. They are just names to me at the moment.


UPDATE: So… There is a bug in Axis which dumps these files into the file system. The files can be deleted as long as they are not current.

BbWorld Presentation Redux Part I – Automation

Much of what I might write in these posts about Vista is knowledge accumulated from the efforts of my coworkers.

I’ve decided to do a series of blog posts on our presentation at BbWorld ’07, on the behalf of the Georgia VIEW project, Maintaining Large Vista Installations (2MB PPT). I wrote the bit about tracking files a while back in large part because of the blank looks we got when I mentioned in our presentation at BbWorld these files exist. For many unanticipated reasons, these may not be made part of the tracking data in the database.

Automation in this context essentially is the scheduling of tasks to run without a human needing to intercede. Humans should spend time on analysis not typing commands into a shell.

Rolling Restarts

This is our internal name for restarting a subset (consisting of nodes) of our clusters. The idea is to restart all managed nodes except the JMS node, usually one at a time. Such restarts are conducted for one of two reasons: 1) have the node pick up a setting or 2) have Java discard from memory everything. The latter is why we restart the nodes once weekly.

Like many, I was skeptical of the value of restarting the nodes in the cluster once weekly. Until, as part of the Daylight Savings Time patching, we provided our nodes to our Systems folks (hardware and operating systems) and forgot to re-enable the Rolling Restarts for one batch. Those nodes starting complaining about issues into the second week. Putting back into place the Rolling Restarts eliminated the issues. So… Now I am a believer!

One of my coworkers created a script which 1) detects whether or not Vista is running on the node, 2) only if Vista is running does it shut down the node, 3) once down, it starts up the node, and 4) finally checks that it is running. Its pretty basic.

Log cleanup to preserve space

We operate on a relatively small space budget. Accumulating logs infinitum strikes us as unnecessary. So, we keep a months’ worth of logs for certain ones. Others are rolled by Log4j to keep a certain number. Certain activities can mean only a day’s worth are kept, so we have on occasion increased the number kept for diagnostics. Log4j is so easy and painless.

We use Unix’s find with mtime to look for files 30 days old with specific file names. We delete the ones which match the pattern.

UPDATE 2007-SEP-18: The axis files in /var/tmp will go on this list, but we will delete any more than a day old.

Error reporting application, tracking, vulnerabilities

Any problems we have encountered, we expect to encounter again at some point. We send ourselves reports to stay on top of potentially escalating issues. Specifically, we monitor for the unmarshalled exception for WebLogic, that tracking files failed to upload, and we used to collect instances of a known vulnerability in Vista. Now that its been patched, we are not looking for it anymore.

Thread dumps

Blackboard at some point will ask for thread dumps at the time the error occurred. Replicating a severe issue strikes us as bad for our users. We have the thread dumps running every 5 minutes and can collect them to provide Blackboard on demand. No messing with the users for us.

Sync admin node with backup

We use rsync to keep a spare admin node in sync with the admin node for each production cluster. Should the admin node fail, we have a hot spare.

LDIS batch integration

Because we do not run a single cluster per school and the Luminis Data Integration Suite does not work with multiple schools for Vista 3 (rumor is Utah has it working for Vista 4), we have to import our Banner data in batches. The schools we host send the files, our expert reviews the files and puts them in place. A script finds the files and uploads each in turn. Our expert can sleep at night.

Very soon, we will automate the running of the table analysis.

Anyone have ideas on what we should automate?

Software Sucking So Seeking Something

So many to cover let me number them.

  1. Trillian has been my IM client of choice since their beta. I’ve used Gaim and any number of programs who do multiple IMs from one client but found them lacking. Trillian was even the first software for which I was willing to pay my own money. (Otherwise its been work who has paid the price. 😀 ).Lately, we have been coordinating through AIM chat rooms. Trillian was working fine for a while but in the past couple weeks has demonstrated issues with these chat rooms. It doesn’t send or receive invites for the rooms. I have an account with Meebo, so when Trillian didn’t work and Meebo did, I knew it was me. So, Pidgin (formerly Gaim) is installed to see if it works and rule out a firewall or something similar. Too much work…
  2. Much of my work involves using Unix-style shells to connect to a machine and type things at the command line. This is the way things are done. When I first started this kind of work, I was using a Sun Ultra 5. It worked really well. Trying to work similarly on Windows 95 in TerraTerm was not the same at all. Ugh. Things in Windows have gotten better… Sorta. Its still not the same.I finally have gotten so fed up with the ssh software from ssh.com I’ve started looking at alternatives.

Or… Maybe the stress of work is getting to me? I’m taking it out on my software? Wow… what psychobabble!

UPDATE (3007-AUG-13: Naturally, PuTTY is at the top of my ssh.com replacement list. I’ve used it before. However, over the weekend I used PenguiNet. It did pretty well. I’ll probably be playing with a bunch of different software over the few months.

Also, a reader pointed out that if I really want to get back to the feel of the Ultra 5, I ought to go Linux. Its an excellent point. I’ve been thinking since 1996 it would be good to go play with it, but I never seem to actually do. One day… One day…

links for 2007-07-18

.

Another Nail in the Coffin of UNIX?

If you have an audio copy of “Taps” play it now….

SGI – Services & Support: End of General Availability for MIPS® IRIX® Products:

SGI launched the MIPS IRIX family of products in 1988. Since then, this technology has powered servers, workstations, and visualization systems used extensively in Manufacturing, Media, Science, Government/Defense, and Energy. After nearly two decades of leading the world in innovation and versatility, the MIPS IRIX products will end their general availability on December 29, 2006.

Support will stick around until 2013? Still, another victory for Linux.