Mail Delivery Background Jobs

Only 8 years into running this product and I still learn something new about it.

Monday there was an event. Two nodes became responsive at about the same time. The other ten nodes did their jobs and transferred session information to the nodes taking on the sessions. Most were so busy they did not respond to monitor requests. There was lots of slowness. But we did not lose sessions. Nor did we lose the cluster.

Somehow we did lose the Mail tool. (Think internal email, but it can forward messages to email.)

In WebCT Vista 3 we diagnosed this by going to Weblogic, finding the email queues, and restarting some things to email would start flowing again. I was not able to find it that way. Apparently now, we go to the Background Jobs as a server administrator. The waiting mail jobs show up in Pending Jobs view.

Once I restarted the cluster, the blocking mail job was changed to Retried as soon as the JMS node came online. Retried only shows up in the All Jobs view. All the other views do not show it. Which makes sense because each view shows the status of the view name. So the Cancelled Jobs view only shows jobs with the Cancelled status. Any jobs with a Retried status should only show in the (non-existent) Retried Jobs and (existing) All Jobs views. It was bad assumption on my part that all potential statuses have a view.

Hindsight being 20/20, what we need is a Nagios monitor to detect is Pending jobs exceeds maybe 20-50 jobs. Normally this table appears empty. But I could see cases where it normally grows fast then quickly clears.

But then again, we have less than a year on this product. What are the odds this will happen again?

Verify Node Has War Update

Last week I posted on how to verify the cookie domain on nodes not just looking in the Ear file on the admin node. That same concept has another cool use: Verifying which VSTs the nodes actually have.

Typically one-off updates from Blackboard has us run the updateWebctEar to replace classes in the war file. In a nutshell it…

  1. opens the ear
  2. opens the war
  3. updates the class at the path
  4. regenerates the war
  5. regenerates the ear

Simple enough by hand, but very convenient in a script. I was glad when Bb gave us this functionality.

cd $WL_DOMAIN
./webct.sh updateWebctEar __some_class.class webct.war/WEB-INF/path/to/class

Verifying that this is in the Ear file is rather easy, but it takes a while manually. Also, one of the steps is to open the ear then search the webct.war. Well, the managed nodes cache the Ear so it kind of saves a step to look there. We run 14 clusters and worry about inconsistencies between various development clusters and between development and production. Plus, as this is a sanity check, why not check all of them? (Also, we use dsh, so checking all 140 production nodes with one command and not having to login to each saves the most time.)

The command…

  1. changes directories to a location in the cache area,
  2. confirms the location
  3. uses the Java ARchive (JAR) tool to list (-t) verbosely (-v) the webct.war file (-f)

All put together…

cd$WL_DOMAIN/servers/node_name_regex/tmp/_WL_user/webct/
&& pwd && $JAVA_INSTALL/bin/jar -tvf */webct.war WEB-INF/path/to/class/__some_class.class

The output looks something like below. The time stamp appears to be when the server administrator added the file to the

118864 Fri Jul 23 11:25:42 EDT 2010 WEB-INF/path/to/class/__some_class.class

I’m thinking this could also be useful to scan for what updates the nodes have in the War just after we’ve installed it.

Most Recent Data

One of the common complaints instructors have about CE/Vista is the Tracking reports don’t have recent enough data. They are shown this for selecting the date range.

Select a Date Range for the Report
Select a Date Range for the Report

Including here the most recent time the tracking was processed (which the application already displays to the server administrator in background jobs) would help the instructor know whether the data is as recent as 4:00 am or 1:00pm.

Maybe when Tracking will run again ought to be displayed to the instructor so he or she knows it will run within the hour or the next morning. That might cut down on instructors running it again and again expecting it to magically show data which won’t be available until many hours later.

Administrators some times have to pick the best operational time to run Tracking. We have direct login checks running several times per hour. When Tracking is run every hour and these checks run at the same time, the time these direct login checks took spiked. Users also complained about poor performance. So we have these run in the wee hours of the morning when users are not generally on the system.

Mail From Address

It appears CE/Vista has several locations for defining the email addresses it uses for SMTP.

  1. $WEBCTDOMAIN/config/config.xml:
    mail.from=
    From address for messages sent.
  2. $WEBCTDOMAIN/customconfig/startup.properties:
    WEBCT_ADMIN_EMAIL=
    Some internal errors have a mailto: prompt to contact the server administrator.
  3. $WEBCTDOMAIN/serverconfs/log4j.properties:
    log4j.appender.EMail.To=
    Report fatal errors.
  4. $WEBCTDOMAIN/serverconfs/log4jstartup.properties:
    log4j.appender.EMail.To=
    Report fatal errors.
  5. $WEBCTDOMAIN/webctInstalledServer.properties:
    WEBCT_ADMIN_EMAIL=
    Installer picks up this value for populating #2 and possibly #3 and #4.
  6. $WEBCTDOMAIN/webctInstalledServer.properties:
    MAIL_ORIGIN=
    Installer picks up this value for populating #1.

What really disturbs me is the Vista 8 installer created log4j properties files with the  SMTP server set up for miles.webct.com and sending from vista.monitor@webct.com? I cannot seem to find anything in the Vista 8 documentation or wiki or Google index about the “Vista Trap Notification” subject line, from address, or SMTP address which the log4j appender appears to be designed to send.

This Vista Trap Notification appears designed to send an email to the address any time a fatal error is encountered. That’s fine. Just use the smtp host and From address requested in the installer.

Don’t get me started about giving end users a mailto: prompt to report errors.

IMS Data Going to Wrong Place

I should know better than to trust documentation over my own intuition. Or to change based on what others tell me.

I followed:

Log in to Vista Enterprise as a Server Administrator or Institution Administrator.
NOTE: To set glcid, you must log in as a Server Administrator.

From the Administration tab, click the Utilities tab.
Click Settings.

Under System Integration, click System Integration API IMS….

Enter values to configure settings. See the table that follows, Standard and IMS Adapter
details on each value.

Click Save Values. The Settings screen appears and the settings are configured.

Standard and IMS Adapter Settings
The following table describes the parameters you can set using the administration user interface.
Setting Description
GlcId

• Stands for global learning context identifier.
Set by Server Administrator only.
Required to run IMS and Standard adapter
commands.

• Identifies the institution in which the adapter
command runs
• Automatically assigned by Vista Enterprise
upon creation of an institution

Of course, it doesn’t say which Glcid, right? After all, every learning context has a Glcid. Since, at the time I only had one institution (before I created the 54 others), I set the Glcid to the one for that institution. Should it be the Glcid for the server or domain learning context? If so, then couldn’t Blackboard just pre-populate it at the time of install? Why do I need to put it there?

At the same time, I didn’t believe it necessary because I had seen IMS imports work without the Glcid set at the server learning context. They worked because the command used to run the IMS import has the glcid.

The result? My imports went into the the institution with the Glcid set at the server learning context, despite the defining in the command I ran to use a different Glcid. Removing the Glcid from the server learning context settings allowed the command to work as I thought it should.

So much for a pristine, clean database.

Tale of Defeating the Crazy Woman

Babies are fascinated by me. When the two of us are in a room, they often find me the most interesting thing in the room. Usually, it is mutual.

So, a mutual friend of a friend, Mojan has a fantastic blog. The past year or so has been about being pregnant and most recently figuring out how to be a parent for the first time. Well, a crazy woman set up a ‘blog” which hotlinks images from Mojan’s blog and falsely represents the child in the photos. Ick. I offered to help with this identity theft issue.

Once upon a time, I was annoyed with people taking images from my last employer’s web site. Since I was the campus web designer, I created an image which said, “All your image are belong to VSU.” Also, as the web server administrator, I figured out how to defeat hotlinking with .htaccess by using mod_rewrite to give them my annoyance rather than their content. For the next couple days I watched the perpetrators try and figure out what was wrong. The hate mail I got was fantastic! I recommended Mojan do the same. When she agreed, I went researching to do what I did once upon a time. This is the .htaccess file I recommended she try.

# Basics
Options +FollowSymlinks
RewriteEngine On

# Condition is true for any host other yours
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?mojansami\.com/ [nc]

# What to change gif, jpg, png to which target. In this case does not exist.
RewriteRule .*\.(gif|jpg|png)$ http://mojansami.com/images/stolenpic.jpg [nc]

My directions were not all that specific. So the next thing I know, her site is sporting an Internal Server Error. *headdesk* She used Dreamweaver to create the .htaccess file and upload it to her site. She reported the file she uploaded disappeared. Eventually, it did occur to me to look for the error.log and see what it said. The log complained about DOCTYPE in the .htaccess file in the home directory. A file which did not show in the FTP listing. So, replacing the bad .htaccess file with a blank one fixed the Internal Server Error.

The .htaccess file in the right place, of course, resolved the issue with the crazy woman hotlinking.

Nothing can fix the pain of another person committing identity theft against you or your loved ones. I really hope Mojan doesn’t become discouraged and abandon blogging entirely. Between moderation and authentication she might find a better balance.

Do you have any stories of online identity theft?

Certification Courses

Today I received an email announcing new certification courses from Blackboard. Every administrator around our system did as well. We have a large body of certified trainers and administrators with multiple individuals at every campus who have been through the certification plus all the functional support have been through both training and admin certification.

I’m not a big fan of certifications as a way of determining what people know. I do think having so many people go through the certification has allowed us to use jargon consistently. Pre-cert, we’d have to waste time explaining what the jargon means. Post-cert, everyone seems to understand these terms. Sure, we are constantly expanding the terms used, but its nice to have a common basis.

Back to the email…. Here are the certifications mentioned in the email:

  • Blackboard Learning System™ GUI Administrator Certification
  • Blackboard Learning System™ Server Administrator Certification (NEW)
  • Campus Edition GUI Administrator Certification
  • Vista GUI Administrator Certification

The points of the message were, I think to announce the new “server administrator” certification and remind people about the existing ones. Initially I was interested because I thought it meant there is a new certification for Vista, whose license name is Blackboard Learning System™ – Vista Enterprise Edition (say that 3x fast), but no… Its just for Academic Suite.

I think Blackboard’s marketing department should do a better job of understanding which products their customers use. Highlighting to over 60 people here in Georgia they are offering cool things to the clients who were not spoils of the WebCT buy isn’t helping Blackboard.

tag: , Blackboard Inc, WebCT