WebCT

Let’s nevermind why I am working on this in the first place. Namely…

1. the Blackboard Learning Environment Connector introduced using the hostname and port for applet URLs in Vista 8 Blackboard,
2. Blackboard dropped WebCT’s support for using a different port for an application when behind a load balancer.

It has been a hectic week. A recap…

Java certificate fix - Yesterday, August 23rd, the certificate distributed in various Java applets expired. The community discovered the issue and informed Blackboard who put out a fix for the more current products on August 15th. Many customers are leery of having such little lead time to test, verify, and install a fix. Well, Vista 3.0.7.17 was also reported to have the problem, but Blackboard didn’t provide a fix until the 20th after I got my TSM to verify it really still is a problem on the 18th. (The corrected 3.0.7.17.8 version was provided August 21st. Why is in the next paragraph.)

The fix for Vista 3 required us to be on 3.0.7.17.8 (hotfix 8 which we had not yet applied), had references to the “webctapp” directory (in Vista 3 it is applications), and distributed a webct.sh script to add updateWar which didn’t work with Vista 3. FAIL. Thankfully we have modified War files in the past, so adding the updates was more work and accomplished before Blackboard provided a corrected version.

To see the Java certificates in Windows: Control Panel > Java > Security > Certificates. The Blackboard ones are verified by Thawte (the Certificate Authority). The old one is issued to Blackboard. The new one is issued to dc.blackboard.com.

Vista 3.0.7.17.8 - This hotfix was released a couple weeks ago. However, since the priority has been the migration to Vista 8, this was on hold. The previous problem made us step up and throw this into production. The testers went to heroic efforts to get this and the certifcate fix tested. Testing was mixed.

1. Losing session cookie because of Office 2007 in Internet Explorer. Happened less often post fix, but still happens in some cases.
2. Autosignon MAC2. Mode to allow insecure MAC works to give the one school using it time to correct update their portal to use MAC2. Originally the plan was to let them work out MAC2 in test.

Slammed by our users…

1. systemIntegrationApi.dowebct - The school using the autosignon wanted to have the correct consortiaId to create the MAC. Some time back in January they started calling this any time users tried to login because a handful (guess was ~12) have had their username changed. So the autosignon failed. Yes, they were sent us 25,000 requests in a busy day (about 20% of the queues were working on these during the day) to handle potential 12 problems in a term. FAIL.
2. pmSelfRegister.dowebt - One of the clusters started to have issues. Two nodes went crappy. I looked at the Weblogic console and found all of the failing nodes had no free spots in the queues. 90% of the queues were working on these. Much of this is because the requests were hanging around for at least 4800 seconds (an hour is 3600 seconds). At about 6000 seconds the cluster recovered when the queues cleared.I think the queues cleared because I changed to false a couple settings:
   • Allow users to register themselves as a Student in a section = false
   • Allow users to register themselves as an Auditor in a section = false

   As I recall, we only had about 22 queue spots open (out of 308) across the whole cluster. We got lucky.

John made a good point… While telling Blackboard about this is pointless, the community at large ought to be aware of another undocumented workspace issue. I found an 8GB .bak in the /u01/app/nodeA/weblogic81/webctbackup on the active JMS node. Taking out user accessible nodes is okay in my book as with 18-20 of them in our clusters, we can lose one and no client would ever know. Mail, chat, learning context administration and other services in CE/Vista fail without a functional JMS node.

An administrator did a template reassignment with “Force archive before template reassignment” set to true. For some reason the file was placed on the JMS node. It should have been deleted. However, it was not. I caught it in time as another large file was dropped within 10 minutes of me deleting the first. I only caught it time because I was at my desk working (not in meetings, at home, or asleep).

This came within one GB of completely filling up the file system. We do not have huge hard drives on these nodes, just 3 times the size we need except for this. Nor do we allow the nodes accrue a ton of logs or junk.

Maybe this is something Blackboard has resolved this for future versions like Vista 4 or 8. Maybe one day we will have official or unofficial documentation about this kind of stuff.

The answers I anticipate from Blackboard:

1. This is functioning as designed. I bet composing the archive requires something from the JMS node, so it must reside there. The JVM is too small as is /var/tmp, so the file system is the best place.
2. Use a bigger hard drive.
3. Set “Force archive before template reassignment” to false.

Even if Blackboard agrees this is bad, then it might get fixed on Vista 8. Certainly it will not get fixed in the officially supported  Vista 3.

If you want to confirm if you have the potential for this problem, then you should have a $NODENAME/weblogic81/webctbackup or a $NODENAME/weblogic92/webctbackup directory. We only have them on all four JMS nodes, but have have seen them on four (out of 76) other nodes. The other 72 nodes lack this directory. While you are at it, make sure you know about the other undocumented work spaces I have mentioned.

So, I got this error…

Unknown WebCT username or invalid password. Please verify the admin user and password in the IMS Settings.

I assumed the username and password were probably right. I had to find my error somewhere else.

The error turned out to be one missing character out of the 57 character long glcid. Totally my fault.

I wonder how long I would have spent dorking around with the password trying to get it work and thinking I must be typing something wrong.

On the WebCT Users email list (hosted by Blackboard) there is a discussion about a mysterious directory called unmarshall which suddenly appeared. We found it under similar circumstances as others by investigating why a node consumed so much disk space. Failed command-line restores end up in this unmarshall directory.

Unmarshalling in Java jargon means:

converting the byte-stream back to its original data or object ¹

This suspiciously sounds like what a decryption process would use to convert a .bak file into a .zip so something can open the file.

This is fourth undocumented work space where failed files site for a while and cause problems and no forewarning from the vendor.

Previous ones are:

1. Failed UI backups end up in the weblogic81 (Vista 3, does this still happen in Vista 8?) directory.
2. Failed tracking data files end up in WEBCTDOMAIN/tracking (Vista 3, apparently no longer stored this way in Vista 4/8 according to CSU-Chico and Notre Dame)
3. Web Services content ends up in /var/tmp/ and are named Axis####axis. These are caused by a bug in DIME (like MIME) for Apache Axis. No one is complaining about the content failing to arrive, so we presume the files just end up on the system.

#3 were the hardest to diagnose because of a lack of an ability to tie the data back to user activity.

Is this all there are? I need to do testing to see which of these I can cross off my list goring forward in Vista 8. Failed restores are on it indefinitely for now.

References:

1. http://www.jguru.com/faq/view.jsp?EID=560072

This is the second time I have worked on making Vista integration work with Banner. The first was 2005 in Vista 3.0.3 at Valdosta State. The production here at GeorgiaVIEW was set up by Harold, Jill, and Amy years ago and integrated into the install scripts or part of the cloned databases.

So now I am working on getting it to work in Vista 8. The IMS imports worked the first time like a charm. When I turned to using the Luminis adapter, the person records worked fine but the group contexts failed in Vista 8 and worked fine in Vista 3. So the “siapi.sh luminis import restrict” works fine.

Command-line

We have 41 institutions in Vista 3 currently. So imports are automated to some degree to preserve the sanity of Jill (and to a lesser degree Amy and myself). Rather than put in the UI all the settings, we have a properties file defining the location, glcid, sourcedid.source and sourcedid.id for each institution. This allows us to easily pass the values when importing at the command-line.

My first approach was to leave the settings identical to what I used to create persons and group records with IMS. This essentially uses the glcid of the institution and sourcedid of the institution. This is what resulted in the person records working and groups not. Fail.

I realized my error in logic must be the lack of a division-to-group relationship as the error described the groups cannot be related to an institution. So I changed the properties to use the division values for the sourcedid. Fail.

So I went looking in “Guide to Integration with the SunGard Luminis Data Integration Suite” for what I ought to use at the command-line. I didn’t find a solution. Just the same command-line lacking even the glcid and sourcedid.

XML

Giving up on the command-line approach for now, I added the relationship element to the XML so the group would become a child of one of the divisions I created with IMS. It sorta worked! The groups all imported but the course failed with the exact same error the groups formerly succeeded. To add insult to injury, simply running the import again on the exact same file had the courses import.

Mistakes

A mistake I made was reading the documentation: “Guide to Integration with the SunGard Luminis Data Integration Suite”.

Sungard Libraries:

1. Page 8 says imq.jar and mbclient.jar do not come with CE/Vista and must be obtained from Sungard. All three of us thought in Vista 3.x these were automatically placed so we didn’t need to place them. Best I can tell, these were installed by Vista. I found $WEBCTDOMAIN/customconfig/startup.properties references both files in CUSTOM_CLASSPATH and setEnv.sh references CUSTOM_CLASSPATH. (This document has notes for what CE customers need to do and no note about CE users needing to go get them from Sungard.)
2. Those who believe the last note would keep reading and find on Page 9 instructions to deposit the files in $WEBCTDOMAIN/serverlibs/. Assuming I am wrong about item #1, the startup.properties expects them in $WEBCTDOMAIN/serverlibs/luminis/ and would not find them where the document says to put them.

I laughed at reading this one.

Dear Blackboard: If you include icons in your interface, they should f’ing well be clickable. Everyone but you knows this. jazzmodeus

I thought this might refer to the new item icons. Jason works for Emory (doing instructional design) and taking classes at Florida State. Both use Academic Suite. So its probably not what I thought….

In Blackboard CE/Vista, the “course list” [1] can show icons to alert about new things to do. These can be about waiting assessments, discussion, mail, etc. If users click on the icon, then they can see the items causing the notice. At least, when left at the defaults.

One of the schools we host discovered when students entered a tool by clicking on these icon, the subsequent activity would not be tracked. The work around was to turn off the link rather than the icons entirely.

We agreed with the school and labored to convince Blackboard this was a major security problem. Unfortunately, the people who post the support bulletins have yet to post something about this problem. Its not a major item unless you are the student being accused of cheating because your activity doesn’t show appropriately.

[1] course list - This name bugs me….

• The name is a hold over from when instruction took place in courses. In this system they take place in sections. So why not section list?
• MyWebCT is dumb. MyBlackboard is dumber. “My” is 2004-ish portal cutesy, personalization name buzzword. Similarly, “e” and “i” are similarly dumb.

Summize provides a great way to troll for what people are saying. Beyond just searching for a term, it provides RSS feeds for terms. I follow several, such Blackboard and WebCT. The WebCT one netted me the following tweet:

annoyed with how clunky webct can be at times - it had to have been designed circa 2000 - amandakern

WebCT products, whether CE or Vista, have always been clunky. Ease of use has always been a problem with the products. Any improvements Vista made were offset by so many more tools and options to make it the net effect more clunky. I’ve seen some sales people and Dr. Cs whip through the navigation like it is easy to use Vista. Practice makes perfect. Too bad the developers can’t be perfect.

Whenever I see schools pick a product, I think the ones who have Ease of use on their list probably have been using WebCT legacy products for years as opposed to Blackboard products. They and their faculty are scarred enough they cannot afford to get it wrong on ease of use again.

A student wants Blackboard Vista to not reveal his or her last name. The student has already gone to the Registrar and gotten a confidentiality flag placed on the record. As I understand it, this flag in Banner is a FERPA protection to prevent the record from being provided to parties external to the university. It does not provide anonymity within the university. That electronic systems are being scrubbed of the student’s last name means something more than just confidentiality.

We only create new and not update from our student information system (SIS). So in general, the last name should not revert.

The instructor must know who the student is in order to correctly assign grades. If grades were automatically sent back to the SIS, then it would match the IMS id to the what is in the SIS. The user name or any other name is immaterial and not a confounder to the process. Unfortunately, our faculty has to manually transfer the grades. Some rely on the WebCT id / username. Others rely on the first and last name. I guess without names, this latter group is going to have to deal with relying on the WebCT id.

Only username, first and last name, and role are populated into the grade book. So moving the last name to another name field (like other, prefix, or suffix) would not help.

The last name appears to be part of their scheme for creating usernames, so they will likely need to change the username if the point is to not let anyone know what it is. The school in question does not appear to populate their Vista user records with a school email address. So I don’t know if the same would need to be done with it as well.

Blackboard Vista 3.0.7 does have issues with renaming the last name. While many things are immediately updated (good), some things are not. This is not a comprehensive list.

1. The last name in the grade book was not updated. Removing the user from the section and restoring it to the section changed the name to the correct one.
2. The last name in discussions was not updated.

So while renaming the account is easy to do, not everything takes place as quicklly as we would like.

Be more secure! Upgrade today.

Want better functionality? Upgrade today.

Save a developer! Upgrade today.

The save a developer thing is the impetus for this post.

The upgrade today mantra annoys me.

1. Software rarely spends enough time in alpha and beta cycles to to identify all the issues.
2. People have been so burned by using software in alpha and beta cycles, they are hesitant to try upgrades and help determine the issues.
3. This lack of attention to the problems ensure, versions 1.0, 2.0, n.0 typically have a ton of unknown problems or are even less secure at times.

Unfortunately, the vendor who makes the application platform we run, Blackboard, has a philosophy to look at new web browsers while they are in beta but not actually work towards fixes for the new browsers until after the products are released. With most releases of Java or supported web browsers (Internet Explorer or Mozilla Firefox), Blackboard heard the complaints by the early adopters and released within a couple months an update which resolved the reported issues.

The students and faculty members fail to understand the issue. I think I do. Blackboard (like WebCT prior) understands there are differences between beta and final. Some of us argue these differences are usually minor. However, this is all asking someone to predict the future which we know is haphazard at best.

Long alpha and beta cycles allow more users to get involved, give those back to the developers, have them fixed before the version release. Burning users with buggy software ensures their lack of faith.