Vista

GeorgiaVIEW

1. SMART (Section Migration Archive and Restore Tool) created for us by the Georgia Digital Innovation Group seemed well received. I’m glad. DIG worked tirelessly on it on an absurdly short schedule.
2. Information is strewn about in too many places. There isn’t one place to go for information. Instead between Blackboard, VistaSWAT, and GerogiaVIEW about 29. I amazed I do find information.
3. Blackboard NG 9 is too tempting for some.
4. Vista does DTD valdiation but not very well. We need to XML validation before our XML files are run. As we do not control the source of these files and errors by those creating the files cause problems, we run them in test before running in production. I am thinking of something along the lines of validating the file and finding the errors and reporting to the submitter the problems in the file. Also, it should do XML schema validation so we can ensure the data is as correct as possible before we load it.

I got this error while trying to run an XML IMS import using the WebCT / Blackboard CE/Vista siapi.sh script…

A unit of type Institution cannot have a unit of type Campus as a child.

Guess being on vacation last week spaced my neurons. Normally, I ignore Blackboard errors as meaningless. This time I listened to the error and sent myself off in the wrong direction trying to figure out what was wrong with the typevalue element of the XML was wrong. Typevalue defines the level of the context. So while my level was set to 30 in the XML, for some reason it must be ignoring it, right?

Wrong! The problem was really that the relationship.sourcedid.source was set to “DBA IMS IMPORT USG_COL” instead of “DBA SIAPI Import USG_COL”. So it was unable to find the parent object. So the error makes no sense to me now.

Can I have my two hours back?

Let’s nevermind why I am working on this in the first place. Namely…

1. the Blackboard Learning Environment Connector introduced using the hostname and port for applet URLs in Vista 8 Blackboard,
2. Blackboard dropped WebCT’s support for using a different port for an application when behind a load balancer.

It has been a hectic week. A recap…

Java certificate fix - Yesterday, August 23rd, the certificate distributed in various Java applets expired. The community discovered the issue and informed Blackboard who put out a fix for the more current products on August 15th. Many customers are leery of having such little lead time to test, verify, and install a fix. Well, Vista 3.0.7.17 was also reported to have the problem, but Blackboard didn’t provide a fix until the 20th after I got my TSM to verify it really still is a problem on the 18th. (The corrected 3.0.7.17.8 version was provided August 21st. Why is in the next paragraph.)

The fix for Vista 3 required us to be on 3.0.7.17.8 (hotfix 8 which we had not yet applied), had references to the “webctapp” directory (in Vista 3 it is applications), and distributed a webct.sh script to add updateWar which didn’t work with Vista 3. FAIL. Thankfully we have modified War files in the past, so adding the updates was more work and accomplished before Blackboard provided a corrected version.

To see the Java certificates in Windows: Control Panel > Java > Security > Certificates. The Blackboard ones are verified by Thawte (the Certificate Authority). The old one is issued to Blackboard. The new one is issued to dc.blackboard.com.

Vista 3.0.7.17.8 - This hotfix was released a couple weeks ago. However, since the priority has been the migration to Vista 8, this was on hold. The previous problem made us step up and throw this into production. The testers went to heroic efforts to get this and the certifcate fix tested. Testing was mixed.

1. Losing session cookie because of Office 2007 in Internet Explorer. Happened less often post fix, but still happens in some cases.
2. Autosignon MAC2. Mode to allow insecure MAC works to give the one school using it time to correct update their portal to use MAC2. Originally the plan was to let them work out MAC2 in test.

Slammed by our users…

1. systemIntegrationApi.dowebct - The school using the autosignon wanted to have the correct consortiaId to create the MAC. Some time back in January they started calling this any time users tried to login because a handful (guess was ~12) have had their username changed. So the autosignon failed. Yes, they were sent us 25,000 requests in a busy day (about 20% of the queues were working on these during the day) to handle potential 12 problems in a term. FAIL.
2. pmSelfRegister.dowebt - One of the clusters started to have issues. Two nodes went crappy. I looked at the Weblogic console and found all of the failing nodes had no free spots in the queues. 90% of the queues were working on these. Much of this is because the requests were hanging around for at least 4800 seconds (an hour is 3600 seconds). At about 6000 seconds the cluster recovered when the queues cleared.I think the queues cleared because I changed to false a couple settings:
   • Allow users to register themselves as a Student in a section = false
   • Allow users to register themselves as an Auditor in a section = false

   As I recall, we only had about 22 queue spots open (out of 308) across the whole cluster. We got lucky.

John made a good point… While telling Blackboard about this is pointless, the community at large ought to be aware of another undocumented workspace issue. I found an 8GB .bak in the /u01/app/nodeA/weblogic81/webctbackup on the active JMS node. Taking out user accessible nodes is okay in my book as with 18-20 of them in our clusters, we can lose one and no client would ever know. Mail, chat, learning context administration and other services in CE/Vista fail without a functional JMS node.

An administrator did a template reassignment with “Force archive before template reassignment” set to true. For some reason the file was placed on the JMS node. It should have been deleted. However, it was not. I caught it in time as another large file was dropped within 10 minutes of me deleting the first. I only caught it time because I was at my desk working (not in meetings, at home, or asleep).

This came within one GB of completely filling up the file system. We do not have huge hard drives on these nodes, just 3 times the size we need except for this. Nor do we allow the nodes accrue a ton of logs or junk.

Maybe this is something Blackboard has resolved this for future versions like Vista 4 or 8. Maybe one day we will have official or unofficial documentation about this kind of stuff.

The answers I anticipate from Blackboard:

1. This is functioning as designed. I bet composing the archive requires something from the JMS node, so it must reside there. The JVM is too small as is /var/tmp, so the file system is the best place.
2. Use a bigger hard drive.
3. Set “Force archive before template reassignment” to false.

Even if Blackboard agrees this is bad, then it might get fixed on Vista 8. Certainly it will not get fixed in the officially supported  Vista 3.

If you want to confirm if you have the potential for this problem, then you should have a $NODENAME/weblogic81/webctbackup or a $NODENAME/weblogic92/webctbackup directory. We only have them on all four JMS nodes, but have have seen them on four (out of 76) other nodes. The other 72 nodes lack this directory. While you are at it, make sure you know about the other undocumented work spaces I have mentioned.

One of the challenges of having 42 institutions is managing the administrators. (Actually we created some 12 spare institutions but why is another  post.) So my challenge was how to not drive myself insane trying to enroll the 6 same admin users to each institution. The best way in my opinion is to create the users in an IMS XML file for each institution and import the data. Creating the users was easy. Next was doing the enrollments.

Naturally, I turned to the Vista 8 Enterprise System Integration Guide on pages 66-67 and 95-96 where it describes which roles can be enrolled at which learning contexts. They go from the lowest most common enrollments at the section level up to the division level. Yeah, there was not anything for the institution level. It even had a comment before the table on pages 66-67:

NOTE: Roles not specified in the table can only be added through the Vista Enterprise administrative interface.

So, because the Institution Administrator is not listed, I could not enroll users to it through the import? It depressed me for about a week. A flash of inspiration had me check the Vista 3 documentation. Sure enough, on page 49 of the Vista 3 System Integration Guide, Institution Administrator is listed. (Admin roles at domain and server contexts, designer roles at instiution and domain contexts are also listed.)

The XML is easy enough to write. Normally, when writing this XML, I just need to refer to the SourcedId for objects I create, so I know their values. However, with this, I need to the know the SourcedId.Id for the institution.

Fortunately, we have collected the properties to a institution.properties and parse it to generate what to run at the command line. Rather than by hand copying the files into place one-by-one, I created a script to take a template, check this institution.properties and place the files in the correct place. In order to make each object unique, a portion of the SourcedId.Name was changed to the name of the destination folder.

Now I just need to add to the script a portion to change the SourcedId.Id for the Institution to the source_id value in the institution.properties. That is easy. Much easier than figuring out where to look in the documentation to find what is correct.

On the WebCT Users email list (hosted by Blackboard) there is a discussion about a mysterious directory called unmarshall which suddenly appeared. We found it under similar circumstances as others by investigating why a node consumed so much disk space. Failed command-line restores end up in this unmarshall directory.

Unmarshalling in Java jargon means:

converting the byte-stream back to its original data or object ¹

This suspiciously sounds like what a decryption process would use to convert a .bak file into a .zip so something can open the file.

This is fourth undocumented work space where failed files site for a while and cause problems and no forewarning from the vendor.

Previous ones are:

1. Failed UI backups end up in the weblogic81 (Vista 3, does this still happen in Vista 8?) directory.
2. Failed tracking data files end up in WEBCTDOMAIN/tracking (Vista 3, apparently no longer stored this way in Vista 4/8 according to CSU-Chico and Notre Dame)
3. Web Services content ends up in /var/tmp/ and are named Axis####axis. These are caused by a bug in DIME (like MIME) for Apache Axis. No one is complaining about the content failing to arrive, so we presume the files just end up on the system.

#3 were the hardest to diagnose because of a lack of an ability to tie the data back to user activity.

Is this all there are? I need to do testing to see which of these I can cross off my list goring forward in Vista 8. Failed restores are on it indefinitely for now.

References:

1. http://www.jguru.com/faq/view.jsp?EID=560072

This is the second time I have worked on making Vista integration work with Banner. The first was 2005 in Vista 3.0.3 at Valdosta State. The production here at GeorgiaVIEW was set up by Harold, Jill, and Amy years ago and integrated into the install scripts or part of the cloned databases.

So now I am working on getting it to work in Vista 8. The IMS imports worked the first time like a charm. When I turned to using the Luminis adapter, the person records worked fine but the group contexts failed in Vista 8 and worked fine in Vista 3. So the “siapi.sh luminis import restrict” works fine.

Command-line

We have 41 institutions in Vista 3 currently. So imports are automated to some degree to preserve the sanity of Jill (and to a lesser degree Amy and myself). Rather than put in the UI all the settings, we have a properties file defining the location, glcid, sourcedid.source and sourcedid.id for each institution. This allows us to easily pass the values when importing at the command-line.

My first approach was to leave the settings identical to what I used to create persons and group records with IMS. This essentially uses the glcid of the institution and sourcedid of the institution. This is what resulted in the person records working and groups not. Fail.

I realized my error in logic must be the lack of a division-to-group relationship as the error described the groups cannot be related to an institution. So I changed the properties to use the division values for the sourcedid. Fail.

So I went looking in “Guide to Integration with the SunGard Luminis Data Integration Suite” for what I ought to use at the command-line. I didn’t find a solution. Just the same command-line lacking even the glcid and sourcedid.

XML

Giving up on the command-line approach for now, I added the relationship element to the XML so the group would become a child of one of the divisions I created with IMS. It sorta worked! The groups all imported but the course failed with the exact same error the groups formerly succeeded. To add insult to injury, simply running the import again on the exact same file had the courses import.

Mistakes

A mistake I made was reading the documentation: “Guide to Integration with the SunGard Luminis Data Integration Suite”.

Sungard Libraries:

1. Page 8 says imq.jar and mbclient.jar do not come with CE/Vista and must be obtained from Sungard. All three of us thought in Vista 3.x these were automatically placed so we didn’t need to place them. Best I can tell, these were installed by Vista. I found $WEBCTDOMAIN/customconfig/startup.properties references both files in CUSTOM_CLASSPATH and setEnv.sh references CUSTOM_CLASSPATH. (This document has notes for what CE customers need to do and no note about CE users needing to go get them from Sungard.)
2. Those who believe the last note would keep reading and find on Page 9 instructions to deposit the files in $WEBCTDOMAIN/serverlibs/. Assuming I am wrong about item #1, the startup.properties expects them in $WEBCTDOMAIN/serverlibs/luminis/ and would not find them where the document says to put them.

I laughed at reading this one.

Dear Blackboard: If you include icons in your interface, they should f’ing well be clickable. Everyone but you knows this. jazzmodeus

I thought this might refer to the new item icons. Jason works for Emory (doing instructional design) and taking classes at Florida State. Both use Academic Suite. So its probably not what I thought….

In Blackboard CE/Vista, the “course list” [1] can show icons to alert about new things to do. These can be about waiting assessments, discussion, mail, etc. If users click on the icon, then they can see the items causing the notice. At least, when left at the defaults.

One of the schools we host discovered when students entered a tool by clicking on these icon, the subsequent activity would not be tracked. The work around was to turn off the link rather than the icons entirely.

We agreed with the school and labored to convince Blackboard this was a major security problem. Unfortunately, the people who post the support bulletins have yet to post something about this problem. Its not a major item unless you are the student being accused of cheating because your activity doesn’t show appropriately.

[1] course list - This name bugs me….

• The name is a hold over from when instruction took place in courses. In this system they take place in sections. So why not section list?
• MyWebCT is dumb. MyBlackboard is dumber. “My” is 2004-ish portal cutesy, personalization name buzzword. Similarly, “e” and “i” are similarly dumb.

Summize provides a great way to troll for what people are saying. Beyond just searching for a term, it provides RSS feeds for terms. I follow several, such Blackboard and WebCT. The WebCT one netted me the following tweet:

annoyed with how clunky webct can be at times - it had to have been designed circa 2000 - amandakern

WebCT products, whether CE or Vista, have always been clunky. Ease of use has always been a problem with the products. Any improvements Vista made were offset by so many more tools and options to make it the net effect more clunky. I’ve seen some sales people and Dr. Cs whip through the navigation like it is easy to use Vista. Practice makes perfect. Too bad the developers can’t be perfect.

Whenever I see schools pick a product, I think the ones who have Ease of use on their list probably have been using WebCT legacy products for years as opposed to Blackboard products. They and their faculty are scarred enough they cannot afford to get it wrong on ease of use again.