DSID-0C090334

Working with our clients on LDAP configuration almost invariable starts with SSL certificates. Self-signed, intermediate, and take up a while. The two tools, openSSL and keytool have become my friends. Working with a network admin for the client, I finally saw the legitimate certificate correctly signed by the intermediate certificate not the self-signed. This means I finally saw this new I error I have never before seen.

javax.naming.AuthenticationException: [LDAP: error code 49 – 80090308: LdapErr: DSID-0C090334, comment: AcceptSecurityContext error, data 525, user@host.domain.tld:    at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3041)

Research on the error code DSID-0C090334 led to indications the LDAP search username was incorrect. The Blackboard CE/Vista LDAP client lacks capabilities many clients have to make it easier to use such as searching deeper into a tree or across branches. In this case our clients configured the user as “cn=account”. We looked at other clients who had something like “cn=account,ou=group,dc=domain,dc=edu”. When presented with this discrepancy as likely a problem, the client suggested a path for us to try like the latter. I entered it, tried our test user.

It worked. They also confirmed it worked. Something to add to the wiki, I guess.

OpenSSL Handshake

Chain

One of the questions we ask our clients initiating an engagement to help them setup external authentication from our LMS to their server is, “What is the certificate authority for your SSL certificate?” We have been burned by people purchasing certificates from authorities Java does not support. (And the support is indeed limited compared to say, Mozilla.)

We were given the name of an intermediate certificate which set off warning klaxons. There are none of these in the cacerts file, the list of root CAs Java uses.

So the clients setup to test. Failures. The error:

javax.naming.CommunicationException: hostname.domain.tld:port [Root exception is javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

From what I was able to find, the error meant the certificate was not understood. Framed into thinking the intermediate CA was the cause I started looking at how to make it work. The two potential routes were get the client to add the intermediate CA to their server or test ways to complete the chain by adding the intermediate to my client.

More failures.

Amy suggested looking at the certificate on the foreign server by connecting with openssl to get a better idea where it said there was a problem. The command looks like:

openssl s_client -connect hostname:port

The return was pretty clear that it could not understand or trust a self-signed certificate. The “i:” in the last line below is the Issuer. This made it clear the certificate was not signed by the intermediate CA we were told. It was a self-signed certificate. Doh!

depth=0 /CN=hostname.domain.tld
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /CN=hostname.domain.tld
verify error:num=27:certificate not trusted
verify return:1
depth=0 /CN=hostname.domain.tld
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=hostname.domain.tld
   i:/DC=tld/DC=domain/CN=domain-NAME-CA

It is clear I need to make checking the certificate on the foreign host part of the standard practice. Did some spot checking of previous setups to test against LDAP and every one has a good certificate chain.

LDAP Fiasco

Last week I logged into the ticket management system to look at updates to cases overnight and saw a pop-up for a “widespread issue”, basically two school, involving LDAP. So I looked up the case. The two schools were on the same cluster. Most likely the problem was on my end, which sucks.

Security people like to change firewall settings to block us. At least that is the most likely fault for these LDAP stopped working cases. From one and then all the nodes for their cluster I could connect to both school’s LDAP servers. So not the firewall.

Next most likely fault was the IP changed or failure in load balancing the LDAP servers. Would two completely unrelated institutions change their LDAP server IPs at the same time? Or have load balancing issues? Both of these seemed unlikely.

So I checked the webct.log on a node and found an error I had never previously seen that there were 50 open LDAP connections so it could not open a new one. Cases were there for both schools. Having never seen the error before, my best guess was the something happened to cause connections to open but not close. A good client would close after less than 300 seconds and ideally less than 30.

So I restarted the node. Users started logging in fine. So I kicked off a rolling script to restart all the nodes one-by-one in the background for no end user impact. It took about an hour to restart all ten. Everyone was happy that resolved the issue.

In retrospect a strategy I might have used could have been to narrow the available nodes to users to those which had already restarted. It took about 6 minutes for each to restart and the restarts happened one by one. The first four also run chat. The next six where regular nodes. Blocking access to this latter set would force users to the first ones and additional nodes as they were fixed. Time to users definitely getting a good node could have dropped to 18 minutes rather an hour. Though… With enough users, this could have overwhelmed the cluster. Before 9:30am, there was not enough traffic to overwhelm three nodes. Maybe better safe though.

New Root CA

One of our clients introduced a new LDAP server for authentication. Like a good partner, they implemented it in the test environment, found it did not work and alerted us to the problem. They also informed us the problem would be the new Thawte Root CA was not implemented on many operating systems and applications. Indeed our research pointed out Sun/Oracle had not implemented the Root CA in their list of valid ones.

Normally we do not add root CAs. A specific case which comes to mind is another client “bought” a free SSL certificate for higher education from a place in Barcelona. In this case, we decided to go ahead and add it. However, we wanted to do so in a supported manner, so I opened a ticket with Blackboard.

The recommendation was to use the keytool to apply the new Root CA. (I would not do this without having an open ticket with Blackboard as if anything goes wrong, then at least you can get faster support.) After a couple modifications of what they gave me, the keytool command worked. Namely, the cacerts file is installed with read only permission, so it needs write to be edited. Plus the -keypass  The process to use if we need to do this again will be:

# Navigate to where the root cert is stored.
cd ${JAVA_HOME}
cd ../lib/security/
# Copy the existing cacerts file to a backup in case we need to revert.
cp cacerts cacerts.bak
# Copy the existing cacerts file to a working copy so not affect running processes while editing.
cp cacerts cacerts.new
# Java sets permissions on the file to 444. Change to 644 to edit.
chmod 644 cacerts.new && ls -l cacerts*
# Set the variable for the name of the file to import.
NEWCERT=newthawteroot1
# Run the import keytool. Will ask for password. Ask the system admin or try the Java default.
${JAVA_HOME}/keytool -import -trustcacerts -alias ${NEWCERT} -file ${NEWCERT}.pem -keystore cacerts.new
# Verify change took.
${JAVA_HOME}/keytool -list -v -keystore cacerts.new | grep -A 10 ${NEWCERT}
chmod 444 cacerts.new && ls -l cacerts*
chmod 644 cacerts && cp cacerts.new cacerts && chmod 444 cacerts && ls -l cacerts*
# Another verify but on  cacerts would not hurt to make sure it has the new root ca.

At this point I asked for the institution test on the node where I made the change. (It happens that test cluster has a single node in a special pool. I had made this change on that node. It also meant other institutions testing on that cluster were not affected.) We still want to verify introducing this does not affect others using LDAP. I don’t think that it will, but that never comes across to people as reliable when something completely unexpected causes a problem.

Now to write a script to push this change to the other 180 nodes. Should be easy enough as I copied the new cacerts to our mounted file system. I just need a script to navigate to the where the file is stored, make a backup, chmod, copy, chmod, and verify.

Email Harvesters

Good Sign I missed the story about brothers convicted of harvesting emails the first time. Well, I noticed a followup.

Back around 2001, the CIO received complaints about performance for the web server. So, I went log trolling to see what the web server was doing. A single IP dominated the HTTP requests. This one IP passed various last names into the email directory. Some quick research revealed Apache could block requests from that IP. That calmed things down enough for me to identify the owner of the IP. The CIO then bullied the ISP to provide contact information for the company involved.

Previous little adventures like this landed me a permanent job, so I jumped at similar challenges.

Well, a few years later, it happened again. This time my boss had made me develop a script for the dissemination of the anti-virus software package to home users. Basically, it used email authentication for verification if someone could get the download link. So, I applied the same technique to the email directory. Well, this upset some people who legitimately needed email addresses. So the human workers would provide email addresses to people with a legitimate need.

I’m glad since I’ve left, VSU no longer looks up email addresses for people. (I thought some of the requests questionable.) Also, my little email authentication script was before LDAP was available to the university. I think the new solution much better.

One the more vocal complainers about my having stopped non-VSU access to the email directory was my current employer. We apparently list email addresses for employees freely. Which makes me wonder how much spam we get is due to the brothers described at the beginning of this story? Or other email harvesters? Just hitting the send button potentially exposes the email address.

No worries. I’m sure Glenn is protecting me. 🙂

Preserving CE/Vista Settings

I’ve been asked for notes about this a few times. So here’s a blog post instead.
🙂

A coworker is working on scripting our updates. We lost the Luminis Message Adapter settings in applying the patch to the environment we provide to our clients. Fortunately, those settings are maintained by us not our clients. So I pushed those settings back very easily. Unfortunately, it points to the need to capture the settings for the potential purpose of restoring the settings.

In Oracle databases, this is pretty easy. As the schema user, run the following. It does some intentional things. First, we have multiple institutions, so the breaks make identifying which institution easier. Second, the same label for multiple forms gets confusing, so I am sorting by setting description id under the theory these ids are generated at the time the page is created, so the same tools will float together. (The last modified time stamp is probably unnecessary, I used it in an earlier version and left it just in case Vista for whatever reason added a new setting for the same label instead of modifying the existing one.) This can be spooled both before and after the upgrade. Use diff or WinMerge to compare the versions. Anything lost from the before version should be evaluated for inclusion adding back to the settings.

col lc_name format a50
col setting_value format a80
col label format a80
col lock format 999
col child format 999

clear breaks computes
break on lc_name skip 1

select learning_context.name lc_name, settings_description.label, settings.setting_value,
settings.locked_flag “lock”, settings_description.inheritable_flag “child”
from learning_context, settings, settings_description
where settings.settings_desc_id = settings_description.id
and settings.learning_context_id = learning_context.id
and learning_context.type_code in (‘Server’,’Domain’, ‘Institution’,’Campus’,’Group’)
order by learning_context.name, settings.settings_desc_id
/

An example of the multiple forms issue is external authentication. CE/Vista provides an LDAP (A) and an LDAP (B). The settings_description.label for both is contextmgt.settings.ldap.source. The settings_description.name for both is source. It looks like each of the two identical labels has a different settings.settings_desc_id value depending on whether it is A or B. To me it seems lame to use the same label for two different ids.

The most vulnerable parts of the application to lose settings during an update are the System Integration settings. A mismatched Jar on a node will wipe all the settings associated with that Jar.

However, I can see using this to capture the settings as a backup just in case an administrator or instructor wipes out settings by mistake. Yes, this is scope creep. Create a backup of the settings table to actually preserve the settings.

create table settings_backup_pre_sp2hf1 tablespace WEBCT_DATA as select * from settings;

Contexts: As a server admin, I maintain certain settings and push those down. Each client has control over some other settings and may push those down from the institution context. Maybe some are creating division and group admins? Maybe some instructors are changing things at the course or section levels. I may end up capturing everything?

Restoration: The whole purpose of preserving the settings is to restore them later. There are a couple methods in theory:

  1. Providing the settings to a human to re-enter. The labelling issue makes me question the sanity of trying to explain this to someone.
  2. Update the database directly would just need settings.id ensure it is the right location. Maybe dump out the settings in the format of an update command with labels on each to explain the context? Ugh.

If settings were not so easily lost, then this would be so much easier.

View: Another table of interest is the settings_v view. (Redundant?) The only reason I don’t like this view is it reports the values for every learning context which makes reporting off it much, much longer. For example, the encryption key for a powerlink is listed 8 places in settings/settings_description and 18,769 places in settings_v.

LMS Security

This morning there was a flurry of effort to locate an article called “Hacking WebCT.” My coworker was able to locate it. We were disappointed. 

The main points of the article were:

  1. Lazy administrators make compromising user accounts easy.
  2. Lazy instructors make getting questions for assessments easy.

These apply to any LMS. So, here is some advice to counter the issues raised in this article.

 

Accounts

Default passwords are the bane of any system. Make users change them. (Yes, this increases support tickets.) This usually comes about because the administrators did not integrate the LMS authentication with LDAP, Kerberos, or CAS  which allows for central management of accounts. Central management of accounts means fewer accounts are likely to sit around with easily guessed intially imposed credentials. 

Linking many services together also raises the exposure should one account account me compromised. Enforce decently strong passwords. Too strong and frequently changed password will encourage users to employ means of remembering passwords which defeat the point. Passwords probably should not ever be just birthdays.

Not sure what advice to provide about the potential of a student installing a keylogger on a computer in a classroom?

 

Assessment Cheating

A long availability period (like a week) provides opportunities for enterprising students to exploit the issues with passwords to see and research questions in advance. Instead, a quiz with a short availability period like an hour means less time to go look at the other account, record the questions, research them, then go back into the proper account and take the assessment.

Instructors should use custome questions. Students can obtain questionss provided by publishers in ePacks or with textbooks from previous students, the same textbooks the instructor received, or even web sites online which sell the information. 

High stakes testing ensures students are looking to cheat. When the value of questions is high, these easier methods than knowing the material ensures a war between students and instructors over cheating. Of course, lowering the value of the questions increases the workload of the instructor. 
🙁

MH Operations

We have been getting these messages for months. Some good folks at Blackboard have caused them to stop for a while, but they just resume. Probably it is multiple cases of the same issue.

LDAP Issues: Dear Valued Customer, The MH Operations team has determined that your MH application server(s) are not able to communicate with your local LDAP server infrastructure. Please notify your TSM when LDAP service is available so that we can confirm that there are no other issues with your hosted Blackboard environment.

The last time, alerts for Valdosta State’s Transaction System (a different product than we run) were being sent to us. A DBA for Vista was listed as a technical contact for TS which she doesn’t run or even know she was considered a technical contact. No one working with TS at Valdosta State had been getting these alerts. So they were excited at the prospect of knowing when there was a problem! I put Blackboard in touch with the people at Valdosta.

In a way, I understand my organization is involved in purchasing the licensing. However, Blackboard needs to do a better job of making sure their customers are informed.

  1. Put the name of the affected product in the notice.
  2. Put the name of the affected URL or IP in the notice.
  3. Put the name of the customer’s organization in the notice.