assessments

You are currently browsing articles tagged assessments.

This morning there was a flurry of effort to locate an article called “Hacking WebCT.” My coworker was able to locate it. We were disappointed. 

The main points of the article were:

  1. Lazy administrators make compromising user accounts easy.
  2. Lazy instructors make getting questions for assessments easy.

These apply to any LMS. So, here is some advice to counter the issues raised in this article.

 

Accounts

Default passwords are the bane of any system. Make users change them. (Yes, this increases support tickets.) This usually comes about because the administrators did not integrate the LMS authentication with LDAP, Kerberos, or CAS  which allows for central management of accounts. Central management of accounts means fewer accounts are likely to sit around with easily guessed intially imposed credentials. 

Linking many services together also raises the exposure should one account account me compromised. Enforce decently strong passwords. Too strong and frequently changed password will encourage users to employ means of remembering passwords which defeat the point. Passwords probably should not ever be just birthdays.

Not sure what advice to provide about the potential of a student installing a keylogger on a computer in a classroom?

 

Assessment Cheating

A long availability period (like a week) provides opportunities for enterprising students to exploit the issues with passwords to see and research questions in advance. Instead, a quiz with a short availability period like an hour means less time to go look at the other account, record the questions, research them, then go back into the proper account and take the assessment.

Instructors should use custome questions. Students can obtain questionss provided by publishers in ePacks or with textbooks from previous students, the same textbooks the instructor received, or even web sites online which sell the information. 

High stakes testing ensures students are looking to cheat. When the value of questions is high, these easier methods than knowing the material ensures a war between students and instructors over cheating. Of course, lowering the value of the questions increases the workload of the instructor. 
:(


Related posts

I laughed at reading this one.

Dear Blackboard: If you include icons in your interface, they should f’ing well be clickable. Everyone but you knows this. jazzmodeus

I thought this might refer to the new item icons. Jason works for Emory (doing instructional design) and taking classes at Florida State. Both use Academic Suite. So its probably not what I thought….

In Blackboard CE/Vista, the “course list” [1] can show icons to alert about new things to do. These can be about waiting assessments, discussion, mail, etc. If users click on the icon, then they can see the items causing the notice. At least, when left at the defaults.

One of the schools we host discovered when students entered a tool by clicking on these icon, the subsequent activity would not be tracked. The work around was to turn off the link rather than the icons entirely.

We agreed with the school and labored to convince Blackboard this was a major security problem. Unfortunately, the people who post the support bulletins have yet to post something about this problem. Its not a major item unless you are the student being accused of cheating because your activity doesn’t show appropriately.

[1] course list – This name bugs me….

  • The name is a hold over from when instruction took place in courses. In this system they take place in sections. So why not section list?
  • MyWebCT is dumb. MyBlackboard is dumber. “My” is 2004-ish portal cutesy, personalization name buzzword. Similarly, “e” and “i” are similarly dumb.

Related posts

Blackboard Vista tracks student activity. This tracking data is viewed as a critical feature of Vista. Our instructors depended on the information until we revoked their ability to run reports themselves due to performance issues. Campus administrators can still generate reports (though some still fail). We doubt the solution to this is Blackboard improving the queries to create the reports. We favor deleting tracking data (data preserved outside of Vista) to resolve the performance issues.

We developed SQL reports to look at the tracking data where the user in question was not a student. Yes, the data is limited, but in determining when and where a user was active, can help determine where to look in logs. When we hit the performance issues we started using these reports where the user interface reports failed to generate.

My understanding was the user interface and SQL reports on tracking were the same. Both looked at the same data. The user interface reports were just sexier wrapped in HTML and using icons. I compared a user interface report to a SQL report. Just prior to doing this, I was thinking, WebCT was stupid for not tracking when students look at the list of assessments. Turns out “Assessment list viewed” was tracked in the user interface all along but was missing in our sqlplus queries. WTF?

The data has to be there. The problem has to be our approach in sqlplus is inadvertently excluding the information from the reports. Because these reports must be accurate, I’ll crack this nut… Or become nuts myself.

CRACKED THE NUT: So, part of the data WebCT collected was the name of pages. There is a page name table which was inner joined to the user action table. So pages without a name were not reported. George suggested an outer join. I placed it on the page name table which now lets us see the formerly missing tracked actions. For the specific case where I found this, I now get all the missing actions.

Considering a Blackboard (it’s their problem now) feature request to ensure every page in the application has a title. I consider it developer laziness (someone else said worthlessness) that some pages might not have something so core and simple.

ANOTHER TRICK: Oracle’s NVL function displays a piece of text instead of a null value. Awesome for the above.


Related posts

This is intended to be a more thoughtful response to Laura regarding Course Management Systems and the need for innovation.

Currently, Course Management Systems are bloatware. They got this way by trying to provide everything to everyone. One instructor wants a feature, the university presses for this feature, the CMS programmers put in the feature. Okay, maybe not even 1/2 the time, but given that we have about 15,000 instructors, even a tenth getting a tenth of what they want adds up very quickly. Where they overlap is where companies feel the pressure to add these features.

In my experience, people have found CE and Vista clunky and difficult to use since 2001ish. Basically, that was when the shiny newness wore off at Valdosta State. If anything, then its gotten worse over time. Personally, I think this is the case because its not easy to use. Part of this lack of ease is because of the sheer number of possible actions required to accomplish frequent tasks. Another part is the overwhelming possible branches one might take [1] in the decision tree. Part of what makes us intelligent is visualizing the goal and taking the steps necessary to get is there. When software is not easy to use, the users feel stupid because they cannot figure out how to get to the goal.

Think about the complaints we have been seeing about CE6 from people using CE4. They are griping about features they are used to using disappearing. No one wants to lose the features or options they frequently use. They also wish the features or options they never use would disappear.

From what I’ve seen, instructors will make use of what the university
provides. When universities don’t provide what instructors want, then
these instructors will find what they want elsewhere and make use of
it. Large companies take a long time to integrate new features. By the
time they figure out the user base wants something, incorporate it,
release it, and customers implement it, the users have become used to
using it elsewhere are not attracted to a feature they’ve been using
for years elsewhere. So then we invoke FERPA and whatever to move them
to the CMS which is more clunky than what they were using already.

So enough with my griping… What is the solution? Well, maybe we should think about what a Course Management System should do?

  1. Course management: This means it provides the university administration means by which they can control access to classes. Its not for the faculty so much as provosts, vice presidents, and registrars to be comfortable the university is not allowing students to take something without paying the institution.
  2. Learning: Specifically, these are communication of concepts and evaluation of concept comprehension.

In a nutshell, #1 is the course list and administration screens while #2 is the course internals. If our focus is recreating the university in an online environment, then the CMS is the right approach. By importing the data from the student information system, we build a hierarchy just like the course catalog and put students into virtual representations of these classes. This mindset is where instructors want to build classes that consist of their lectures, the assignments, and the assessments. Its the face-to-face class online. Thankfully, online classes are moving to using tools to better utilize the advantages of the WWW. However, the focus is more towards improving peer discussion.

Maybe this approach isn’t the best one for learning? Last month I read a few articles off a web site advocating a different model: students gathering and creating information themselves (Personal Learning Environment). The instructor in this model becomes more of a mentor like independent study or how universities functioned at the time of our Founding Fathers. I’ve been hearing this is the direction education ought to take for over a decade now. However, I think its unlikely as its easier on the instructor to use the bird shot approach. :)

My Approach: The CMS is only an integration framework to provide access to tools. It doesn’t try to provide these tools at all. There are hundreds of wiki products who are better at some things depending on how its used. Why should the CMS think it can do it better than all of them? Same thing applies to blogs, social bookmarking, file sharing, etc. This means universities will provide a number of these tools and support dozens of different applications and integrate them all. We will have to better understand data flow, security, how all these pedagogically work well together. It’ll be a nightmare.

[1] One of things I unfortunately still do is recreate the user’s actions by figuring out what they clicked on in the recorded session. Much of the problems we see are user error, probably through not understanding the ramifications of the action.


Related posts

Harold Powers, Georgia State University

  • Template
  • Copy files in File Manager
  • Learning Module – Only content pages and assessments are exported in a learning module. Other tools such as discussion, chat, urls, etc. are not exported.

Related posts

.

  • Timeline:
    • Oct 2006: Faculty committee selected Sakai over Blackboard Vista
    • Jan 2007: Developed a roll-out plan.
    • Jun 2007: Pilot
    • Aug 2007: Production
    • Still: Some classes still running on CE4.1, being phased out of use.
  • Needs – no more than 5% of code custom written by GA Tech or professional services.
    • Integration with Banner.
    • Grade book
    • Assignments – no resubmit. Professor had built a section aggregation tool which combined with assignments started killing the servers. Ate all the database connections. Dead in the water Monday afternoon to noon on Tuesday until a code change was implemented.
    • Assessments
    • Course lists would not show unpublished sections. Students were concerned registration failed to take place. Wrote a “More” link to show students the course exists but not yet available.
  • t-square
    • GA Tech – 30,000 users and 10,000 sections. 16,000 users login.
  • Sakai out of the box not very good.
    • Java Server on Tomcat.
    • User administration sux. No way to see what classs a student is taking without an outside application. Created an administrative console, available to machine room admins. Monitors services, processes. Admin console Perl connecting to APIs.
    • Admin role can access every course. Built in admin console to link directly to courses to go help troubleshoot.
  • Unicon – professional services
    • Built t-square implementation.
    • Created some tools.
  • Staffing
    • Systems – OS and hardware – 1 person
    • Application support – 1 person
    • Code developers, Quality Assurance – 3 people
    • Database Administrator – 0.5 people (spends part of time on other projects)
    • Instructional technologists – 2 people
  • Why Sakai? Faculty hate WebCT. Possibilities of integration (aren’t there possibilities for this with Blackboard?).
  • Costs – refused to say. Rumor is 2x our costs for making available for 200,000 students for Blackboard.

Kinda weird. Suggested it was nigh impossible to create accounts except through Banner in WebCT CE. Except the same APIs which create Banner accounts can create guest accounts?


Related posts

I really have to stop listening to the same song played over and over. It may affect my thinking….

We had another node crash due to the Sun JVM issue. Our start script failed to make a file in /var so the node did not become fully operational as expected. While waiting for those with permission to delete some stuff to free up space, I went looking for what I could delete myself. Naturally /var/tmp seemed a likely place. I found 1,171 files named Axis#####axis. (Replace the #s with well… numbers.) They used up only 42MB. Most were small. Looking across all our machines there are thousands of these dating back to February of this year.

I love the Unix file command. It will tell you what kind of files are there. So I used file | sort -k 2 to sort by the type. Almost all of the files were either plain text or JPEG or GIFs. One file, called a “c program file” turned out to be a JavaScript (based on the C syntax). I downloaded a JPEG file locally, renamed it to have the .jpg extension, and opened it in an image viewer. It opened correctly. Seems its a graphic of a table.

It would seem our Blackboard Vista 3 has been collecting these files for months. They do not take up very much space. There are not nearly enough files to represent a download of content by all users. Our /var would fill up hourly in that case.

Axis is an Apache SOAP project. Vista’s exposed APIs use Axis, I believe. So, the running hypothesis is several of our campuses are using a product which is contacting the APIs to upload content. Its spread out enough that all four clusters are affected. Its something that started about February.

Suspect #1 Respondus – Chosen because we know it hits the APIs to upload content. Discounted because the content is lecture materials. Respondus works with assessments (aka quizzes, tests, exams).

Suspect #2 Impatica – Chosen because the JavaScript file references PPT. Impatica compacts PowerPoint (aka PPT) files and allows them to play without needing a PPT player. Their support pages teach users how to use the Campus Edition 4 user interface to upload content into a course. O-kay….

Suspects #n Softchalk, Diploma, Microsoft .Learn, etc. – I haven’t really investigated any of these. They are just names to me at the moment.


UPDATE: So… There is a bug in Axis which dumps these files into the file system. The files can be deleted as long as they are not current.


Related posts

So, you are a teaching an online class. Students cheating naturally is a concern. How does one prevent them from stealing answers?

  • Code in the online class system? Unfortunately, the makers of the learning management systems lag behind the creativity of cheaters. Plus, they can only control their systems. How do they enforce security in the web browser, desktop / laptop, cell phone, classroom, or any other environment?
  • JavaScript? This is the most laughable solution. I’ve known how to disable JavaScript in browsers I use since 1999. I’ve never met a JS security solution I could not beat by simply turning off JavaScript. With Firefox and the Web Developer toolbar it literally takes two clicks. People like it because its cheap. I guess you get what you pay for in this case.
  • Code in the operating system? Dozens of software applications are designed to prevent cheating by controlling what can be done with the desktop or laptop. Certainly this appears to be the most comprehensive solution. However, it often means students go to a proctored environment. What’s the point of taking a class online if I have to go to a classroom?
  • Cameras? The only solution that deals with the possibility of face-to-face or cell phone type collusions. These operate by the students exhibiting suspicious behavior. Students will have to figure out how to act naturally.

The better the solution, the more expensive and less likely to be purchased. Instead, we’ll use cheesy JavaScripts because students are dumb. They’ll never figure it out. Unless by never you mean with a simple Google search.


Related posts