Accounting Predictions

In my Prediction Accountability, I ranted on how no one really knows whether predictions are accurate and ended with it really does not matter because no one is going to really stop using these services because they are usually wrong. Basically, I thought it futile to even try. In retrospect that is probably the perfect reason to do it.

So I came up with a scoring system:

    • Good Recommendation= 3 points
    • Not interested= -1 points
    • Wishlist/Queue= -2 points
    • Dislike= -3 points

Would you score these differently? Why?

My reasoning goes something like this. Something I agree I should watch should equal the inverse number of points of something I know I will dislike from previous experience. Anything I am not really interested in definitely is not a win, so it should be a negative, but not too close to a dislike. Suggesting something already on that company’s records that I am interested in wastes my time because they already know I am interested in it, so lose two points.

First pass, Amazon sent me an email today saying,

Are you looking for something in our <x> department? If so, you might be interested in these items.

One item I have thought I should watch based on TV ads but not put on my wishlist yet, so I agree with Amazon, I might be interested in it. It gets three points. (3) Five items already were in my wishlist so that is negative two points each. (3 -10= -7) One item is the 6th season of a television series I have only seen part of the first season and not gotten around to completing even that so not interested and negative one point. (-7 -1= -8) Another item is the 3rd season of a TV series I where I have not watched even the first yet. If the recommendation had been the first, then I would count it as a good one so instead I’ll award halfway between good and not interested (-8 + 1 = -7) Out of eight items in the email, the score is a -7. That is just one email. I track this for a couple months and see where it goes. And do the same for Netflix.

I think this exercise points out the possibility that these “predictions” are basically nudges more to buy something.

If your Learning Management System vendor claimed they have a 90% plus correct prediction rate for whether students will fail a class, then how would you assess it? The obvious start would be track the predictions for classes but do not provide the predictions to instructors. Compare the predictions to actual results. Of course, these things are designed around looking at past results. What is the investment company statement they have to put in so they do not get sued for fraud? Oh, right, “Past success does not guarantee future performance.” So I would not rely too much on just historical data. I would want a real world test the system is accurately working.

LMS Brackets

I don’t really follow basketball. So, it is odd that I even registered this tweet from Dennis Kavelman, @dkavelman:

Dennis Kavelman  Team D2L has 4 schools in sweet sixteen! Go #marquette, #arizona, #msu, #osu! @desire2learn

He is excited because four universities who are clients of his company are in the major NCAA basketball championship contest. If only four are Deseire2Learn customers, then that means the other twelve are not. That made me wonder what are the Learning Management Systems used by these sixteen schools?

    1. Louisville : Blackboard
    2. Oregon : Blackboard
    3. Michigan State : D2L
    4. Duke : Sakai
    5. Wichita State : Blackboard
    6. La Salle : Blackboard
    7. Arizona : D2L
    8. Ohio State : D2L
    9. Kansas : Blackboard
    10. Michigan : Sakai
    11. Florida : Sakai
    12. Florida Gulf Coast : Angel (Blackboard) and Canvas in 39 days
    13. Indiana : Sakai
    14. Syracuse : Blackboard
    15. Marquette : D2L
    16. Miami (FL) : Blackboard

Looks like the breakdown is:

Blackboard 8
Desire2Learn: 4
Sakai: 4

This is an interesting grouping. I kind of knew Sakai tended to be the product of choice for well off schools with the money to spend on customization. So, schools with strong athletics probably are more likely to have something like Sakai. Of course, I expect Canvas to be better represented too as it is hot of late. While Moodle tends to be favored by really small schools without a budget, I still figured it would have some representation (really just FGCU).

Which LMSes will be involved with those in the Final Four?

The Loss of Tech Support

I found a statement in Twitter is your IT support interesting:

For reasons I won’t go in to, I haven’t been able to get [a WordPress install with the FeedWordPress plugin] done at the Open University, despite trying since last July. I’ve spoken to people at others unis and it isn’t isolated to the OU, it seems to be this low-level, experimental type of IT support is increasingly difficult to find.

Do you know who I think the culprit is? The VLE. As universities installed VLEs they became experts at developing enterprise level solutions. This is serious business and I have a lot of respect for people who do it. The level of support, planning and maintenance required for such systems is considerable. So we developed a whole host of processes to make sure it worked well. But along the way we lost the ability to support small scale IT requests that don’t require an enterprise level solution. In short, we know how to spend £500,000 but not how to spend £500.

(For those of you non-British/European readers, VLE are Virtual Learning Environments which are often also called Learning Management Systems on this side of the Atlantic.)

It is true the higher education IT has change with online class systems, but I think that part of the symptom and not causal. Chief Information Officers, Chief Academic Officers, and presidents all get recognition for big things. Enterprise level solutions are sexy because it is something that makes them look decisive and effective. Employees who report to them know this, so enterprise level solutions have the priority. Everything else fits into the dwindling extra work time.

What extra time?

The good news though is the small things have gotten much easier for anyone to go off on their own. At my last job, I sat as an ex-officio member of the Faculty Senate technology committee. One of the hot topics one year was a couple faculty members taught students how to use the LMS adopted by another college system in the state. It was two courses. Should we spend $20,000/yr and take up a significant amount of my time running a second LMS? Or should they continue to pay $800/yr for Blackboard to do it? The answer ultimately was to continue with Blackboard. Now days, they probably would be directed at CourseSites. At the time my to-do list was several pages long and hundred plus hour weeks were not uncommon just to keep top and high priority items timely done. The ETA for anything not top or high priority was over a year.

I prefer working with innovative technologies. Custom solutions that require creative thinking and problem solving make me feel like I accomplished something special. They give the biggest rush. Enterprise level software is steak and potatoes, so it is the core. The enterprise is the minimum. I just wish I more time to devote to achieve going beyond the minimum than I did. Well, do. This is a top level decision. Improve staffing and flexible team management so that people can spend time working on the things that make them happier.

One of Many

The Learning Management System (LMS) has been a despised technology by some ever since I started working with one, WebCT, in 1999. At the time it was deemed crappy technology that had to improve or die. So today in 2012, about 13 years later, I have to roll my eyes at the pundits writing about how the current technology has not significantly changed in a decade (really more than a decade) because it still offers the same tools and will die unless it adapts.

My first few years, 2006-2010, of working at GeorgiaVIEW, our active user counts doubled every 1.5 years. We plateaued at around 290,000 and grow a few thousand a year. Numbers of actions in the system still doubles every 1.5 year. That is insane growth. Growth unlikely fueled by people despising use of the tool. Right now, we are getting pressure to migrate Summer 2012 content for the Fall 2012 start in Desire2Learn1 because instructors roll over the classes from term-to-term. That speaks of long term consistent loyal use not occasional only as little as have to use. For something on the verge of death, it is hard enough keeping the users happy.

I am a database administrator not a faculty member (or dean or vice president for academic affairs or provost). It seems to me though no one would say, “When you teach a class, the white board in the room is the only tool you can use.” Instead, the push would be to add to the available tools in a neverending pursuit of finding better ones. So we see pressures to integrate the LMS with a variety of similar specialized services. Many are textbook replacements or supplementary services designed specifically for student needs. Others are social media. More and more the LMS is just a portal: a place to organize where students really go to learn.

Also, as an IT guy, I think it is important to have a plan B. Things sometimes fail. As a student I was always annoyed when the instructor had to leave the room for 20% of the class to go track down a piece of chalk because the remaining ones were too small to write. I applauded once in my junior year because the instructor happened to have a piece of chalk in her purse just for that contingency. Similarly, faculty members and even students should think about what to do when the LMS is not there. Heck, what should they do if everything the university IT runs like the web sites, email, portal, and network all disappear. It can happen.

When the university bureaucracy selects and administrates a tool, they will adhere to university policy which adheres to higher education laws. When a faculty member selects and administrates a tool, they should do the same. Unfortunately, that means the faculty member becoming familiar with policy and law. Another challenge is running into different interpretations. An example: a user following @VSUENGL1101 on Twitter could be reasonably expected to be a student at Valdosta State University enrolled in the subject English class 1101. Some say that violates the Family Education Rights and Privacy Act. Some disagree, so it is being debated. The law is old and did not likely anticipate social media, so naturally there is movement towards an update.

I doubt the LMS will simply die because there is something better. Instead it will remain one of many tools for years to come. Like the land line, television, JavaScript, still camera, WiFi, non-smartphone, and (God forbid) pagers.

Note 1: Desire2Learn objects to their product being called an LMS. They prefer Learning Environment on the grounds it integrates with so many other tools.

P.S. This totally is from a sustaining technology perspective. Guess I should write this from a disruptive technology perspective.

TED Talk: Pursuit of the Perfect Spaghetti Sauce

This is an older TED Talk by Malcolm Gladwell just before his book Blink dropped. He tells the story about how the food industry figured out that there is not a single perfect product that meets everyone’s needs. Instead there are a number of different clusters that meets the majority of wants.

We find this in the Learning Management System world where there is no perfect product for every student, teacher, department head, dean, or administrator. Maybe just maybe the solution is many different approaches to tools for online learning that meet the needs. The LMS, the Personal Learning Environment, digital textbooks, publisher sites, synchronous communication tools, and anything else an instructor can successfully achieve students mastering course objects probably is good for students. This would suggest the rigor part of Laura Gekeler’s post on LMS Evaluations the most important. But not to arrive at a single LMS but to identify the clusters of needs and what tools best help meet their needs.

Even worse, people will self-report what they like as being what they think we want. So identifying their needs probably is not looking at what they say they need. Analytics probably tells a better story of what is effective.

The mind knows not what the tongue wants. — Dr. Howard Moskowitz

Textarea Backup

I am going through my software installed on my work computer in order to transfer to a new one. This came to my attention as something potentially relevant to others.

A common problem we hear doing web-based learning management system is the web browser crashed before the user could submit a form. The complaints we hear usually are because an assignment was lost so the student received a 0 for a major grade. The ones who managed to redo the assignment in time generally never reach us. Nor do the mail messages or discussions or anything else not for a grade. The causes are many. Naturally the blame lies with us for running such a crappy product. Smart applications like WordPress post/page editor automatically save these boxes. Unfortunately, 99.99% are not smart.

An interesting Greasemonkey script, Textarea Backup, will preserve information written into a textarea form element. When the browser restarts and returns to the page, the information written into the textarea will be there.

Google Chrome does native support for Greasemonkey scripts. Mozilla Firefox still requires the Greasemonkey add-on.

With Greasemonkey installed, one can just hit the install button on a scripts page at userscripts.org and click through the various confirms one really wants to download or install it. Pretty simple to install.

Do colleges or universities actually encourage add-ons like Textarea Backup to students? Or are they left to figure out stuff like this on their own?

Pick Up Line

(I will never use.)

My name’s Vista. Can I crash at your place tonight?

Noticed at geekpickuplines.

Especially funny for me because the product I run is the Blackboard Learning Management System Vista Enterprise. We just call it “Vista”. (Yes, very confusing when Windows Vista users want to know the compatibility of Vista with Vista. The answer: barely.)

Muzzled

For over a month now my team has been heads down to provide some sandbox environments for the University System of Georgia Learning Management System Transition Task Force. The evaluators are looking at a sandbox for each contender. Various technical teams are also determining how the product fits with our experience and our operations. Growing new skills and abilities is probably a good thing as long as it fits the organization.

My inclination is to blog about every discovery whether good, bad, or ugly. Yet this whole processes is overshadowed by fear. Fear of the loser initiating a lawsuit and anything I write being taken out of context to support a case is the main reason I have muzzled myself about it. Probably even private blog posts on a private blog could be requested by a subpoena.

I guess there will be plenty of time to gripe about it all when the decision is made and we surge towards meeting an absurdly short timeline to implement a production environment. That is a whole other blog post I probably should never write.

Apples to Oranges

My web hosting service, Dreamhost, happens to have a one-click-installer for Moodle. So I installed one for my own personal sandbox. In looking at the available roles, it suddenly occurred to me…. Comparing any LMS to another is like comparing an apple to an orange. The industry is like the Tower of Babel. Each product has its own jargon covering much of the same ground in absurdly different ways. How could you have an Internet if all the servers talked to each other so differently? Yet in technology created for higher education every system has a different name for the person who teaches a class or even if the name is the same, the capabilities differ.

Sure, there are some commonalities, even apples and oranges are both fruit, but the developers had different conceptual models in mind. The same word meaning different things really is quite annoying. Another example is a course in Vista is a container for the type of place where learning takes place whereas in Learn a course is where teaching takes place. Teaching in Vista takes place in sections. So… Vista : section :: Learn : course.

These different conceptual models are why the faculty get so irate about change. It is hard enough to have to learn new places to click and how to accomplish what you used to accomplish. For some period of time they have to have two vocabularies and maybe even years later they still cannot call it the correct term. (WebCT CE/SE called where teaching takes place courses and both former and current coworkers 6 years later still call sections “courses”).

One would think standards organizations like the IMS Global Learning Consortium would help solve this. Every product adhering to a standard should end up adopting consistent terminology, conceiving of objects similarly, and conceiving of processes similarly. This make comparing the two easier. Except the standard adoptions appear to be in the integration components or database not the main product.

I really feel bad for the instructional technologist who has to support more than one learning management system.

Also, selecting a new LMS seems like an insanely difficult task when trying to learn a dozen vocabularies enough to ascertain whether it has what you need.