LMS Non-Negotiables

I listened in on the first town hall meeting for our USG LMS Transition Task Force on Thursday. There are 3 more town halls this week and a final one December 9th. It sounds like the task force is looking for what items are non-negotiable, extremely important, nice to have. Here are the non-negotiable items from the list. Here are my thoughts.

  • Security: Agree. Student data is critical information to keep away from those who ought not see it while giving access to those who should. I would include in this an audit log of administrative actions such as changing passwords, resetting virtual classrooms, or anything else which possibly could be abused.
  • Scalable: Agree. We’ve seen fantastic usage growth other the years. When I started with this project four years ago, we had only around 100,000 active users. We now approaching 300,000 active users. Even each user does more now than then. There is no reason we will see an end to usage growth.
  • Integrates with enterprise systems (i.e. Banner): Agree. There is a need for a relatively easy way to ensure the faculty and the students have accounts which are placed in the correct virtual classrooms. I’ve seen a desire for real-time integration. The Luminis Data Integration Suite always looked to cause more problems than it would solve.
  • 508 Compliance: Agree. Every user ought to be able to get the information in the class. However, to truly meet this I would think that would include fixing faculty uploaded content so that is accessible.
  • Don’t go backwards (features and functionality meet or exceed current functionality): Unsure. I’m not aware of an LMS option which meets every feature we currently have in Vista 8. The only way to meet this one is to negotiate which are the non-negotiable features.
  • Cross-platform and cross-browser support: Could not agree more. Most web sites I visit work in any web browser I choose. Vista 8 has limited supported operating system and browser combinations. Don’t forget the cantankerous Java Applets multiple versions of Java behave erratically and prior to Java 1.6.0_11 left in place older versions. Also sometimes new versions of Java suddenly do not work.
  • Ease of use and good user interface (student, instructor, administrator): Agree. More is not always better. I sense a frustration about a lack of efficiency accomplishing tasks.
  • Timely support and response: Agree. I understand this one to mean fix the problem in 1-2 weeks not a year plus.
  • Good communication regarding downtime: Unsure of the intent. Vista 8 has a pretty good announcements tool. Does it mean be more aggressive in telling the users when the system will go down next for a scheduled maintenance? I wonder if it means my organization (hosting) ought to take a firmer hand rather than continue to depend on the campuses in letting end users know.
  • Back up and restore capability (minimum 1 year – nice to go back farther)/archiving/back-up without significant downtime: Unsure of the intent. Our system backups are daily without any downtime involved. My best guess is it means something like a wiki history for all content and tools and maybe the whole virtual classroom. Should something bad happen the faculty member ought to be empowered to fix it and not depend on going to an administrator every time. While Vista 8 allows faculty to make their own backups, this was disabled to avoid performance issues. Also, the restore overwrites everything and not selective enough to ensure the faculty would not lose other data trying to retrieve something specific. Imagine losing 10 weeks of work in order to retrieve an accidentally deleted file. (Administrators have unintentionally done this.)
  • Ability to bring in guests to the system (i.e. collaboration): Agree. In a bricks-and-mortar classroom, the faculty can just ask a guest to come to the right room in a building. With Vista, the virtual classroom is more like a fortress requiring the faculty member to complete some kind of paperwork/memo to get an id so the guest can pass through security.

For those of you in similar searches, does this list look similar to yours? What would you add?

Some things I am surprised are not non-negotiable.

  • Better grade book: The existing one in Vista 8 is cumbersome, especially the grade calculator. A key use of the LMS is for students to understand their performance in the class. However, keeping up with the calculated grade at any given point is a lot of work for the faculty.
  • Reporting and analytics: The faculty, advisors, and tutors need to know which students are having difficulty.  Department heads and deans need to know which instructors are failing to spend enough effort teaching a class. People composing budgets need to know how much the LMS and auxiliary software are used.
  • Administrator becomes another user: Similar to *nix’s “su – user”, some problems only become apparent when using the correct account. Rather than change the password, take a look, and give the user the new password, administrators need an easier way of reviewing.

Night School

I noticed a couple weeks back there are interesting spikes in the evening hours of Sunday through Wednesday. Just like morning/afternoon usage, the evening spikes diminish but even more so by comparison.

As I recall for Monday through Wednesday, when I first started, the evening traffic almost flatlined at 5pm and then dropped off at 11pm. Over time the spike has grown to the point we have more users active in the evening than during “business hours”.

In this graph, the numbers across the bottom are the week of the year. The numbers along the left side are the number of users active within the last 5 minutes.

Yaketystats

Really I have no data to say why the change in trend. (We are not 100% online and the majority of the classes we host are supplemental to face-to-face, with hybrid and totally online fighting for second place.) I hope the days of instructors teaching in a computer lab and having students follow along died a hard painful death. If so, then the amount of activity during the day would lessen some. Students and faculty would still go online during the day between classes. However, more student access to broadband at home would empower them to go online more often in the evening and increase the difference between day and evening user activity.

Identifying where each individual IP resides is hard. Doing so for many is more time than I would want to invest in the question. Campus vs. residential vs. corporate is relatively easy. However, “home” for a student could be on campus or residential. Maybe someone else knows better than me.

I guess this means we really ought to look at our automated operations which kick off at 10pm. WebCT recommended they be run when user activity is light or they could impact performance.

State of the LMS

Watched an informative WebEx about The State of the LMS: An Insitution Perspective presented jointly by Delta Initiative and California State University. An true innovator in this market could become the leader.

Market share numbers annoy me. These are always self-reported numbers from a survey. The sample sizes are almost always not very impressive and when broken down doesn’t really represent the market. DI didn’t post a link to where they got the numbers just the name of the group. Some digging and turned up this Background Information About LMS Deployment from the 2008 Campus Computing Survey. For background information it is woefully lacking in important information such as sample size, especially the breakdown of the types of institutions in the categories.

The numbers DI quotes of CC are very different for the same year the Instructional Technology Council reports: Blackboard market share 66% (DI/CC) vs 77% (ITC). An 11% difference makes is huge when the next largest competitor is 10% (DI/CC).

Other missing critical information: Are these longitudinal numbers, aka the same respondants used participate in every year the survey quotes? Or is there a high turnover rate meaning an almost completely different set of people are answering every year so the survey completely relies on the randomness of who is willing to answer the survey? So the numbers could shift just because people refuse to answer giving Blackboard reduced market share only because Moodle customers are more willing to respond to questions about it?

Most of the major LMS products on the market started at a university or as part of a consortium involving universities. I knew the background of most of the products on in Figure 1. Somehow I never put that together.

Will another university take the lead and through innovation cause the next big shakeup? I would have thought the next logical step to address here in the DI presentation would be the innovative things universities are doing which could have an impact. Phil described Personal Learning Environments (not named) as potentially impacting the LMS market, but he was careful to say really PLEs are an unkown. The were no statements about brand new LMSs recently entering or about to enter the market.

Figure 1: Start year and origin of LMSes. Line thickness indicates market share based on Campus Computing numbers. From the DI WebEx.

Network Recording Player - State-wide LMS Strategy 8262009 90839 AM-1

When people use my project as an example, it gets my attention. GeorgiaVIEW was slightly incorrectly described on page 26 Trends: Changing definition of “centralization”.

  1. We do not have an instance per institution which has a significantly higher licensing cost. We do give each institution their own URL to provide consistency for their users. Changing bookmarks, web pages, portals, etc everywhere a URL is listed is a nightmare. So we try to minimize the impact when we move them by a single unchanging URL.We have 10 instances for the 31 institutions (plus 8 intercampus programs like Georgia ONmyLINE) we host. Learn 9 will not have the Vista multiple institution capability, so should we migrate to Learn 9 an instance per institution would have to happen.
  2. We have two primary data centers not have a primary and a backup data center. By having multiple sites, we keep our eggs in multiple baskets.

The primary point about splitting into multiple instances was correct. We performed the two splits because Vista 2 and 3 exhibited performance issues based on both the amount of usage and data. With ten instances we hit 20,000 4,500 users (active in the past 5 minutes recently) but should be capable of 50,000 based on the sizing documents. We also crossed 50 million hits and 30 million page views. We also grow by over a terabyte a term now. All these numbers are still accelerating (grows faster every year). I keep hoping to find we hit a plateau.

Figure 2: LMS consortia around the United States. From the DI WebEx.

Consortia Nationwide

All this growth in my mind means people in general find us useful. I would expect us to have fewer active users and less data growth should everyone hate us. Of course, the kids on Twitter think GeorgiaVIEW hates them. (Only when you cause a meltdown.)

UPDATE: Corrected the active users number. We have two measure active and total. 20,000 is the total or all sessions. 4,500 are active in the past 5 minutes. Thanks to Mark for reading and find the error!

Pointless

So I wanted to open a support ticket. However, in thinking about what I can ask for the company to do arrayed against what they are willing to offer for support, I realized… I am not going to get a resoultion for the ticket.

  1. It is functioning as designed.
  2. They are just going to tell us the workarounds we have already implemented.

So, what is the point? Other than distracting employees of the company with something they are never going to solve, I get no benefit. I just get to be the passive-aggressive, CYAer, paper pusher who gets to point at the fact I opened a pointless support ticket to justify my employment.

Yes, the problem could trigger a cascade of events which would result in the failure of services for about 3,000 active users. We stood at the brink twice yesterday and the day before. Because we DBAs are responsive, we saved it. The next time we will do the same.

The company is not going to release another patch for the product unless forced to do so (aka glaring security hole). So even if we could convince them of a bug, then no resolution would be provided in this version. I’ll have to replicate to see if the same problem exists in a newer version they do adequately support. If so, then I would have justification in opening a ticket.

Now… how to I identify an 8GB section archive…

Page View Metric Dying

First Metricocracy measured hits. Pictures and other junk on pages inflated the results so Metricocracy decided on either unique visitors or page views. Now, the Metricocracy wants us to measure attention. Attention is engagement, how much time users spend on a page.

What do we really want to know? Really it is the potential value of the property. The assumption around attention is the longer someone spends on a web site, the more money that site gains in advertisement revenue. The rationale being users who barely glance at pages and spend little time on the site are not going to click ads. Does this really mean users who linger and spend large amounts of time on the site are going to click more ads?

This means to me attention is just another contrived metric which doesn’t measure what is really sought. I guess advertisement companies and the hosts brandishing them really do not want to report the click through rates?

My web browsing habits skew the attention metric way higher than it ought to be. First, I have a tendency to open several items in a window and leave them lingering. While my eyes spent a minute looking the content, the page spent minutes to hours in a window… waiting for the opportunity. Second, I actively block images from advertisement sources and block Flash except when required.

As a DBA, page views also has debatable usefulness. On the one hand we could use it because it represents a count of objects requiring calls to the database and rendering by application and web server code. Hits represent all requests for all content, simple or complex, so is more inclusive. Bandwidth throughput represents how much data is sucked out or pushed into the systems.

We DBAs also provide supporting information to the project leaders. Currently they look at the number of users or classrooms who have been active throughout the term. Attention could provide another perspective to enhance the overall picture of how much use our systems get.

Cat Finnegan, who conducts research with GeorgiaVIEW tracking data, measures learning effectiveness. To me, that is the ultimate point of this project. If students are learning with the system, then it is successful. If we can change how we do things to help them learn better, then we ought to make that change. If another product can help students learn better, then that is the system we ought to use.

Ultimately, I don’t think there is a single useful metric. Hits, unique users, page views, attention, bandwith, active users, etc., all provide a nuanced view of what is happening. I’ve used them all for different purposes.

VLE vs PLE

PLEs reflect the needs of a student information gatherer. VLEs (aka LMSs) reflect the classroom and institution online. (PLEs vs VLEs) Blackboard Vista is a VLE. Its competitors are all VLEs. Vista is the worst VLE of them all as its designed to host multiple institutions in one site. The GeorgiaVIEW project takes the VLE approach to the extreme by getting a number of institutions to collaboratively host together (though the technology doesn’t scale out to 170,000 active users).

The questions I have though is: Are VLEs worse than PLEs? Would students learn more through PLEs?

Back when I had direct instructor contact, instructors wished to combine multiple classroom meeting times into one WebCT CE or Vista online section. The universal reason being so all the students studying the same topics could discuss these topics. This was before social networks… unless you want to count the now defunct sixdegrees.com? The idea being that student interaction is where they would learn.

Bill Huitt describes the classroom model as old world, without foundation, and eventually will collapse to the dismay of educational administrators everywhere. The student-teacher relationship is the key to the learning.