Page View Metric Dying

First Metricocracy measured hits. Pictures and other junk on pages inflated the results so Metricocracy decided on either unique visitors or page views. Now, the Metricocracy wants us to measure attention. Attention is engagement, how much time users spend on a page.

What do we really want to know? Really it is the potential value of the property. The assumption around attention is the longer someone spends on a web site, the more money that site gains in advertisement revenue. The rationale being users who barely glance at pages and spend little time on the site are not going to click ads. Does this really mean users who linger and spend large amounts of time on the site are going to click more ads?

This means to me attention is just another contrived metric which doesn’t measure what is really sought. I guess advertisement companies and the hosts brandishing them really do not want to report the click through rates?

My web browsing habits skew the attention metric way higher than it ought to be. First, I have a tendency to open several items in a window and leave them lingering. While my eyes spent a minute looking the content, the page spent minutes to hours in a window… waiting for the opportunity. Second, I actively block images from advertisement sources and block Flash except when required.

As a DBA, page views also has debatable usefulness. On the one hand we could use it because it represents a count of objects requiring calls to the database and rendering by application and web server code. Hits represent all requests for all content, simple or complex, so is more inclusive. Bandwidth throughput represents how much data is sucked out or pushed into the systems.

We DBAs also provide supporting information to the project leaders. Currently they look at the number of users or classrooms who have been active throughout the term. Attention could provide another perspective to enhance the overall picture of how much use our systems get.

Cat Finnegan, who conducts research with GeorgiaVIEW tracking data, measures learning effectiveness. To me, that is the ultimate point of this project. If students are learning with the system, then it is successful. If we can change how we do things to help them learn better, then we ought to make that change. If another product can help students learn better, then that is the system we ought to use.

Ultimately, I don’t think there is a single useful metric. Hits, unique users, page views, attention, bandwith, active users, etc., all provide a nuanced view of what is happening. I’ve used them all for different purposes.

Finding Sessions

Clusters can making finding where a user was working a clusterf***. Users end up on a node, but they don’t know which node. Heck, we are ahead of the curve to get user name, date, and time. Usually checking all the nodes in the past few days can net you the sessions. Capturing the session ids in the web server logs usually leads to finding an error in the webct logs. Though not always. Digging through the web server logs to find where the user was doing something similar to the appropriate activity consumes days.

Blackboard Vista captures node information for where the action took place. Reports against the tracking data provide more concise, more easily understood, and more quickly compiled. They are fantastic for getting a clear understanding of what steps a user took.

Web server logs contain every hit which includes every page view (well, almost, the gap is another post). Tracking data represents at best 25% of the page views. This problem is perhaps the only reason I favor logs over tracking data. More cryptic data usually means a slower resolution time not faster.

Another issue with tracking is the scope. When profiling student behavior, it is great. The problem is only okay data can be located for instructors while designers and administrators are almost totally under the radar. With the new outer join, what we can get for these oblivious roles has been greatly expanded.

Certainly, I try not to rely too much on a single source of data. Even I sometimes forget to do so.

User Interface v. SQL Reports and Tracking

Blackboard Vista tracks student activity. This tracking data is viewed as a critical feature of Vista. Our instructors depended on the information until we revoked their ability to run reports themselves due to performance issues. Campus administrators can still generate reports (though some still fail). We doubt the solution to this is Blackboard improving the queries to create the reports. We favor deleting tracking data (data preserved outside of Vista) to resolve the performance issues.

We developed SQL reports to look at the tracking data where the user in question was not a student. Yes, the data is limited, but in determining when and where a user was active, can help determine where to look in logs. When we hit the performance issues we started using these reports where the user interface reports failed to generate.

My understanding was the user interface and SQL reports on tracking were the same. Both looked at the same data. The user interface reports were just sexier wrapped in HTML and using icons. I compared a user interface report to a SQL report. Just prior to doing this, I was thinking, WebCT was stupid for not tracking when students look at the list of assessments. Turns out “Assessment list viewed” was tracked in the user interface all along but was missing in our sqlplus queries. WTF?

The data has to be there. The problem has to be our approach in sqlplus is inadvertently excluding the information from the reports. Because these reports must be accurate, I’ll crack this nut… Or become nuts myself.

CRACKED THE NUT: So, part of the data WebCT collected was the name of pages. There is a page name table which was inner joined to the user action table. So pages without a name were not reported. George suggested an outer join. I placed it on the page name table which now lets us see the formerly missing tracked actions. For the specific case where I found this, I now get all the missing actions.

Considering a Blackboard (it’s their problem now) feature request to ensure every page in the application has a title. I consider it developer laziness (someone else said worthlessness) that some pages might not have something so core and simple.

ANOTHER TRICK: Oracle’s NVL function displays a piece of text instead of a null value. Awesome for the above.

Better Way to Count

Our awesome sysadmins have put the user agent into our AWStats so we are tracking these numbers now. They discovered something I overlooked. Netscape 4.x is 10 times more used than 7.x or 8.x. Wowsers! Some people really do not give up on the past.

Back in the Netscape is dead post, I used this to count the Netscape 7 hits.

grep Netscape/7 webserver.log* | wc -l

Stupid! Stupid! Stupid! The above requires running for each version of Netscape. This is why I missed Netscape 4.

This is more convoluted, but I think it its a much better approach.

grep Netscape webserver.log* | awk -F\t ‘{print $11}’ | sort | uniq -c | sort -n

It looks uglier, but its much more elegant. Maybe I ought to make a resolution for 2008 to be elegant in all my shell commands.

This version first pulls any entries with Netscape in the line. Next, the awk piece reports only the user agent string. The first sort puts all the similar entries next to each other so the uniq will not accidentally duplicate. The -c in the uniq counts. The final sort with the -n orders them by the uniq’s count. The largest will end up at the bottom.

RE 2007: GeorgiaVIEW Meeting (Pre-Conference)

I am blogging from the pre-conference GeorgiaVIEW meeting @ Rock Eagle yesterday afternoon and this morning. I enjoy connecting with people around the state of Georgia who use our Vista system. Most of them do not make it to BbWorld. Some hot topics:

  • Alternatives to Blackboard Vista
  • Training
    • Content repository
  • Returning Reports and Tracking to instructors.
    • Some reports still failing. One approach may be to remove tracking data from Vista database and make it available elsewhere.
  • Upgrade to Vista 4. People want a timeline, access to a training instance ASAP, please not do an in-place upgrade.
    • Limited shelf life on internals of Vista 3 / 4.0 – 4.1.2
    • More of customers have moved or are moving to Vista 4 / CE 6 than a year ago.
    • Can take advantage of new tools available in Vista 4.
    • Data retention – policy, reponsibilities (faculty, campus, OIIT)
    • Phased approach – parallel environments, at some point Vista 3 goes away and no longer available.
    • End of Fall 2008 or Spring 2009.
  • People are both quite happy we are going to Vista 4 and disconcerted at the prospect of having to move to Vista 4 in even over a year from now (at the worst by April 2009).
    • Export / import of non-SIS created users.
    • Training

Lovely (yeah a real person and she is) says Lovely Freelove would be one of the best names ever.

BbWorld Presentation Redux Part I – Automation

Much of what I might write in these posts about Vista is knowledge accumulated from the efforts of my coworkers.

I’ve decided to do a series of blog posts on our presentation at BbWorld ’07, on the behalf of the Georgia VIEW project, Maintaining Large Vista Installations (2MB PPT). I wrote the bit about tracking files a while back in large part because of the blank looks we got when I mentioned in our presentation at BbWorld these files exist. For many unanticipated reasons, these may not be made part of the tracking data in the database.

Automation in this context essentially is the scheduling of tasks to run without a human needing to intercede. Humans should spend time on analysis not typing commands into a shell.

Rolling Restarts

This is our internal name for restarting a subset (consisting of nodes) of our clusters. The idea is to restart all managed nodes except the JMS node, usually one at a time. Such restarts are conducted for one of two reasons: 1) have the node pick up a setting or 2) have Java discard from memory everything. The latter is why we restart the nodes once weekly.

Like many, I was skeptical of the value of restarting the nodes in the cluster once weekly. Until, as part of the Daylight Savings Time patching, we provided our nodes to our Systems folks (hardware and operating systems) and forgot to re-enable the Rolling Restarts for one batch. Those nodes starting complaining about issues into the second week. Putting back into place the Rolling Restarts eliminated the issues. So… Now I am a believer!

One of my coworkers created a script which 1) detects whether or not Vista is running on the node, 2) only if Vista is running does it shut down the node, 3) once down, it starts up the node, and 4) finally checks that it is running. Its pretty basic.

Log cleanup to preserve space

We operate on a relatively small space budget. Accumulating logs infinitum strikes us as unnecessary. So, we keep a months’ worth of logs for certain ones. Others are rolled by Log4j to keep a certain number. Certain activities can mean only a day’s worth are kept, so we have on occasion increased the number kept for diagnostics. Log4j is so easy and painless.

We use Unix’s find with mtime to look for files 30 days old with specific file names. We delete the ones which match the pattern.

UPDATE 2007-SEP-18: The axis files in /var/tmp will go on this list, but we will delete any more than a day old.

Error reporting application, tracking, vulnerabilities

Any problems we have encountered, we expect to encounter again at some point. We send ourselves reports to stay on top of potentially escalating issues. Specifically, we monitor for the unmarshalled exception for WebLogic, that tracking files failed to upload, and we used to collect instances of a known vulnerability in Vista. Now that its been patched, we are not looking for it anymore.

Thread dumps

Blackboard at some point will ask for thread dumps at the time the error occurred. Replicating a severe issue strikes us as bad for our users. We have the thread dumps running every 5 minutes and can collect them to provide Blackboard on demand. No messing with the users for us.

Sync admin node with backup

We use rsync to keep a spare admin node in sync with the admin node for each production cluster. Should the admin node fail, we have a hot spare.

LDIS batch integration

Because we do not run a single cluster per school and the Luminis Data Integration Suite does not work with multiple schools for Vista 3 (rumor is Utah has it working for Vista 4), we have to import our Banner data in batches. The schools we host send the files, our expert reviews the files and puts them in place. A script finds the files and uploads each in turn. Our expert can sleep at night.

Very soon, we will automate the running of the table analysis.

Anyone have ideas on what we should automate?

Its 10:34 Do You Know Where Your Tracking Files Are?

One of my co-workers says about tracking, “One of the big selling points to [Blackboard] Vista is the wealth of tracking data for auditing, grade challenges, and catching cheaters.” Certainly, Reports and Tracking in Blackboard’s Vista 3 is one of the more favorite tools. So making sure the data gets there is critically important.

The tracking data is not immediately written to the database. Instead, its staged on each node and applied to the database on the schedule provided by the UI’s System Administrator role. The schedule can be hourly or daily. We normally have these set to upload daily at 5am which is a slow part of the day. Though, going into splitting our data sets, we had temporarily set them to hourly to ensure the data would be uploaded. Coming out the split, we lost 3 nodes in 3 hours for the first day of classes due to Java issues. Going back to daily upload of the tracking data brought down the rate to about 2-3 nodes a day. Much better!

This data is staged in the form of files on each node. In the domain directory, locate the tracking directory. In it should be files with the date and hour and hour in the name. Each is rolled at the top of the hour. A file called .active-log contains the name of the file being written to and which the node not to upload the data. The data in these files is a little difficult to interpret. However, you can get the timestamp (Java epoch), learning context id, node, role, action, location, and person id in these logs. Working against the reporting interface or tracking data in the database would yield better results as this information is matched against more useful information like the WebCT ID, names, etc.

So, its possible for these files not to get written to the database for one reason or another. One error causes the rejection of the whole file (not just the offending entry). However, attempting to process the same file again does not result in duplicate entries. A short list of problems:

  • Database can only accept entries of a certain length.
  • Improper use of doublequotes.
  • Special characters.
  • Sudden node failure.

Database can only accept entries of a certain length. When an event’s length is too long, the database encounters an error and cannot place the entry into the tracking data. Problems is Vista 3’s SP7 caused the problem length to drop to about 590 characters. An update in hotfix 2 increased the event length to 1151.

Improper use of doublequotes. Designers using doublequotes in object names (even properly paired) cause issues. Its possible a hotfix intercepts these as I have not seen a case of this in a really long time.

Special characters. The usual special characters work fine. So your amperstands and colons are fine. Its the  which bother me. Monday, Amy got a report a tracking file had not uploaded. It had one event with a length of 1154 which failed (lines of 1151 length did not have an error). The 1154 length one had a text block that looked like “   ” but when I copy that text locally to Windows, it looks like “”. I think the extra spaces bumped the length from 1151 (just under the fail point) to 1154 (just over) and caused the error.

Sudden node failure. As well as we manage things, unexpected problems happen. We had once incident where we lost 6 nodes at once. One of the aftermaths of that even was a tracking file would not get written to the database for two of the six. It turns out an error was written to the end of the current tracking file at the time. Once the error was removed, the files were processed correctly.

The last two always make me think, “Weird.”

Cancelled

You’d think a city would make life hell on venue owners and fine advertisers for making them spend thousands of employee man hours cleaning up advertisements pasted onto walls of buildings, benches, etc. Punishment is only effective with the subject notices and can tie back to the cause-effect relationship. Tracking down, fining, and hearing appeals of “who is responsible” takes time as well. This… This is so simple its elegant!

Glasgow’s clean-up squads are tackling illegal bill posters with a little creative vandalism of their own…. the city’s litter wardens are cancelling illegal ad sites by pasting up stickers over unlicensed ads.

A few seconds with a paintbrush is enough to render the ad useless until a clean-up team arrives to tackle the site. They have already had an impact on some rogue promoters who have been inundated with complaints from music fans. People who have bought tickets to some of this summers big gigs have complained, thinking that an event, rather than the advert, had been cancelled.

The source said: “If people start phoning concert promoters complaining that they thought the gig had been cancelled, then the promoters have no-one but themselves to blame for having the posters put up in the first place.”

Best Virus of the Day

From: customercare@newegg.com
Subject: Order Confirmation number: 37679041
Body:
Dear Customer,

Thank you for ordering from our internet shop. If you paid with a credit card, the charge on your statement will be from name of our shop.

This email is to confirm the receipt of your order. Please do not reply as this email was sent from our automated confirmation system.

Date : 08 Oct 2006 – 12:40
Order ID : 37679041

Payment by Credit card

Product : Quantity : Price
WJM-PSP – Sony VAIO SZ370 C2D T7200 : 1 : 2,449.99

Subtotal : 2,449.99
Shipping : 32.88
TOTAL : 2,482.87

Your Order Summary located in the attachment file ( self-extracting archive with “37679041.pdf” file ).

PDF (Portable Document Format) files are created by Adobe Acrobat software and can be viewed with Adobe Acrobat Reader. If you do not already have this viewer configured on a local drive, you may download it for free from Adobe’s Web site.

We will ship your order from the warehouse nearest to you that has your items in stock (NY, TN, UT & CA). We strive to ship all orders the same day, but please allow 24hrs for processing.

You will receive another email with tracking information soon.

We hope you enjoy your order! Thank you for shopping with us!
(Attached is a zip file)