Institution Analytics #USGRockEagle13

Janice Hill, Columbus State University

  • Process:
    • Define KPI’s : grades,  starting degree, ending degree, and many more.
    • Design and Implement : ODI integrator
      • Subject area example : summation helps reports only pull one row per student.
      • Updating : degrees awarded only loaded at end of term.
    • Validation of data : Work with Institutional Research to figure out where wrong. Consulting with individuals who think data did not look right.
    • Production release : Start a new cycle.
  • Data elements:
    • Banner, PeopleSoft, Excel spreadsheets
    • student head count, student attempted credit hours, and about 30 others.
  • Dashboards : 8 in production, 2 in completed validation, 2 subject areas ready to be built. Changes to a dashboard not saved across sessions, so users need to export to a file.
  • Structure of Dashboard : Level prompts : College, department, program, major, term. Analysis. Footnotes.
  • Users with access : President, VP, Deans, Dept heads.
  • Export types : PDF, Excel, Web,
  • Errors: BI data loaded at 6am, so local data pulled at 9am WILL result in very small differences.
  • Progression dashboard : credit hours by term, avg GPA by class, avg GPA vs credit hours earned, demographic breakdowns, grades by academic level, grades by section
  • Retention and Graduation dashboard : after 1 year, after 6 years. Use both counts and percentages.
  • Talk with faculty about their data needs so can show it exists or build it into a report.
  • Individualized training. Understanding how to filter is a challenging concept.
  • User tracking enabled, so know how long they stay on a dashboard, filters used, the SQL used.
  • Try to use as little filters as possible. Her job to get the data. User’s job is to interpret.
  • Decisions and policy affected by this data.
  • Trying to get grade data to improve early warning.
  • What are the products for which they want analytics?
  • Using University System of Georgia requirements for retention, so pegged to Fall enrollment. “Some times you have to go past what makes sense to you and implement the rule.”

Excellent session!

Muzzled

For over a month now my team has been heads down to provide some sandbox environments for the University System of Georgia Learning Management System Transition Task Force. The evaluators are looking at a sandbox for each contender. Various technical teams are also determining how the product fits with our experience and our operations. Growing new skills and abilities is probably a good thing as long as it fits the organization.

My inclination is to blog about every discovery whether good, bad, or ugly. Yet this whole processes is overshadowed by fear. Fear of the loser initiating a lawsuit and anything I write being taken out of context to support a case is the main reason I have muzzled myself about it. Probably even private blog posts on a private blog could be requested by a subpoena.

I guess there will be plenty of time to gripe about it all when the decision is made and we surge towards meeting an absurdly short timeline to implement a production environment. That is a whole other blog post I probably should never write.

Organization Relationships

A friend of mine who I used to work with once remarked (2007-ish) the University System of Georgia does not really work like a system so much as a loose confederation fighting over money. Given I have no access to budgets, I would not know. GeorgiaVIEW works remarkably well given there are only a few people running the system and hoards of people administrating it for their campus. There is a mostly correct mix of grassroots and top down pressure.

The Board of Regents Information Technology Services have fostered a culture of “help requests must go through the tickets”. Tickets allow the team to better triage issues. Tickets show leaders we are helpful. The unintended consequence is weakening the relationships we have. Tickets indicate we are too busy to be helpful. Relationships are accountable so an individual shows vulnerability to me by admitting not understanding, breaking, or other problems. My part of the relationship is to console, advise, or fix the problems. Tickets make all this harder because they are less personal.

When I talk with my coworkers, we covet the connections we hold across the system for they are the true value. How do we develop these relationships inside the formality of processes which fail to incentivise them?

We have email lists, instant messages, weekly Wimba sessions, etc., but there is obviously  a problem when the same people who have these things only tell me about things when they see me in person. I’m reminded of the ITS CIO spending time going to campuses to talk to them about their needs. Maybe that should something we do throughout the organization especially at my level? Also, when I was at Valdosta State, my best information about the needs of faculty members and students came from visiting them not the technology I developed to encourage reporting issues.

Technology is not magic. It does make those who are not communicating start. It just shifts the form and potentially makes it more difficult. Ideally the difficulty will be so slight no one will notice. One can make communication easier by going from a more difficult technology to a more easy form. Still… It is not as good as being there with the person.

USG Annual Computing Conference

Some of you may have noticed me posting on Twitter using the #usgre10 hashtag. This was the recommended tag to use when posting about the conference.

In talking to a director at a university in the University System of Georgia, he said something interesting which had been said to my CIO, “More good for the USG will be accomplished here at Rock Eagle in these two days then the rest of the year.” (This sounds like When Ideas Have Sex or Where Good Ideas Come From.) This conference had been canceled due to lack of funding from both internal and corporate sponsorship. Due to demand from many universities, the conference was restored.

First, one-on-one conversations happen which might not otherwise occur. My former boss at Valdosta State asked me about a decision my group had made which his assistant director kept pushing back as unacceptable. He explained what he’d understood. I explained what I understood. Suddenly it made more sense to him. I was then able to explain it to the assistant director so she understood. A huge problem went away from 15 minutes of conversation? That is a huge win-win for everyone.

Second, getting to see sessions on the work being done at other schools in the system I wish I knew was being done. UGA developed a tool called El Cid which accomplishes many of the needs we have with one “institution” with 43 different administrators because multiple schools participate in various programs. The administrators were provided rights I disagree are appropriate because their needs are not available at the level where they do have access. El Cid could allow them to do those things for their areas without having the rights to mess up other areas.

Third, criticism which might not otherwise be expressed. As much as it pains me to hear it, I do need to hear the complaints people have about the products we run, the service we provide, and the planned directions. With the phone calls, tickets, emails, surveys, and other communication we do, it seems like what is being done is okay. However, get those same people into a room and the criticism comes flooding forth. This is the food we need understand so we can make improvements.

UPDATE: 2010-OCT-22 at 17:12

Fourth, the wishlists which might otherwise languish. I suspect people are hesitant to put requests in writing which might be negative. We like tickets because they can be tracked and provide a history. However, we also put requirements on opening a ticket like the section, the users, and the time. These requirements mean people may not open a ticket because they do not have enough information. They also may not open a ticket because these requirements make it sound like the bar is extremely high to warrant of spending the effort. The act of speaking to me eliminates the filter.

Fifth, while we have email, phone, instant messenger, wikis, Twitter, (and soon Sharepoint and Office Communicator,) etc., the reality is none of these methods establish the strong social bonds we get from face-t0-face. A strong community has social bonds as the foundation. These tools work well when the social bonds are already there.

Regent NeSmith

I knew some things about William “Dink” H. NeSmith, Jr. a relatively new member of University System of Georgia Board of Regents through a friend and former coworker, Andy Fore, who personally knows Dink.

  • Jesup, Georgia
  • publishes newspapers
  • nice guy

Dink dropped by to tour our facility and answer questions.

One of the more interesting answers to a question about expanding distance learning had to do Dink’s belief online is the direction of the future and with the University of Phoenix operating in our state. He would rather see the money students give them come to us instead. The sense I get is Georgia ONmyLINE intends to help Georgians locate the online class options available to students. The project I work on, GeorgiaVIEW, provides the online class infrastructure. Another project I help intends to provide a more seamless integration between schools for those registering with Georgia ONmyLINE. Guess we are cutting edge?

Blackboard iPhone App

People have been contacting me all day about the Blackboard iPhone App. Both Blackboard and the Chronicle of Higher Education posted blogs about its release.

I find it interesting Jessica mentioned a Georgia student is the inspiration in the Bb blog post. There are over 200,000 students in Georgia who cannot use this application because it relies on Blackboard Sync which only operates for Academic Suite (Classic) products. Blackboard says the Sync product isn’t available to the CE/Vista products used by all but a few schools in the University System of Georgia.

The odds are good the poor student who needs the app can’t use it.

Also, the USG is exactly the kind of client who Blackboard says should wait and see before migrating to Learn.

Most Wired Teacher

“Who is the most wired teacher at your college?” (A Wired Way to Rate Professors—and to Connect Teachers)

Although the university runs workshops on how to use Blackboard, many professors are reluctant, or too busy, to sit through training sessions. Most would prefer to ask a colleague down the hall for help, said Mr. Fritz.

Professional support is too intimidating, cold, careless. Support fixes the problems of others who created problems for themselves:

  • choices made in software to use
  • configuration choices
  • mistakes logic in processing

The concept of identifying the professors who most use the system is a good one. We already track the amount of activity per college or university in the University System of Georgia. The amount of data (think hundreds of millions of rows across several several tables)  would make singling out the professors a very long running query. Doesn’t mean it is a bad idea. Just don’t think it is something we would do with Vista 3. We probably could with Vista 8 which uses a clean database.

I’d like to see two numbers:

  1. Number of actions by the professor
  2. Number of actions by the all classes the professor teaches

Ah, well, there are lots of other reports which need to be done. Many more important than this one. 

Some questions from the article: “Will colleges begin to use technology to help them measure teaching? And should they?” At present, to create such reports, IT staff with database reporting or web server skills are needed. Alternatively, additonal applications like Blackboard Outcomes System can provide the data. The real problem is the reliability and validity of the data. Can it really be trusted to make important decisions like which programs or employees are effective.

Graduation Rates

Systemwide, only 48 percent of the students in Georgia’s four-year public institutions earn a degree within six years of enrollment, compared to a national average of 54 percent. Even allowing eight to 10 years, the percentage of students earning degrees climbs only slightly. In contrast, the Georgia high school graduation rate is between 56 and 60 percent.
–AJC, Degrees are a critical goal

I kept reading this article to find some insight about the problem. All I got was Maureen’s opinions about the thoughts and opinions of universities without even so much as a quote to at least confirm them. For example: “Often times, colleges blame the lack of degree completion on students’ academic shortcomings….” Such a bold accusation has to deserve a report, a quote, something!

I realize the AJC focuses on paper sales, but would it kill you to put a link to the 2005-2006 University System of Georgia Annual Report Card you cited since it is on a web site? As the only documented source, it would be good for others to eyeball the same information. I want too much information not too little. 🙂

The annual report cited has several sections. It covers three measures of the Regents’ Exam, three different kinds of graduation rates, and the freshman retention rates.

I know while I was working at Valdosta State University, freshman retention (report) was huge. A great effort was put towards making sure students learned as much as they could to be successful. Across the system, the freshman retention numbers look pretty good compared to the graduation rates. However, they are for the 2004 cohort and not the 1999! In such a report, I’d like to see the data on a single cohort to be able to compare.

So, what are some of the nuances in all this? It seems a simple Google search led me to an Insidehighered.com piece in 2005 titled, “Missing the Mark: Graduation Rates and University Performance“. It points out:

  • Students may transfer between schools. In Georgia, this is usually thought of as from the smaller schools to the flagships: UGA and GT.
  • Students who take classes part-time while working full-time may take much longer than even 6 years to graduate.

From the same article:

When we use graduation rates to compare campuses and infer from these comparisons that one campus is doing a better job of educating students than another, we exceed the value of the data point. It may be that the campus with a high grad rate is simply giving away its degrees (after all, a diploma mill has a 100 percent graduation rate). It may be that the campus with a low rate is working with high risk students. Or, it may be that the high rate campus is doing a superb job of advising and the low rate campus is ignoring its students.

The grad rate doesn’t tell which answer is right. The right answer requires a detailed analysis of what happened to the students who dropped out, and that takes more than one data point. If the campus improves its own graduation rate, that is probably a good thing, but comparisons of grad rates among campuses usually produce more distortion and misunderstanding than progress.

If we overemphasize graduation rates, colleges may suffer the temptation to do whatever it takes to get the high percentages celebrated by rating agencies and trustees. One way to help improve the rate, after all, is to inflate grades so that all students are above average, and water down the curriculum so everyone can get a degree — not outcomes of much interest to the policy makers, board members, or politicians who tout graduation rates as the touchstone of college performance.

My brother is hurting his school’s graduation rate. His university doesn’t have the degree he seeks. Where he intends to transfer and get that degree doesn’t have the degree his fiancee seeks. Rather than they spend time apart at separate schools, the plan is for her to finish where they are both going to school while he part-times out his degree. After they get married, they will go live where he can finish his degree at the other school.

UPDATE: I like this College Results Online tool to make comparisons.