Automate Yourself Out Of A Job

At VSU, my boss got a promotion when his boss retired. Then a shuffling of jobs gave me a promotion, but it also sent my old job to another group. That left the web services group going from 2.5 full-time positions down to 1.5.

We did not have that much free time. If anything in the early 2000s, the responsibilities were growing which is how we ended up with 2.5 people. My only way to restore sanity was by automation. Admittedly, I love scripting and schedulings, so my approach to things at that time was to write scripts to handle jobs. The change gave more motivation to ensure that anything that could be automated was. Or I would drown in the work.

What made it hard was, even as I automated these jobs, more things were coming to the web. The needs grew faster than I could develop the tools to handle it. It was a fantastic experience, though.

The Importance of Student Workers to A University

Gille believed that [UGA] Transit could not succeed without its stable of student employees. She said the campus-centered transportation is best fulfilled by students who are on campus nearly every day, not individuals in the community who rarely otherwise come in contact with the University of Georgia campus. It’s easier to acclimate hundreds of students to campus driving routes than to find the same number of non-students willing to learn the routes. The Importance of Student Workers to A University

My first day at Valdosta State University as a student, I also applied for and got a job working in the library. (Yeesh, I think that means I’ve been working for the same employer-ish for 22 years.) I loved the public and school libraries growing up. And I did some of my research for the middle school science fairs in the college library. I love books, so why not?

My final summer, they hired me a temporary staff to fill-in at the reference desk. Normally, a faculty member librarian did that work, but I was being entrusted to do when they were at half capacity. That seemed to seal the deal: I would go to library school for my master’s degree and become a librarian. (Fate intervened by running into my future boss the next fall who convinced me to come work in IT.)

As staff at VSU IT, I supervised a handful of students near the end. They were invaluable for keeping Web Services running. Yes, they were cheap labor. They also hopefully learned some skills that made their careers. Student labor is what made the school operate. Hiring good students is just as important for any staff position because they represent the university, they do the work that allows it to run, and they ensure the quality of almost anything except maybe the professor vocalizing to a classroom. Students do not get the respect of staff, but they for many areas are most of the staff. The departments might not exist without student workers.

 

Why you need a pre-performance routine

As he moves toward the OR, McLaughlin is running through a precise series of thoughts and visualizations, which he calls the Five Ps. First is a Pause: He tries to forget what’s happened earlier in the day and focus only on the present. Next, he thinks deeply about the Patient. “This is a seventy-three-year-old man, and we need him to come out of this pain-free and able to walk more easily,” he says to himself. He reviews his Plan, mentally rehearsing the surgery step-by-step. Then he offers some Positive thoughts: “You were put on this Earth to do this operation,” he says. Finally, as he steps toward the table, he says a quick Prayer. “It’s very ritualistic, and I’m very focused,” he says.

Back when I had to do Friday night maintenance work as a GeorgiaVIEW database administrator, I had something like this. I would do the Pause to quiet my mind to become fully present. Then I would think about the systems involved. Then I would mentally step through the plan for the maintenance.

 

Stealing the T

The Georgia Institute of Technology has a tradition of stealing the letter T. I first ran across this in a local news media story where the letter T was stolen from signage. The main tradition is stealing the T from Tech Tower which has “TECH” on each of the four sides. (They return it during halftime of the homecoming American football game.)

It occurred to me that it would be especially hilarious for some GT alumni to steal the Ts in Trump Tower in Manhattan. It would become Rump Ower.

#FirstSevenJobs

There is a Twitter thing running around where people post their first seven jobs. I do not think mine would fit in a tweet, so I put it here. This title should show up there as a hashtag and be my contribution.

If you count by employers, then I have had 1-3.

  1. University System of Georgia
    1. Valdosta State University
      1. Odum Library
      2. Information Technology
    2. Board  of Regents

Valdosta State is part of the USG. One perspective is I have only had one overall employer. VSU is just a bigger unit than say the library. One could say I have had three employer entities.

If you count by position codes, then I think the list is (not counting repeats in the same position):

  1. Student worker: Reference book shelver
  2. Casual laborer: Reference book shelver, Inter-Library Loan, Government Documents
  3. Student worker: Government Documents
  4. Casual laborer: Reference desk manager
  5. Student worker: Peer Reference desk
  6. Casual laborer: Webmaster Cooperative Education intern
  7. Casual laborer: Assistant Webmaster (CSSII)

Wow, those are all the crazy positions I held before become permanent staff. The next job in the list is the first permanent staff position. In total all seven were just over 5 years.

 

Learning Tech

I learned electronics as a kid by messing around with old radios that were easy to tamper with because they were designed to be fixed.

Lee Felsesnsteinin The Innovators: How a Group of Inventors, Hackers, Geniuses, and Geeks Created the Digital Revolution by Walter Isaacson

The story I tell about how I ended up working in information technology is about having a computer all my life as a child, breaking it, and most importantly knowing that I had to fix it before my parents found that I had. The typical takeaway is that I was intelligent, talented, etc. But, really that reveals the wrong assumption.

More correct takeaway is by this point in computer history, people designed computers to be fixed. The above quote suggests radios were initially custom built, which made them expensive to fix. To accomplish mass production, modular components make it easier to assemble but also as a side benefit easy to swap failed parts. Computers followed the same path but not only on the hardware side but the also software. Modularity to software is how we can patch, install new software, change settings, etc to fix issues.

Even today, I see people look appalled that smartphones can be successfully sold without an easy way for the owner to replace the battery or a microSD slot to add storage. We like to be able to fix our stuff. Maybe it is our Do-It-Yourself cultural biases at play.

Making things fixable lowers the bar to tinker with it. Tinkerability makes something more accessible to learn where, when, how, why it behaves the way it does. Those experiences in turn make a user self-taught into a power user and eventually into a computer administrator who really is just a power user given the keys to offlimits parts.

healthcare.gov

As an information technology professional, when a web site has performance problems, I sigh, gnash my teeth, and gripe just like everyone else. However, twenty minutes later I realize I have been there and feel bad for those having to deal with the mess. Also, should I feel hurt that I am not among the nation’s brightest IT minds since I was not asked to help?

GeorgiaVIEW, one of the projects on which I work, has about four thousand active users on average and with topping out around 5-6 thousand week days and eight during an abnormal event. When users are having problems, they tend to come back which gives them a new session yet the old one has not expired, so the system deals with more and more sessions compounding a performance problem. Some of the descriptions people gave about having problems with healthcare.gov sounded like they came back over and over trying to enter.

The most annoying thing about the healthcare.gov problems though are the pundits. Early on, I heard they should have hired Silicon Valley companies to build the site as though IT people only come from there. They specifically named companies famous for their high profile meltdowns to build the health care exchange as experts in building huge sites without problems. Later came the small companies who build web sites for others, but not at this scale.

It is extremely difficult to build a site to the perfect scale. Overbuilding is expensive, so there is pressure to scale back. Business workflows are murky at best because until people use it, they really are unsure what it is they want. (They just know what was built is not right and why.)

TED Talk: What we’re learning from online education

MOOCs are still the buzz in 2013. The best quote I have heard about them is that they replace an in-person class like Facebook replaces a social life. Of course, Facebook is my main social life….

I do sense a hope that MOOCs will replace a whole education or at least credits (think AP courses).

If the video below does not work, then try Daphne Koller: What we’re learning from online education.

Anatomy of a Mistaken Shut Down

We intensely monitor our servers. We want to know things before a work ticket reaches us.

So a  month ago one morning I saw notifications where a couple servers failed login checks. (A process does a login and logout for each server multiple times an hour.) These go to the servers directly. Another check comes in the front door like a regular user. It also was failing, which is super bad.

Project 365: Day 014

My first instinct was to find if there was a running process for our shutdown script. There was and I killed the process. Then I found the crontab entry that started this and removed it.

At this point there was a hard decision to make very fast:

    1. Recover this one.
    2. Make sure the others instances are not affected.

I ended up doing the latter. In retrospect, I guess I wanted to ensure I did not have multiple fires. If others were doing it too, then I would ask coworkers to help. If just the one, then I could handle it. And it was only a couple minutes to check by checking the dates in the crontab of certain hosts for the shutdown script. This one of the ten was the only one affected.

So I resumed the recovery. The first thing the shut down script does is flip a flag in a file that tells the load balancer whether to allow traffic to the servers. I reversed that first. Half the servers started picking up the traffic and ended the outage. Then I started up the 5 of 10 servers that had shut down.

From start of the outage to when users were back in was about 14 minutes.

Usage was pretty light because the term ended a few days prior.

Probably this was a holdover from doing upgrades the year prior. Crontab does not have year, just month/day or weekday. So we have to make sure we remove things targeted for a specific day. (Or start using at more.)