Rote Loading

With this specific application, we can import data, but there are limitations due to its 2000-era handling of XML files.

  1. HTML forms uploading files have to…
    1. Have all the packets be received by the server.
    2. Process the file with the browser connection still open.
    3. The server has to tell the browser everything was received and is done.
  2. All this has to happen as one action within a 5 minute window.

A better method would allow just uploading the files to a page. Background processes would monitor that location and process the files independent of the browser. Notifications can be sent to alert the user the processing is done.

Or… recognize the echo XML file because you took too long and prevent it from being loaded or remove the data.

I ended up figuring out that if we split the files at about 5,000 records, then it should take about half the 5 minute window. I am pleased that for most I have seen that is true and about one in ten take so much longer that if I had cheated and gone with closer to the 5 minutes, I would be deleting duplicates. (This last is because some files are 50MB and others 100MB.)

The grumbling of this post is that I am on the 25th of 58 files. This is tedious. I am lamenting not creating a curl script to do this part for me. Automation is perfect for things like this.

 

TED Talk: Our Future is Awesome

Automation is here. When I was in college, for most of my time there, I thought Industrial-Organizational Psychology was my career path. It was not until my last semester when I started working as a student for IT that it all changed. Little did I know that people efficiency was dying. Automation, aka the efficiency of technology (computers and robots), is where industry is making its money. Or that really the anger boistering both the Republican and Democratic presidential front-runners is due to automation. People are frustrated with the economy and how it affects them.

Edward Elser explains why:

Wages no longer follow productivity.

A whole lot of people are unhappy. Some people blame the soulless corporation. Others blame workers who want $15 an hour for a marginal job. But, here is the deal. Here is where I think the root of the discontent is: the system does not need unskilled labor anymore. Industry simply does not need unskilled or marginalized workers.

It used to be when productivity went up, so did wages. But somewhere in the 1970s a disconnect happened. The value of labor stopped following productivity. And it is my opinion this trend is not going to change; it is going to accelerate..

Our Future is Awesome | Edward Elser | TEDxAugusta

March of the Machines (Automation)

Saw a tweet about and interesting piece in ABC News Australia Digital disruption: How science and the human touch can help employees resist the march of the machines. Basically, many jobs are going away due to automation. W.I.R.E.D. has a similar story: Robots Will Steal Our Jobs, But They’ll Give Us New Ones.

One of the long struggles I have ever pushed in my career is automation of machines. My approach falls along the line of: if it is going to be done more than once or will take a really long time by hand, then it needs to be automated. This is hard to do. The temptation is to do it by hand once, see how it went, then write a script which does it for the next time. The trouble being that if this is done between having completed the first one and the second, then there is little incentive. Best is to make the automation part of doing it the first time, the second time can include any remediation necessary to make it more perfect.

All this automation makes us more effective employees. My team of three managed hundreds of web servers and dozens of database servers for ten sites. Without automation that would have been a nightmare. The replacement product was more difficult to automate so with fewer servers we needed more people. Yet the drive to better automation is making lives easier. (Technically I left that program about a year ago when my replacement was hired and took over my spot in the on-call rotation.)

A fear I hear about automation is that people will lose their jobs. It reminds me globalization and manufacturing moving overseas to China. Highly repetitive, mindnumbing jobs were the most at risk and as those work forces got better, what was at risk moved up the complexity ladder.

The fear of both globalization and automation led to books like A Whole New Mind. The idea is that if your job is highly repetitive or analytical, then it is at risk to these forces. Becoming the person who designs, describes, coordinates, or finds meaning in stuff (aka “right brain” activities) is the way to survive the coming storm. This book very influenced how I started thinking about my work.

Back in 2003, I automated everything I could because I was overwhelmed with work and little resources beyond great computers and my own skill to make it better. My supervisees focused on meeting with the clients to talk about the web site they wanted and build that. I wrote code to report about or fix problems to prevent people needing to call or email about problems.

Where I wish we would head is more like You Really Don’t Need To Work So Much. I meant to send this to my boss (maybe he’s reading this blog)? All our efficiencies should mean we have less to do not more, so why do we work so hard?

The past fifty years have seen massive gains in productivity, the invention of countless labor-saving devices, and the mass entry of women into the formal workforce. If we assume that there is, to a certain degree, a fixed amount of work necessary for society to function, how can we at once be more productive, have more workers, and yet still be working more hours? Something else must be going on.

From my experience, the to-do list gets ever larger. Not because there is more to do, but because more is possible. I’d just rather spend more of my time on solving hard problems than easy repetitive tasks.

P.S. This post really only exists because I loved the phrase “March of the Machines” enough I wanted it as a title for something on this blog.

Slackers and IT

Go read “Science Fiction Is for Slackers.”

As a rule, science fiction may be the laziest of all genres, not because the stories themselves are too facile—they can be just as sophisticated and challenging as those of any other genre—but because they often revel in easy solutions: Why walk when you can warp? Why talk when you’re a telepath? Technology in such stories typically has more to do with workarounds than it does with work.

I do love science fiction. From robots/AI to star travel to virtual reality. I love it all. I may even love it BECAUSE of the laziness. I’d love to have all these things to make my life better. And much of science fiction influences technologists into making decisions to make the fiction a reality.

The How Shatner Changed the World (mock) documentary talks about the technologies of Star Trek and how scientists work towards making these things reality. Faster than light travel and cybernetics are still aspirant. But cell phones and personal computers were influenced by technologists familiar with the show and movies.

At times I worry about automation putting me out of a job, but then I remember my career goal is always to replace myself with a tiny shell script. Why click when I can script? Why script when I can tell an AI to handle it? Sure it takes away some of my responsibilities, but what I am supposed to do has always changed. And I get better challenging work when I free myself from mundane tasks.

Guess this is why I told Puppet Labs my job is an Automation Evangelist. It’s not universal. I have allies, but convincing people of the good in automation is much like changing their religion.

Back in college I was encouraged to become a librarian. More specifically, people thought I should become an automation librarian. I guess the automation part stuck?

10.2 Upgrade

Later this week both of our production systems will get upgraded from Desire2Learn 9.4.1 to 10.2. This will be an epic effort. Over several days a few dozen people will work on various aspect of the databases, application servers, and testing. Good ‘ole clean IT fun just in time for Christmas. Finding the magical gap between classes and end of term was hard enough when our upgrades took a day. These multi-day ones are much tougher.

Back at FUSION a coworker and I attended a presentation by the team lead for D2L’s Automation group. Thankfully the excellent work from there will be used as part of the upgrade to identify problems earlier. Also, our client testing appears to take a leap forward to automating testing. (We need to take a light year leap forward with application level monitoring.)

Six months of work will be complete.

Then we get to turn around and do it again for May.

Why Ten

The question of why we run ten clusters came up recently. Off the top of my head, the answer was okay. Here is my more thoughtful response.

Whenever I have been in a conversation with a BEA (more recently Oracle) person on Weblogic, the number of nodes we run has invariably surprised them. Major banks serve ten times the number simultaneous users we have on a half dozen managed nodes or less. We have 130 managed nodes for production. Overkill?

There are some advantages they have.

  1. Better control over the application. WebCT hacked together an install process very much counter to the way BEA would have done it. BEA would have had one install the database, the web servers, and then deploy the application using either the console or command-line. WebCT created an installer which does all this in the background out of sight and mind of the administrator. They also created start and stop scripts which do command-line interaction of Weblogic to start the application. Great for automation and making it simple for administrators. It also lobotomies the console making many advanced things one could normally do risky. So now the console is only useful for some minor configuration management and monitoring.
  2. Better control over the code. When there is a performance issue, they can find what is the cause and improve the efficiency of the code. The best I can do is point out the inefficiencies to a company who chose as a priority a completely different codebase. If you do not have control over the code, then you give the code more resources.
  3. As good as Weblogic is at juggling multiple managed nodes, more nodes does not always equal better. Every node has to keep track of the others. The heart beats communicate through multicast. Every node sends out its own and listens for the same from all the others. Around twenty nodes they would miss occasional beats on their own. Thrown in a heavy work load and an overwhelmed node can miss enough missed beats it becomes marked as unavailable by the others. Usually at this point is when the monitors started paging me about strange values in the diagnostics. Reducing the number of nodes helped.

More resources means more nodes. We had two clusters with about 22 nodes (44 total) each when we hit a major performance wall. They were split into four clusters with 15 nodes each (60 total). Eventually these grew to over 22 nodes each again. At this point upgrading was out of the question. A complete overhaul with all new databases and web servers meant we could do whatever we wished.

The ideal plan was a cluster per client. Licenses being so expensive scrapped that plan.

Ten clusters with 13 managed nodes each was a reasonable compromise. More nodes while also using smaller clusters achieved both needs well. Empty databases also gave us a better restarting point. The databases still have grown to the point certain transactions run slowly just for 4 terms later. (I was hoping for 6.) Surviving the next two years will be a challenge to say the least. I wish we got bonuses for averting disasters.