Listening to the Radiolab Bit Flip episode. If you haven’t listened, then you really should first.
Hire the people who will automate themselves out of a job, then just keep giving them jobs.
— jessie frazelle (@jessfraz) December 16, 2017
At VSU, my boss got a promotion when his boss retired. Then a shuffling of jobs gave me a promotion, but it also sent my old job to another group. That left the web services group going from 2.5 full-time positions down to 1.5.
We did not have that much free time. If anything in the early 2000s, the responsibilities were growing which is how we ended up with 2.5 people. My only way to restore sanity was by automation. Admittedly, I love scripting and schedulings, so my approach to things at that time was to write scripts to handle jobs. The change gave more motivation to ensure that anything that could be automated was. Or I would drown in the work.
What made it hard was, even as I automated these jobs, more things were coming to the web. The needs grew faster than I could develop the tools to handle it. It was a fantastic experience, though.
For a long while, I have thought Gmail was smart enough to see emails I receive and make a calendar entry. Apparently, the truth is I forgot about creating an IFTTT applet to look for emails with the group subject tag and make a calendar entry for me.
It worked so well, that I guess I had no reason to pay attention.
With this specific application, we can import data, but there are limitations due to its 2000-era handling of XML files.
A better method would allow just uploading the files to a page. Background processes would monitor that location and process the files independent of the browser. Notifications can be sent to alert the user the processing is done.
Or… recognize the echo XML file because you took too long and prevent it from being loaded or remove the data.
I ended up figuring out that if we split the files at about 5,000 records, then it should take about half the 5 minute window. I am pleased that for most I have seen that is true and about one in ten take so much longer that if I had cheated and gone with closer to the 5 minutes, I would be deleting duplicates. (This last is because some files are 50MB and others 100MB.)
The grumbling of this post is that I am on the 25th of 58 files. This is tedious. I am lamenting not creating a curl script to do this part for me. Automation is perfect for things like this.
Automation is here. When I was in college, for most of my time there, I thought Industrial-Organizational Psychology was my career path. It was not until my last semester when I started working as a student for IT that it all changed. Little did I know that people efficiency was dying. Automation, aka the efficiency of technology (computers and robots), is where industry is making its money. Or that really the anger boistering both the Republican and Democratic presidential front-runners is due to automation. People are frustrated with the economy and how it affects them.
Edward Elser explains why:
Wages no longer follow productivity.
A whole lot of people are unhappy. Some people blame the soulless corporation. Others blame workers who want $15 an hour for a marginal job. But, here is the deal. Here is where I think the root of the discontent is: the system does not need unskilled labor anymore. Industry simply does not need unskilled or marginalized workers.
It used to be when productivity went up, so did wages. But somewhere in the 1970s a disconnect happened. The value of labor stopped following productivity. And it is my opinion this trend is not going to change; it is going to accelerate..