Rote Loading

With this specific application, we can import data, but there are limitations due to its 2000-era handling of XML files.

  1. HTML forms uploading files have to…
    1. Have all the packets be received by the server.
    2. Process the file with the browser connection still open.
    3. The server has to tell the browser everything was received and is done.
  2. All this has to happen as one action within a 5 minute window.

A better method would allow just uploading the files to a page. Background processes would monitor that location and process the files independent of the browser. Notifications can be sent to alert the user the processing is done.

Or… recognize the echo XML file because you took too long and prevent it from being loaded or remove the data.

I ended up figuring out that if we split the files at about 5,000 records, then it should take about half the 5 minute window. I am pleased that for most I have seen that is true and about one in ten take so much longer that if I had cheated and gone with closer to the 5 minutes, I would be deleting duplicates. (This last is because some files are 50MB and others 100MB.)

The grumbling of this post is that I am on the 25th of 58 files. This is tedious. I am lamenting not creating a curl script to do this part for me. Automation is perfect for things like this.

 

TED Talk: Our Future is Awesome

Automation is here. When I was in college, for most of my time there, I thought Industrial-Organizational Psychology was my career path. It was not until my last semester when I started working as a student for IT that it all changed. Little did I know that people efficiency was dying. Automation, aka the efficiency of technology (computers and robots), is where industry is making its money. Or that really the anger boistering both the Republican and Democratic presidential front-runners is due to automation. People are frustrated with the economy and how it affects them.

Edward Elser explains why:

Wages no longer follow productivity.

A whole lot of people are unhappy. Some people blame the soulless corporation. Others blame workers who want $15 an hour for a marginal job. But, here is the deal. Here is where I think the root of the discontent is: the system does not need unskilled labor anymore. Industry simply does not need unskilled or marginalized workers.

It used to be when productivity went up, so did wages. But somewhere in the 1970s a disconnect happened. The value of labor stopped following productivity. And it is my opinion this trend is not going to change; it is going to accelerate..

Our Future is Awesome | Edward Elser | TEDxAugusta