Content Migration Progress Tracking

When moving hundreds of thousands of courses between WebCT Vista and Desire2Learn, keeping track of what made it through which stage seems like an obvious hindsight thing to do. I added that last bit because we started to notice where things fell between the cracks starting to pile up. The basic process…

    1. Through Oracle SQL we populate a middleware database with those courses which meet acceptable criteria.
      CRACK: With the wrong criteria, courses are not even selected.
    2. Through Oracle SQL we generate XML listing in 50 count sets of the courses.
      CRACK: A subset of data loaded into the database may be extracted.
    3. A shell script automates running the WebCT command-line backup process to create a package for each course.
      CRACK: The command-line backup fails on some courses.
    4. Desire2Learn scripts pick up the files and convert WebCT formatted packages to Desire2Learn.
      CRACKS: Too big fail. Too long paths fail. This step can fail to create CSV files for the next step.
    5. Converted packages are placed in a queue to be imported into Desire2Learn.
      CRACKS: WebCT Vista courses can have 1,000 characters in the name and D2L fails if there are more than 50. Courses named the same as a previously loaded one but with a different file name loads both into the same course.

So, there are apparently five different stages and eight potential failures to track and no end-to-end tracking to even know what is missing. Which means inventing something to check logs for errors. 

First thing, I tried writing SQL to create an ordered list of the courses that are available.

The WebCT backups were a little tougher to convert into a useful list. The file names follow the format of Course_Parent_1234567890.bak. They were also in various directories, so I ended up doing something like this to get a list of the files, strip off the parents & time stamps, strip off the directories, and order it.

ls */*.bak | awk -F_ ‘{print $1}’ | awk -F\/ ‘{print $2}’ | sort

So I have two ordered lists. Anything in the D2L one and not in the WebCT one ought to be the work of clients making their own stuff. Anything in the WebCT one and not in the D2L one ought to be my missing ones. Only almost every line is a mismatch.

Visually comparing them, I realized the same courses had in effect different names. All the spaces were converted to underscores. So I ended up adding a sed to convert spaces to underscores after the sort.

Then I wrote some more scripts.

    • Go through the logs from #4 and #5 to display the errors. With it I was able to compare my list of missing with the errors and confirm why they did not come through.
    • Some of the cracks can be addressed by making new import files. So I wrote a script to add those lines to a redo.csv file. Touch up the problems and go.

Basically at this point it only covers 3-5. At some point I am going to have to check steps 1-5. 

Digital Legacy

A book on time management in talking about long-term goal planning suggests we define the legacy we wish to leave. Coming from academia, I typically think of a legacy as a name on a building, an applicant with an alum for a parent, or a scholarship. However, the artifacts left behind by previous cultures are also a legacy.

Our digital footprints both could be part of this legacy or easily lost. I lean toward all this data we spew about the Internet will be lost eventually. I have seen floppy disks and hard drives die, taking with them the only copy of critical data. I have seen companies report their hard drives stolen from their machines in co-location as why customers lost their data. I have seen companies close web sites because they ran out of money. Let’s not forget natural disasters like earthquakes and floods.

So we keep backups.

Who will preserve these backups once we are gone? Are you able to read the data from computers 40 years ago? Maybe we’ll be better about being able to read the data from past when we reach 40 years into the future?

Not likely.

CE / Vista Undocumented Workspaces

On the WebCT Users email list (hosted by Blackboard) there is a discussion about a mysterious directory called unmarshall which suddenly appeared. We found it under similar circumstances as others by investigating why a node consumed so much disk space. Failed command-line restores end up in this unmarshall directory.

Unmarshalling in Java jargon means:

converting the byte-stream back to its original data or object 1

This suspiciously sounds like what a decryption process would use to convert a .bak file into a .zip so something can open the file.

This is fourth undocumented work space where failed files site for a while and cause problems and no forewarning from the vendor.

Previous ones are:

  1. Failed UI backups end up in the weblogic81 (Vista 3, does this still happen in Vista 8?) directory.
  2. Failed tracking data files end up in WEBCTDOMAIN/tracking (Vista 3, apparently no longer stored this way in Vista 4/8 according to CSU-Chico and Notre Dame)
  3. Web Services content ends up in /var/tmp/ and are named Axis####axis. These are caused by a bug in DIME (like MIME) for Apache Axis. No one is complaining about the content failing to arrive, so we presume the files just end up on the system.

#3 were the hardest to diagnose because of a lack of an ability to tie the data back to user activity.

Is this all there are? I need to do testing to see which of these I can cross off my list goring forward in Vista 8. Failed restores are on it indefinitely for now.
🙁

References:

  1. http://www.jguru.com/faq/view.jsp?EID=560072

Using An LMS As a Network Drive

Ran across a video describing how to get the WebDAV info in CE6 (aka Blackboard Vista 4 Lite) for the purpose of using CE6 as a network drive.

The narrator says this is a good idea because if the site has good policies, then backups are being made. In the event of a site disaster, you can recover your files from it.

This is a HORRIBLE idea.

  1. An LMS is unlikely to be sized in such a way to store backups of all user content. The IT administration will end up buying more expensive storage for CE6 than for other desktop backup solutions.
  2. By placing content unrelated to classes, you will contribute to making the CE6 site slower.
  3. The IT administration will not be able to recover a single file for you should you make a mistake. They will have to restore the whole database, place a CE6 web server in front of it, and get the one file for you. Its a more expensive investment in time to recover your content.

Use a backup system to do backups. Use a online instruction system to instruct.

RRRv302 REPOST: Did A Stupid Thing

Originally posted January 15, 2004.

v4 means its the fourth incarnation of this blog. This post was in v3. Thankfully, MovableType writes the content to files meaning there is a lasting archive. That reminds me… Need to put on the calendar to do regular backups of this blog.

Back to the post:

Did A Stupid Thing

No, really stupid. About the stupidest thing I have done in years. So stupid that it proves my idea that I am the luckiest person ever. No, really!

Soft sand cannot support weight. Small Japanese cars do not have the traction or capability of getting out of it. So even to attempt to turn around on a dirt road where there is a gate and soft sand between is about the dumbest thing ever. I knew it before I tried it. However, I had just spend 1/2 an hour driving around the middle of nowhere to find a friend’s house using only my recollection of the directions and map from several days earlier.

So my car was stuck and my efforts to get it out probably were only making it worse. This is the worst part in terms of my abject stupidity. Decided to ask for help. Walked down the dirt road to the first house and knocked on the rail leading to the mobile home. The people inside obviously did not hear so I stomped on the step and yelled an inquiry as to whether anyone was home. The lady of the house turned on the light, saw me, and freaked. Her husband was not quite as skittish, but still pretty nervous as he asked me to show him my hands and inquired about the availablity of weapons on my person and car. His dad next door had a tractor that might be able to help so he went to ask. While waiting for his dad, he asked all kinds of more questions.

I did lie here. Told him that I was visiting my friend earlier. On my way home, I realized I left something at the house and was going to turn around to go back over there. Because of the hills and speeds of cars on the highway, I didn’t want to turn around in the middle of the road (this sentence is true). Didn’t want to let these people know that I was out here essentially kind of lost and helpless.

The dad came, asked more grilling questions and asking for specifics. Stuff like my name, where I live, where I work, what I do. Turns out he has a web site. They pushed the car enough back with it in reverse that it was able to get traction. I turned around in their driveway and went home.

Called my friend to let him know what happened and that I am okay.

In talking with one of my assistants this morning, I found out that someone had used a ploy of asking for help to murder a family not all that far from there. Their timidity was certainly understandable. Their bravery in assisting me seems so much more impressive. The guy could have seen me there, killed me and probably not gone to prison. That is why I am still the luckiest guy walking on this planet.

Posted by Ezra at January 15, 2004 11:27 AM

Comments

Good story, though. Why didn’t you go to their front door and knock?

Posted by: lacey at January 15, 2004 05:58 PM
Knocked on the railing because I didn’t want to be extremely close to the door. As a black male, I know that white people tend to be a little skittish around “my type”. Them looking out the window and seeing me right there might have been a little nerve wracking.

I did notice that the husband stayed in his truck quite a bit with his right hand on the seat. I really think he might have had something there to take care of me should I have been a threat.

Posted by: ez at January 15, 2004 06:15 PM

Quotes to Make You Think

These also make great signature files! Additional collected quotes are under the blog’s quotes category. Drop a comment to suggest something.

Organized alphabetically by first major word.

Anonymous/Unknown Author

A

Accomplishing the impossible means only the boss will add it to your regular duties.

According to my calculations, the problem doesn’t exist.

Adult: A person who has stopped growing at both ends and has started growing in the middle.

Advice is free: The right answer will cost plenty.

After all is said and done, more is said than done.

All the darkness in the world cannot extinguish the flame of a single candle.

Almost anything is easier to get into than out of.

Always remember that you’re unique. Just like everyone else.

Always try to be modest. And be darn proud of it!

Ambition is a poor excuse for not having enough sense to be lazy.

B

Be nice to your kids. They’ll choose your nursing home.

Before you criticize someone, you should walk a mile in their shoes. That way, when you criticize them, you’re a mile away and you have their shoes.

The best part about procrastination is that you are never bored because you have all kinds of things that you should be doing.

Blessed are the pessimistic for they hath made backups.

By the time a man realizes that maybe his father was right, he usually has a son who thinks he’s wrong.

C

Children may close their ears to advice, but they keep their eyes open to example.

A clean desk is a sign of a cluttered desk drawer.

Cleaning the house while your kids are still growing is like shoveling the walk before it’s stopped snowing.

A clear conscience is a sign of a bad memory.

A company is known by the people it keeps.

A conclusion is the place where you got tired of thinking.

Confidence is the feeling you sometimes have before you fully understand the situation.

A conscience is what hurts when all your other parts feel so good.

The customer is always right but not always pleasant.

D

Don’t be irreplaceable; if you can’t be replaced, you can’t be promoted.

Don’t hate yourself in the morning – sleep till noon.

Duct tape is a lot like the Force. It has a Dark Side, it has a Light Side, and it binds the galaxy together…

E

The early bird may get the worm, but the second mouse gets the cheese.

Endeavor to so live that when you die even the undertaker will be sorry.

Ever stop to think, and forget to start again?

Every time I think I’ve hit the bottom, someone lends me a shovel.

Everyone seems normal until you get to know them.

Experience is a hard teacher because she gives the test first, the lessons afterward.

Experience is something you don’t get until just after you need it.

F

Few women admit their age, Fewer men act it!

A filing cabinet is where papers get lost alphabetically.

Four-word story of failure: Hired, tired, mired, fired.

G

Generally speaking, you aren’t learning much when your lips are moving.

Go the extra mile. It makes your boss look like a slacker.

A good test for conversation: if you wouldn’t write it and sign your name to it, don’t say it.

Good resolutions are often checks drawn on an account with insufficient funds.

The great essentials of happiness are something to do, something to love and something to hope for.

H

Hard work has a future payoff. Laziness pays off NOW.

The Haves and the have-nots can often be traced back to the Dids and the Did-Nots.

The highest reward for man’s toil is not what he gets for it but what he becomes by it.

How many of you believe in telekinesis? Raise my hand….

I

I don’t exaggerate. I just remember bigger than most people.

I don’t suffer from insanity, I enjoy every minute of it!

I have a mind like a steel trap – rusted open.

I haven’t lost my mind. It’s backed up on tape somewhere.

I pretend to work. They pretend to pay me.

If at first you don’t succeed, destroy all evidence that you tried.

If at first you don’t succeed, give up, no use being a damn fool.

If at first you don’t succeed, skydiving is not for you.

If both of us thought alike, one of us would not be necessary.

If everything seems to be going well, you have obviously overlooked something.

If nothing ever sticks to TEFLON, how do they make TEFLON stick to the pan?

IF Pro is the Opposite of Con, What Is The Opposite of Progress? Congress!!

If you’re going in the wrong direction, God allows u-turns.

If you ask a question you don’t want an answer to… expect an answer you don’t want to hear.

If you don’t learn from your mistakes, what’s the point of making them?

If you find a path with no obstacles, it probably doesn’t lead anywhere.

If you tell the truth, you don’t have to remember anything.

If you think nobody cares if you’re alive, try missing a couple of car payments.

If you were somebody else, would you want to be friends with you?

Indecision is the key to flexibility.

It has been determined that research causes cancer in rats.

It is harder to conceal ignorance than to acquire knowledge.

It is when we forget ourselves that we do things that are most likely to be remembered.

It takes less time to do a thing right than to explain why you did it wrong.

It will do no good to get on the right track if you are headed in the wrong direction.

J

Just because someone doesn’t love you the way you want them to doesn’t mean they don’t love you with all they have.

K

Keep your ideals high enough to inspire you and low enough to encourage you.

L

Language is a wonderful thing. It can be used to express our thoughts, to conceal our thoughts, or to replace thinking.

Laugh alone and the world thinks you’re an idiot.

Leaders go down in history — some farther down than others.

Leadership has been defined as the ability to hide your panic from others.

Life is a sexually transmitted disease with a 100% mortality rate.

Life is complex: part real, part imaginary.

Love may be blind, but marriage is a real eye-opener.

M

Many people quit looking for work when they find a job.

The measure of man’s real character is what he would do it he knew he would never get caught.

The mistake a lot of politicians make is forgetting they’ve been appointed and thinking they’ve been anointed.

The more people I meet, the more I like my cat.

The more you know, the more you know you ought to know.

N

Never miss a good chance to shut up.

No matter how far money goes, it still can’t go as far as the next paycheck.

No matter how much I care, some people just don’t care back.

No one is listening until you make a mistake.

O

Of all the things you wear, your expression is the most important.

One of the mysteries of life is how the boy who wasn’t considered good enough to marry the daughter can be the father of the smartest grandchild in the world.

One reason folks get into trouble is that trouble usually starts out being fun.

One trouble with the world is that so many people who stand up vigorously for their rights fall down miserably on their duties.

The only time the world beats a path to your door is if you’re in the bathroom.

P

People who do the world’s real work don’t usually wear neckties.

People will accept your ideas much more readily if you tell them that Benjamin Franklin said it first.

Poor planning on your part does not constitute an emergency on my part.

Practice makes perfect, so be careful what you practice.

A professor is one who talks in someone else’s sleep.

The quickest way to get a lot of undivided attention is to make a mistake.

People who wonder where the younger generation is headed for would do well to consider where it came from.

The problem with the gene pool is that there is no lifeguard.

Q

R

The really productive ups and downs are getting up in the morning and down to work.

The reason we never see ourselves as others see us is that we’d never believe what we saw.

A reckless driver is a person who passes you on the highway in spite of all you can do.

S

Save time… see it my way.

Some people get lost in thought because it’s unfamiliar territory.

Some people spend the first six days of each week sowing wild oats; then go to church on Sunday and pray for a crop failure.

The sooner you fall behind, the more time you’ll have to catch up.

A sure-fire formula for making a good speech: have a good beginning and a good ending–and keep them as close together as possible.

To steal ideas from one person is plagiarism; to steal from many is research.

A synonym is a word you use in place of one you can’t spell.

T

Tact is the ability to close your mouth before somebody else wants to.

There can’t be a crisis today, my schedule is already full.

There is absolutely no substitute for a genuine lack of preparation.

There is nothing so comforting as the patter of little children’s feet about a home, because the moment the sound stops one knows that they are up to something they shouldn’t be.

There is nothing that upsets a person quite as much as having company drop in and see the house looking as it usually does.

There needs to be fewer Republican Senators and fewer Democratic Senators, and many more United States Senators.

Thinking is when your mouth stays shut and your head keeps talking to itself.

This isn’t an office. It’s Hell with fluorescent lighting.

Those who complain about the way the ball bounces are usually the ones who dropped it.

Time may be a great healer, but it certainly is no beauty operator.

Trains stop at train stations
Buses stop at bus stations
On my desk is a workstation.

A true friend is one who thinks you’re a good egg even though you’re slightly cracked.

U

Usually I try to take it one day at a time, but lately several have attacked me at once…

V

W

Warning: Dates in Calendar are closer than they appear.

We hang the petty thieves and appoint the great ones to public office.

What most people want these days is less to do, more time to do it, and more pay for not getting it done.

When adults act like children, they are called silly. When children act like adults, they are called delinquent.

When computing, whatever happens, behave as though you meant it to happen.

When our children are old enough not to say or do anything in public to disgrace us, they have reached an age when the things we do and say embarrass them.

A wise man learns from the mistakes of others, a fool by his own.

Why is ‘abbreviation’ such a long word?

Wild horses couldn’t drag a secret out of most women. Unfortunately, women seldom have lunch with wild horses.

With every right, there is a responsibility. Just once, I wish someone would demand his responsibility.

A woman wants one man to meet her every need. A man wants every woman to meet his one need.

A woman will pay $1 for a $2 item she doesn’t want. A man will pay $2 for a $1 item he DOES want.

Work is the easiest thing man has ever invented to escape boredom.

Work like you don’t need money,
Love like you’ve never been hurt,
And dance like no one’s watching.

X

Y

Years wrinkle the skin, but lack of enthusiasm wrinkles the soul.

You are only young once. After that, you have to think up some other excuse.

You can’t keep people from having bad opinions about you, but you can keep them from being true.

You cannot build a reputation on the things you are going to do.

You cannot make someone love you. All you can do is be someone who can be loved. The rest is up to them.

You only live once, but if you live right, once is enough.

Z

Misc.

Comment from a correspondent in Australia on the U.S. political scene: “Thank God you got the Puritans while we only got the convicts.”