Search Within Files

I keep some logs in a directory just in case I need to reference them later. The kind of data that has saved my bacon on a handful to times. Of course, it has over 3,000 files (16GB) in the directory. Of those less than a hundred were potentially relevant. And in the end only a couple dozen had the data I sought.

Windows Explorer used to make this easy to search. I could put in the pattern I wanted and tell it to search the text within them. It would give me the file name for each containing the search string. For whatever reason Windows 7 had to make it more difficult.

So, I wrote the easiest of Powershell scripts:

$filelist=D:\path\to\*files*.log
Get-Content $filelist | Select-String -pattern “Search String

Good thing too because apparently I need to to go through my Indexing Options and identify every file extension I want to search to index file contents. What a royal pain. My guess is doing so would also blow up the EDB data file from its currently 2GB to something way larger. 10GB? 50? 100? Yuck.

 

Windows Module Installer

A pain in my side over the past year finally forced me into addressing it. Windows Module Installer runs as TrustedInstaller.exe and for most cases just does its job which is to keep in touch with the Windows Update service and apply the updates sent to it.

Occasionally they develop a memory leak and consume RAM until someone intervenes. We have about 140 servers. About 22 over the past two months about 20 showed this behavior. Only when it uses about 2GB of the 10GB we allocated to these servers do I usually have to intervene. That has been about 3 times over the past 2 months and ten over the past year.

Using Yaketystats to see the trend was far worse than I had noticed, I decided we needed to do one of two things.

  1. Shut it down. Start them when we need them. Shut them down again when we do not. Benefit is we do not have to worry about them getting out of control consuming resources. Unfortunately those wanting to push out updates will have to add a step to start them before pushing them.
  2. Recycle. Routinely shut them down and start back. Relatively easy to automate, so set it and forget it. Recycle

Well, it gets much worse. First, running the commands work inconsistently. For example, I ran

Set-Service TrustedInstaller -startuptype “Automatic”

against every host in a development system. As is my habit, I ran a check to make sure it worked. It did on two of the five. So I ran it again. The other three were fixed. So I did that same process on another development system with five hosts. Three of the five worked the first time and the other two the second. The pattern held true for another three systems all with five servers each. Setting the startuptype to Manual worked the same inconsistent way.

My check:

Get-WmiObject -ComputerName $computer win32_service -Filter “name = ‘trustedinstaller'”

Second, stopping and starting them does not appear to stick. Several minutes after I have stopped all of the services they appear to back in the prior state. Those who were not running stay not. Those who were running are again. And if I start all of them, then at some point those who were not running stop again.

Guess I have a lot of research ahead of me. 🙁

Just Get Rid of Java

Apparently there are security flaws in the current version of Java allowing the installation of malicious software through web browsers unknown to the user. The known attacks using this flaw work on Windows, OSX, and Linux. According to Reuters:

Java was responsible for 50 percent of all cyber attacks last year in which hackers broke into computers by exploiting software bugs, according to Kaspersky. That was followed by Adobe Reader, which was involved in 28 percent of all incidents. Microsoft Windows and Internet Explorer were involved in about 3 percent of incidents, according to the survey.

The Department of Homeland Security recently said computer users should disable Java. At first this seems odd. The vulnerability in question is only in Java 7. So why not go back to Java 6? Well, Java 6 has vulnerabilities too, which is why DHS and others have recommended getting to 7. Also, starting in 7, the automatic upgrades are more aggressive. So going backwards is probably not a great idea. (If just happens I had to go backwards to get a tool I needed to work and forgot to go back forward.)

Also, for a similar situation back in August the recommendation was to make the browser prompt before allowing Java to run. The strategy is just stop Java entirely. Apple has removed Java browser plugins. That could work too. Except for bad, bad software like ours (sorry, sarcasm if you could not tell) which makes use of a few applets. In the last week I have gotten a request to add another applet.

A fix to Java 7’s vulnerabilties should be available in a couple days.

OrgCode Duplicate Filter

I was asked to work my “Unix magic”.

The problem? Duplicate courses were spooled and converted from the WebCT format to the Desire2Learn. The conversion process creates an import file using the WebCT SourcedId.Id as the OrgCode. The first time the OrgCode is used, it creates a course. The next and subsequent times, it duplicates content. So these duplicate converted courses gave us a situation where we were screwed.

Fortunately our partners at Desire2Learn intercepted the problem before it got worse.

Out of 1,505 still to be imported, there were 468 duplicates. Yes, 31% duplicates.

D2L asked me to filter the imports to remove the duplicates. I said I am too much of a n00b with Windows to pull it off.

The reply was to use Unix.

Boy do I love Bash shell scripting. In two hours I solved it, though after the high of solving something I had no idea how to write this morning in two hours, there must be something wrong with it.

First, my general idea was to read the file line by line and write those lines with OrgCodes that do not yet exist to a filtered.csv file. I started out looking to exactly duplicate my existing file in another file by reading it line by line.

A while loop which reads each line and records the whole line in a variable.

INPUTFILE=/path/to/file.txt
exec<$INPUTFILE
while read LINE
do
     stuff...
done

I quickly discovered though that since Windows uses the backslash, that foiled the ability of echo to exactly write every line to a file. The backslash escapes the next character. Neither double nor single quotes helped the situation. Oops. So I decided to use sed to make a temporary copy to duplicate the backslashes. A first backslash escapes the next character, in this case a second backslash.

sed -e 's|\\|\\\\|g'

As an error check, the last thing the script does is a diff -u to compare source and new files. At this stage nothing means perfect. I like the -u to give me easier to read results.

So I was able to get an exact copy of the original. All that was next was to get the OrgCode, check it against my filtered file, and if it did not exist, then add it to the end of the filtered file.

ORGCODE=`cat $LINE | awk -F, '{print $1}'`
IF_EXISTS=`grep $ORGCODE $FILTERFILE`
if [ -z IF_EXISTS ] ; then
     echo $LINE >>  $FILTERFILE
fi

Easy. Too easy?

The checks against my work confirmed it worked.

    1. A sorted version of the source run through this and compared in diff -u consistently showed the correct lines were excluded.
    2. Counts for the number of duplicates and the difference of lines missing works.
    3. A check for the number of duplicate OrgCodes returns nothing on the filtered file.

Pick Up Line

(I will never use.)

My name’s Vista. Can I crash at your place tonight?

Noticed at geekpickuplines.

Especially funny for me because the product I run is the Blackboard Learning Management System Vista Enterprise. We just call it “Vista”. (Yes, very confusing when Windows Vista users want to know the compatibility of Vista with Vista. The answer: barely.)

Supported

(This is an post I wrote back in November but didn’t publish…. Until now. Have fun!)

Mitigated speech gets a lot of use by people trying not to offend. All too often, people who have been hurt because of mitigated speech question what isn’t being told as though the omission or gaps are intentionally deceptive.

What are or are not supported browsers came up again. The trick here is the mitigated speech used with the levels of support. I assume the intent is clarity.

  • Certified – supported with complete testing done.
  • Compatible – supported with some testing done.
  • Provisional – supported with some testing done before official release.

Certified is taken as supported by all parties. Compatible and Provisional are interpreted as not supported because the complete testing has yet to be done. I think Blackboard’s intent was to mark them as supported but qualify how customers might encounter issues due to not fully testing. This means Blackboard is interested in learning about the problems encountered in order to address them.

At least that is my interpolation. Mmmmmm the Kool-Aid is good.

Linux Adventure Part 2

Linux Adventure Part 1Linux Adventure Part 3 [SOLVED]

So far into the story, I tried repairing Windows Vista which failed to actually give me a working entry into the operating system. The Linux Live CDs were non-committed forays into Knoppix, CentOS, and Ubuntu. All failed to turn on the wireless. An ethernet cord would have gotten me online.

So I was stuck with pretty much a brick.

My next step was to venture out to the store and buy a hard drive. The Ubuntu CD included an installer, so I used it to install a local copy. Continued research revealed my problem probably was the fact my computer came with a Broadcom 4312 card. (My brother said my problem was trying use wireless with Linux.)

Without an ethernet connection, I ended up installing Linux STA drivers from source by downloading them and transferring them by FTP.  No good. Multiple times. I never got it to recognize them. Other options called for installing a firmware update on the wireless card. The idea of a firmware update to the wireless card making me stuck on Linux worries me.

Thankfully I got home to where I have ethernet cords. By this point, I had so completely hosed things, so I reinstalled Ubuntu to start over fresh. Now seeing the Internet through the LAN, Ubuntu offered me “restricted” hardware drivers. The b43 set didn’t do anything. The STA set did enable the Wireless option. Even dhclient referenced eth2! However, the wifi status light doesn’t turn on when I enable wireless. Ugh. So the drivers work better but not enough to get it working.

Also, (based on the time stamp of the file I was able to find in a backup of the problem laptop) I haven’t connected a computer to my home network since February, so I didn’t remember what was the password for the network. Finding which computer or external drive contained the information took a few hours. Yay for backups.

Linux Adventure Part 1Linux Adventure Part 3 [SOLVED]

Linux Adventure Part 1

For about a week now I’ve been without my personal laptop as anything much more than a brick. I think tonight I am going to copy off the pictures and other important information to my desktop. From there, anything I do to make the situation worse will no longer matter as much.

Monday night, I shutdown the laptop. Microsoft Vista Automatic Updates said it was working on some updates post-logout. Rather than babysit, I went to bed. I should have babysat it.
🙁

The next morning, Tuesday, starting the computer told me I had a corrupted or missing \boot\BCD. The Boot Configuration Data file is pretty important, as without one the Windows operating system doesn’t even give me a command prompt. After some research I found out I needed my Windows installation DVD only 250 miles away. This caused me so much distress I even forgot I had a spare computer with me.

So I decided to download a Linux Live CD and use that while stuck away from home. At least I would be able to research the problem and possibly fix it later. The first Live CD I tried was a downloaded iso flavor called Knoppix, I remembered from many years ago. Ick. Knoppix Adriane is intended for the visually impaired slipped by me, so the computer reading everything got annoying extremely quickly. Finally turned off the reading stuff, but I had a new problem. Wireless wasn’t working.

Macintosh LC III … And I was out of CD-Rs.

So a newer memory was a few years ago, a friend with a barely functioning Macintosh LC III (pictured right) wanted to get her stuff off it. She brought it up again a few times since, the most recent occasion to ask me to explain why her Windows computer cannot just read 3.5″ floppies from the Mac without any computer-ese. A coworker mentioned a Live CD of CentOS could mount the drive and transfer the data.

So, I downloaded an iso of the CentOS Live CD while I went to the store to get some disks to burn. While starting up CentOS, I downloaded Ubuntu just in case this second Live CD failed. It was a good thing because the CentOS Live CD was prettier without any improvement in getting on the wireless.

Nor was the Ubuntu Live CD any better.

By this point, I had found a site offering a torrent to a Vista Recovery CD. The quandary was to go back to Windows or stick with Linux. The recovery CD off a random web site could just not work or at worst infect the non-functioning computer. So I installed BitTorrent and downloaded the recovery CD. I tried the Startup Repair, System Restore, and Command Prompt (to manually rebuild the booter). Since this failed, I decided Windows Vista was dead.

So I started looking into how to make Ubuntu work for me.

Linux Adventure Part 2Linux Adventure Part 3 [SOLVED]

Odd Tracking File Recording

Every time a Vista 3 node is shut down without going through the initiated shut down process, there is a chance of incorrect data written to the tracking files (in NodeA/tracking/). Normally it leaves strange characters or partial lines at the end of the file. This is the first time I have seen it write the contents of another log instead of the tracking data.

click – 1.0 – 1244228052889 – 1135588340001 – “nova.view.usg.edu-1244227762853-6288” – SSTU – discussion – “compiled-message-viewed” – “page name” – 558711383 –

click – 1.0 – 1244228052891 – 15.0; .NET CLR 1.1.4322)”

2009-04-23      20:58:35        0.0030  xxx.xxx.xxx.xxx    JxH1zg4fZT1LTGcpmyNW    200     GET     /webct/libraryjs.dowebct        locale=en_US    0       “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)”

Even better. The node went down on June 5th at around 3pm. The lines from the other log were from April 23rd at 8:58pm.

Why am I surprised to see new incorrect behavior? Especially when the node was really confused?