Tuesday, December 16, 2008

tunneling ssh through proxy server

I needed to create a secure SSH connection to my Fedora box at home. However, I am behind a proxy server at work and SSH doesn't have switches that allow it to get through a proxy. So in combination with ntlmaps and the corkscrew program, Open Source helped me do this in a quick and easy way.

Here's a sample .ssh/config file that works for corkscrew:
sodo@linux-z6tw:~> cat .ssh/config
ProxyCommand /usr/local/bin/corkscrew 5865 %h %p

Note the This is because ntlmaps is listening on all ports:
sodo@linux-z6tw:~> NTLM authorization Proxy Server v0.
Copyright (C) 2001-2004 by Dmitry Rozmanov and others.
Now listening at linux-z6tw on port 5865

sodo@linux-z6tw:~> netstat -na | grep "LISTEN "
tcp 0 0* LISTEN
tcp 0 0* LISTEN
tcp 0 0* LISTEN
tcp 0 0* LISTEN
tcp 0 0* LISTEN

Here is the corkscrew home page:

Here is a great instruction set for corkscrew:
and here is a second link, just in case:

Now I am able to securely connect and grab files from my home PC. Yeah!

Wednesday, October 29, 2008

helpful svn commands

Here are some useful subversion commands I'd like to remember. But instead, I'll write them down, cause I'm always forgetting.

The simplest being the ubiquitous "checkout the latest version of the source", using ffmpeg as the example:
svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg

Here's an example of a command to checkout a specific version of code, where "co" is shorthand for "checkout" and "-r" is the parameter to specify a particular revision number:
svn co -r 12000 svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg

After you've downloaded that version, you can check the date stamps on the files by using the following command. The log option shows you dates/time/version numbers of the files in the svn repository. It sorts the code in the repository by date. I then pipe the output to awk to print only the date column and then remove blank lines with sed:
svn log ffmpeg | awk -F"|" '{print $3}' | sed '/^$/d'

that's all I have for now. I will update when I have time,

Tuesday, October 28, 2008

formatted date in Windows script

I find the output of the full date command in Windows lacking. Thus, I was trying to find a way to date/time stamp a logfile in a Windows batch script.
C:\WINNT>date /t
Tue 10/28/2008

I only wanted YYYYMMDD_HHmm, where
YYYY = four digit year
MM = two digit day
DD = two digit month
HH = two digit hour
mm = two digit minute

So, I needed to combine outputs from both the Windows date and time functions.

My batch script ended up looking like this:
set dt=%date:~-4%%date:~4,2%%date:~7,2%_%time:~0,2%%time:~3,2%
bash "siteBackupAll.sh" | tee log%dt%.txt

In the example above, the resulting script outputs a logfile named:

The Windows help for SET is pretty cryptic, much like any Unix man page. You can see all the options for Windows shell programming (variables, operators, expressions, etc) if you type
SET /?

at the command prompt.

Variable expansion in Windows is a little weird. Here is a snip from the help, but essentially, you can act on the expansion of a variable (ie, pluck parts from the output) by using a numbering scheme that will start counting from the left or right, depending on whether the number you specify is positive (start from left) or negative (start from right). In essence, the numbering is much like any programmatic substr (substring) function:

May also specify substrings for an expansion. %PATH:~10,5%would expand the PATH environment variable, and then use only the 5characters that begin at the 11th (offset 10) character of the expandedresult. If the length is not specified, then it defaults to theremainder of the variable value. If either number (offset or length) isnegative, then the number used is the length of the environment variablevalue added to the offset or length specified.

%PATH:~-10%would extract the last 10 characters of the PATH variable.
%PATH:~0,-2%would extract all but the last 2 characters of the PATH variable.


Tuesday, October 14, 2008


This will be a new section on the site that will summarize articles that describe recent exploits that I've noticed occurring on my website. Most of the articles will detail the exploits and how to recognize them. Over time, this post will grow. Pay attention to the date and time of the post to find out when it was last updated.

ISC: SQL Injection hacks


Saturday, October 04, 2008

vnc on linux won't connect to mac

I wanted to logon to my Mac via my Fedora 7 installation. Unfortunately, connecting to a mac from a linux box using RealVNC doesn't work:

After a bit of Googling, TightVNC seemed to be the ticket. Unfortunately, the VNC and TightVNC libraries conflict:
[root@ogre ~]# rpm -ivh tightvnc-1.3.9-1.i386.rpm
Preparing... ########################################### [100%]
file /usr/share/man/man1/vncviewer.1.gz from install of tightvnc-1.3.9-1 conflicts with file from package vnc-4.1.2-20.fc7

Therefore, I yanked out the RealVNC libraries:
[root@ogre ~]# yum remove vnc*
Loading "installonlyn" plugin
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package vnc-server.x86_64 0:4.1.2-19.fc7 set to be erased
---> Package vnc.x86_64 0:4.1.2-20.fc7 set to be erased
---> Package vnc-libs.x86_64 0:4.1.2-19.fc7 set to be erased
---> Package vnc-reflector.x86_64 0:1.2.4-3.fc7 set to be erased
---> Package vnc-ltsp-config.noarch 0:4.0-3 set to be erased
---> Package vnc-libs.i386 0:4.1.2-19.fc7 set to be erased

..downloaded Fedora Core 6 tightvnc rpm from here:

..and installed the Core 6 libraries (though I run Fedora 7) just fine:
[root@ogre ~]# rpm -ivh tightvnc-1.3.9-1.i386.rpm
Preparing... ########################################### [100%]
1:tightvnc ########################################### [100%]

Now my Fedora box can logon to my Mac using TightVNC:
[root@ogre ~]# vncviewer macwired
Connected to RFB server, using protocol version 3.8
Performing standard VNC authentication
Authentication successful
Desktop name "MACLT"
VNC server default format:
32 bits per pixel.
Least significant byte first in each pixel.
True colour: max red 255 green 255 blue 255, shift red 16 green 8 blue 0
Warning: Cannot convert string "-*-helvetica-bold-r-*-*-16-*-*-*-*-*-*-*" to type FontStruct
Using default colormap which is TrueColor. Pixel format:
32 bits per pixel.
Least significant byte first in each pixel.
True colour: max red 255 green 255 blue 255, shift red 16 green 8 blue 0
Using shared memory PutImage
ShmCleanup called


Thursday, September 18, 2008

Netgear WNR3500: disappointing

Here is my short review of the Netgear WNR3500.

I initially purchased a Netgear WNR3500 as a replacement for my aging Netgear WGT624. Unfortunately, I had two problems, one of them major, necessitating the return of the device:
1) poor installation process
2) wireless communication drops

The installation process took a lot longer than expected: about 1 1/2 hours. It failed twice: once from an older XP notebook PC and second, from a new Vista machine. I could understand from an older notebook, but the install simply failed on the new Vista box. After resetting my network a number of times, the install finall worked from Vista. Resetting the router to its default settings can be a major pain. I learned a few things from this page:

Secondly, as I host a website from my wireless PC, the wireless communication drops were unacceptable. I tried three different versions of the firmware with the same result, wireless network drops. As my older Netgear WGT624 worked from the same location (upstairs about 20 feet away from my wirelessly connected MacBook), I have had to rollback to use the WGT624 and return the WRN 3500. Read more about the ongoing drops and DNS proxy errors here:

On a more positive note, wired connections seemed to work just fine. Also, the administrative interface to the router was 95% the same as my older WGT624, so Netgear did not make me learn new admin commands.

However, my two major issues left me with a sour taste in my mouth and disappointed me, given my positive experience with my older Netgear WGT624. Too bad. I was looking forward to blazing RangeMax, 802.11n wireless speeds. What I got was network drops. Blecch!

A Bummed Cacasodo

Sunday, September 14, 2008

install and review of Yougle for VMC

One of the major things I wanted to do was integrate all my media as well as Internet TV like Joost, YouTube, Veoh, Brightcove, etc, into Vista Media Center. An industrious programmer from Australia has been working on such a program called Yougle (http://push-a-button.com/products/youglevista/) that integrates a variety of online video sources, as well as audio sources (like Live365 Internet radio), picture sites (Flickr, of course) and animations. From my perspective, he hits the big one by getting YouTube working within VMC.

The install of the January 2008 build of the software was relatively painless (described in more detail below). Be aware that you will need to install ffdshow for some necessary codecs.

Update 9/17/2008
The install for the new WIP build seems to hang for about a minute or two on the initial timebar of the install. I then let it sit for those few minutes. The install then continues and installs normally. Not sure what the installer is doing at this point to be hanging like that. If you experience problems, perhaps just let the installer sit for a few minutes to see if that helps. FYI - My box is a P4, 3.2Ghz box w/2GB mem & 500GB SATA internal

Another caveat is that since I've installed TV Pack 2008 for VMC, I needed to fake the Yougle installer into thinking I was on an older version of Media Center. TV Pack changes the version (identity) of Media Center from 5.0 to 5.1. If you have TV Pack installed, the thread below shows the registry edit you'll need to make to get this working:

For video, I've explored YouTube, MSN Soapbox and South Park. Initially, I had some problems with YouTube videos hanging after the first couple of vids loaded. Reading some of the Yougle Forums, there is an "embedded" setting within Yougle. Switching to "embedded" seemed to fix the problem. Here is how to tell whether your videos are playing natively or embedded:

Update 9/18/2008
I also noticed that I lost my sound in Yougle/YouTube for the simple, silly reason that I inadvertantly pressed "mute" on the YouTube flash player. Stupid mistake, but it kept me busy troubleshooting for about 15 minutes.

Otherwise, YouTube and MSN are working as advertised. Which is great.

Update 9/17/2008
Actually, the most stable YouTube player option in Yougle is "YouTube Embedded w/ Minimize". I played around for a few hours last night without losing the YouTube player behind VMC. Awesome!

One drawback is that searching YouTube within Yougle doesn't seem to pull up the same results as a YouTube search executed within a browser. The search is based off of YouTube's RSS feeds. Unfortunately, when I used the search function, I couldn't find some of my favorite videos. The developer was kind enough to research this conundrum. Also, when I added South Park as a video source, none of the South Park videos played properly. Specifically, I waited and waited for the embedded player to move from the rotating timer icon to actually start playing the video, but this never happened. A similar thing also happened with the animations. I assume these problems aren't due to a slow Internet connection, as I am hard wired into a 6MB Comcast connection. I will explore these problems later.

For audio, Live365 is spotty at best. I could only tune in one out of ten stations that I tried. I tested various Internet radio stations after midnight on Monday morning, but I don't think the timeframe should have any impact. I will explore this more in-depth later.

For pictures, the Flickr integration is excellent. High resolution imagery from Flickr looks great on an HDTV.

In sum, Yougle isn't perfect, but it is a great start to be able to view YouTube vids in VMC. I'm psyched to start using this on an ongoing basis. Of course, I will update this thread with more information as I go through the discovery process.

Kudos to Push-a-Button for some fine work!

The install process is relatively simple:
1) install ffdshow
2) install Yougle

* if you've installed TV Pack 2008 for VMC, you'll need to make a registry tweak to get Yougle working
** note that support for 64-bit Vista is in test mode as of 9/15/2008:

Here's a bit more about the install process:
1) Install ffdshow
- I decided to install the latest stable release, from 12/01/2006
2) Download the older, more stable version of Yougle here:

or download the most recent Work In Progress from the Announcements section on the Forum:

I chose to download the more stable version for now.

ffdshow install process
Initial download of v.610 (20061201) here:

Click "Download"

Run Installer

Accept default directory for install

Accept default components

Select Start Menu folder

Select the default Additional Tasks

Select default speaker setup, unless you have something other than two channel stereo

Ready to Install!

Installed! Go ahead and click Finish

Yougle install process and registry edit needed with TV Pack 2008
Download page

Click Download

TV Pack 2008 error

Registry Key for TV Pack 2008..set Ident to 5.0

Once key is edited, restart Yougle setup..now it runs!

Select default directory for install

Vista may show a user account control error message like this one..press Allow!

Yougle install runs and is finished rather quickly! Click finish.

Reset MediaCenter version back to 5.1 in regedit


Thursday, September 11, 2008

command line date time change

Or how to set the time in Linux.

Here's another thing I always forget: how to change the date or time at the command line, preferably using the Bourne shell in Linux.

To change the date/time on a Linux box to September 10, 2008 at 5:37pm, enter the following:
date 0910173708

The ordering of the fields is a bit odd, so pay attention.

where the date/time format is the following:
MM - two digit month
DD - two digit day
HH - two digit hour
mm - two digit minute
YY - two digit year

All are zero padded if date/time values are below 10.

Changing the date time was never easier!


Tuesday, September 09, 2008

500 errors from IIS and Oracle

For the past couple of years, the Microsoft Win2K/Win2K3 website that I support had been plagued with 500 errors and subsequent IIS resets. Our website is a combination of custom ASP and ASP.net code running on 32-bit Windows, connecting via ODBC driver to an Oracle backend on Solaris. The side effect to our customers of IIS blowing up is that for a short period of time (less than an hour, usually), customers would see a number of these 500 server errors from the server that panicked. Usually, this was not much of a problem, as it happened on one or two servers once or twice per day and the number of errors was low. The workaround was to reset IIS.

Recently, we added more servers into our web farm and the problem happened more often, with a greater number of 500 errors being spit out when one of these newer, more powerful machines' IIS process loses its mind. So, we contacted Microsoft and used DebugDiag to setup a crash/hang dump to find out what memory threads were in play when IIS died.

Sending this information to Microsoft, Microsoft engineers pointed to an interaction between IIS and Oracle. Specifically, there were pointers to the Oracle ODBC driver that we use to connect our web application to our database. In the past, we have had very little luck with Oracle being able to solve any driver related issues for us. But we know that the first thing Oracle Support will ask for is "do you have the latest driver set installed." Knowing that Oracle was going to require this of us, we started the process of upgrading the driver from 9i to 10G ( in our development environment. More specifically, we needed the following:
- the Oracle 10G client (1GB)
- the 6810189 patch set for 32-bit Windows (1GB)
- the 7218676 patch set for 32-bit Windows (67MB)

The install is quite large. Our very basic install took about 500MB of disk space. Once we got the new 10G drivers and patch set installed in development, we put it through a gamut of tests. It seemed to work fine, but development is no substitute for production traffic on a site that gets millions of hits every day. So, we started rolling out the new drivers to the most problematic web servers.

The first week was very tense, as we let the drivers cook on one server. Previously, this server blew up at least once or twice a day. With the new drivers in place, two days went by without IIS experiencing a problem. Three days went by. Then an entire week went by without a problem! We were psyched! We started rolling out the driver to the rest of the eighteen web servers, two servers every two days.

After a week went by, we could tell that the number of 500 errors and IIS resets were diminishing. After a second week went by, we were about half way through the lot, with every decreasing numbers of 500 errors. Best of all, the boxes that were patched weeks back had not reset themselves. This was unheard of! After using Oracle drivers for eight years, this was the only time in the history of our website that we were not seeing 500 errors! Fabulous!

We finished upgrading from to a couple weeks later and are EXTREMELY happy to report that we no longer are experience IIS resets and blasts of 500 errors.

Thank you Oracle, for finally fixing this issue!

Sunday, August 17, 2008

performance problems with HD viewing in Media Center resolved

This is a follow up to my first three posts:
Vista View Saber DA-1N1-I tuner and Media Center
Vista 32-bit, First Impressions
Making Sense of Home Theatre PCs

I had quite a battle over the past couple of days. The goal was to get a working Vista Media Center that did not choke while displaying live, HDTV content. First, I have to say that Microsoft has done a kick ass job with Media Center. It has taken a number of years, but the Media Center interface is really slick and easy to use, the content and functionality deep and the possibilities pretty open, given the way they've designed it. You can even download an SDK for it, which is cool.

In practice, however, getting Media Center to work on older equipment is challenging and frustrating for the newcomer. The interesting thing is the base system I started with ("started with"..yuk, yuk) was no slouch:

  • Dell 400SC, 3.2Ghz
  • 2GB PC3200 mem
  • 128MB ATI All-in-Wonder 9800 Pro AGP 4x card
  • 120GB 7200RPM IDE system drive

The Problem
The main problem was that when I viewed live HDTV channels, I would get drop outs and compression artifacts.

As you know, I had already completed the base Vista install, installed aVista View Saber DA-1N1-I tuner card and layered the latest and greatest Media Center with TVPack update on top of it.

There were a couple links to the Media Center TV Pack Unofficial Download on Engadget's site here. I've plucked out a couple of links to download TV Pack 2008 here, but I'm not sure how long they will be valid:

The neat thing about TVPack with the Vista View tuner is that, among many other things it did, was resolve tricky problems with displaying the correct listings in the TV Guide if you have multiple channel sources (ie, cable and antenna signal inputs), as well as add support for clear QAM (quadrature amplitude modulation) channels. QAM channels are the HD channels that a TV provider will send "in the clear" over the cable line. In my case, Media Center sees about 14 clear QAM channels from Comcast.

Tuner Setup
I have a three way splitter in my upstairs loft that splits the cable signal three ways: one to the analog, NTSC connection on the TV tuner card, one to the digital, ATSC connection on the tuner card and one to the cable modem itself. Because I was getting drops, I thought that I was just getting a weak signal. I removed the splitter and plugged the feed directly into the digital connection on the tuner card. No luck, as the signal still broke up.

Initial Performance Stats
Taking a look at the new Reliability and Performance Monitor in Vista (quite nice, really) while an HDTV show was being displayed in MC, I saw that CPU was being driven close to 100% on my hyperthreaded system. Peaking on CPU is never good. I thought the problem could be my video card not offloading enough load from the CPU. Maybe the ATI card was unpowered to the task? I removed it and installed an NVidia 7600GS that I had laying around. Symptoms were the same with the new card.

So far, the evidence showed that the problem was not in the source signal or video card since:
1) the input source signal was clean
2) a capable video card was choking in the same way as an older one

Reviewing Performance Monitor, the disk seemed to be quite busy. I assumed this was due to the live tv buffer that MediaCenter keeps while you watch programs. To explain, MediaCenter has the same ability as a regular consumer DVR, in that it buffers live tv in order to allow the viewer to playback the last few minutes of a program. This rewind feature is supported by the system writing huge files to disk.

I should have pulled up some disk queue stats to confirm, but I had a gut feeling that the problem was in the disk subsystem. Looking at the disk reads and writes in Performance Monitor, I saw that about 4MB of data were being written to and read from the hard drive every second while an HD program was being shown. My current hard drive was a 120GB IDE. IDE throughput is pretty weak by today's standards. Hmmm..this old drive has to go. So, I was off to the nearest store to get a hard drive.

Throwing Money At It Will Solves the Problem!
In a pinch, Staples is my closest source of computer parts. Of course, it is not the best choice for prices. Oh well, I was on a mission. Seagate 500GB SATA..$130. Eeesh. That is a little too much. I patiently waited for the computer department rep to finish his conversation with another customer. I'm glad I did. I overheard that Staples supported price matching. I scrambled to find a machine with an open internet connection. Finding one, I Googled "site:bestbuy.com seagate 500GB SATA". Lo and behold, they had the very same drive for $90! Great! At the register, the clerk had to confirm the price. I noticed his point of sale system had web browsing directly built in. But he confirmed the price..$40 saved!

I ran home and installed the drive. Reading through the installation manual, I saw that Seagate has Disk Wizard, which allows you to format your new drive. It also had an option I hadn't seen before..to clone an older drives' contents onto a newer one. Psych! I was expecting to have to reinstall Vista, but if this Acronis clone software actually worked, I wouldn't have to go through that pain.

New Drive Fun
I installed the drive in the PC and got Disk Wizard running. Selecting the appropriate clone options, clone was to copy the old drives' content, sets the master boot record and reboot. Hitting "go", I saw that the clone ran very quickly, almost too quick, and then asked me to reboot. I figured the process must do the cloning on reboot, but when the box rebooted, the OS did not boot. Oh sh1t. I hope it hadn't blown away my Vista install. Googling for people with similar problems, I learned from someone else's issue that clone changes the master boot record, so my system trying to read off the new disk. Since the clone process didn't go as expected, there was no operating system for the master boot record to access. Luckily, clone did not touch the original system disk, so after unplugging the new one, I was able to verify my base Vista system was still alive.

Starting Disk Wizard again, I saw it had a bootable install version of the clone process. Since the initial Vista install process did not work, I thought I'd give this a shot. I burned an ISO to a CD and started up the clone process. This time, I saw progress on the clone procedure meter. It took about 17 minutes to clone 24GB of data from a 120GB IDE to my new 500GB Seagate. After the clone process was complete, the box rebooted. While the box rebooted, I made sure to shut down the box entirely and take the original system disk offline. After doing this, I restarted the system and was overjoyed to see Vista start booting. New drive works with my old install of Media Center! Yay!

Did the New Drive Help?
Moment of truth. I pulled up Media Center with some live HDTV. I still had dropouts even with the new, faster drive. Damn! OK. Since you can't turn off Live TV easily in MediaCenter (why would I want to, with Olympic Volleyball coming up?), I decided to meet the burden head on by using my old IDE RAID card and setup a stripe set of old hard drives. I had two 120GB lying around (the one old system disk plus another), so I installed the card, setup the RAID set in the BIOS and formatted the new RAID set when it came up in Vista. Voila! Lots of fast disk capacity on hand! Now to repoint the destination directory for the live hd temporary files to the RAID set.

Move those Temp Files!
In Media Center, I saw that there was no easy way to turn off the Live TV feature. So I figured the location of the temp file directory was in the registry. Reviewing the Disk statistics of Reliability and Performance Monitor more closely, PerfMon actually shows the location of the directory where disk access (reads and writes) are occurring. Sweet! Looking at the details of the offending process, I saw that this directory was the culprit of most of the reads and writes:
C:\Users\Public\Recorded TV\TempRec\TempSBE

Cool. I searched on this string in regedit, found the appropriate location:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Media Center\Service\Video\Tuners\DVR\IO\Writer

Then, I tweaked the values to match a new directory that I setup on the D:. I restarted Media Center for the change to take effect and waited patiently for live HD to come up. Watching it, I saw that HD was definetly smoother, but not perfect. It still hiccupped from time to time. DAMMIT! Looking at the top CPU consumers in Task Manager, I saw that dwm.exe (Desktop Window Manager) kept popping up, albeit with low CPU consumptions of 5-7%. More performance tuning necessary.

Optimizing Media Center Performance
The key to getting Media Center working right is optimizing performance in Vista. Here are a few articles that helped me do that, in order of most useful:
ExtremeTech article
Andrew Grant's blog
APCmag.com article
Jwyse's blog

Also, the following article has some registry tweaks to get your Media Center setup just the way you like it:
MediaCenter.com.au article

One feature that I had kept running in Vista was the Aero interface with a sidebar of gadgets (clock, RSS news feed and rotating pictures). Apparently, the process dwm.exe controls the display of these visual effects and gadgets. In Control Panel -> Personalization -> Window Colors and Appearance, I disabled the Aero interface by selecting "Windows Standard" as the color scheme. After the change, I checked live HD. There were less drops, but still an occasional one every fifty seconds or so. Damn! On a hunch, I disabled the RSS news feed that was in the sidebar under the clock. Restarting Media Center, I watched for any live HD dropouts. One minute went by..no problem. Two minutes..no problem. Three minutes..no drops. Awesome! I think I've got the problem licked!

Right now, I've had live HDTV (the Olympics) running for four hours. There seems to be a barely noticeable lag sometimes when replaying the live HDTV buffer. I believe this is due, again, to disk access. I may add a third 7200RPM 80GB drive to my stripe set to improve performance. I will let you know how that goes.

In general, if you are skirting the edge of recommended hardware for HD viewing, Media Center seems VERY sensitive to any processes taking up CPU or mouse movements. Disabling any CPU performance hogs is essential in order to have live, HDTV in Vista Media Center run properly on an older system. If I want a perfectly clear HD viewing experience with no dropouts, I will not open ANY programs. Also, I don't have any programs running that update the display, like Task Manager or Performance monitor. In fact, I won't even touch the mouse if I want undisturbed video. Media Center a bit touchy, eh?

I assume if I had a new dual or quad core box with a fast CPU, the latest memory, a 1333Mhz front side bus, PCI Express video card and all SATA, I wouldn't have these problems. But I didn't feeling like paying an arm and a leg for a new box. With my trusty old Dell 400SC, I may be able to get by adding a $100 tuner and $100 hard drive only. Time will tell!

In sum, once Media Center is running well, it IS pretty sweet. Now to find a usable remote for VMC. Any suggestions?


Tuesday, August 12, 2008

Vista View Saber DA-1N1-I tuner and Media Center

This is a follow up to my first two posts:
Vista 32-bit, First Impressions
Making Sense of Home Theatre PCs

Last night, I spent a few hours experimenting with Windows Media Center on Vista and the new TV Pack, with the goal of getting digital cable from Comcast integrated into Media Center. As suggested from the article linked above, I reinstalled Vista (painfully slow). I then installed the TV Pack and associated hotfixes. That day, I had bought a Vista View Saber DA-1N1-I tuner card. It is PCI, to fit in my older Dell 400SC. I let Vista select the driver from the internet and I saw that it found the latest driver.

Setup in WMC was straightforward, similar to Ben's experience. I connected the card's NTSC connection to my comcast line. After the requisite asking of where Ilived, downloading of my area's TV guide and inspecting my tuners' connection, WMC got me to the point where I could select my area's cable provider. Since I live in NY Metro, there were about ten choices. I selected Comcast in Jersey City. So sure enough, channels 1-99 on Comcast appeared.

I was happy to see I had basic analog connectivity, but wondered where the digital channels went. At the time, I didn't realize that this was regular analog cable from Comcast. I went through the discovery process again, and this time I selected Comcast Digital as my provider. Well, this didn't work, because the cable was plugged into the analog, NTSC connection. In my haste, I hadn't realized that the card I bought is only good for ATSC, over the air HD and NTSC analog cable. It won't get me what I wanted, which was the ability to watch and record unencrypted, clear QAM channels via my Comcast connection. Duh.

Taking my idiocy in stride, I moved on to the ATSC output. When I connected my cable connection to ATSC, as expected, WMC didn't find any channels. I thought I might be able to use the cable coming from the back of the PC as an antenna, so I disconnected the opposite end of the cable from the PC (the splitter end), reran the automatic setup and I was nicely surprised to see that the WMC picked up the local over the air HD channel selection, about 14 channels in my case, directly across the river from Manhattan. Performance was a bit choppy, as my signal wasn't very good in my computer closet. I'm sure with a proper antenna, the signal would be great.

In both cases, analog cable and over the air, WMC picked the correct guide. The guide picked up all the shows, was really nice to navigate through and had lots of supplemental information. Very slick.

Today, I went back to the drawing board and picked up a Hauppauge WinTV HVR-1600 . I'll let you know how this install goes tomorrow.

Long story short, though, if you are looking for a card that works for NTSC and ATSC over the air signals out of the box with this latest update for WMC, the Vista View Saber seems to fit the bill.

UPDATE 8/14/2008
Last Tuesday, VistaView released a QAM patch (a simple registry tweak) for MediaCenter with TVpack to allow the Saber DA-1N1-I to allow you to tune QAM channels on ATSC cable TV: http://www.vistaview.tv/index.php?option=com_docman&task=cat_view&gid=31&Itemid=46

I downloaded the patch and rebooted. I split my cable line to go to both the analog NTSC and digital ATSC jacks on the back of the Vista View card. The positive: during the tuner setup, Media Center now sees both the analog tuner and a new digital cable tuner:

ViXS PureTV Analog TV Tuner

Continuing through the tuner setup, I can select my provider's (Comcast) channel selection and guide. Checking the source of the channels under Guide -> Edit Channels, I get a mix of the OTA HD channels (about 14 channels) on the Digital Cable connection, but the rest (70 or so) are coming through the analog tuner. The digital channels seem to break up a lot. I imagine using an HD antenna instead of the digital cable would help here, since the digital cable is only getting me those 14 channels. If I hook up only one cable to the ATSC input, I only get those 14 channels. I have had no luck seeing any channels I would normally see with the Comcast Motorola box above 99.

From the Guide -> Edit channels, I see that most of the available channels that are disabled have padlocks next to them and that the source of them is the digital tuner. I imagine the padlocks mean that they are encrypted.

Tonight allowed me to get a bit farther, but am perplexed as to why I'm not seeing any channels above 99. I am a bit of a newbie to this MediaCenter/cable integration stuff. It seems as if Comcast locks everything above 99..not sure right now. Looking at the clear qam guide for my zip code here https://www.silicondust.com/hdhomerun/lineupui?Cmd=LocationProgramsWeb&Country=US&Postcode=07302, I notice that the higher range (above 99) channels that I enjoy watching like Military/VH1 classic/Encore/etc are missing from the list. I guess they are not in clear QAM. Maybe I'm just out of luck?

So, a bit of progress, a bit of disappointment.

UPDATE 8/15/2008
One follow up: I'm a bit disappointed in the performance of HDTV in Media Center w/TV Pack. I've got a Dell SC400 (probably four 1/2 years old) P4, 3.2Ghz, 2GB w/120GB IDE drive and ATI all-in-wonder 9800 pro. Not the latest and greatest, sure, but no slouch box either. Viewing standard def, Media Center drives the box to run at 50% CPU without a problem. Running live HDTV (1280x720), MC runs the box to 100%, at which point I get drop outs. Also, I notice SearchIndexer runs constantly, thus sucking up more CPU.

Granted, the system is a few years old, but this is unacceptable. I will look for ways to tune or upgrade the system to see if I can relieve the 100% spiking.

Andrew Grant has an excellent post on optimizing Windows Media Center to improve performance. I will try his suggestions out tonight. A more technical article on the same subject is on APCmag.com.

UPDATE 8/16/2008
Today, I wasted about an hour tracking this high CPU problem down. I accidentally scheduled a recording and wondered why Media Center Receiver service was taking up so much CPU. This was because it was recording a show and I didn't know it! I went into my Recordings and deleted the scheduled recording.

Getting Started with Windows Media Center in Vista
Getting Started with HDTV in Media Center
Great intro to channel tweaking in Media Center
Tweaking Media Center's HDTV Lineup
Media Center Receiver Server Technical Discussion


Saturday, August 09, 2008

Vista Ultimate 32-bit first impressions

This is a follow up to my first post
Making Sense of Home Theatre PCs

Last night, I experimented with Vista Ultimate, 32-bit for the first time. My intention was to use Windows Media Center to access all my digital media and to feed Media Center my Comcast HDTV connection. As I was going to install Vista on a 1.8Ghz Pentium IV with 1.5GB of memory, I knew the system was underpowered for Vista and that the install didn't have much of a chance of running smoothly.

I was correct for a number of reasons:
1) First, the Vista install did not recognized my Sil0680 IDE RAID card.
This was a bummer, as I thought I could make up for the lack of CPU horsepower by running a stripe set. During the install, Vista is smart enough to allow you to stop and look for drivers. It also allows you to search for drivers on a USB flash drive, so I found Sil 680 drivers from here:

and then copied them to the USB. However, when I loaded them on the box, Vista did not recognize the drivers as the correct drivers for the Sil RAID card. This is doubly disappointing as the Ultimate install DVD I had was SP1. Argh.

So, I had to back off and install the base OS to a regular IDE drive that I had floating around. That was problematic too, as the primary partition was not marked as active. So, I had to boot up with a floppy, mark it as active and give it a base format. Surprisingly, Vista still did not recognize the partition as usable. So, just to see if an older XP install would recognize the partition, I booted with XP. Luckily, XP did see the partition. In fact, though fdisk reported only one partition on the drive, the XP install disk saw two partitions. Strange. Using the XP installer, I deleted both partitions, created a new one and reformatted as NTFS. This seemed to do the trick and the XP install continued and installed XP.

After the XP install finished correctly, I reinserted the Vista install DVD and gave the install another shot. Because I had fixed the drive partition problems with XP, Vista now saw the partition correctly and I was able to install the OS successfully.

2) The install process took interminably long and did not give you adequate feedback about where it was in the process and when that part of the process was going to be finished.

3) ATI's All-in-Wonder Radeon 8500DV does not work with Vista.
This was also a great bummer, as I wanted to try out some of the TV functionality with Windows Media Center. Reading the install README from the Catalyst download section on ATIs website:

it seems that older card is not supported under Vista. I was hoping to take advantage of my older equipment. With Vista, again, this was not to be.

4) Vista is a resource hog.
Once I got Vista running, the OS did seem to run fairly smoothly on this low-end equipment. However, Vista is heavy:
- base install requires about 10GB with the default install selections
- memory usage is about 540MB with nothing running

So, Vista is a hog. And it is designed for the latest and greatest equipment. I would not recommend the operating system for users who want to take advantage of older equipment they have lying around. Vista just won't run it. This might be Microsoft's way of weaning old equipment off the supported hardware list. Which is here if you need it:
Vista Hardware Compatibility List

As I experiment more with Vista, I will let you know my impressions of the OS. Right now, I'm a bit saddened that Microsoft has kicked the old folks (READ: old equipment) out the door in favor for the new.

Aren't we supposed to care for our elders?

Thursday, August 07, 2008

making sense of media center/home theatre PCs

I bought an Apple iPod Touch 32 GB over the weekend (a post is coming) and it has led me down some interesting paths. It is wonderful to be able to scroll through a good chunk of my music, video and photos easily, all in one little handheld device.

Even more impressive than its storage and playback capabilities are all the applications and connectivity options available via the App Store. The apps will really lengthen the life of both the iTouch and iPhone for years down the road.

iPod Touch Extensibility
One app that I particularly liked was Remote, which allows one to control iTunes from the iTouch/iPhone. Plus, its free!

While surfing for applications, I also came upon the TouchPad Pro. The handy little application can be used to control a Windows Vista Media Center PC with the iPod Touch!

Unfortunately, as of this writing (08/05/2008), the TouchPad Pro is not available for the iTouch's 2.0 firmware:

Finally, Mocha VNC Lite is a free app that connects you via to any VNC Server (Mac/PC/Unix) in your environment:


Home Theatre PC (HTPC) Possibilities
In playing around with the iTouch, I started thinking about the possibilities of having the iTouch become my remote for not just my media, but for my HDTV and Home Theatre, too. This got me very excited, as I would love to have all my music, video, recorded HD, podcasts, everything integrated in one solution, with a super easy-to-use remote via the TouchPad Pro. This is the Holy Grail of the so-called "Home Theatre PC!"

From my reading, I seem to have three choices:
  • AppleTV
  • Linux MCE (MythTV)
  • Windows Vista Media Center

Here's one gentleman's comparison of Linux and Windows MCEs. Though biased, it is interesting:

Here's what I'm thinking in terms of requirements:
-should be relatively simple to setup: oops, this rules out MythTV
-needs to play cable TV (both std/high def): oops, this rules out Apple TV
-needs to play most common media formats MP3s/MPEGs/DIVX/Quicktimes/DVDs/Blu-ray
-needs to integrate iTunes*

* Yes, you may be saying that Windows Media Center does not integrate iTunes and you'd be correct. However, thinking about how this would work, it should be a relatively minor step to switch applications from Media Center to iTunes via whatever controller or remote you'd choose to use. Also, further investigation showed me that solutions from both sides of the fence (Linux/Windows) seem to have some basic level of iTunes integration or at least being able to access the media files at the system level. More on this later.

From this short list of requirements and some reading that I've done over the past couple of days, I'd say the most likely contender is Media Center running on Windows Vista. I'm no huge Microsoft fan, but if a software's features and functionality fit the bill, then you should use it.

Here is a nice walkthrough to Windows Media Center features:

How to Connect Your TV to a Windows Media Center PC:

Finally, here's five things you didn't know you could do with Windows Media Center:

The Problem
One of the stickiest wickets in that requirements list is how best to integrate your cable SD/HDTV viewing and recording. It would also be nice if this was simple to setup. From my reading so far, it doesn't look like any PC-based home theatre/media centers are easy to setup by any means. At least computer companies are starting to address this by using a PCI card that integrates with your cable TV providers service, also known as a digital cable tuner card. A CableCard is a PCMCIA like card built into new-generation televisions and PCs that allows digital cable reception without a set-top cable box. Here is a nice primer on CableCards:

Here is some basic information about cable TV tuners via HP forum:

This was a very informative read about the regulatory background of the CableCard:

And Chris also has a good FAQ on the CableCard. Be aware that in its current form, CableCard does not allow you to use On Demand or Pay Per View services. Apparently, this is because those features require two-way communication that is not present in today's CableCard solution. This is unfortunate, because it means that today, in August of 2008, we still do not have a complete Home Theatre solution. (Well, I guess I can live without ONE feature).

Possible Solution: Dell XPS 420
From this article, I found that the Dell XPS 420 might be a solution that integrates a CableCard and might fit the bill.

Here's Dell's full specs:

An excellent FAQ on the XPS 420:
Chris Lanier's blog

As a cable card is only distributed to consumers if they purchase an entire new system, the unfortunate thing is that the CableCard solution locks individuals into buying a preassembled machine versus building their own. But given some evidence from intrepid individuals like Keith Combs, this might not be all bad. Here is his very informative blog entry:

Also, his six month checkup is a nice follow through:

Another solution: HP m9300t
Another solution is from HP, the HP m9300t Series

Here is a page on HPs site devoted to more general information about the HP m9200 line of products.

Decisions, decisions
Before purchasing any solution, I plan on building out a Vista box, just to see how far I can get, without the cable tv tuner part.

Let you know how I do in a follow-up post,

Update 8/9/2008
Vista Ultimate install and first impressions

Update 8/8/2008
Digital Tuners
I happen to be perusing blog articles about people who have built their own media centers and I see this Hauppauge digital tuner card has come up a number of times. The FAQ page has some good info on it.
For more basic information on what TV Tuners can and cannot do, take a look at this post from AVS forum/Overclockers.net:

Update 8/15/2008
Vista View Saber card, Media Center and TV Pack install

Update 9/15/2008
Viewing YouTube Within VMC

Sunday, August 03, 2008

best windows shortcut chart

This chart showing Windows keyboard shortcuts is so good it merits its own post:
Awesome list of windows shortcuts!

Thursday, May 15, 2008

Verisign Class 1 CA Individual Subscriber cert expires

Yesterday, 5/14/2008, we were rudely awakened to errors in our SSL checkout process. Basically, the dialog box that informs a person browsing the website that a problem has occurred showed up. Investigating our IIS 5.0 web servers, we found that none of the SSL certificates had expired. Digging deeper, we then looked at the certificate store on the machine. In the certificate store, we found that the Intermediate Certificate called Verisign Class 1 CA Individual Subscriber certificate had expired. Yuck!

For Windows 2000
We called Verisign and confirmed that the expired certificate was affecting our site. Verisign was able to give us a link to download the latest certificate:

We followed the install and verfication instructions to install the cert on each server in the farm. After making those changes, I verified in Firefox that the SSL dialog box no longer appeared. Thank God!

For Windows 2003
Win2K3 was slightly different. We actually needed to make the certificate appear in our Intermediate Certificate list:

This was accomplished by downloading the latest intermediate cert that works with your chosen level of SSL certification from here:

For us, we use Non-EV Premium SSL certificates. So we used that link from the main page. Once we installed them by following the directions on the linked page,
http://www.verisign.com/support/verisign-intermediate-ca/secure-site-pro-intermediate/index.html, we saw the intermediate cert appear (date of 10/24/2011) in the proper certificate store:

Finally, a reset of IIS was needed. Yuk!

The one thing I do not have clarity on is exactly what certificates does a typical web server running ASP and ASP.net need to run correctly. I know that there is a hierarchy of certs:
1) Root certificates
2) Root cert signs Intermediate
3) Intermediate signs Personal (www.mysite.com)

We had a similar problem back a couple of years ago and I had been too busy to research the question then. I will try to research it and give an update to this page.

Good luck, folks!

Wednesday, April 16, 2008

Ubuntu 7.10 install in VMware Server gotcha

Trying to install Ubuntu 7.10 in VMware Server 1.04, I came upon a lot of these messages as Ubuntu was trying to install itself into the virtual machine's hard drive:
sd 2:0:15:6 rejecting I/O to offline device

A caveat to installing most Linux distributions within VMware Server for XP is that you need to set the SCSI driver to Lsilogic in the configuration file for the virtual machine. The config file will be a file with the extension ".vmx" in the VM's directory. For example, the config file for my Ubuntu install was "Other.vmx"

Here is the line you need to add to your .vmx file:
scsi0.virtualDev = "lsilogic"

My entire .vmx file ended up looking like this:
config.version = "8"
virtualHW.version = "4"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "1024"
ide0:0.present = "TRUE"
ide0:0.fileName = "Other.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "auto detect"
ide1:0.deviceType = "cdrom-raw"
floppy0.fileName = "A:"
Ethernet0.present = "TRUE"
displayName = "Ubuntu 7.10"
guestOS = "other"
priority.grabbed = "normal"
priority.ungrabbed = "normal"
ide0:0.redo = ""

ethernet0.addressType = "generated"
uuid.location = "56 4d 32 f1 73 c7 b5 15-26 b3 bf 4b eb 77 4f c6"
uuid.bios = "56 4d 32 f1 73 c7 b5 15-26 b3 bf 4b eb 77 4f c6"
ide1:0.autodetect = "TRUE"
ethernet0.generatedAddress = "00:0c:29:77:4f:c6"
ethernet0.generatedAddressOffset = "0"
numvcpus = "2"

Of course, the line you add may differ if you have more than one SCSI device. Once the line is added, you'll need to restart your virtual machine for the change to take effect.

Good luck!

Tuesday, April 08, 2008

black screen/no video in VLC

I noticed a problem in VLC on XP the other day. While watching a music video, I starting getting sound but no video. I thought it was some sort of display problem, but then tried QuickTime and found I could watch the video in QuickTime. Just to see if it would help, I logged off and back on. Still giving the same problem. I didn't change any settings, so I thought it was a hardware problem. So I shutdown the box and turned it back. Same problem. Argh.

I then realized that it must be some issue with VLC itself, so I started digging through the Settings menu; specifically, Preferences. Looking in Preferences -> Audio with "Advanced options" checked, I saw that "Audio desynchronization compensation" had reset to -10000000! Woah! How did that happen? I reset this value to 0 (zero) and played back my original video. Voila! The video played perfectly.

I don't know what key combination or bug I may have triggered, but that is a wacky one. Hope this helps someone in similar distress.


Friday, March 21, 2008

authenticated FTP through browser

I always forget the syntax for accessing a password protected FTP site through a web browser

If you know the username and password of the FTP site, you should be able to access the FTP site through your web browser. You need to construct a URL with the following syntax:

For example:

Make sure if you are accessing via a proxy server, that FTP is configured in your proxy settings.


Friday, March 07, 2008

recording video off Motorola 6412

I had been wanting to copy some unencrypted shows off of my Comcast cable box, a Motorola DCT-6412, for quite some time. By "unencrypted", I mean your basic cable. This does not include premium channels like HBO, SHO or MAX. Also, you won't be able to grab pay-per-view content.

I finally got around to trying this out last week by using my MacBookPro and a standard 6-pin male-to-male firewire cable. GXW on http://www.macosxhints.com/ has provided some excellent instructions for doing this.

OS X10.4.9 or above
Apple's FireWire SDK (downloadable from Apple's Developer website)
VLC (Video Lan player for Mac) or MPEG Streamclip for playback

1) connect Mac to Motorola
2) verify physical connectivity via System Profiler
3) connect and verify firewire connection with AVC Browser
4) replay show and grab stream with Virtual DVHS

You'll need Apple's firewire SDK in order to do this. Create a free Apple ADC Online account from here:

Once I got the SDK installed and verified my connectivity as above (see GXW's fine article for the details), I was able to successfully pull content off of my Motorola 6412 HD cable tuner. However, the way it works is that you have to capture live content as it is playing or replay a stored recording in order to capture it to your Mac. In other words, this method does not allow you to access the hard drive in the STB to get at the stored recordings directly. Therefore, it will take some time if you have a bunch of shows stored in order to replay and capture them to a file.

Hint: Don't fast-forward or reverse while recording a saved program to your Mac's hard drive. This will screw up the audio and video synchronization of the destination file.

I was able to pull both regular (SD) format and HD format content off the Motorola. HD shows are recorded to MPEG-TS (transport stream) format. Also, I just realized I didn't try recording content off of the analog channels, I only recorded shows off of the digital channels. So I can't speak to that.

Once I get a workflow in place, it will be nice to have my favorite movies and shows as files to stream from a media server. Yeehoo!


Wednesday, March 05, 2008

understanding CSS using Firebug

Understanding CSS is not trivial stuff. But the Firebug plugin (http://www.getfirebug.com) for Firefox can help. Here's a video I produced that shows how to inspect your CSS with Firebug. The context is understanding Blogger templates:

Also depicted in the video are cross-hairs from the Orca screen magnifier, an application for disabled people that is installed with the Fedora 7 distribution. I used the cross-hairs because the mouse cursor disappears when I capture the video with Cinelerra.


Thursday, February 28, 2008

command line email in Cygwin

As mailx is not part of the Cygwin distribution, you don't have that many choices if you want to send an email from a shell script. In this regard, Jeremy Reed has kindly provided a nice substitute script that can help us out:

If you have Cygwin installed, you'll need to do the following things:
1) install ssmtp
2) configure ssmtp with ssmtp-config
3) install Jeremy's mailx shell programs

There are a couple of gotchas with step 2 above.
1) make sure you have the directory /etc/ssmtp created
- for some reason, the ssmtp-config program doesn't create it
2) make sure you define a fully qualified hostname
- if not, you'll get the following error
$ /usr/bin/mailx -s "test" cacasododom@gmail.com < test.txt
3) make sure that you have a symbolic link created for /usr/sbin/sendmail that links to /usr/sbin/ssmtp and that /usr/sbin is in your PATH. Otherwise, you'll get this error:
$ mailx.sh -s "test" joe.user@yahoo.com < test.txt

That's it.

I preferred to use Jeremy's program over nail, because it gives me the same arguments as mailx did.

Thanks Jeremy!

Tuesday, February 12, 2008

Site Overlay in Analytics is lame

We can only look a gift horse in the mouth so often. So when Google stumbles with a small feature in the free toolsets that they provide us, we shouldn't yell at them.

Case in point: the Site Overlay feature in Google Analytics.

Site Overlay *should* be very cool, giving us a visual representation of the traffic to our websites by overlaying the percentage of traffic directly on the page itself. For my site, it looks something like this:

Notice all those 0%'s? Unfortunately, Site Overlay only works with static pages. It chokes on dynamic content, CSS, javascript links, redirects and almost any recent technology used to power a website. Here is the company line from a rep I emailed regarding what Site Overlay can and cannot do:

Thank you for your email. I understand you are concerned about zero datain your Site Overlay reports. Please note that the site overlay report uses cURL to request a web page, and displays clicks on standard links that are embedded on that page. Its functionality is currently limited to static pages with unique links to content located elsewhere on the website.

The site overlay is not currently able to work with Javascript links, CSS content, Flash navigation, downloadable files, outbound links, frames, or automatic redirects.

In the above cases, it's common to see missing overlay bars, or bars with zero content. If there are multiple links on a page that all point to the same place, Analytics will total the clicks to on all those links and display the total number for each overlay bar on each of those common links.

If you have a filter that changes your Request URI, the Site Overlay report may display 0 in your report.

Oh well. If Google updates this feature to allow for the use of dynamic content, I'll let you know.

Saturday, February 09, 2008

understanding CSS using Firebug

Blogger templates are dense with Cascading Style Sheets and can pretty difficult to understand if you need to edit them. I didn't really understand them until I picked one apart using Firebug (http://www.getfirebug.com), the developer's plugin to Firefox. I have to tell you, Firebug really kicks ass and makes understand the CSS of any HTML page very clear. Enjoy the video!

Wednesday, February 06, 2008

upgrading Cygwin: dos2unix

I recently converted from a very old version of Cygwin Beta 20 (circa 1999) to the latest 1.5.25-7 version of the Cygwin DLL. I noticed when I tried to execute some of my existing scripts that had executed under the old beta version, I now would receive a lot of these errors:
./script.sh: line 40: $'\r': command not found
./script.sh: line 5: syntax error near unexpected token `$'do\r''

Looks to me like the command interpreter on the new version of Cygwin is a lot more picky than the old one. Specifically, Cygwin is choking on the carriage returns that Windows XP Notepad inserts at the end of each line of the file. Yuk.

I wondered if this was something configurable with an environment variable (http://www.cygwin.com/cygwin-ug-net/setup-env.html), so I first tried setting the CYGWIN environment variable to nobinmode (http://www.cygwin.com/cygwin-ug-net/using-cygwinenv.html). Unfortunately, this change had no effect. I did find a bit of background here:

I had a bit more success using one of the nice little utilities Cygwin has called "dos2unix". Dos2Unix strips out any carriage return or line feed characters in text files and automatically saves the file. The command is simple:
dos2unix [filename]

where [filename] is the name of the file you wish to convert. Dos2Unix has no man page, so don't bother looking for it.

Unfortunately, I had quite a few scripts in various directories, so I just went through each directory and typed:
dos2unix *.sh

I could have written a script to interate through the directories, but there were only about fifteen, so I got lazy and just ran the command manually. After running the command, I was able to execute the scripts without error.


Saturday, February 02, 2008

tag cloud for Blogger blog labels

Phydeaux3 has posted some very easy-to-follow instructions on how to create a tag cloud for labels on your Blogger blog:

A tag cloud looks like this:

You can see a working implementation of a label cloud here:

The one thing that Phydeaux doesn't provide is an overview of the steps necessary in order to create the tag cloud. Here is the overview:
  1. Back up your blog template
  2. Add the label cloud stylesheet to your blog template (before the end skin tag)
  3. Add javascript variables for the cloud to your template before the end HEAD tag
  4. Replace the template code for your existing Label widget with the custom Label Cloud code
Finally, Phydeaux adds some troubleshooting tips as well as some hints for editing the label cloud text sizes and colors:

Nice job, Phydeaux!!

ps - If you're new to Blogger, I've posted a number of videos here that you may find useful:

Thursday, January 31, 2008

cygwin cron on Win2K/XP

Well, setting up Cygwin (http://www.cygwin.com/) cron on Win2K Server and my XP workstation is a complete bitch. Simply put, my scripts would not start properly. I used the latest version of the Cygwin DLL, v.1.5.25-7.

In a nutshell, my problems came down to three issues:
1) cron would only work as my local domain account
-lots of good information here: http://www.andrewlouie.com/wordpress/?page_id=88
-and a little bit of info here: http://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-switch
2) cron needed to be installed using "smbntsec" privileges for my SSH-based scripts to work
3) finally, c:\cygwin\bin needed to be the last entry in my PATH enviroment variable

Installation Steps Added 3/1/2008
Most of the following information comes from these gentlemens' fine posts:

The overview is that you'll have to do quite a bit to get cron under cygwin running properly. Please refer to the two docs above, but here is the short list of steps:
1) decide on a user that cron will run under
- we have a domain account used for scheduled tasks, so I used that
2) add the user to /etc/passwd
- setup a proper home directory in /etc/passwd
- if you do not have a proper home directory, you'll see "Can't cd to $HOME" errors from the output of cronevents.exe
- copy valid .bashrc/.bash_profile files to your users' home directory
3) under Control Panel -> Administrative Tools -> Local Security Policy -> Local Policies -> User Rights Assignment -> tweak your Win2K/XP Local Security Policy settings to add the chosen user to
- (Adjust Memory Quotas for a Process) Increase Quotas
- Create a Token Object
- Log on As A Service
- Replace Process Level Token
4) create a crontab for your user
- crontab -e
5) verify that the permissions on the command(s) to be cron'd will allow the user you've chosen to execute the command(s)
6) run cron-config to enable the cron daemon
- you may get execute and write errors related to permissions on /var, /var/run, /var/log, /usr/sbin/sendmail, /usr/bin/cronlog. I took the easy route and fixed them with:
chmod 766 [file or directory]
chmod 755 [file or directory]

Also, make sure the user that you will be using to execute cron jobs is setup in /etc/passwd.

6) once the cron daemon is running, validate that your script is running properly by looking at cronevents.exe and $HOME/cron.log
- if you have not applied the Local Policy settings correctly, you may see the "can't switch user context" error in the log output
- at one point, I received this error:
C:\cygwin\usr\sbin\cron.exe: *** fatal error - could not load user32, Win32 error 1114

I resolved this by moving my user from the "guest" group to the "Administrators" group in /etc/passwd. Of course, you may not want to do this based on your security needs.

I learned a few things along the way:
1) after installing the cron package into Cygwin, you want to use cron-config to setup cron
2) after enabling your cron, cronevents.exe is a good thing
3) occasionally, $HOME/cron.log provides some useful info
4) the CYGWIN environment variable can be a nasty beast:
5) the HOME environment variable must be set properly:
http://blog.spikesource.com/crontab.htm suggests running "env" in your cron out to a file to find differences between your user environment and the cron environment. Good idea!
6) don't forget your scripting basics:
7) documentation can help:

1) after you install the cron packages from the CygWin setup program, make sure to use "cron-config" to add or remove the cron daemon as an NT service, and perform configuration of cron.

I needed to run the service as myself, with the security tweaks described in both these places:

Here's a sample of the installer dialog:
$ cron-config
Cron is already installed as a service under account DOMAIN\USER.
Do you want to remove or reinstall it? (yes/no) yes
OK. The cron service was removed.

Do you want to install the cron daemon as a service? (yes/no) yes
Enter the value of CYGWIN for the daemon: [ntsec smbntsec] smbntsec

The service can run either as yourself or under a privileged account.
Running as yourself allows better access to network drives,
but does not allow to run the crontab of other users.
Do you want to the cron daemon to run as yourself? (yes/no) yes

Please enter the password for user 'SODO':
Running cron_diagnose ...
... no problem found.

Do you want to start the cron daemon as a service now? (yes/no) yes
OK. The cron daemon is now running.

In case of problem, examine the log file for cron,
/var/log/cron.log, and the Windows event log (using /usr/bin/cronevents)
for information about the problem cron is having.

Examine also any cron.log file in the HOME directory
(or the file specified in MAILTO) and cron related files in /tmp.

If you cannot fix the problem, then report it to cygwin@cygwin.com.
Please run the script /usr/bin/cronbug and ATTACH its output
(the file cronbug.txt) to your e-mail.

Occasionally, cron-config may bark at you for permissions errors on /etc/group and /etc/passwd. Fix them and then re-run cron-config.

Event Log Errors
If I ran the cron daemon under the SYSTEM user context, I got the following Application Event log error:
The description for Event ID ( 0 ) in Source ( /usr/sbin/cron ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: /usr/sbin/cron: PID 5300: (CRON) error (can't cd to HOME).

After a proper install, I still see a running set of Information Alerts in the Application logs. These don't seem to hurt, as my jobs still runs:
The description for Event ID ( 0 ) in Source ( /usr/sbin/cron ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: /usr/sbin/cron: PID 5016: (SODO) CMD (/scheduledtasks/scripts/load.sh ).

2) Setup your cron using "crontab -e". This puts you into the edit window of the text editor that you installed with Cygwin.

Once a cron is setup, view the output of cronevents.exe to see what cron jobs executed:
$ cronevents.exe tail
2008/01/31 16:53:01 [SODO] /usr/sbin/cron: PID 5992: (SODO) CMD (/scripts/load.sh >> /scripts/load.txt )
2008/01/31 16:53:01 [SODO] /usr/sbin/cron: PID 6652: (SODO) CMD (/scripts/connScript.sh >> /scripts/conn.txt)
2008/01/31 16:54:01 [SODO] /usr/sbin/cron: PID 8132: (SODO) CMD (/scripts/load.sh >> /scripts/load.txt )
2008/01/31 16:54:01 [SODO] /usr/sbin/cron: PID 4708: (SODO) CMD (/scripts/connScript.sh >> /scripts/conn.txt)
2008/01/31 16:55:01 [SODO] /usr/sbin/cron: PID 8164: (SODO) CMD (/scripts/connDiff.sh)
2008/01/31 16:55:01 [SODO] /usr/sbin/cron: PID 3372: (SODO) CMD (/scripts/connAlert.sh )
2008/01/31 16:55:01 [SODO] /usr/sbin/cron: PID 4460: (SODO) CMD (/scripts/load.sh >> /scripts/load.txt )
2008/01/31 16:55:01 [SODO] /usr/sbin/cron: PID 7976: (SODO) CMD (/scripts/connScript.sh >> /scripts/conn.txt)
2008/01/31 16:56:01 [SODO] /usr/sbin/cron: PID 6424: (SODO) CMD (/scripts/load.sh >> /scripts/load.txt )
2008/01/31 16:56:01 [SODO] /usr/sbin/cron: PID 2676: (SODO) CMD (/scripts/connScript.sh >> /scripts/conn.txt)

3) Viewing the cron logs are important! The cron.log is place in $HOME/cron.log:
$ cat /cygdrive/h/cron.logThis file was written by the /usr/bin/cronlog script on 20080131_165003
From: root (Cron Daemon)To: SODOSubject: Cron
SODO@SODO02 /scripts/connDiff.sh

X-Cron-Env: ALLUSERSPROFILE=C:\Documents and Settings\All Users
X-Cron-Env: COMMONPROGRAMFILES=C:\Program Files\Common Files
X-Cron-Env: COMPUTERNAME=SODO02X-Cron-Env: COMSPEC=C:\WINDOWS\system32\cmd.exe
X-Cron-Env: CYGWIN=smbntsecX-Cron-Env: NUMBER_OF_PROCESSORS=4
X-Cron-Env: OS=Windows_NTX-Cron-Env: PATH=/cygdrive/c/Program Files/PHP/:/cygdrive/c/oracle9i/bin:/cygdrive/c/oracle9i/jre/1.4.2/bin/client:/cygdrive/c/oracle9i/jre/1.4.2/bin:/cygdrive/c/Program Files/Oracle/jre/1.1.8/bin:/cygdrive/c/WINDOWS/system32:/cygdrive/c/WINDOWS:/cygdrive/c/WINDOWS/System32/Wbem:/cygdrive/c/Program Files/TeaLeaf:/cygdrive/c/Program Files/Python25:/cygdrive/c/Program Files/QuickTime/QTSystem/:"C:/Program Files/Symantec/Norton Ghost 2003/":/cygdrive/c/Program Files/Common Files/Adaptec Shared/System:/usr/bin:/cygdrive/c/Program Files/SSH Communications Security/SSH Secure Shell:/bin
X-Cron-Env: PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 3, GenuineIntel
X-Cron-Env: PROGRAMFILES=C:\Program FilesX-Cron-Env: SYSTEMDRIVE=C:
X-Cron-Env: TMP=/cygdrive/c/DOCUME~1/SODO/LOCALS~1/Temp
X-Cron-Env: TEMP=/cygdrive/c/DOCUME~1/SODO/LOCALS~1/Temp
X-Cron-Env: WINDIR=C:\WINDOWSX-Cron-Env: SHELL=/bin/sh
X-Cron-Env: HOME=//SAN/Users$/SODO

4) My scripts do such things as SSH into other boxes and grab HTTP connection counts. The scripts only ran using the CYGWIN environment variable set to smbntsec.

5) the HOME environment variable must be set properly: http://blog.spikesource.com/crontab.htm suggests running "env" in your cron out to a file to find differences between your user environment and the cron environment. Good idea!

6) don't forget your scripting basics:

7) documentation can help:

Or whatever the latest version happens to be..

Mail That Cron Sends

Change the email that cron sends error messages to:
run cron-config
Add/edit the "From:" address in /etc/ssmtp.conf

Hope these tips help!
Feel free to drop me a line or ask me a question.