digging down into a linux process

So I inadvertantly exited out of my Fedora X server without saving or exiting my VMware Player's open virtual machine:

This left my vm in an unknown state. You could still see the lock files in the Virtual Machine's directory:
[sodo@ogre ~]$ ll Virtual\ Machines/Windows\ 7\ x64/
total 10976188
drwxrwxr-x 3 sodo 4096 2011-01-07 12:53 caches
-rw-r--r-- 1 sodo 320659 2011-01-18 14:41 vmware-0.log
-rw-r--r-- 1 sodo 359599 2011-01-18 14:14 vmware-1.log
-rw-r--r-- 1 sodo 565465 2011-01-08 00:14 vmware-2.log
-rw-r--r-- 1 sodo 317488 2011-01-18 17:33 vmware.log
-rw-rw---- 1 sodo 8684 2011-01-18 14:42 Windows 7 x64.nvram
-rw------- 1 sodo 2103836672 2011-01-18 17:35 Windows 7 x64-s001.vmdk
-rw------- 1 sodo 2121203712 2011-01-18 17:35 Windows 7 x64-s002.vmdk
-rw------- 1 sodo 2145255424 2011-01-18 17:35 Windows 7 x64-s003.vmdk
-rw------- 1 sodo 2145976320 2011-01-18 14:49 Windows 7 x64-s004.vmdk
-rw------- 1 sodo 955 2011-01-18 14:41 Windows 7 x64.vmdk
drwxrwxrwx 2 sodo 4096 2011-01-18 14:41 Windows 7 x64.vmdk.lck
-rw-rw---- 1 sodo 1073741824 2011-01-08 00:14 Windows 7 x64.vmem
-rw-rw---- 1 sodo 0 2011-01-07 12:37 Windows 7 x64.vmsd
-rw-rw---- 1 sodo 182610705 2011-01-18 10:18 Windows 7 x64.vmss
-rwxrwxr-x 1 sodo 2477 2011-01-18 14:42 Windows 7 x64.vmx
-rw-rw-r-- 1 sodo 1645 2011-01-07 12:53 Windows 7 x64.vmxf
drwxrwxrwx 2 sodo 4096 2011-01-18 14:41 Windows 7 x64.vmx.lck

But after some Googling, there seemed to be no way to restart the orphaned vm without killing the process that was hanging out there. Before I killed the vm process, I researched it to find out more about it. First, I did a search on the process:
[sodo@ogre ~]$ ps -ef | grep vmx
sodo 4629 1 13 10:18 ? 00:31:48 /usr/lib/vmware/bin/vmware-vmx -ssnapshot.numRollingTiers=0 -sRemoteDisplay.vnc.enabled=FALSE -s vmx.stdio.keep=TRUE -# product=8;name=VMware Player;version=3.1.3;buildnumber=324285;licensename=VMware Player;licenseversion=6.0; -@ pipe=/tmp/vmware-sodo/vmxb90ce351150180d7;readyEvent=90 /home/sodo/Virtual Machines/Windows 7 x64/Windows 7 x64.vmx

I saw that the process number was 4629. The command that started the process was vmware-vmx:
[sodo@ogre ~]$ ps -p 4629
PID TTY TIME CMD
4629 ? 00:31:48 vmware-vmx

Digging into the process directory, I saw the status of the process was sleeping:
[sodo@ogre ~]$ cat /proc/4629/task/4629/status
Name: vmware-vmx
State: S (sleeping)
Tgid: 4629
Pid: 4629
PPid: 1
TracerPid: 0
Uid: 500 500 0 500
Gid: 500 500 500 500
Utrace: 0
FDSize: 256
Groups: 500
VmPeak: 3281396 kB
VmSize: 3131036 kB
VmLck: 0 kB
VmHWM: 1409588 kB
VmRSS: 1343972 kB
VmData: 2727812 kB
VmStk: 288 kB
VmExe: 6784 kB
VmLib: 134888 kB
VmPTE: 3212 kB
Threads: 1
SigQ: 0/80092
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: fffffffe7ffbfeff
SigIgn: 0000000000301000
SigCgt: 0000000193c9eeef
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 4744582
nonvoluntary_ctxt_switches: 7620

Alas, the vmware service offered no consolation..I could not shutdown the daemon gracefully:
[sodo@ogre ~]$ sudo /etc/init.d/vmware restart
Stopping VMware services:
VMware USB Arbitrator [ OK ]
At least one instance of VMware Player is still running.
Please stop all running instances of VMware Player first.

VMware Authentication Daemon [FAILED]

And so, good readers, I was forced to kill the process. And hard, as I had to add the -9 switch to kill, which effectively says "kill the process and don't do any cleanup". "Kill that sucker, but good!"

[sodo@ogre Windows 7 x64]$ kill 4629

[sodo@ogre Windows 7 x64]$ ps -ef | grep 4629

[sodo@ogre Windows 7 x64]$

Yes, this harsh kill command did the trick. And then I had to explain to my poor Win7 vm why I had pulled the plug so harshly:


Oh cruel fate, why dost thou mock me?
TAG

Reference
http://aplawrence.com/SCOFAQ/FAQ_scotec6killminus9.html

removing unnecessary startup programs in 2003/XP

Or better yet: "How to find and destroy rogue processes"

One thing that that makes me angry is when unnecessary processes take up too much memory or kernel space. This especially infuriates me when the lack of memory or kernel space ends up bringing my web server down! So this morning, I had this situation happen on my recently reimaged XP notebook, a Dell Latitude 600C. This is a work laptop and as such, is locked down by my security department. I always like when security locks things down, so then I can find ways around the lockdown! :)

In this case, IIS crashed because of there were too many programs using kernel memory. How do I know this? Well, after the crash, I inspected the event logs. In the system event log, I found the following error:
"Symantec antivirus auto protect could not scan file c:\windows\system32\xxx.dll for viruses due to low kernel stack"

A quick search on Google provided a few URLs that gave me hints as to what was going on:
http://service1.symantec.com/SUPPORT/ent-security.nsf/pfdocs/2002071208532048?Open&docid=2002071208532048&nsf=ent-security.nsf&view=docid
http://techrepublic.com.com/5208-11192-0.html?forumID=52&threadID=162216

As usual, I decided to take the low road and resolve my problem in the quickest way possible. The resolution was to disable the myriad of useless programs that execute when my system starts up. These include:
ctfmon.exe:
http://support.microsoft.com/kb/282599
igfxtray.exe
http://www.liutilities.com/products/wintaskspro/processlibrary/igfxtray/
hkcmd.exe
http://www.neuber.com/taskmanager/process/hkcmd.exe.html
pctspkr.exe
http://www.neuber.com/taskmanager/process/pctspk.exe.html

Also, I chose to disable a few unnecessary media and pdf programs that started on bootup.
iTunesHelper/qttask/Acrobat Assistant

So how did I find out what was running and how do disable them from starting up? I used Mark Russinovich's great Process Explorer program
http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx

Microsoft has bought him out and rightly so, because this guy REALLY knows the Windows OS inside and out. He's made his expertise available through a number of very useful programs available here:
http://www.microsoft.com/technet/sysinternals/default.mspx

My first task was to identify the programs that were hogging up memory unnecessarily. I downloaded and installed Proc Explorer from the link above and ran it. Running the program gives you a screen like this:



If you roll down to the bottom of the Process Explorer display, you'll see a bunch of processes hanging off of the "explorer.exe" process. Under that main branch, you'll see a ton of running processes, some of which you may recognize if you've ever looked at the running processes using Task Manager. I looked at this list of running processes and decided which ones that I could safely disable. A number of them were just helper apps, like the above mentioned media player related utilities. I also noted other helper apps that I didn't need:
trackpad (SynTPLpr.exe/SynTPEnh.exe/point32.exe)
fax (faxctrl.exe)
graphics card (igfxtray.exe)
keyboard (hkcmd.exe)
modem (pctspk.exe)

I just hate all that crap getting loaded and wasting memory.

One great thing about Process Explorer is that you can kill rogue processes right then and there within the program. Just identify the process that you'd like to kill in the list, right-click and select Kill or Kill Process Tree if the rogue process has spawned a number of child processes:


To round out the discussion on Process Explorer, PE includes an information-rich task manager system information window you can access via the menu or just by pressing CTRL-I. It has more information that the graphic Task Manager than Microsoft gives you and shows you real time updates of top processes while the graphs scroll by:


So how did I actually disable these programs from executing on start up? For that, msconfig is your nearest and dearest friend, especially if your XP Taskbar has been locked down by Security! MSconfig is a program provided by Microsoft that allows you to disable startup programs and generally wreak havoc on your system, so be damn careful when you use it. To start MSconfig, click Start -> Run -> type in "msconfig" and press enter. You should see a screen like this:



Click the Startup tab. You'll see a bunch of startup items listed, the command that started them and the registry location of where they are set to run. MSconfig makes it easy for you to disable these programs by simply unchecking the programs you don't want to run on Startup. Here is the list after I was done deselecting the unnecessary programs on my machine:



After you deselect these memory hogs, go ahead and click OK. You'll then get a message to reboot now or later. If you are overzealous and have clicked too many startup items off, your PC may not come up correctly. So be careful to not go too crazy and remove everything from automatically starting up. Of course, an ounce of prevention is worth a pound of cure. And if you do screw something up, you'll probably learn something in the process of fixing it. So there is a reason that things happen the way they do.

Go ahead and reboot. Hopefully, when the system restarts, you'll have a bit more free memory and probably a faster machine if you've killed processes that were hogging your CPU. Good luck!

PS - If you want to do things the good old fashioned way (ie, edit the registry!), edit this key in regedit:
My Computer -> HKEY_LOCAL_MACHINE -> Software -> Microsoft -> Windows -> CurrentVersion -> Run

PPS - Of course, you'll need to click Start -> Run -> "regedit" and hit enter to get into regedit first!

PPPS - Here's the great Mark Russinovich on a bug hunt using Process Explorer. Learn from the master!