measuring performance while using VMware Server

My first performance posting regarding VMware Server’s love of large fast drives (/2007/06/performance-note-for-vms-they-love-fast.html) was sufficient, but not detailed enough for some readers. So I thought I’d give you guys a bit more technical information on measuring performance while using VMware Server.

Configuration
I recently created a rather large vm. It is a 45GB XP Professional guest OS built using VMware Converter 3.0.0 build 39557 and runs under the latest VMware Server version 1.0.3 build-44356. VMware Server is running on my desktop, XP Professional running on a Dell Precision 670 workstation with dual 3Ghz Xeons and 3.2GB RAM.

Before I started the test, I already had a couple of virtual machines already running:
- a Windows 2000 Advanced Server vm
- a Fedora Core 6 vm

Monitoring Performance
On my host OS, I started Performance Monitor with the following counters:
* Physical Disk Object -> Avg. Disk Write Queue Length
* Physical Disk Object -> Avg. Disk Read Queue Length
* Physical Disk Object -> Avg. Disk Queue Length
* Processor Object -> % Processor Time

Average Disk Queue Length is a sum of the number of read and write requests (Read/Write Queue Lengths) queued for the selected disk. Obviously, Average Disk Read Queue Length is the number of read requests queued for the selected disk and Average Disk Write Queue Length is the number of read requests queued. Such things as disk speed, disk cache size, i/o bus speed and RAID configuration (if any) affect disk throughput, which is the amount of read and write requests that the disk system can handle at any one time.

I also use the memory counters available in Task Manager:


The Test
I saw that my system slowed down considerably while using my big XP virtual machine, so I started Performance Monitor and selected the key indicators shown above plus “% Processor Time”, a measure of the amount of CPU used. After I started up my XP vm, it booted up to the logon prompt and I logged in. Immediately, I noticed my system slowed down to a crawl. If I tried to open any applications, those applications would take five minutes to start. So where’s the problem?

CPU Not a Problem
For this time period, I first looked at my CPU data in Performance Monitor. I note that since CPU is low, between 5-35%, CPU is not the cause of any system slowdown:


Is Memory the Problem?
Next, I took a look at the stats in Task Manager. I notice that my 1GB pagefile is being utilized:


However, Total Physical Memory does not exceed Total Commit Charge (shown in the above graphic). This would indicate insufficient system RAM. From these stats, it looks like my system is not memory bound.

Disk Queuing Through the Roof!
Looking at the Disk Queue stats, I noticed that disk queuing went through the roof:


In the Performance Monitor graph shown above, the black line is Avg. Disk Queue Length. You’ll notice that if you add the values of Avg. Disk Write Queue Length and Avg. Disk Queue Read Length together at any one point in time, that the sum of those values is equal to Avg. Disk Queue Length.

In a system performing optimally, there should be little disk queuing happening. According to the help menus in Performance Monitor, the recommended values of Avg. Disk Queue Length should be the number of spindles plus 2. In this case, I have a stripe set (RAID0) of two drives. There are two spindles corresponding to the spindle in each drive of the stripe set, so the optimum Avg Disk Queue Length of my system should be less that four.

In the image above, you can see values of between 25 to 40 (note the scale) for Disk Queue Length. There is some pretty heavy queuing going on here! I note that my machine is still barely responsive when I try opening any applications. You'll notice that there are breaks in the Performance Monitor graph. This indicates that my system is so bogged down that the Performance Monitor program is freezing. Youch! This screen was captured after about ten minutes since I started the VM. After about 20 minutes, the disk queuing stops and I am able to open applications normally:


You can see in the above graphic that the high disk queue condition stops after I’ve logged into my XP vm and the guest OS fully loads. For the 45GB XP guest OS running on my dual proc Dell Precision workstation, this took a full twenty minutes! Wow. That is a LOT of disk queuing. Again, CPU and memory stayed fairly constant during this experiment. CPU was roughly 30% and there was a little less than 2GB of RAM available during the process.

One note: if you don't like the default scale of 100:1 on the Performance Monitor chart, you can change that default scale by:
1) right-clicking on the statistic in the legend of the chart
2) select Properties
3) click the Data tab
4) choose another scale under the Default Scale dropdown menu

Conclusions
Perhaps the VMware engineers can explain why the VM took so long to free up my disk resources, but I suspect that it is simply the fact that the virtual machine is so big (45GB) and that VMware Server cannot handle a vm of that size efficiently. I’d really need a server class machine with super fast disk i/o to handle the intense read/write activity.

Perhaps you have an interesting VMware performance story? If so, drop me a line at cacasododom@gmail.com or just comment below.

Have a good weekend everybody!
'sodo

performance note for VMs: they love fast drives

Because virtual machines are essentially big files, you will benefit if the disks they are stored on are fast. So, if you stripe a couple SATA or SAS drives, this should really help speed things up. Enterprise users will obviously have access to better firepower (RAID 1+0, RAID 5, etc), but I'm framing this in the context of the tech dabbler just getting his feet wet with the technology.

I bring this issue up because I put an old, circa 1999 20GB spare drive in my XP workstation and used that drive as the conversion destination of my local XP Professional Workstation to vm. After the conversion, which was successful, I started the vm and the performance was horrible! It was almost like the vm hung or was frozen. In truth, the vm was just extremely slow because the IDE drive was circa 1999 and had little buffer for the 8GB+ vm. I didn't immediately know what was happening, so I used XP's Performance Monitor to view average disk queue length.


Average disk queue length is a rough measure of disk performance, but it is a useful gauge of i/o problems. For me, I find that values of over 20 indicate performance issues. Your mileage may vary.

One note: if you don't like the default scale of 100:1 on the Performance Monitor chart, you can change that default scale by:
1) right-clicking on the statistic in the legend of the chart
2) select Properties
3) click the Data tab
4) choose another scale under the Default Scale dropdown menu

UPDATE 7/6/2007: I've expanded my discussion of performance monitoring VMware Server here:
/2007/07/measuring-performance-while-using.html

Also, I noticed my performance seems to suffer if you use the "Split Disk Into 2GB files" option. As well, I selected "Allocate all disk space now" for better performance.

In regards to this option, I found a bug in the VMware Converter gui. When I initially went to configure the conversion of the local machine, I was not able to deselect the "Split disk into 2GB files" option for the destination drive (a second local hard disk) that I wanted to use.


The checkbox was greyed out and unselectable. Trying to fix the problem, I returned to the destination dialog box, changed my destination drive to a network share and the option suddenly became available. On a hunch, I then went back to the destination location window, changed the destination drive to the original local drive and the "Split disk into 2GB files" option was now available.

I don't know why the option was made unavailable at first, but I was glad to be able to route around the error.