using NetQoS to diagnose network congestion

We use a backend connection from our corporate offices to our hosting provider. On Monday towards the end of the day, I noticed a big slowdown in our RDP (Remote Desktop) connectivity to our Windows servers at the hosting provider. Monday is normally a very busy day for network traffic in the office as well as our main website. So I wondered where the traffic was coming from. Luckily, our network engineers had recently purchased a suite of network management tools from NetQoS (http://www.netqos.com/).

The engineers setup NetQoS monitors on the backend pipe to allow us to see the traffic going through our T1 speed, Frame Relay connection. The nice thing about NetQoS is the ability to break down the traffic going back and forth through the pipe by protocol and port. In the below graph labeled "Stacked Protocol Trend In" for this past Monday, you'll see a green blob after 15:00.

"Trend In" is data inbound to the monitored router. In our case, inbound is into our corporate network. "Trend Out" is data going outbound from the monitored router. In like fashion, outbound is out to our managed hosting provider.

As indicated by the key on the right side of the diagram, this is a spike in SSH (port 22) traffic:


Interesting! The funny thing is that a similar network performance degradation happened the previous week. I changed the date range for the NetQoS reports and saw an even larger, longer lasting spike in SSH traffic:


OK. So it seems we have a problem, but what is the cause?

We typically use SSH to monitor various Unix servers in the environment. One of the administrators has a bunch of X sessions that display performance statistics from a number of servers at the hosting facility. This X session traffic is done over SSH port. He told me that while the high SSH traffic issue was happening, one of his X session graphs was not refreshing properly. I suspected that since he was the main administrator of the servers and that he had about twelve X sessions open, that there is a bug in the X session software that was making one or more of his monitoring sessions create this spike in traffic.

At this point, he still has his X sessions open, but we will wait until the next spike in SSH traffic to try to determine if it is this admins' X sessions that are the culprit. At that time, we will probably shut down his sessions to see if that fixes the problem. I will update the blog next week to let you know how it goes.

Anyway, I'm glad that NetQoS helped us troubleshoot the problem. It gives us some good insight into network traffic by TCP/IP application.

UPDATE: another great tool to diagnose your local computer's network traffic is IPTraf (http://iptraf.seul.org/). I will review this software which runs under Linux and give a basic description of how to use it in an upcoming post.

cheers!

Mac RDP / killing a process under Mac OSX

It seems that RDP for Mac has a tendency to hang if you Apple-Quit out of the program. Since I'm relatively new to my MacBook Pro 17", I was rebooting in order to free up RDP for another session.

Tired of my own laziness, I decided to do things the right way and find out how to Force Quit an application in OS X. I'd love to take credit for such a well-written document, but the folks at Ohio State beat me to it and put together a great little page describing how to kill a process under Mac OSX. I couldn't have done it better:
http://8help.osu.edu/1253.html

Be aware that after killing RDP using Force Quit, it does take a couple of tries in order to login again to the remote server. Also, you'll experience less RDP hangs if you gracefully logout of your Windows session by using Start -> Disconnect, rather than bluntly Apple-Quit'ting out of RDP. Word to the wise..go gently into that good night.

Also, you can only run ONE RDP session on your MacBook Pro at a time, so its use is limited if you need to login to many servers. One idea to get around this limitation is from your Mac, login to one jump server that will allow you to get to all other boxes you need to support. This way, you can have your Mac Cake and eat it too!

Finally, if you've left a session open and another RDP session is started from another server going to the same remote host that your Mac previously had control over, RDP on the Mac tends to hang. Obviously, this necessitates a Force Quit again.

right-mouse click for XP/Vista on MacBook / Remote Desktop for Mac

As I run both XP, Vista and Fedora in VMware Fusion on my MacBook Pro 17", I was struggling with the fact that since the MacBook only has one button below the trackpad, I did not have access to a right mouse button to get the contextual help and properties menus in XP and Fedora.

Well, if I had looked longer and harder in the System Preferences -> Keyboard and Mouse -> Trackpad properties of my MacBook, I would have found an obscure option called "Place two fingers on trackpad and click button for secondary click."


This option is an advanced finger technique applied to the trackpad. To enable your context-sensitive right-mouse click, just stick two fingers on the trackpad and click an object in XP or Fedora. Voila! You should see your right-mouse option menu appear! Sweet!


Enabling this option allowed me to do everything I needed to do in XP and Fedora, as well as in Remote Desktop Connection (RDC) for Mac:
http://www.microsoft.com/mac/otherproducts/otherproducts.aspx?pid=remotedesktopclient

Nice!

One caveat, though, is that apparently, this option is not available in MacBook 15" models. I don't know why this is, but if I have time, I will find out.

Finally, I learned about three additional key combinations while editing this post. First was using Function (fn) key in combination with the directional keys in the lower right corner of my MacBooks' keyboard. The Function (fn) key is in the lower left hand corner of the corner. The directional keys on the opposite site of the keyboard are PageUp PageDown, Home and End.

Secondly, Fn-delete works as a normal delete key within a Windows session when using Mac RDC. Finally, press the Mac delete key alone to function as a backspace in a Windows RDC session.

Both of these keystrokes are very useful and available cross-platform when you use the freely available Remote Desktop Connection from Microsoft.

7/30/07 Update: I put a more definitive list of keyboard shortcuts and key symbol names here for your enjoyment:
/2007/07/macbook-pro-keyboard-shorcuts-and.html

Here is the most awesome list of key bindings for the mac:
http://support.apple.com/kb/HT1343

cheers!
'sodo