tech answer guy: gnuplot

Showing posts with label gnuplot. Show all posts

Wednesday, November 17, 2010

setting up IIS to use shell scripts as CGI scripts

Because I'm old school and love shell scripts, I wanted to kick off a shell script from my web browser. In this case, my shell script would sort through a text file, use gnuplot to output a graphic as a png and then display the script in the browser.

Setup IIS to execute shell scripts
I won't belabor how to setup your web server to execute shell scripts, as the link below Installing CGI Applications in IIS 6.0 explains most of the steps. However, I will call out one caveat that was tricky to setup. I use Cygwin to write my shell scripts, so I wanted IIS to use Cygwin's sh.exe as the shell script command interpreter. The nut here is that I needed to add a special configuration in order to have IIS execute Cygwin's script interpreter. Here's how to do this.

right-click on the cgi folder
click Properties
select the Virtual Directory tab
click Configuration
click Add and type the following:

C:\cygwin\bin\sh.exe "%s" "%s"

Create a Simple Shell Script as a CGI
It is useful to setup a simple script in order to test that IIS is calling sh.exe properly. Here is mine:
#!/bin/sh
echo "HTTP/1.0 200 OK"
echo "Content-type: text/html"
echo
echo "<HTML><HEAD><TITLE>environment variables test</TITLE></HEAD>"
echo "<BODY>"
/usr/bin/env ¦ /usr/bin/sort
echo "</BODY></HTML>"
exit

Remember that this script is essentially hand-rolling an HTTP response. Breaking it down:
1. Send the header:
echo "HTTP/1.0 200 OK"
echo "Content-type: text/html"

2. Make sure you send a blank line (\n\r )
echo

3. After the header and the blank line, send out well-formed HTML
echo "<HTML><HEAD><TITLE>environment variables test</TITLE></HEAD>"

4. Plop some shell functions in there to verify that sh.exe is interpreting commands correctly. In this example, I am printing out and sorting the results of the "env" command. The "env" command prints out the IIS working set of environment variables. Make sure to put the full path in your command string:
/usr/bin/env ¦ /usr/bin/sort

5. Close your HTML
echo "</BODY></HTML>"

6. Exit the script
exit

Reference the script in the HTML of a test page:
<a href="/cgi-bin/env.sh">click me</a>

Here is the output of the script

This is a very simple script, but you can see that the sucker works. As it took a good two hours of finagling, I thought it worthwhile to post these instructions and links to further reading.

My gnuplot script is much more involved. If interested, drop me a line and I can send you that script or maybe post another blog article about how to create it.

Good luck!
TAG

References
Common Gateway Interface (CGI)
Short list of HTTP status codes
Installing CGI Applications in IIS 6.0
Writing CGI Scripts for a Web Server

Monday, August 13, 2007

scripted gnuplot graphs

One of my many tasks as a webmaster is to monitor the traffic to and from our web servers. As a follow up to my quick start guide to gnuplot: /2007/07/quick-start-guide-to-gnuplot-plotting.html
I will expand upon that posts' themes and give you a simple script that will output a nice PNG graphic of a data set without having to use gnuplot interactively.

As in the previous post, my goal is to plot HTTP connections. I want the script to generate the graph for the previous days' activity. Should be simple enough!

As review, my data file looks like this:
20070727151200 215 210 5
20070727151259 191 186 5
20070727151401 207 201 6
20070727151500 186 180 6
20070727151600 165 156 9
20070727151700 167 161 6
20070727151801 182 174 8
20070727151900 197 190 7
20070727152001 197 191 6
20070727152100 187 181 6

* The first column is date/time stamp.
* The second column is total HTTP connections.
* The third column is HTTP connections from the client browser to the cache.
* The last column is HTTP connections back to the origin server.

Here is the script:
!#/bin/bash
TITLE="Caching server HTTP connections"
TODAY="`date +%Y%m%d`000000"
YESTERDAY="`date +%Y%m%d --date="-1 day"`000000"
YMAX=350

gnuplot -noraise <<-finis

set xdata time
set timefmt "%Y%m%d%H%M%S"
set yrange [0:$YMAX]
set ytics 50
set bmargin 3
set grid xtics mxtics ytics
set title "$TITLE"
set xrange ["$YESTERDAY":"$TODAY"]
set term png medium
set output "webconn.png"
plot 'webconn.txt' using 1:3 title "HTTP_WEB" with lines,\
'webconn.txt' using 1:4 title "HTTP_OWS" with lines
finis

You can see I have a few constants up top that set a title and the date range. Note the neat little "date -1" trick in the date command for the YESTERDAY constant. That has come in handy more than once!

I then set a maximum value for the Y axis.

Using the -noraise switch tells gnuplot to not raise the graph window. Next, I use the input redirector (<<) to feed in parameters to gnuplot. The "-finis" is simply a label to the input that I call at the beginning and the end of my input. I then feed in as many parameters as I need. When I am done, I just call "finis" without the hyphen and this turns off the input.

You can see I feed in a lot of the parameters that I had to manually enter during the last post regarding interactive gnuplot. I've thrown a few more in with this example:

set yrange sets the maximum range of the y axis
set ytics sets the spacing of the bars in the y axis
set bmargin sets the spacing of the bottom margin in pixels
set grid xtics mxtics ytics sets your grid preferences
set term sets the output type; in this case, a medium sized png graphic
set output specifies the name of the file

The rest of the commands you already know from my first post on the subject. At the end of all the parameters I set, I make sure to close the input by entering the keyword "finis" that I defined at the beginning of the input redirection. You can use any non-shell script word for the keyword. I just chose "finis" because it means "The End" and has no other special meaning to the Unix shell.

Finally, here is the lovely output:

Hope you liked it!
'sodo

Friday, July 27, 2007

quick start guide to gnuplot: plotting web connections

Early this morning, I was a Total Beginner to gnuplot (http://www.gnuplot.info/), but now I am happy to tell you that I have advanced to the state of simple Beginner. As shown on the home page link above, you can plot pretty much any data set with gnuplot and do it quite prettily. But gnuplot is not the easiest program to get started with and the documentation is pretty cryptic, so I thought I'd share with you what I learned about it via a specific task.

The Task
I wanted to plot the minute by minute HTTP connection data from a web caching server. The connection counts are generated by the output of "netstat -na grep ESTABLISHED" and stored in a simple text file. There is a bit more to that script, but for now, I will limit this discussion to how to get a simple chart out of gnuplot.

A word about caching web servers. As implied by the word "cache", a caching web server stores copies of web pages from an origin web server in order to alleviate the load off that origin server. Upon the first request for a web page, the caching server requests the page from the origin web server and stores that page in memory or on disk. The cache can then fulfill any subsequent client requests for the same page, thus taking the hit and alleviating the load of the request off of the origin server.

Since a caching server functions as an intermediary (proxy) server, I am interested in plotting both the connections from the client browser to the caching server, as well as the connections from the caching server to the origin web server on the backend.

The content of the text file looks like this:
20070727151200 215 210 5
20070727151259 191 186 5
20070727151401 207 201 6
20070727151500 186 180 6
20070727151600 165 156 9
20070727151700 167 161 6
20070727151801 182 174 8
20070727151900 197 190 7
20070727152001 197 191 6
20070727152100 187 181 6

* The first column is date/time stamp.
* The second column is total HTTP connections.
* The third column is HTTP connections from the client browser to the cache.
* The last column is HTTP connections back to the origin server.

As you can see from this small amount of data, it looks like the cache handles roughly 97% of the requests for the web site and 3% are directed back to the origin servers.

So let's make a nice graph of this data, dammit!

Install gnuplot
If you're on Fedora, "yum install gnuplot" will do the trick. On Debian, "apt-get install gnuplot gnuplot-doc" should do it for you. Otherwise, download the source from http://www.gnuplot.info/ and you'll be off and running.

You can run gnuplot interactively or via command shell. For the purpose of this discussion, I will discuss the interactive interface for gnuplot. Once the software is installed, type "gnuplot" to start the program:

[root@computer ~]# gnuplot
gnuplot>

Getting Help
If you type "help" at the prompt, you'll get a list of help topics available:

Help topics available:
    batch/interactive bugs              commands          comments
    coordinates       copyright         environment       expressions
    glossary          graphical         help-desk         introduction
    line-editing      mouse             new-features      old_bugs
    plotting          set               show              startup
    substitution      syntax            time/date

To get information about a particular topic, simply type "help <topicname>" like so:

Help topic: plotting
 There are three `gnuplot` commands which actually create a plot: `plot`,
 `splot` and `replot`.  `plot` generates 2-d plots, `splot` generates 3-d
 plots (actually 2-d projections, of course), and `replot` appends its
...

You can press the return key for more text and if you're at a "Subtopic of.." menu, you can get a list of available options by typing a question mark, like so:

Subtopic of plotting: ?
Subtopics available for plotting:
    styles

My First Plot!
The easiest thing I did is try a test plot of some data. I typed the following:
gnuplot> plot 'webconn.txt'

This gave me a rather ugly plot of the first column of data using red crosshairs for every observation:

I learned a bit about customizing my plots from here:
http://www.duke.edu/~hpgavin/gnuplot.html
And here:
http://www.gnuplot.info/docs_4.0/gpcard.pdf

Let's Get Fancy
I didn't like the red crosshairs, so I found an option to change the graph to a line graph:
gnuplot> plot 'webconn.txt' with lines

And then a more advanced graph specifying the exact columns for use (the x and y axes) with the "using" parameter. I also threw in the axis title for good measure:
gnuplot> plot 'webconn.txt' using 1:3 title "HTTP_WEB" with lines

And finally, multiple data columns:
gnuplot> plot 'webconn.txt' using 1:3 title "HTTP_WEB" with lines,'webconn.txt' using 1:4 title "HTTP_OWS" with lines

You can also shorten the syntax of the command by typing "u" for "using", "w" for "with", "t" for "title" and "l" for lines, like so:
gnuplot> plot 'webconn.txt' u 1:3 t "HTTP_WEB" w l,'webconn.txt' u 1:4 t "HTTP_OWS" w l

The Plot Thickens
Oooh. Getting somewhat cryptic now, eh? But something is wrong with the data in the graph. Why does it have these strange lines every so often?

After digging through my data, I saw that these ugly, flat lines occurred at each change in the day. Oh! gnuplot is interpreting the date/time string as a number! At the change of the day, the hour jumps from 200707212359 to 200707220000. Numerically, this is a jump of 7641 (220000-212359). I needed to make gnuplot think my xaxis is a date format. I found the solution with a combination of the following commands:
set xdata time - this sets the data format of the xaxis to a time format
set timefmt "%Y%m%d%H%M%S" - this sets the time format to match the timestamp in my data file

Once I set these two parameters, I got quite a lovely graph using the last command:
gnuplot> plot 'webconn.txt' u 1:3 t "HTTP_WEB" w l,'webconn.txt' u 1:4 t "HTTP_OWS" w l

Finally, I wanted to zoom into a specific day. There was another nice command I found:
set xrange ["200707250000":"200707260000"]

This made searching on a specific day very easy!

I hope you enjoyed this short tutorial. I will endeavour to add more information as I learn more about this exciting tool.

Reference
Great examples herein

Cheers!

Cacasodo