protecting your data, locally and globally

I've spent a number of years trying various backup methods for my Linux box.  I think I finally have a pretty good one down.  The main idea is to setup my system in order to make backup and restore easier. This setup involves two components:
- a data source: my documents, videos, audio files and pictures are stored locally on a redundant  hardware RAID5 set
- the separation of system and data partitions on different physical devices (hard drives)

Backup Strategy
My actual backup strategy comes in three parts:
- an archive solution: fsarchiver to backup the system and data partitions
- network backup: a network backup device such as a NAS or in my case, a Drobo
- global backup storage solution: unlimited CrashPlan account

This backup strategy has been working well, though not without hiccups along the way.  It protects my important data by providing redundancy at multiple levels.  At a high level, here is how this is implemented:
- disk redundancy via RAID5 set
- local redundancy via network addressable storage
- global redundancy via CrashPlan if my house is destroyed

More specifically:
- my Linux system drive, Fedora 17, is one physical SATA drive
- my data drive is a hardware RAID5 unit using a 3Ware 9650SE with four physical SATA drives
- when I install a new OS, I use symbolic links (screen cap) in my user's home directory to point to my data, explained below

The bottom line is that no single backup method should be your entire backup strategy.  If you only have two of these methods implemented, you're better off than most people.

System Setup
Symbolic links are the key to segment the system partitions from your data partitions.  Segmenting is important because it separates your system from your data.  With this separation, it is much easier (from a Linux perspective), to upgrade and try different versions of Linux on your system drive, while your data drive stays essentially untouched and less prone to upgrade or experimentation tragedies.  Welcome to Linux!

Technically, here is how segmentation is implemented.  On my system drive, I mount my data partition.  In this example, I'm using /mnt as the mount point for my ext4 data partition:
[sodo@computer ~]$ cat /etc/fstab
/dev/mapper/vg_computer-lv_root /                    ext4    defaults        1 1
/dev/mapper/vg_computer-lv_home /home                ext4    defaults        1 2
/dev/mapper/vg_computer-lv_swap swap                 swap    defaults        0 0
/dev/mapper/vg_ogre-lv_root /mnt                     ext4    defaults        0 0

I have my content folders on the data partition that I will symbolically link from my system drive:
[sodo@computer mnt]$ ls -ltr /mnt
total 36
drwxr-xr-x. 16 sodo sodo 4096 Dec 21 19:04 MusicLibrary
drwxrwxr-x.  9 sodo sodo 4096 Jan 12 12:07 videos
drwxr-xr-x.  8 sodo sodo 4096 Feb  1 10:57 doc
drwxrwxr-x.  8 sodo sodo 4096 Mar  2 09:26 pictures

I then create symbolic links from my user's home directory to the equivalent directories that I've setup on my data partition:
[sodo@computer ~]$ ls -l | grep '^l'
lrwxrwxrwx.  1 sodo sodo    8 Oct 23  2011 Documents -> /mnt/doc
lrwxrwxrwx.  1 sodo sodo   18 May 30  2011 Music -> /mnt/MusicLibrary/
lrwxrwxrwx.  1 sodo sodo   14 Sep  2  2011 Pictures -> /mnt/pictures/
lrwxrwxrwx.  1 sodo sodo   12 May 28  2011 videos -> /mnt/videos/

With the symbolic links in place, I've made the link from my system to my data.

The Archive-Backup Process
I am going to use the terms "archive" and "backup" synonymously.  The overview is that I'll show how I back up my data partition using fsarchiver and then I'll copy those backups to both my local network and global storage solutions.  FSarchiver is a bit different than regular backup systems in that it archives entire filesystems only.  It is not a file-based backup method.  So when you go to save or restore a filesystem, you specify a filesystem to backup and have limited ability to exclude (but not include) directories or files with the "exclude" switch only (as of 5/2016).

Below, I show the archive process for the data partition.  Feel free to extrapolate the information herein to do the same for your system partition.

The Core Archive Process
1) check the used space on the source partition to be archived, as well as the available space on the destination/target for the backup (fill in missing info).  From the below example:
a. source filesystem (the data partition): vg_ogre-lv_root
b. destination partition (for backup storage): vg_computer-lv_home
[sodo@computer ~]$ df -H
Filesystem                       Size  Used Avail Use% Mounted on
/dev/sda1                        508M   80M  403M  17% /boot
devtmpfs                         5.3G     0  5.3G   0% /dev
/dev/mapper/vg_computer-lv_root   53G   15G   36G  30% /
/dev/mapper/vg_computer-lv_home  2.0T   .2T  1.8T  67% /home
/dev/mapper/vg_ogre-lv_root      4.5T  1.4T  2.9T  32% /mnt

2) verify available space on the destination filesystem
The source partition is using 1.4TB on vg_ogre-lv_root and I have 1.8TB available on the destination for the backup, lv_home.  So..good to go.

3) if there is enough space on the target filesystem, prepare to run fsarchiver and unmount the data partition.
In order to keep the filesystem from being updated during the archive process, fsarchiver asks to unmount the target filesystem before making the backup.  Like so:
[sodo@computer ~]$ sudo umount /mnt/

The nice thing about the split system-data partition setup is that it is unnecessary to load a Live CD in order to backup the data partition.  Normally, one has to boot with a LiveCD in order to backup the system partition.

4) Once the filesystem is unmounted, run fsarchiver.  As one of the destinations for the backup is the cloud, use the -c option to encrypt with a password:
[sodo@computer ~]$ sudo fsarchiver -j8 -c [password] -o savefs ~/f17backup/backup_lv_root.fsa /dev/mapper/vg_ogre-lv_root

The archive of my 1.2TB drive took five hours on my eight-core, 1.6Ghz Dell SC1430.
[sodo@computer ~]$ ll ~/backup/backup_lv_root.fsa 
-rw-r--r--. 1 root root 1186229328445 Mar  5 08:32 /home/sodo/backup/backup_lv_root.fsa

Copying the file to network-based storage
I have a 2.5TB CIFS (Windows share) created on my Drobo.  On my Linux box:
1) I mount the Drobo filesystem:
[sodo@computer ~]$ sudo mount -t cifs //drobo/Linux /mnt/drobo -o credentials=/home/sodo/smb.credentials

2) Copy the archive to it:
[sodo@computer ~]$ sudo cp -rp ~/f17backup/ /mnt/drobo/

Copying the file over my home network took about 16 hours.

3) Review the archive info (the -c switch allows you to enter a password for the archive):
[sodo@computer ~]$ fsarchiver -c - archinfo /mnt/drobo/f17backup/backup_vg_ogre-lv_root.fsa
Enter password: 
====================== archive information ======================
Archive type: filesystems
Filesystems count: 1
Archive id: 5131fa94
Archive file format: FsArCh_002
Archive created with: 0.6.15
Archive creation date: 2013-03-05_01-20-04
Archive label:
Minimum fsarchiver version: 0.6.4.0
Compression level: 3 (gzip level 6)
Encryption algorithm: blowfish

===================== filesystem information ====================
Filesystem id in archive: 0
Filesystem format: ext4
Filesystem label:
Filesystem uuid: 3ffe7328-8f96-4028-bd79-5f644a030fc2
Original device: /dev/mapper/vg_ogre-lv_root
Original filesystem size: 4.02 TB (4416677830656 bytes)
Space used in filesystem: 1.22 TB (1338969956352 bytes)


A Word About the Drobo
The Drobo has been one of the easiest storage and backup solutions that I've ever used.  It integrates seemlessly with Time Machine for my MacBook and I've created a Windows share on the device in order to copy my Linux archive.

Over the past few years, I've expanded the drives within it about three times now.  I went from four 500GB drives, to four 1TB drives and one 500GB to my configuration as it is today, five 1TB drives.  The drive upgrades were easy, though time consuming.  I removed each of the older drives one at a time and replaced them with the larger drives.  Each time a drive was upgraded, the Drobo would automatically and non-destructively rebuild it's storage protection.  The Drobo's storage protection is called BeyondRAID, Drobo's own custom algorithm on top of RAID.

The integration with Mac is seemless; however, the Windows/CIFS file share can be a bit wonky, as the share has a tendency to become unavailable for whatever reason.  The resolution is to shutdown and restart the Drobo and that seems to fix the problem.

Cloud-Based Storage
A last layer of protection above and beyond the local and network copy of my data is to copy the encrypted archive to a cloud-based solution.  The purpose is to protect my data in case of a natural disaster that destroys all my local storage media.  With the increasing amount of natural and man-made disasters happening these days, I've recently invested in a data protection plan with www.crashplan.com.  I got an unlimited plan to store my 1.2TB of data to the cloud.  Most of that data is audio, video and image files.

After tweaking the CrashPlan app to pump more data through my local and wide area network (from 1280KB and 2560KB, respectively, to about 6400KB for both), it took about two weeks to upload this amount of data to CrashPlan's cloud!

In sum, if you have a lot of data, all these procedures take time.  From backing up, local network copy and then cloud copy.  If you're using Linux, you probably have the stomach for all this.  In the end, though, you'll have a backup solution that is pretty solid and relatively easy to implement unlike custom scripted solutions.

Love to hear any comments on how you backup your systems.

ciao!
TAG

References
http://crazedmuleproductions.blogspot.com/2010/02/fsarchiver-good-backup-for-ext4.html
http://www.drobo.com/how-it-works/beyond-raid.php
http://support.crashplan.com/doku.php/recipe/stop_and_start_engine

Here are some of my earlier articles on fsarchiver and a review of the Drobo.

capturing a live stream from the GoPro Hero 2 to a PC

I wanted to make a multi-camera video using Linux of my new drum set, the Roland TD-30K.  Also, I didn't want to fuss with storage media; in other words, the dance of:
- record video to a device
- pull out the storage media  from the device (Compact Flash/tape/SDHC card, what have you)
- connect the storage media to the PC
- copy the files to the PC

What a pain that sh1t is. So the second goal was to record all three streams direct to the PC.  Now, I have an older JVC HD10U 720P video cam.  I knew I could grab the stream over firewire.  Also, I have a Logitech C910 webcam.  With the Logitech, I knew I could grab its video as well.  I had two other cameras as potential for this experiment:
- my lovely Canon 5D or
- the GoPro Hero 2

The Canon wouldn't work, as there is no streaming option by default.  I'd need some hardware HD streaming solution to do that:
http://oliviatech.com/asus-wicast-streaming-wireless-hd-video-from-camera
http://www.fcp.co/dslr/rigs/81-stream-video-from-your-dslr-over-wifi
http://www.blackmagic-design.com/products/intensity/

However, with the release of the GoPro iPhone app, the GoPro guys have coded a way to stream; it was just a question of whether or not I could hook into it.  After doing some digging on this great thread over at the GoPro User Forums (http://goprouser.freeforums.org/howto-livestream-to-pc-and-view-files-on-pc-smartphone-t9393.html) and with a healthy dose of experimentation, I was able to do it.  More for my notes than anything else, here's how you can stream from your GoPro Hero 2 to your PC.

Requirements
- GoPro Hero 2
- GoPro WIFI BacPac
- PC with FFmpeg installed (my test results were conducted on Fedora 17 Linux)
- Validate that your Hero 2 and WIFI BacPac are up-to-date with the latest firmware versions. As of this writing on 2/17/13, those versions are:
Hero 2: 8.12.222
BacPac: 3.4.1

If the software on these devices is not up-to-date, the below instructions won't work for you.  Also, I don't know if these commands work on a Hero 3, as I don't have one.  Though someone apparently got it to work here:
http://goprouser.freeforums.org/gopro-hero-3-wifi-streaming-and-browser-access-t10531.html

1) Update the firmware on your BacPac and GoPro Hero 2 if they are not updated!
2) turn on the BacPac
3) turn on the GoPro Hero 2
4) When turned on, press the menu button on the WIFI BacPac.  The screen of the Hero 2 will give you the option to connect to a Phone or Tablet.  Select this option.

This is interesting because when you select the option, the BacPac creates it's own wireless access point at the IP 10.5.5.9.  I created an /etc/hosts file entry for the BacPac so that I didn't have to type the IP all the time:

[sodo@computer ~]$ ping gp
PING gopro (10.5.5.9) 56(84) bytes of data.
64 bytes from gopro (10.5.5.9): icmp_req=1 ttl=64 time=1.81 ms
64 bytes from gopro (10.5.5.9): icmp_req=2 ttl=64 time=0.742 ms

If you do a network map of the 10.5.5.0/24 network, you'll find two devices:
[sodo@computer ~]$ nmap -A 10.5.5.0/24
Starting Nmap 6.01 ( http://nmap.org ) at 2013-02-17 11:38 EST
Stats: 0:01:11 elapsed; 254 hosts completed (2 up), 2 undergoing Service Scan
Service scan Timing: About 85.71% done; ETC: 11:39 (0:00:11 remaining)

Nmap scan report for gopro (10.5.5.9)
Host is up (0.0060s latency).
Not shown: 998 closed ports
PORT     STATE SERVICE VERSION
80/tcp   open  http?
8080/tcp open  http    Cherokee httpd 1.2.101b121204_
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at http://www.insecure.org/cgi-bin/servicefp-submit.cgi :
SF-Port80-TCP:V=6.01%I=7%D=2/17%Time=51210783%P=x86_64-redhat-linux-gnu%r(
SF:GetRequest,C6,"HTTP/1\.0\x20\xff\xfbAllow:\x20GET\x20\r\nAccept-Ranges:
SF:\x20bytes\r\nCache-Control:\x20no-cache\r\nCache-Control:\x20no-store\r
SF:\nConnection:\x20Keep-Alive\r\nServer:\x20GoPro\x20Web\x20Server\x20v1\
SF:.0\r\nContent-Type:\x20text/plain\r\nContent-Length:\x202\r\n\r\n")%r(F
SF:ourOhFourRequest,C6,"HTTP/1\.0\x20\xff\xfbAllow:\x20GET\x20\r\nAccept-R
SF:anges:\x20bytes\r\nCache-Control:\x20no-cache\r\nCache-Control:\x20no-s
SF:tore\r\nConnection:\x20Keep-Alive\r\nServer:\x20GoPro\x20Web\x20Server\
SF:x20v1\.0\r\nContent-Type:\x20text/plain\r\nContent-Length:\x202\r\n\r\n
SF:");
Service Info: OS: Unix

And

Nmap scan report for 10.5.5.109

Host is up (0.0011s latency).

Not shown: 995 closed ports

PORT      STATE SERVICE                VERSION

22/tcp    open  ssh                    OpenSSH 5.9 (protocol 2.0)

| ssh-hostkey: 1024 84:a4:7c:c7:42:e5:53:f3:64:a2:89:2c:5e:97:26:60 (DSA)

|_2048 a2:27:68:40:d4:9f:90:72:f2:6c:0b:99:75:67:c3:76 (RSA)

111/tcp   open  rpcbind (rpcbind V2-4) 2-4 (rpc #100000)

| rpcinfo: 

|   program version   port/proto  service

|   100000  2,3,4        111/tcp  rpcbind

|_  100000  2,3,4        111/udp  rpcbind

888/tcp   open  ssl/hbase-master       Apache Hadoop Hbase 2.0

|_sslv2: server still supports SSLv2

|_http-methods: No Allow or Public header in OPTIONS response (status code 404)

|_http-title: 3ware 3DM2 - computer - Summary

| ssl-cert: Subject: commonName=computer/organizationName=3ware Inc./countryName=US

| Not valid before: 2011-10-28 18:44:48

|_Not valid after:  2021-10-26 18:44:48

902/tcp   open  ssl/vmware-auth        VMware Authentication Daemon 1.10 (Uses VNC, SOAP)

16001/tcp open  fmsascon?

Service Info: Host: computer; Device: storage-misc

Note that if you perform the nmap command, you may lose connectivity to the GoPro and will have to restart both the WIFI BacPac and the main body of the GoPro.

5) Connect your PC to the BacPac's wireless access point (WAP) at 10.5.5.9.  The default password is GoProHero.  Important note:

*** In order to send commands to the GoPro via its Cherokee Web Server, you need to connect the computer on which you have a running browser to the wireless access point that the GoPro creates in step 4 ***

6) After you've connected to the BacPac WAP, you'll need to instruct the GoPro to start streaming.  You can do this by sending commands to the webserver that is running on port 80.  Here is the command to start the stream:

http://10.5.5.9/camera/PV?t=WIRELESSPASSWORD&p=%02

You'll need to substitute the password (default or your own custom password) for the GoPro where I've written WIRELESSPASSWORD.  The %02 says.."start streaming".  You can turn off streaming by sending the same command, only using %01 on the end:

http://10.5.5.9/camera/PV?t=WIRELESSPASSWORD&p=%01

Someone compiled a list of commands that you can send to the GoPro using the web interface.  I've listed those commands at the bottom of this post.  Also, an enterprising gentleman has created a Windows app to send the various commands to your GoPro while it is in wireless tablet mode: http://cam-do.com/WiGo/.

7) If you've started streaming, validate that the little files that represent the streaming are posted to the Cherokee webserver by visiting this URL:

http://10.5.5.9:8080/live/

You should see the files listed in the browseable directory.  This means that the stream is running and that you can capture it:

8) Once the stream is running, use FFplay (a media player bundled with FFmpeg) to validate the stream works:

[sodo@computer ~]$ ffplay -i http://10.5.5.9:8080/live/amba.m3u8

ffplay version 0.10.6 Copyright (c) 2003-2012 the FFmpeg developers

  built on Dec  1 2012 12:21:08 with gcc 4.7.2 20120921 (Red Hat 4.7.2-2)

...

[applehttp @ 0x7fa0600008c0] Estimating duration from bitrate, this may be inaccurate

Input #0, applehttp, from 'http://10.5.5.9:8080/live/amba.m3u8':

  Duration: N/A, start: 144.977167, bitrate: N/A

    Stream #0:0: Video: h264 (Main) (HDMV / 0x564D4448), yuv420p, 432x240 [SAR 1:1 DAR 9:5], 29.97 tbr, 90k tbn, 59.94 tbc

 154.92 A-V:  0.000 fd=   0 aq=    0KB vq=   21KB sq=    0B f=0/0   0/0 

I've bolded some interesting things about the stream:

* the format is Apple's HTTP Live Streaming (HLS) format (FFmpeg calls this "applehttp"

* the stream is H264 compression

* color model is yuv420p

* size of the stream is 432x240

* framerate is 29.97

Here's a snap of the output

Holy crap..it works!  The best thing about this is now you can capture the output to a file like so:
ffmpeg -y -i http://10.5.5.9:8080/live/amba.m3u8 -c:v copy -an test.mp4

The above command captures the stream and copies it (-c:v copy) in the same format it was sent.  For my purposes, this was best as I did not have to re-encode the stream.  I simply "copied" the stream to the hard drive of my Linux box.  This saved CPU processor for the other two streams I captured.

I put this stream together along with the other two streams in a sample video of my new drum kit.  Only bummer is that the GoPro stream of my kick pedals is that it is so low rez:


If anyone has a clue as to how to edit the firmware to increase the resolution of the stream that gets saved to webroot of the Cherokee web server, please let me know.  I'd love to be able to suck up a video stream that is higher rez than 420P.

So that's it.  I'd really like to be able to tweak the firmware of the GoPro Hero 2 in order to increase the resolution of the stream to capture, but it looks like that hasn't been fully accomplished yet unlike Magic Lantern for the Canon 5D.  Though folks are trying!

Enjoy,
TAG

References
http://ffmpeg.org/ffplay.html
http://ffmpeg.org/ffmpeg.html
http://goprouser.freeforums.org/howto-livestream-to-pc-and-view-files-on-pc-smartphone-t9393.html
http://cam-do.com/WiGo/
http://goprouser.freeforums.org/wifi-backpack-sniffing-t7993-10.html
http://0fyi.wordpress.com/2007/02/23/live-capture-of-a-digital-video-stream-using-dvgrab-and-ffmpeg/
http://chdk.setepontos.com/index.php?topic=5890.0
http://goprouser.freeforums.org/gopro-hd-bus-interface-discussion-and-projects-f23.html

Apple's HLS Streaming Format
http://www.streamingmedia.com/Articles/Editorial/What-Is-.../What-is-HLS-%28HTTP-Live-Streaming%29-78221.aspx
http://developer.apple.com/library/ios/#documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/UsingHTTPLiveStreaming/UsingHTTPLiveStreaming.html
http://www.iis.net/learn/media/live-smooth-streaming/apple-http-live-streaming-with-iis-media-services
http://www.3ivx.com/technology/windows/metro/http_live_streaming.html

Commands you can send to the GoPro while Phone/Tablet Streaming is active
Update 2013/04/19
Added the correct syntax for password entry..my original cut-and-paste included HTML characters that broke the display of the commands in Blogger
*** end update ***
The general query structure is: 
http:////?t=[password]&p=

Where: can be bacpac or camera.

Queriable Information
Turn on camera : http:///bacpac/PW?t=[password]&p=%01
Turn off camera : http:///bacpac/PW?t=[password]&p=
Change mode : http:///bacpac/PW?t=[password]&p=%02

Start capture : http:///bacpac/SH?t=[password]&p=%01
Stop capture : http:///bacpac/SH?t=[password]&p=

Preview
On : http:///camera/PV?t=[password]&p=%02
Off : http:///camera/PV?t=[password]&p=

Mode
Camera : http:///camera/CM?t=[password]&p=
Photo : http:///camera/CM?t=[password]&p=%01
Burst : http:///camera/CM?t=[password]&p=%02
Timelapse : http:///camera/CM?t=[password]&p=%03
Timelapse : http:///camera/CM?t=[password]&p=%04

Orientation
Head up : http:///camera/UP?t=[password]&p=
Head down : http:///camera/UP?t=[password]&p=%01

Set Resolution of Video Recorded to the Device
WVGA-60 : http:///camera/VR?t=[password]&p=
WVGA-120 : http:///camera/VR?t=[password]&p=%01
720-30 : http:///camera/VR?t=[password]&p=%02
720-60 : http:///camera/VR?t=[password]&p=%03
960-30 : http:///camera/VR?t=[password]&p=%04
960-60 : http:///camera/VR?t=[password]&p=%05
1080-30 : http:///camera/VR?t=[password]&p=%06

FOV
wide : http:///camera/FV?t=[password]&p=
medium : http:///camera/FV?t=[password]&p=%01
narrow : http:///camera/FV?t=[password]&p=%02

Photo Resolution
11mp wide : http:///camera/PR?t=[password]&p=
8mp medium : http:///camera/PR?t=[password]&p=%01
5mp wide : http:///camera/PR?t=[password]&p=%02
5mp medium : http:///camera/PR?t=[password]&p=%03


Photo Resolution in Black Edition
12mp w : http:///camera/PR?t=[password]&p=%05
07mp w : http:///camera/PR?t=[password]&p=%04
07mp m : http:///camera/PR?t=[password]&p=%06
05mp m : http:///camera/PR?t=[password]&p=%03



Delete (may only work on Black Edition..not confirmed)
Last
http:///camera/DL?t=[password]&

All (Format)
http:///camera/DA?t=[password]&


Timer
0,5sec : http:///camera/TI?t=[password]&p=
1sec : http:///camera/TI?t=[password]&p=%01
2sec : http:///camera/TI?t=[password]&p=%02
5sec : http:///camera/TI?t=[password]&p=%03
10sec : http:///camera/TI?t=[password]&p=%04
30sec : http:///camera/TI?t=[password]&p=%05
60sec : http:///camera/TI?t=[password]&p=%06

Localisation
On : http:///camera/LL?t=[password]&p=%01
Off : http:///camera/LL?t=[password]&p=

Bip Volume
0% : http:///camera/BS?t=[password]&p=
70% : http:///camera/BS?t=[password]&p=%01
100% : http:///camera/BS?t=[password]&p=%02

what window manager am I running?

Recently, I've installed quite a few new Linux distributions as virtual machines. They all use different window managers and it is difficult to tell what window manager I have installed. The "wmctrl" program clears this fog. Run the command like this:
sodo@linux-nie4:~> wmctrl -m
Name: Metacity
Class: N/A
PID: N/A
Window manager's "showing the desktop" mode: OFF

Et voila! You have the name of your window manager. Also, you could grep your process list for a known window manager like metacity or compiz like so:
sodo@linux-nie4:~> pidof metacity
5134
sodo@linux-nie4:~> ps -ef | grep metacity
sodo 5134 4984 0 10:36 ? 00:00:01 /usr/bin/metacity
sodo 30798 5467 0 11:35 pts/0 00:00:00 grep metacity

Also, it might be helpful to know your X session setup. This can be determined by looking at your environment variables:
sodo@linux-nie4:~> printenv | egrep 'MANAGER|SESSION' | sort
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-1UXtH7rCgE,guid=9b7438cbc749781f920116e54d973474
DESKTOP_SESSION=gnome
GDMSESSION=gnome
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
SESSION_MANAGER=local/linux-nie4:@/tmp/.ICE-unix/4984,unix/linux-nie4:/tmp/.ICE-unix/4984
WINDOWMANAGER=/usr/bin/gnome
XDG_SESSION_COOKIE=74cab8c1ad6aab7a0cf6c38f4d8dc6fa-1301754995.529512-1092139338
XSESSION_IS_UP=yes

A KDE Example
sodo@linux-z6tw:~> printenv | egrep 'MANAGER|SESSION' | sort
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-mzeA1Sx0fF,guid=43e3b1dcb893bbe684e0bdfe00001fe6
DESKTOP_SESSION=default
KDE_FULL_SESSION=true
KDE_SESSION_UID=1000
KDE_SESSION_VERSION=4
KONSOLE_DBUS_SESSION=/Sessions/1
SESSION_MANAGER=local/linux-z6tw:@/tmp/.ICE-unix/11242,unix/linux-z6tw:/tmp/.ICE-unix/11242
SHELL_SESSION_ID=985449ee0b7140008fe9978a32662d10
WINDOWMANAGER=/usr/bin/startkde
XDG_SESSION_COOKIE=6b050dcfcaf094a4a7c7a30e000002ae-1301545596.870464-842889003
XSESSION_IS_UP=yes

sfrase@linux-z6tw:~> wmctrl -m
Name: KWin
Class: kwin
PID: 11247
Window manager's "showing the desktop" mode: OFF

cool.
TAG

fun (?) with Enlightenment window manager

As much as I'm used to it, I've finally bored of Metacity, the default window manager in Fedora. Metacity is very nice if you live in the Windows world from 9-5 like I do..the keybindings are very similar. So I decided to upgrade..

For the past three days, I've spent a good amount of time with Enlightenment, a visually striking, light(-er)weight window manager for Linux (and other OSs). The initial install was simple, as Enlightenment is in Fedora's repos:
[sodo@computer ~]$ sudo yum info enlightenment
[sudo] password for sodo:
Loaded plugins: presto, refresh-packagekit
Installed Packages
Name : enlightenment
Arch : x86_64
Version : 0.16.999.050
Release : 5.fc12
Size : 10 M
Repo : installed
From repo : fedora
Summary : Highly optimized and extensible desktop shell
URL : http://enlightenment.org/p.php?p=about/e17&l=en
License : MIT
Description : Enlightenment 0.17 is desktop shell based on Enlightenment Foundation
: Libraries. It's highly optimized and provides extensive theming capabilities.
: A Desktop shell means it's a window manager plus a file manager, plus
: configuration utilitys all in one. It works reasonably fast even on old and low
: range computers, providing eye-candy environment.

Documentation
As a new user, I've found the documentation and resources for Enlightenment to be difficult to navigate. The best source for a new user is to visit the Wiki and choose "User Guides":
http://trac.enlightenment.org/e/wiki

Module Installation
Also, module installation is buried in the Wiki:
http://trac.enlightenment.org/e/wiki/Gadgets

Update 2011/02/12
I've since installed from source to give me the latest and greatest. The source install was relatively painless downloading the latest E packages from here:
http://www.enlightenment.org/p.php?p=download&l=en

Install the sources in this order:
eina-1.0.0
eet-1.4.0
evas-1.0.0
ecore-1.0.0
embryo-1.0.0
edje-1.0.0
efreet-1.0.0
e_dbus-1.0.0
eeze-1.0.0
enlightenment-0.16.999.55225

For each of the sources, after extracting the source code via "tar xvfz ", run:
./autogen
make
sudo make install

The only occasional problem that I had was a missing library here and there, like liblua. I resolved this by installing the -devel package. A second problem I encountered was that I needed to set:
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
***end update***

Enlightenment is very cool, very pretty, but it takes quite a bit of getting used to especially if you're coming from the world of Metacity.

I think I like it, but the jury is out due to all the configuration I've had to do.

A battle, as always
So here are a few of the problems I encountered, and some workarounds:
1) ALT-PrntScreen didn't work

I had to rebind the PrtScn key to gnome-screenshot, the default Gnome screen capture program.

Binding keys
The key bindings interface in Enlightenment is a little funky. You get there by clicking on the desktop, go to Settings, Settings Panel, Input category, Key Bindings:


The left part of the screen is the key and the right hand side shows the Action you want to perform. In my case, ALT+Print was not in the list, so I first had to Add Binding. Then I needed to bind the key to an action, which was to Launch a Command:
/usr/bin/gnome-screenshot -w


The -w parameter screencaps a single window. If you leave off the -w parameter, your entire desktop will be screencapped to a PNG file. Once I created this key binding, all was well and I was able to capture screens again.

2) ALT-space to maximize/restore windows was not bound to anything

Again, I had to bind this command to a Window:State:


The nice thing about the Maximize window state is that it toggles between Maximize and Restore (or Unmaximize in the Enlightenment world).

3) Icons for some of the programs I selected for my Icon Bar were missing:


I had to go and find individual icons by right-clicking on the space in the icon bar, select Change Icon Properties, select Icon, and navigate to the following directory:
/usr/share/icons/apps/scaleable

You can also find icons in any of the sized directories; for example:
/usr/share/icons/apps/32x32


I had to do this for Google Chrome, gnome-terminal, vmware player and my nvidia settings. I'm surprised that Enlightenment did not automatically find these for some of the more popular programs.

4) the Shelf's icon bar seems to start programs in their own environment separate from your user environment

I found this out because when I started Google Chrome in Enlightenment, Google Chrome no longer respected any of my proxy settings. The workaround was to create a simple shell script that exports my proxy environment variables and starts Chrome:
[sodo@computer ~]$ cat googleStartup.sh
#!/bin/bash
export http_proxy=http://http-proxy:8080/
export https_proxy=http://http-proxy:8080/
export no_proxy=localhost,127.0.0.0/8
/opt/google/chrome/google-chrome &

I then put that script in the Executable section of the icon. Don't forget the ampersand on the end of the command. If you don't have it, you may see an Enlightenment dialog popup saying something like "Enlightenment was unable to run the application..the app failed to start" when exiting out of the app.

5) The "Remember" feature:

..to remember size, position and settings of a windowed program (in this case, gnome-terminal) plain ol' just doesn't work. Haven't got a workaround for this yet.

Other bothersome things I have yet to fix
1) it is meant as a convenience, but it is irritating when the cursor jumps to the Save dialog
* this was corrected via Settings -> Settings Panel -> Windows -> Window Focus -> Advanced -> uncheck "Slide pointer to a new focused window"


2) desktop switches to the adjacent one when your mouse is next to a desktop's far edge
* this was corrected via Settings -> Settings Panel -> Input -> Edge Bindings and removing the binding for the Right Edge. Erg!!


3) ctrl-shift left or right does not select text, it brings you to the adjacent desktop
* ARGH..still no fix

4) clicking on an application does not make the applications in front of the clicked application disappear..erg!
* this was corrected via Settings -> Settings Panel -> Windows -> Window Focus -> "Click Window to Focus".


5) I miss the GNOME system monitor panel widget. Weep.


It's always a tough row to hoe in the Linux world..nothing great or tasty or beautiful comes without some work.

Keep you posted..
TAG

References
E User Guide
http://jeffhoogland.blogspot.com/2010/10/e17-basics-faq.html
http://jeffhoogland.blogspot.com/2010/12/more-enlightenment-faq.html
good info on features and keybindings of Metacity: https://github.com/tthurman/metacity
themes http://exchange.enlightenment.org/

digging down into a linux process

So I inadvertantly exited out of my Fedora X server without saving or exiting my VMware Player's open virtual machine:

This left my vm in an unknown state. You could still see the lock files in the Virtual Machine's directory:
[sodo@ogre ~]$ ll Virtual\ Machines/Windows\ 7\ x64/
total 10976188
drwxrwxr-x 3 sodo 4096 2011-01-07 12:53 caches
-rw-r--r-- 1 sodo 320659 2011-01-18 14:41 vmware-0.log
-rw-r--r-- 1 sodo 359599 2011-01-18 14:14 vmware-1.log
-rw-r--r-- 1 sodo 565465 2011-01-08 00:14 vmware-2.log
-rw-r--r-- 1 sodo 317488 2011-01-18 17:33 vmware.log
-rw-rw---- 1 sodo 8684 2011-01-18 14:42 Windows 7 x64.nvram
-rw------- 1 sodo 2103836672 2011-01-18 17:35 Windows 7 x64-s001.vmdk
-rw------- 1 sodo 2121203712 2011-01-18 17:35 Windows 7 x64-s002.vmdk
-rw------- 1 sodo 2145255424 2011-01-18 17:35 Windows 7 x64-s003.vmdk
-rw------- 1 sodo 2145976320 2011-01-18 14:49 Windows 7 x64-s004.vmdk
-rw------- 1 sodo 955 2011-01-18 14:41 Windows 7 x64.vmdk
drwxrwxrwx 2 sodo 4096 2011-01-18 14:41 Windows 7 x64.vmdk.lck
-rw-rw---- 1 sodo 1073741824 2011-01-08 00:14 Windows 7 x64.vmem
-rw-rw---- 1 sodo 0 2011-01-07 12:37 Windows 7 x64.vmsd
-rw-rw---- 1 sodo 182610705 2011-01-18 10:18 Windows 7 x64.vmss
-rwxrwxr-x 1 sodo 2477 2011-01-18 14:42 Windows 7 x64.vmx
-rw-rw-r-- 1 sodo 1645 2011-01-07 12:53 Windows 7 x64.vmxf
drwxrwxrwx 2 sodo 4096 2011-01-18 14:41 Windows 7 x64.vmx.lck

But after some Googling, there seemed to be no way to restart the orphaned vm without killing the process that was hanging out there. Before I killed the vm process, I researched it to find out more about it. First, I did a search on the process:
[sodo@ogre ~]$ ps -ef | grep vmx
sodo 4629 1 13 10:18 ? 00:31:48 /usr/lib/vmware/bin/vmware-vmx -ssnapshot.numRollingTiers=0 -sRemoteDisplay.vnc.enabled=FALSE -s vmx.stdio.keep=TRUE -# product=8;name=VMware Player;version=3.1.3;buildnumber=324285;licensename=VMware Player;licenseversion=6.0; -@ pipe=/tmp/vmware-sodo/vmxb90ce351150180d7;readyEvent=90 /home/sodo/Virtual Machines/Windows 7 x64/Windows 7 x64.vmx

I saw that the process number was 4629. The command that started the process was vmware-vmx:
[sodo@ogre ~]$ ps -p 4629
PID TTY TIME CMD
4629 ? 00:31:48 vmware-vmx

Digging into the process directory, I saw the status of the process was sleeping:
[sodo@ogre ~]$ cat /proc/4629/task/4629/status
Name: vmware-vmx
State: S (sleeping)
Tgid: 4629
Pid: 4629
PPid: 1
TracerPid: 0
Uid: 500 500 0 500
Gid: 500 500 500 500
Utrace: 0
FDSize: 256
Groups: 500
VmPeak: 3281396 kB
VmSize: 3131036 kB
VmLck: 0 kB
VmHWM: 1409588 kB
VmRSS: 1343972 kB
VmData: 2727812 kB
VmStk: 288 kB
VmExe: 6784 kB
VmLib: 134888 kB
VmPTE: 3212 kB
Threads: 1
SigQ: 0/80092
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: fffffffe7ffbfeff
SigIgn: 0000000000301000
SigCgt: 0000000193c9eeef
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 4744582
nonvoluntary_ctxt_switches: 7620

Alas, the vmware service offered no consolation..I could not shutdown the daemon gracefully:
[sodo@ogre ~]$ sudo /etc/init.d/vmware restart
Stopping VMware services:
VMware USB Arbitrator [ OK ]
At least one instance of VMware Player is still running.
Please stop all running instances of VMware Player first.

VMware Authentication Daemon [FAILED]

And so, good readers, I was forced to kill the process. And hard, as I had to add the -9 switch to kill, which effectively says "kill the process and don't do any cleanup". "Kill that sucker, but good!"

[sodo@ogre Windows 7 x64]$ kill 4629

[sodo@ogre Windows 7 x64]$ ps -ef | grep 4629

[sodo@ogre Windows 7 x64]$

Yes, this harsh kill command did the trick. And then I had to explain to my poor Win7 vm why I had pulled the plug so harshly:


Oh cruel fate, why dost thou mock me?
TAG

Reference
http://aplawrence.com/SCOFAQ/FAQ_scotec6killminus9.html

streaming 1080P using MoCA, ethernet over cable

As you guys know, I recently bought a new AV receiver. In support of that, I setup an audio server that streams lossless AAC from the MacBook to the receiver. This works well and sounds great. FYI - the lossless audio streams at about 1411Kbps according to this spec:

RFC 2586 - The Audio/L16 MIME content type

Secondly, I setup TwonkyMedia server on my Linux video editing box to stream video. I tested streaming 1080P content from that server over wireless using a Linksys by Cisco WET610N Wireless-N Ethernet Bridge to the Playstation 3. It's cool, as the very high def video plays back, but unfortunately, I get stutters. Connecting the PS3 via wired eliminates the stutters. So, the best solution to the problem would be to run an ethernet line from upstairs to the downstairs. However, I live in a condo and it is impractical for me to run ethernet. What to do?

It seems that a standards group has been working on this same problem and that a solution has been around for a few years now:

http://www.mocalliance.org/

I guess I've been sleeping at the wheel! Anyway, MoCA allows me to use the coax cable within my house to stream video. That's great! Here are a few devices that do what I wanted:

Netgear MoCA Coax-Ethernet adapter
http://www.engadgethd.com/2009/06/25/netgear-moca-coax-ethernet-adapter-review/

Motorola NIM100

http://shop.ebay.com/i.html?_nkw=Motorola+NIM100

Actiontec MI424WR (used in Verizon FIOS)

http://www.actiontec.com/products/product.php?pid=189

After reading the below articles, I decided the Moto NIM100s were the best fit for me, as they were an inexpensive and solid solution:

http://lemmingreviews.blogspot.com/

http://www.dslreports.com/forum/r22918152-splitNIM100-configuration

http://www.avsforum.com/avs-vb/archive/index.php/t-951949.html

I purchased two from eBay for $110 (shipping included).

After a few missteps, I got those b1tches working last night. I had trouble early on, as:

1) the devices don't default to DHCP by default

2) I didn't realize you couldn't get to the admin GUI when the coax connection was active

So I fumbled around for two hours until I figured that stuff out.

But now I am able to stream glorious 1080P videos from my Twonky Media Server to my Playstation 3! Of course, its not perfect, as the highest bitrate streams are coming in at greater than 60Mbps! Woah! Thus, I need to gain an understanding of the capacity of the MoCA network within my condo.

To do this, I found a nifty utility called netio that allows you to measure bandwidth between two hosts on your network:

http://www.softpedia.com/progDownload/NETIO-Download-76189.html

You setup a server component on a PC at one end of your network and then a client PC at the other end of the network. The program then sends packets of different size from the client to the server and reports the send and receive rates in between. The output looks like this:

CLIENT SIDE

C:\temp\netio\bin>win32-i386.exe -t 192.168.1.97

NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

TCP connection established.
Packet size 1k bytes: 10345 KByte/s Tx, 10908 KByte/s Rx.
Packet size 2k bytes: 10700 KByte/s Tx, 10954 KByte/s Rx.
Packet size 4k bytes: 11209 KByte/s Tx, 11342 KByte/s Rx.
Packet size 8k bytes: 11157 KByte/s Tx, 10953 KByte/s Rx.
Packet size 16k bytes: 11122 KByte/s Tx, 11044 KByte/s Rx.
Packet size 32k bytes: 11256 KByte/s Tx, 11536 KByte/s Rx.
Done.

SERVER

C:\temp\netio\bin>win32-i386.exe -t -s

NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

TCP server listening.
TCP connection established ...
Receiving from client, packet size 1k ...
Sending to client, packet size 1k ...
Receiving from client, packet size 2k ...
Sending to client, packet size 2k ...
Receiving from client, packet size 4k ...
Sending to client, packet size 4k ...
Receiving from client, packet size 8k ...
Sending to client, packet size 8k ...
Receiving from client, packet size 16k ...
Sending to client, packet size 16k ...
Receiving from client, packet size 32k ...
Sending to client, packet size 32k ...
Done.
TCP server listening.

hope folks find this useful,

TAG

Twonky Media Install notes
(from the HowTo)

Copy all files (keeping the directory structure) into your most 
favourite installation directory, e.g. /usr/local/twonkymedia.

Make sure that the server file, plugins and the cgi scripts have the exe bit set.
If not, try:
"chmod 700 twonkym* cgi-bin/* plugins/*"
Eventually it is necessary to have a multicast route set for the
server by issuinge, e.g.:
"route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0"

If you are running a Debian distribution on a 64 bit architecture, it
is possible that you need to install the 32 bit compatibility
libraries. In this case invoke:
"apt-get install ia32-libs"

Now start the server with a full qualified path (never just "./twonkymedia"),
but "/usr/local/twonkymedia/twonkymedia" . The server will display a startup
message. The server is running now and you can find and use it with
your UPnP client device. Before you continue, you should point the
server to one or more content directories via the configuration
web-page, unless the content is available from subdirectories of the
server' working directory.

To open the server's configuration web-page, open the following url in a web 
browser:

http://127.0.0.1:9000/config

command line date time change

Or how to set the time in Linux.

Here's another thing I always forget: how to change the date or time at the command line, preferably using the Bourne shell in Linux.

Example:
To change the date/time on a Linux box to September 10, 2008 at 5:37pm, enter the following:
date 0910173708

The ordering of the fields is a bit odd, so pay attention.

where the date/time format is the following:
MMDDHHmmYY
MM - two digit month
DD - two digit day
HH - two digit hour
mm - two digit minute
YY - two digit year

All are zero padded if date/time values are below 10.

Changing the date time was never easier!

enjoy,
sodo

using NetQoS to diagnose network congestion

We use a backend connection from our corporate offices to our hosting provider. On Monday towards the end of the day, I noticed a big slowdown in our RDP (Remote Desktop) connectivity to our Windows servers at the hosting provider. Monday is normally a very busy day for network traffic in the office as well as our main website. So I wondered where the traffic was coming from. Luckily, our network engineers had recently purchased a suite of network management tools from NetQoS (http://www.netqos.com/).

The engineers setup NetQoS monitors on the backend pipe to allow us to see the traffic going through our T1 speed, Frame Relay connection. The nice thing about NetQoS is the ability to break down the traffic going back and forth through the pipe by protocol and port. In the below graph labeled "Stacked Protocol Trend In" for this past Monday, you'll see a green blob after 15:00.

"Trend In" is data inbound to the monitored router. In our case, inbound is into our corporate network. "Trend Out" is data going outbound from the monitored router. In like fashion, outbound is out to our managed hosting provider.

As indicated by the key on the right side of the diagram, this is a spike in SSH (port 22) traffic:


Interesting! The funny thing is that a similar network performance degradation happened the previous week. I changed the date range for the NetQoS reports and saw an even larger, longer lasting spike in SSH traffic:


OK. So it seems we have a problem, but what is the cause?

We typically use SSH to monitor various Unix servers in the environment. One of the administrators has a bunch of X sessions that display performance statistics from a number of servers at the hosting facility. This X session traffic is done over SSH port. He told me that while the high SSH traffic issue was happening, one of his X session graphs was not refreshing properly. I suspected that since he was the main administrator of the servers and that he had about twelve X sessions open, that there is a bug in the X session software that was making one or more of his monitoring sessions create this spike in traffic.

At this point, he still has his X sessions open, but we will wait until the next spike in SSH traffic to try to determine if it is this admins' X sessions that are the culprit. At that time, we will probably shut down his sessions to see if that fixes the problem. I will update the blog next week to let you know how it goes.

Anyway, I'm glad that NetQoS helped us troubleshoot the problem. It gives us some good insight into network traffic by TCP/IP application.

UPDATE: another great tool to diagnose your local computer's network traffic is IPTraf (http://iptraf.seul.org/). I will review this software which runs under Linux and give a basic description of how to use it in an upcoming post.

cheers!

finding out the BIOS version in Linux

Here's another little tidbit of useless information that you will no longer need to keep in your heads because I will have told you and all you will have to do is visit http://www.google.com and search for "finding BIOS version in Linux" and this lovely little article will appear magically before you.
:)

Initially, I thought the best way to find out your BIOS version would be to look in /var/log/dmesg and run this command:
root_testserver root > grep BIOS /var/log/dmesg
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000009fffa000 (usable)
BIOS-e820: 000000009fffa000 - 00000000a0000000 (ACPI data)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
mtrr: probably your BIOS does not setup all CPUs
PCI: PCI BIOS revision 2.10 entry at 0xf0094, last bus=9
PCI: Device 00:00 not found by BIOS
PCI: Device 00:01 not found by BIOS
PCI: Device 00:02 not found by BIOS
PCI: Device 00:78 not found by BIOS
PCI: Device 00:7b not found by BIOS
PCI: Device 00:80 not found by BIOS
PCI: Device 00:82 not found by BIOS
PCI: Device 00:88 not found by BIOS
PCI: Device 00:8a not found by BIOS
apm: BIOS not found.
ide0: BM-DMA at 0x2000-0x2007, BIOS settings: hda:pio, hdb:pio
ide1: BM-DMA at 0x2008-0x200f, BIOS settings: hdc:pio, hdd:pio

The important line in the output should look something like this:
PCI: PCI BIOS revision 2.10 entry at 0xf0094, last bus=9

or, eliminating some extraneous information, you can use the enhanced command:
root_testserver root > grep BIOS /var/log/dmesg grep evision

However! As I looked through the online documentation of my Compaq DL380, I saw that the BIOS version is NOT PCI BIOS revision 2.10, but actually an HP specific BIOS named P52! I asked my friendly neighborhood Unix admin and he turned me onto the "dmidecode" command.

Check it:
root_testserver root > dmidecode more
# dmidecode 2.2
SMBIOS 2.3 present.
54 structures occupying 1477 bytes.
Table at 0x000EC000.
Handle 0x0000
DMI type 0, 20 bytes.
BIOS Information
Vendor: HP
Version: P52
Release Date: 04/14/2005
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 4096 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
ESCD support is available
Boot from CD is supported
Selectable boot is supported
5.25"/360 KB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 KB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported

That's all folks! May all your BIOS dreams come true!

Oh..here's a related article on how to determine the version of your operating system in Linux or SunOS:
http://www.techanswerguy.com/2007/06/how-to-get-operating-system-version-in.html