Sunday, December 02, 2012

"yum check" always helps..

A couple weeks ago, I was successful in upgrading my Fedora 15 system to Fedora 17 using PreUpgrade.  It worked well for me, outside of the usual NVidia funkiness on my machine (virtual consoles don't work on my SC1430).  However, I did notice a bit of detritus in my installed programs list when I tried to do my most recent "yum update".

Error:  Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem. Eg.:
       
         1. You have an upgrade for kernel-tools which is missing some
            dependency that another package requires. Yum is trying to
            solve this by installing an older version of kernel-tools of the
            different architecture. If you exclude the bad architecture
            yum will tell you what the root cause is (which package
            requires what). You can try redoing the upgrade with
            --exclude kernel-tools.otherarch ... this should give you an error
            message showing the root cause of the problem.
       
         2. You have multiple architectures of kernel-tools installed, but
            yum can only see an upgrade for one of those arcitectures.
            If you don't want/need both architectures anymore then you
            can remove the one with the missing update and everything
            will work.
       
         3. You have duplicate versions of kernel-tools installed already.
            You can use "yum check" to get yum show these errors.
       
       ...you can also use --setopt=protected_multilib=false to remove
       this checking, however this is almost never the correct thing to
       do as something else is very likely to go wrong (often causing
       much more problems).
       
       Protected multilib versions: kernel-tools-3.6.8-2.fc17.x86_64 != kernel-tools-3.3.4-5.fc17.i686


So I ran "yum check" to find the issues:
[sodo@computer bin]$ yum check

Loaded plugins: langpacks, presto, refresh-packagekit
kernel-tools-3.6.7-4.fc17.x86_64 is a duplicate with kernel-tools-3.3.4-5.fc17.i686
1:kmod-nvidia-2.6.41.10-3.fc15.x86_64-280.13-2.fc15.16.x86_64 has missing requires of kernel-uname-r = ('0', '2.6.41.10', '3.fc15.x86_64')
1:kmod-nvidia-2.6.43.8-1.fc15.x86_64-280.13-4.fc15.6.x86_64 has missing requires of kernel-uname-r = ('0', '2.6.43.8', '1.fc15.x86_64')
netbeans-6.9-3.fc15.noarch has missing requires of netbeans-platform12 >= ('0', '6.9', None)
netbeans-apisupport-6.9-3.fc15.noarch has missing requires of netbeans-platform12 >= ('0', '6.9', None)
netbeans-ide-6.9-3.fc15.noarch has missing requires of netbeans-platform12 >= ('0', '6.9', None)
netbeans-java-6.9-3.fc15.noarch has missing requires of netbeans-platform12 >= ('0', '6.9', None)
xorg-x11-drivers-7.4-8.fc17.x86_64 has missing requires of xorg-x11-drv-nouveau

To clear these errors, I:
- removed netbeans, as I don't use it
- added the nouveau driver
- removed the older, Fedora 15 kmod-nvidia drivers
- removed the older, duplicate kernel-tools (3.3.4-5)

Once that was done, I re-ran "yum update" and no error messages appeared!  I was good to go for launch.
TAG

Thursday, October 04, 2012

shrinking a "growable" VMware disk (VMDK file)

Just a quick note on shrinking VMware VMDK files.  In short, it's an option WITHIN the VM guest, under VMware Tools or through vdiskmanager command line:
http://www.vmware.com/support/ws5/doc/ws_disk_shrink.html

















or a command like:
"C:\Program\VMware\VMware Server\vmware-vdiskmanager.exe" -k "XP.vmdk"

Backstory
I noticed my XP VM had ballooned to 200GB on the filesystem.  Woah!
[sodo@computer ~]$ ll Virtual\ Machines/XP
total 351188880
drwxr-xr-x. 3 sodo sodo        4096 Jul 25  2011 caches
--rw-rw-rw-  1 sodo sodo 199013260658 Oct  4 13:35 XP.vmdk


The size of single logical C: partition was set to 232GB and it utilized about 1/2 of that in actual files (about 120GB).  Wow.  Bloated!  I had to do something, but quick.  Here's what I did:
1) deleted all extraneous files (~45GB)
2) as I'll be migrating to Win7 soon, I deleted all my XP Hotfix uninstall directories.  This yielded an inCREDible 5GB of data..just in uninstall files!
3) defragged my XP guest

Resize NTFS and Create a Partition
At this point, there was about 45GB left in actual files.  I decided to do two things using gParted
1) resize the NTFS partition down as much as possible with room to spare for other files
2) create a second partition out of the blank space left over.  I did this because VMware Tools can't shrink empty partition space.  See a deeper technical discussion under "Disk Space" on this page.

1a) The resize from 232GB down to 74GB:






























2) Create a new partition



























My Shrink!
Once these two steps were completed, I was then able to go back and perform the shrink within the VM guest using VMware Tools, Shrink tab:

















The shrink took about two hours, but once completed, I was left with a 47GB VMDK!

[sodo@computer ~]$ ll Virtual\ Machines/XP
total 51222880
drwxr-xr-x. 3 sodo sodo        4096 Jul 25  2011 caches
--rw-rw-rw-  1 sodo sodo 49013260288 Oct  4 13:35 XP.vmdk


Awesome!
TAG

References

Saturday, September 29, 2012

hacking my Pioneer VSX-53 receiver, part I

Well, not hacking.  Just exploring the Pioneer VSX-53.

I added the IP of the receiver and added it to my hosts file as the hostname "vsx" for ease-of-use.

Nmap with the device in standby:
Nmap scan report for vsx (192.168.1.xx)
Host is up (0.032s latency).
rDNS record for 192.168.1.xx: pioneer
Not shown: 991 closed ports
PORT      STATE SERVICE     VERSION
23/tcp    open  telnet?
80/tcp    open  http        GoAhead-Webs embedded httpd
|_http-methods: No Allow or Public header in OPTIONS response (status code 400)
443/tcp   open  ssl/https?
1024/tcp  open  rtsp        Apple AirTunes rtspd 103.2
| rtsp-methods: 
|_  ANNOUNCE, SETUP, RECORD, PAUSE, FLUSH, TEARDOWN, OPTIONS, GET_PARAMETER, SET_PARAMETER, POST, GET
1900/tcp  open  tcpwrapped
8080/tcp  open  http-proxy?
|_http-open-proxy: Proxy might be redirecting requests
49152/tcp open  unknown
49153/tcp open  unknown
49154/tcp open  unknown
Device type: media device
Running: Bowers & Wilkins embedded, Denon embedded, Marantz embedded
OS details: Audio receiver: Bowers & Wilkins Zeppelin Air, Denon AVR-1912, or Marantz NR1602
Network Distance: 1 hop


Port 80










Port 8080










Port 443
From Firefox: guess Pioneer licenses some tech from Denon:
vsx uses an invalid security certificate.

The certificate is not trusted because it is self-signed.
The certificate is only valid for avr.denon.jp\00
The certificate expired on 2/12/99 7:00 PM. The current time is 9/29/12 11:41 AM.



Port 443 will lead you to the same index file (Download MCACC data).  Interesting that the extension on the default home page is .asp.

HTTP Server response headers
mac:pioneer newuser$ wget --server-response http://vsx/index.asp
--2012-09-30 11:15:03--  http://vsx/index.asp
Resolving vsx... 192.168.1.xx
Connecting to vsx|192.168.1.xx|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.0 200 OK
  Date: Sat Jan  1 00:00:00 2000
  Server: GoAhead-Webs
  Pragma: no-cache
  Cache-Control: no-cache
  Content-type: text/html
Length: unspecified [text/html]


Telnet Port 23
Start the unit by typing "PO" once connected
mac:log newuser$ telnet vsx 23
Trying 192.168.1.xx...
Connected to pioneer.
Escape character is '^]'.
PO
PWR0
LM0701
LM0701
FL0220202020482E4D2E472E20202020
FN26
VTA111111111111111111000000000000
VTA100000111111111110000000000000
GBH08
GCH0010000""
GDH000010000100001
GEH01000"STARTING H.M.G."
GEH02000"Please wait..."
GEH03000""
GEH04000""
GEH05000""
GEH06000""
GEH07000""
GEH08000""
FL02202050434D204449524543542020
FL025354415254494E4720482E4D2E47
GBH08
GCH0110000"Top Menu"
GDH000010000800008
GEH01101"Internet Radio"
GEH02001"Rhapsody"
GEH03001"Sirius"
GEH04001"Pandora"
GEH05001"EyeConnect (MACLTSFRASE)"
GEH06001"Favorites"
GEH07001"Recently Played"
GEH08001"Setup"
FL0392496E7465726E65742052616469
GBH08
GCH0100000"Top Menu"
GDH000010000800008
GEH01101"Internet Radio"
GEH02001"Rhapsody"
GEH03001"Sirius"
GEH04001"Pandora"
GEH05001"EyeConnect (MACLTSFRASE)"
GEH06001"Favorites"
GEH07001"Recently Played"
GEH08001"Setup"
FL03926E7465726E657420526164696F
FL03927465726E657420526164696F20

You can use other commands like "VU" or "VD" for Volume Up (VU) or Volume Down (VD).  Referenced website and PDF show you more.

Command prompt updates when I connect to an Internet radio station via iTunes:

GBH08
GCH0611010""
GDH000010000100001
GEH01020""
GEH02023"0:00"
GEH03021""
GEH04022""
GEH05000""
GEH06000""
GEH07000""
GEH08000""
GBH08
GCH0601010""
GDH000010000100001
GEH01020"AirPlay"
GEH02023"0:00"
GEH03021"Sucessfully connected to network"
GEH04022""
GEH05000""
GEH06000""
GEH07000""
GEH08000""
GBH08
GCH0201010""
GDH000010000100001
GEH01020"AirPlay"
GEH02023"0:00"
GEH03021"Sucessfully connected to network"
GEH04022""
GEH05000""
GEH06000""
GEH07000""
GEH08000""
GBH08
GCH0201010""
GDH000010000100001
GEH01020"wshu-baroque"
GEH02023"0:00"
GEH03021""
GEH04022""
GEH05000""
GEH06000""
GEH07000""
GEH08000""


Interesting.

Next, I want to put a sniffer on the wire to tell me how much bandwidth the Pio uses.  Eventually, it would be great to overwrite the firmware and install a Linux server to show visualizations through Project-M while audio is running through the box instead of the lame, static Pioneer song title display.

Reference
http://blog.raymondjulin.com/2012/07/15/remote-control-your-pioneer-vsx-receiver-over-telnet/

Sunday, March 04, 2012

redirecting ffmpeg output, performing calculations is messy

Today, I tried to capture some output from FFmpeg, the media encoder/decoder and had some trouble.  The context is that I wanted to do a sanity check on the total number of frames of video FFmpeg outputs from joining multiple video files:
http://crazedmuleproductions.blogspot.com/2012/03/joining-concatenating-video-files.html

Seems the output from FFmpeg is actually sent via standard error, and not standard output.  The typical output from FFmpeg looks like this:
[sodo@computer tmp]$ ffmpeg -i 1.mpg -an -vcodec copy -f mpeg2video -y NUL 
ffmpeg version 0.7.11-rpmfusion, Copyright (c) 2000-2011 the FFmpeg developers
  built on Feb 25 2012 08:39:28 with gcc 4.6.1 20110908 (Red Hat 4.6.1-9)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --extra-version=rpmfusion --enable-bzlib --enable-libcelt --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-nonfree --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-x11grab --enable-avfilter --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  libavutil    50. 43. 0 / 50. 43. 0
  libavcodec   52.123. 0 / 52.123. 0
  libavformat  52.111. 0 / 52.111. 0
  libavdevice  52.  5. 0 / 52.  5. 0
  libavfilter   1. 80. 0 /  1. 80. 0
  libswscale    0. 14. 1 /  0. 14. 1
  libpostproc  51.  2. 0 / 51.  2. 0
Input #0, mpeg, from '1.mpg':
  Duration: 00:12:08.26, start: 1.000000, bitrate: 15584 kb/s
    Stream #0.0[0x1e0]: Video: mpeg2video (Main), yuv420p, 1280x720 [PAR 1:1 DAR 16:9], 104857 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0.1[0x1c0]: Audio: mp2, 44100 Hz, stereo, s16, 256 kb/s
Output #0, avi, to 'NUL':
  Metadata:
    ISFT            : Lavf52.111.0
    Stream #0.0: Video: mpeg2video, yuv420p, 1280x720 [PAR 1:1 DAR 16:9], q=2-31, 104857 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop, [?] for help
frame=21827 fps=4494 q=-1.0 Lsize= 1357993kB time=00:12:08.26 bitrate=15275.7kbits/s    
video:1357365kB audio:0kB global headers:0kB muxing overhead 0.046304%

All of that is sent to standard error.  The piece I was interested in capturing was the number of frames (bolded, above).  Now, when FFmpeg runs in a terminal, you see the number of frames incrementing; ie, going up.  I can redirect standard error to standard output and save to a file with this command:
ffmpeg -i 1.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | tee test.txt 

If I cat out "test.txt", and look at the last few lines of the file, I see the frame information I want to capture:
Stream mapping:
  Stream #0.0 -> #0.0
Press [q] to stop, [?] for help
frame=21827 fps=4494 q=-1.0 Lsize= 1357993kB time=00:12:08.26 bitrate=15275.7kbits/s    
video:1357365kB audio:0kB global headers:0kB muxing overhead 0.046304%

If I use awk to search for 'frame' to grab the line with the number of frames:
[sodo@computer tmp]$ awk '/frame/ {print $0}' test.txt
frame=21827 fps=4294 q=-1.0 Lsize= 1357365kB time=00:12:08.26 bitrate=15268.6kbits/s

I see my frames value.  However, when I then pipe that command into awk again to grab the second column of data, I get this:
[sodo@computer tmp]$ awk '/frame/ {print $0}' test.txt | awk -F= '{print $2}'
 2195 fps

What's going on here?  "2195" is not the expected result.  I should be seeing "21827 fps".  Looking at the file in vi shows me the problem.  Look at all the hidden ^M characters that have turned up in the file:
Press [q] to stop, [?] for help
frame= 2195 fps=  0 q=-1.0 size=  138799kB time=00:01:13.20 bitrate=15532.0kbits/s    ^Mframe= 4288 fps=4287 q=-1.0 size=  279176kB time=00:02:23.04 bitrate=15988.3kbits/s    ^Mframe= 6398 fps=4265 q=-1.0 size=  419307kB time=00:03:33.44 bitrate=16092.8kbits/s    ^Mframe= 8400 fps=4199 q=-1.0 size=  559022kB time=00:04:40.24 bitrate=16341.0kbits/s    ^Mframe=11378 fps=4551 q=-1.0 size=  696798kB time=00:06:19.61 bitrate=15036.8kbits/s    ^Mframe=14230 fps=4743 q=-1.0 size=  836173kB time=00:07:54.77 bitrate=14427.8kbits/s    ^Mframe=16331 fps=4665 q=-1.0 size=  979121kB time=00:09:04.87 bitrate=14720.7kbits/s    ^Mframe=18526 fps=4631 q=-1.0 size= 1117252kB time=00:10:18.11 bitrate=14807.1kbits/s    ^Mframe=20475 fps=4377 q=-1.0 size= 1245505kB time=00:11:23.14 bitrate=14935.5kbits/s    ^Mframe=21827 fps=4294 q=-1.0 Lsize= 1357365kB time=00:12:08.26 bitrate=15268.6kbits/s    ^M
video:1357365kB audio:0kB global headers:0kB muxing overhead 0.000000%

Oh boy.  You can easily see the problem in the output.  The reason for this seems to be that when FFmpeg updates the values of frames in the output to the terminal, the control character ^M gets input into my output file, test.txt.  We cannot see the ^M when doing a "cat" or "grep" on the file, but it is in the file nonetheless.  So how do we fix?

The goal again is to pluck that final number of frames from the output of FFmpeg that is stored in test.txt.  I can use ^M as a field delimiter within awk using the ascii equivalent of ^M, \015.  First, I have to look for the line in the output that has many fields delimited by the ^M, the line that corresponds to when FFmpeg is updating the "frames" line:
[sodo@computer tmp]$ awk -F"\015" 'NF>1 {print $0}' test.txt
frame=21827 fps=4294 q=-1.0 Lsize= 1357365kB time=00:12:08.26 bitrate=15268.6kbits/s

We can see the many entries on this line:
[sodo@computer tmp]$ awk -F"\015" 'NF > 1 {print $1"\n\r"$2"\n\r"$3}' test.txt
frame= 2195 fps=  0 q=-1.0 size=  138799kB time=00:01:13.20 bitrate=15532.0kbits/s    
frame= 4288 fps=4287 q=-1.0 size=  279176kB time=00:02:23.04 bitrate=15988.3kbits/s    
frame= 6398 fps=4265 q=-1.0 size=  419307kB time=00:03:33.44 bitrate=16092.8kbits/s 

I really just want the last field of information on this line that will give me the final number of frames that FFmpeg has output.  What I have to do is print out the second to last field in the line.  

I do this by performing a simple calculation: a variable called "secondtolast" will be set to the number of fields in the line (delimited by each occurrence of ^M) minus one (secondtolast=NF-1).  I then print use that value as the field I want to print out ($secondtolast) :
[sodo@computer tmp]$ awk -F"\015" 'NF > 1 {secondtolast=NF-1;print $secondtolast}' test.txt
frame=21827 fps=4294 q=-1.0 Lsize= 1357365kB time=00:12:08.26 bitrate=15268.6kbits/s 

After getting the correct field, I can then extract the frame number:
[sodo@computer tmp]$ awk -F"\015" 'NF > 1 {secondtolast=NF-1;print $secondtolast}' test.txt | awk '{print $1}' | awk -F= '{print $2}'
21827

Relating this back to my original problem, counting the total number of frames in a multiple video file concatenation, this long command works:
[sodo@computer tmp]$ for i in 1 2 3 4 ;do ffmpeg -i $i.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | awk -F"\015" 'NF > 1 {secondtolast=NF-1;print $secondtolast}' | awk '{print $1}' | awk -F= '{print $2}'; done
21827
42264
34750
33697

Wow.  Messy.  Even more messy is encapsulating the previous output within another awk command to get a total number of frames for the entire concatenation routine:
[sodo@computer tmp]$ !! | awk '{print sum+=$1}'
for i in 1 2 3 4 ;do ffmpeg -i $i.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | awk -F"\015" 'NF > 1 {secondtolast=NF-1;print $secondtolast}' | awk '{print $1}' | awk -F= '{print $2}'; done | awk '{print sum+=$1}'
21827
64091
98841
132538

Update 3/4/12
I realized I didn't catch the case where FFmpeg does NOT output multiple "frame=" lines.  This occurs when the video file is so small that FFmpeg can digest it in one fell swoop and outputs one "frame=" line only.  To catch this condition, I rewrote the script a bit, throwing in an else in the case where the last field separated by ^M contains the one and only count of frames in the video.  Also, I fixed the script to account for a space " " in the value for the number of frames:
[sodo@computer tmp]$ for i in 1 2 3 4 ; do ffmpeg -i /tmp/$i.mpg -an -vcodec copy -f mpeg2video -y NUL 2>&1 | awk -F"\015" '{if (NF > 1) {secondtolast=NF-1; print $secondtolast} else {print $NF}}' | grep 'frame=' | awk -F"fps" '{print $1}' | awk -F= '{print $2}';done
13307 
11313 
10603 
 9324 
 9033 
10660 

I'm sure there is an easier, more compact way to do this, but I'll leave that up to people with better awk chops.  I'd be interesting to hear if someone has a better solution.

cheers,
TAG
Reference
Feel free to drop me a line or ask me a question.