Showing posts with label drobo fs. Show all posts
Showing posts with label drobo fs. Show all posts

Sunday, March 17, 2013

backing up my systems: it ain't my day (or month)

OK.  I've been in three weeks of hardware hell, mainly due to the fact that I wanted to get my backups for all my machines (a MacBook Pro, my main Linux video editing workstation and an older Windows Vista digital audio workstation (DAW)) properly backed up.  I detailed my strategy for this in my last post.  This post is more of a rant than anything else, so please excuse the lack of any real mentorship on problem solving, except maybe "Google is Your Friend."

Issue #1: Drobo runs out of space
The Drobo has been a fine unit for me.  But as time goes on, you acquire more media and your available space runs out.  You'd think it would be a simple matter of buying a new disk, putting it in the Drobo and letting the BeyondRAID rebuild it's array.  Well, the first drive I bought, a Western Digital Green 1TB, died after the first rebuild.  That never happened to me before, where a drive failed out-of-the-box for me.  Never having that problem before, I didn't truly believe it was dead.

With my non-belief firmly in place, I tried to use the drive in different capacities.  So as a test, I formatted the disk using my Thermaltake BlacX connected to my Mac.  I was able to copy files over to it (though I didn't copy gigs and gigs worth as a true test).  But when I put the unit back in the Drobo, the Drobo gave an immediate "red" light for that drive bay, indicating the drive was bad.  I switched drives in the Drobo unit around, because I thought it could have been a faulty drive bay.  

Then, I had the bright idea to move the data off my 2TB system drive of my main Linux machine to the new Western Digital, put the 2TB in the Drobo and use the new 1TB (which I really thought was a good, error-free drive) as my Linux system drive. So still thinking that the 1TB drive was good, I would have to do some fancy footwork in order to make this possible as the system drive was a logical volume.  This entailed a week of work to figure out how to shrink a logical volume in order to fit the used space of the 2TB drive (which was less than a terabyte) onto the 1TB.

I learned a lot from that experience, to be detailed in a later post.  Suffice it to say that in the end, the 1TB was truly dead and I ended up getting a new 1TB (a Western Digital Black) from BestBuy and that solved my Drobo storage issue.  Kudos to BestBuy, as they were able to give me the Black at the same price as the Green for my trouble.

Issue #2: Mac Time Machine "the identity of this backup disk has changed" (Sparsebundle Problem)
This was an odd one.  After installing the new disk in the Drobo, Time Machine started showing the error "the identity of this backup disk has changed".  From the below post:

I executed the "chflags" command listed.  This ran for about four hours.  After, I tried to execute the "hdutil" command listed, but the Mac said it had already ran the command.  So testing the result of the chflags command, I shutdown and restarted the Drobo.  When Time Machine started backing up, it no longer gave me the error.  Hooray.  Another one down.

Issue #3: Windows Vista DAW crashes
So after a week spent on #1 and #2, I was ready to start work on a new musical project with some friends.  Firing up my old Dell 400SC running Windows Vista (OK, OK..I know I need to upgrade Win7, but I've got a recording session coming up soon and didn't want to change OS's yet), I was presented with this error:
c\windows\system32\config\system corrupt

Oh, wonderful.  So I popped in the Vista Ultimate DVD and selected "Repair".  After it ran, the system rebooted and I was pleasantly surprised to find that this fixed the problem and that I was able to get back into the system.

Getting back into the system, I reasoned that if the drive was going bad, I'd better make a backup.  So I ponied up $40 for Drobo's PC Backup product, the ugly step-brother of the seemless Drobo integration with Mac Time Machine.  Assuming the PC product worked the same way the Mac product did, I selected the defaults.  Well, the defaults do NOT backup the entire drive.  Only your user data.  My bad for not reading the fine print, but I believe that a Drobo product should be consistent between systems and the default should be to backup your entire drive with all system data included, as long as you have the space on your Drobo.  But that's just me.

The missing data would be crucial for what happened next.

Issue #4: Windows Vista DAW crashes again
After taking a two day hiatus from my backup shenanigans, I fired up the DAW again.  And guess what..a new error appears:
\Windows\system32\winload.exe is missing or corrupt (status 0xc000000f)

Oh great.  Going back to my ritual, I loaded in the Vista Ultimate DVD and selected "Repair".  However, after the reboot, no go..still the same "missing or corrupt" error.  I tried a number of times doing the repair, as the Vista repair process would show slightly different screens every time it booted and recognized the system.  This gave me false hope that the DVD was actually repairing something correctly.  Also, the frustrating part of this process that for whatever reason, the DVD would take 10 minutes to load on my Dell.  I'm not sure what the problem was there.  So I chewed up a few hours doing this multiple times.  

Finally, after reading some Google posts by people with the same issue, I decided to run "chkdsk /r" from the command line, rather than relying on the non-informative Windows Vista screen to run some unknown fix command.  I had to specifically boot into the System Recovery Options screen as shown in the below post:

Once I was there, I selected "Command Prompt" and typed in good ol' "chkdsk /r", the "repair" option to chkdsk.  This time, I was rewarded with an actual status screen that told me "bad clusters found", Windows was marking the clusters as bad and was moving the files located on those clusters to good sectors on the disk.  (Sectors and cluster primer here: http://t.co/DLFjrXAp5C).  This process took about three hours, unlike the half-hearted effort that Windows Vista attempted.  I wonder why Vista did not default to doing a real "chkdsk /r".  That doesn't help anyone who has a failing disk.  Bad default!

After the bad cluster identification and repair, I was really glad to see Vista boot up properly!  But since there were so many bad clusters, I had to make a full backup or clone of that drive but quick!  For this, I popped in an unused 500GB SATA I had lying around.  I repartitioned and formatted this drive.  It had been a second Vista system disk and one point, so I knew the drive's main partition was marked as bootable.  So I was good to go there.  I then dragged all the files from my C: onto the new E: (my DVD being the D:).  However, on bootup, Vista showed an error:
"System volume on disk is corrupt"

I suspected this was a problem with the NTFS boot files on the 500GB drive as they had links from the partition map from the old 256GB drive that was failing.  Luckily, when I ran Vista repair, Vista was able to fix this issue and the system started properly.

Issue #5: Windows Vista continually keeps "preparing your desktop"
After the system came up, I made sure all my applications (Reaper, Drobo PC Backup, etc) were working properly.  Unfortunately, they were not, as Vista continually kept giving me the message "Preparing Your Desktop" when I logged into my profile.  I tried a number of things from Google, but those suggestions did not work.  I didn't have any critical data in the old profile, so I figured I'd bit the bullet and create a new profile.  After doing this, the message disappeared and I was able to save my desktop settings and application preferences properly.

In Sum
Wow.  So this has been three weeks of hell.  I "think" I am back to steady state with my systems.  I was able to reset Drobo PC Backup to a full system backup of my Vista DAW to the Drobo.  The Drobo is backing up the Mac just fine and CrashPlan is encrypting my main Linux box backup to the Cloud.

Maybe now I can go outside and get some sun?
TAG

Monday, August 02, 2010

drobo fs install, review

If all computer technology was like the Drobo, there would be less people complaining that their tech is difficult to use or doesn't work. I've had my Drobo FS for three days, but man, I love the thing. It is such a pleasure to not have to worry about RAID arcana just to have your systems safely backed up.

Got It!
I bought the Drobo FS from Amazon (about $694 with free shipping) and set it up between yesterday and today. I installed the Drobo Dashboard, the utility program for the Drobo, on my MacBook Pro, rebooted the Mac, popped a combination of five 250GB and 500GB drives in, hooked up the power and the ethernet cables and fired it up. Drobo Dashboard saw the unit as it sat attached to my router.

The Basics
Dashboard asked me if I wanted to upgrade the firmware..so I did. There was a small hiccup with Dashboard not seeing the Drobo after doing the firmware, so I had to hard power it down. But second time around, the firmware update worked. Dashboard then asked me if I wanted to upgrade the Dashboard software, which I did. I restarted the Mac. I then went into Advanced Settings -> Tools and configured email alerts to send me an email if Drobo becomes too full or has problems.

Create Shares
Back in Dashboard, I created a couple of shares, one as backup for my main Linux video editing workstation and the other for my MacBook and TimeMachine. I setup my MacBook to use 500GB of storage as a TimeMachine backup. The TimeMachine hook up worked like a charm and started backing up.

Stress Test Drobo!
I decided to test it hard out of the box and ran that TimeMachine backup from the mac (connected wirelessly) coincident with a big copy from my Linux video editing box. On my Linux box, I mounted the Drobo via SMB/CIFS and copied over my 460GB archive. The backup of the Linux archive took about 3.5 hours at about 25MiB/s over wired ethernet. Not bad for over gigE and with contention from the Time Machine backup.

The next morning, I got tired of waiting for the wireless TM backup of 130GB to finish, so I stopped the TM backup and then hooked up the Mac to a hard wired ethernet connection. Still took about five hours to copy over the remaining 80GB.

Update
I recently created a 760GB archive of my video workstation using fsarchiver. Copying the one file from my workstation to the Drobo took about 10 hours (8pm to 6am). That was with contention from a 40GB Time Machine backup that was running at the same time.
***end update***

Drobo Apps
Today I installed a few Drobo Apps (apache/droboutils/dropbear). DroboUtils is an easier way to manage Drobo Apps and as a prerequisite, it requires Apache to be installed. I used Apache for another function, setup of my personal website. I haven't cutover to using the apache web server as my main webserver yet because I need to tweak the apache config to hide some of the directory structures.

There is no telnet client for DroboApps, but DropBear is the SSH server for the Drobo. After installing, it was pretty cool to be able to logon to the device.

syslog entries
Sep 13 17:40:23 MAC sudo[57338]: root : TTY=unknown ; PWD=/ ; USER=newuser ; COMMAND=/sbin/mount_afp afp://user:pass@192.168.1.87/DroboApps /Volumes/DroboFS/0db102670141/1/DroboApps
Sep 13 17:40:24 MAC kernel[0]: AFP_VFS afpfs_mount: /Volumes/DroboFS/0db112312341/1/DroboApps, pid 57338
Sep 13 17:44:09 MAC /System/Library/CoreServices/backupd[57359]: Backup requested by user
Sep 13 17:44:09 MAC /System/Library/CoreServices/backupd[57359]: Starting standard backup
Sep 13 17:44:24 MAC kernel[0]: AFP_VFS afpfs_mount: /Volumes/MacTimeMachine, pid 57360
Sep 13 17:44:24 MAC /System/Library/CoreServices/backupd[57359]: Mounted network destination using URL: afp://user@Drobo.local/MacTimeMachine
Sep 13 17:44:24 MAC /System/Library/CoreServices/backupd[57359]: Backup destination mounted at path: /Volumes/MacTimeMachine
Sep 13 17:44:30 MAC /System/Library/CoreServices/backupd[57359]: Disk image /Volumes/MacTimeMachine/MAC_0017f2ca5f8b.sparsebundle mounted at: /Volumes/Backup of MAC
Sep 13 17:44:30 MAC /System/Library/CoreServices/backupd[57359]: Backing up to: /Volumes/Backup of MAC/Backups.backupdb

Summary
All-in-all, the process was super smooth and the device is everything I hoped for. With the ability to expand drive space at will, I shouldn't outgrow the FS very soon. I'm also looking forward to using more Drobo Apps and letting that be my server, instead of my XP VM on my Mac!

Long live Drobo!

Some Pics of the Process
drobo formatting my stack of drives


updating the firmware on the drobo


drobo temporarily loses its mind during firmware update


don't touch drobo while firmware is being updated!


drobo starting up after successful firmware update


drobo successfully updates its firmware


drobo dashboard download for update


time machine backup using drobo!


25MiBps (Mebibytes per second) network copy speed (with two backups to Drobo running..Time Machine and a backup of a 438GB archive)


Update 2010/08/14
Drobo shell commands

Found some good links on cross-compiling packages for the Drobo:

Ethtool specific links

Drobo Forums
Feel free to drop me a line or ask me a question.