[SASAG] Broken Raid 5 Repair

Jeff Silverman jeffsilverm at gmail.com
Wed Sep 9 13:04:38 PDT 2009


I had a RAID 5 array based on a 3ware 7040 controller with 4 drives.  One
day, I noticed that one of the drives went bad.  But the system kept
running.  Eventually, I bought a new drive and installed it.  I made it a
hot spare and figured I would integrate it into the array after the system
booted with the threeware utilities.  Alas, the ROM BIOS in the 3ware
controller didn't recognize that the three good drives were an array and
refused to export the drive to the OS.  So my machine won't boot.

I had a reasonably good backup of the user data on the server, but no backup
of the OS.  So I took the array apart and rebuilt it with all 4 drives.
Then I reinstalled the OS from CD (Ubuntu server 9.04), did the
customizations of Apache, DNS, NTP from scratch (which took about 4 hours),
restored the user data from backup.  All in all, about 8 hours of work.

My recommendation is that, while your array is still working (it is working,
isn't it?) that you backup all of your user data and everything in /etc
before you begin.

Jeff Silverman

On Wed, Sep 9, 2009 at 10:47 AM, spam at tprophet.org <spam at tprophet.org>wrote:

> The best presentation I've ever seen on this topic was at Defcon. While
> flippant and irreverent (this *was* Defcon after all), here's the slide
> deck:
> http://myharddrivedied.com/Defcon17/RAID-Speech.pdf
> -TProphet
> Eric C. Aiken wrote:
>> I seeking advice on repairing a broken RAID 5 Array.I have a 6 disk RAID 5
>> array that lost a second drive.
>> There are a number of commercial-ware products the report the ability to
>> repair the array.  I'm not sure of their abilities and secondly they all
>> seem windows NTFS focused.  What may complicate the is issue is the
>> filesystem is XFS.
>> There are a few folks who have repaired their arrays by dd'ing the
>> individual disk to  images files and rebuilding from there.  I like to hear
>> anybody else's experience with repair broken arrays.  Any tools (commercial,
>> open source, shareware), processes or any other good's and bad's you might
>> have experienced.
>> Thanks
>> Eric C. Aiken
>> ------------------------------------------------------------------------
>> _______________________________________________
>> Members mailing list
>> Members at lists.sasag.org
>> http://lists.sasag.org/mailman/listinfo/members
> _______________________________________________
> Members mailing list
> Members at lists.sasag.org
> http://lists.sasag.org/mailman/listinfo/members

Jeff Silverman, linux sysadmin
nine two four   twentieth avenue east
Seattle, WA, nine eight one one two -3507
(2O6) 329-1O94
jeffsilverm at gmail.c0m (note the zero!)
Read my book, "Failure is Not an Option: How to build reliable computer
systems from unreliable parts using Open Source software"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sasag.org/pipermail/members/attachments/20090909/f922df15/attachment.html>

More information about the Members mailing list