How to Replace a Hard Drive In a Raid 5 via Ubuntu

Unfortunately I started receiving errors saying that one of the drives in my raid 5 (google exact error message) was messed up.  It held up for quite a while and it was time for an upgrade anyways.  Here are the steps I used to replace a drive within in my raid.

Determine which drive is failing

1. Since I didn’t label the drives I have to check the serial numbers to determine which one is faulty.  So I first installed hdparm. (If you want to get some SMART statistics then I would look into smartmontools)

sudo apt-get install hdparm

2. Once hdparm is installed you can run the command below.  It will list out all the serial numbers for the drives that are currently installed in your system:

hdparm -i /dev/sd? | grep SerialNo

Model=WDC WD5000BEVT-22A0RT0, FwRev=01.01A01, SerialNo=WD-WX20AA92XXXX
  Model=WDC WD20EFRX-68AX9N0, FwRev=80.00A80, SerialNo=WD-WMC1T133XXXX
  Model=WDC WD20EFRX-68AX9N0, FwRev=80.00A80, SerialNo=WD-WMC30059XXXX
  Model=WDC WD80EMAZ-00WJTA0, FwRev=83.H0A83, SerialNo=2SG4XTVW

2. Write down the serial number of the drive that needs to be replaced

3. Verify that the raid drive matches the serial number by replacing the ? with the drive letter

hdparm -i /dev/sdb | grep SerialNo

Model=WDC WD20EFRX-68AX9N0, FwRev=80.00A80, SerialNo=WD-WMC1T1XXXXXX

4. List the drives in the raid:

 thegeekfix@nas:~$ cat /proc/mdstat
  Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
  md0 : active raid5 sdc1[2] sdb1[1] sdd1[3]
   3906763776 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

5. Once you have verified the serial number of the drive you can mark it as failed and then remove it from the raid with the following command:

mdadm --manage /dev/md0 --fail /dev/sde1
mdadm --manage /dev/md0 --remove /dev/sde1

Adding the new drive

1. After replacing the hard drive you’ll want to double check that the system is able to read it.

   hdparm -i /dev/sdb | grep SerialNo

2. Next, you’ll want to partition it.

   parted -a optimal /dev/sde 

3. Finally, add it to the raid.

   mdadm --manage /dev/md0 --add /dev/sde1

4. Depending on how much data you have it can take a couple of hours.  You can view the progress by using the following command:

   cat /proc/mdstat

5. If you are replacing all the drives then you will have to partition and add each one to the raid one at a time.

Growing your raid

In my instance I decided to just replace all of the hard drives with larger ones (2tb -> 8tb) as I was running out of space anyways.  After replacing, partitioning, and adding each drive one by one it was time to expand the raid.  Here is how I did it:

1. I came across some information noting that it is strongly recommended to remove the bitmap prior to expanding so I executed the following commands.  Also note that this process can take a couple of hours depending on how large your raid will be:

   mdadm --grow /dev/mdX --bitmap none
   mdadm --grow /dev/mdX --size=max
   mdadm --wait /dev/mdX
   mdadm --grow /dev/mdX --bitmap internal

2. Once that completes you should be able to see your expanded raid by simply typing the following command.

df