Linux RAID1: How to replace the boot disk of a mirror?

Linux RAID1 (mirroring) is great! However when the boot disk of your MD mirror fails, your system is broken.
This HOWTO describes the steps you need to get your system up and running again.

Warning and Disclaimer!
Don't use this HOWTO if you are not an experienced system administrator!!!
Neither the author nor Integrated Services accept any reponsability for using this HOWTO.
If as a result your system ends up completely broken, it's all up to you! Be warned!!

Skills needed:

Handson experience with Linux commands
vi editing skills (or your favourite editor)
basic knowledge of Linux MDADM

Tools used:

Debian Live CD (or equivalent rescue CD/DVD)
http://live.debian.net

1. Back up your data

Back up all your data.
If your system cannot boot because the boot disk failed completely, first carry out step 2 “Make Disk B bootable”.

2. Make Disk B bootable

2.1 Boot using rescue CD/DVD

Boot up your system from the Debian Live CD/DVD. Choose: “Live 686”. ¹⁾

Get root permissions: ²⁾

$ sudo su -

2.2 Start the RootFS mirror

Bring up the mirror of the Root filesystem.
E.g. if RootFS is on partition 2:

# mdadm --assemble /dev/md2 /dev/sda2 /dev/sdb2

mdadm: /dev/md2 has been started with 1 drive (out of 2).

Check mirror status:

# cat /proc/mdstat

Personalities : [raid1] 
md2 : active raid1 sdb2[1]
      8191936 blocks [2/1] [_U]
      
unused devices: <none>

2.3 Mount the RootFS

# mkdir /md2

# mount /dev/md2 /md2

2.4 Enter the RootFS

Chroot to the RootFS. ³⁾

# mount -t proc proc /md2/proc

# chroot /md2 /bin/bash

2.5 Make Disk B bootable (Grub v.1)

Note: The following commands are for Grub version 1.

Make the necessary devicefiles: ⁴⁾

# cd /dev

# MAKEDEV sda sdb

Install Grub bootloader on disk B:

# grub --no-floppy

grub> device (hd0) /dev/sdb

grub> root (hd0,1)

grub> setup (hd0)

grub> quit

More information on Grub:
http://www.kkoncepts.net/node/198

2.6 Reboot the system using Disk B

Leave the RootFS:

# exit

Reboot:

# reboot

Remove the CD/DVD and boot the system from disk B using the BIOS Boot menu: ⁵⁾ ⁶⁾

BIOS Boot menu
CD/DVD
Disk 0
< Disk 1 >
Select Boot device and press Return to boot.

3. Remove Disk A partition(s) from their mirror(s)

3.1 Flag all Disk A component partitions as "failed"

Each component partition must be flagged as “failed” before they can be removed from their mirror.
We can easily fail all component partitions in one step.
E.g. to flag mirrors md2, md3, md5, md6, md7, md8 as failed:

# for i in 2 3 5 6 7 8
do
  mdadm --fail /dev/md$i /dev/sda$i
done

3.2 Remove all disk A component(s) from their mirror(s)

# for i in 2 3 5 6 7 8
do
  mdadm --remove /dev/md$i /dev/sda$i
done

Verify the result:

# cat /proc/mdstat

Personalities : [raid1] 
md8 : active raid1 sdb8[1]
      10240192 blocks [2/1] [_U]
      
md7 : active raid1 sdb7[1]
      10240192 blocks [2/1] [_U]
      
md6 : active raid1 sdb6[1]
      2048128 blocks [2/1] [_U]
      
md5 : active raid1 sdb5[1]
      2048128 blocks [2/1] [_U]
      
md3 : active raid1 sdb3[1]
      20479936 blocks [2/1] [_U]
      
md2 : active raid1 sdb2[1]
      8191936 blocks [2/1] [_U]
      
unused devices: <none>

3.3 Update the mdadm.conf

# /usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf

4. Replace Disk A

4.1 Physically replace disk A with the new disk

Shutdown the system:

# shutdown -h now

Now physically replace disk A with the new disk and boot up the system.

4.2 Partition new disk A

In order to create an identical partition structure on the new disk A we need to copy the partition structure of disk B to the new disk A: ⁷⁾

# sfdisk -d /dev/sdb | sfdisk /dev/sda

5. Add the new Disk A partitions to the mirror

5.1 Zero MD-superblocks of disk A partitions

If disk A was previously used in a Linux MD array all superblocks should be erased before adding the disk A partitions to their mirrors:

# for i in 2 3 5 6 7 8
do
  mdadm --zero-superblock /dev/sda$i
done

5.2 Add Disk A partitions to the mirror

# for i in 2 3 5 6 7 8
do
  mdadm --add /dev/md$i /dev/sda$i
  sleep 2
done

Verify the result:

Personalities : [raid1] 
md8 : active (auto-read-only) raid1 sda8[0] sdb8[1]
      10240192 blocks [2/2] [UU]
      
md7 : active raid1 sda7[0] sdb7[1]
      10240192 blocks [2/2] [UU]
      
md6 : active raid1 sda6[0] sdb6[1]
      2048128 blocks [2/2] [UU]
      
md5 : active (auto-read-only) raid1 sda5[0] sdb5[1]
      2048128 blocks [2/2] [UU]
      
md3 : active (auto-read-only) raid1 sda3[0] sdb3[1]
      20479936 blocks [2/2] [UU]
      
md2 : active raid1 sda2[0] sdb2[1]
      8191936 blocks [2/2] [UU]
      
unused devices: <none>

5.3 Update mdadm.conf

# /usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf

6. Make Disk A bootable

Note: The following commands are for Grub version 1.

Verify the Grub device map:

# more /boot/grub/devive.map

(hd0)	/dev/sda
(hd1)	/dev/sdb

Install Grub bootloader on disk A:

# grub --no-floppy

grub> root (hd0,1)

grub> setup (hd0)

grub> quit

7. Reboot the system

Normally reboot your system.

Everything should be just fine!

.

¹⁾

You may have to change the BIOS settings in order to boot from CD/DVD.

²⁾

There is no password.

³⁾

Many actions require “/proc” support, so mount /proc first.

⁴⁾

udev is not active on the chrooted FS.

⁵⁾

Consult your BIOS manual to determine how to invoke the Boot menu.

⁶⁾

Yes, the Boot menu shown here is fake!

⁷⁾

Needless to say that the new disk A should at least be of the same size as disk B. Even better if the new disk A is of the same brand and model as disk B.

Inhoud