Replace a disk in RAID
Suppose the server has 2 disks: /dev/sda
and /dev/sdb
. These disks are assembled into a software RAID1 using the mdadm --assemble
utility.
One of the disks has failed, for example, it is /dev/sdb
. A damaged disk must be replaced.
It is advisable to remove the disk from the array before replacing the disk.
Remove a disk from the array
Check how the disk in the array is partitioned:
cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda3[0] sdb3[1]
975628288 blocks super 1.2 [2/2] [UU]
bitmap: 3/8 pages [12KB], 65536KB chunk
md0 : active raid1 sda2[2] sdb2[1]
999872 blocks super 1.2 [2/2] [UU]
unused devices: <none>
In this case, the array is assembled so that md0
consists of sda2
and sdb2
, md1
consists of sda3
and sdb3
.
On this server, md0
is /boot
and md1
is swap and root.
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 0 985M 1 loop
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 977M 0 part
│ └─md0 9:0 0 976.4M 0 raid1
└─sda3 8:3 0 930.6G 0 part
└─md1 9:1 0 930.4G 0 raid1
├──vg0-swap_1 253:0 0 0 4.8G 0 lvm
└─vg0-root 253:1 0 925.7G 0 lvm /
sdb 8:16 0 931.5G 0 disk
├─sdb1 8:17 0 1M 0 part
├─sdb2 8:18 0 977M 0 part
│ └─md0 9:0 0 976.4M 0 raid1
└─sdb3 8:19 0 930.6G 0 part
└─md1 9:1 0 930.4G 0 raid1
├──vg0-swap_1 253:0 0 4.8G 0 lvm
└─vg0-root 253:1 0 925.7G 0 lvm /
Remove sdb
from all devices:
mdadm /dev/md0 --remove /dev/sdb2
mdadm /dev/md1 --remove /dev/sdb3
If partitions from the array are not removed, mdadm
does not consider the disk to be faulty and is using it, so an error will be displayed that the device is in use when it is removed.
In this case, mark the disk as failed before removing it:
mdadm /dev/md0 -f /dev/sdb2
mdadm /dev/md1 -f /dev/sdb3
Run the commands to remove partitions from the array again.
After removing the failed disk from the array, request a replacement disk by creating a ticket with the s/n of the failed disk. The availability of downtime depends on the server configuration.
Determine the partition table (GPT or MBR) and move it to a new disk
After replacing the damaged disk, you need to add the new disk to the array. To do this, you need to determine the type of partition table: GPT or MBR. The gdisk
is used for this purpose.
Install gdisk
:
apt-get install gdisk -y
Execute the command:
gdisk -l /dev/sda
Where /dev/sda
is a serviceable disk that is in RAID.
For MBR, the output will be something like the following:
Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present
For GPT is roughly as follows:
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Before adding a disk to the array, you need to create partitions on it exactly the same as on sda
. Depending on the partitioning of the disk, this is done differently.
Copy partitioning for GPT
To copy the GPT partitioning:
sgdisk -R /dev/sdb /dev/sda
Here, the disk to which the partitioning is copied is written first, and the disk from which it is copied (that is, from sda
to sdb
) is written second. If you mix them up, the partitioning on an initially serviceable disk will be destroyed.
The second way to copy the markup:
sgdisk --backup=table /dev/sda
sgdisk --load-backup=table /dev/sdb
After copying, assign a new random UUID to the disk:
sgdisk -G /dev/sdb
Copy partitioning for MBR
To copy the MBR partitioning:
sfdisk -d /dev/sda | sfdisk /dev/sdb
Here, the disk from which the partitioning is copied is written first, and the disk to which the partitioning is copied is written second.
If the partitions are not visible in the system, you can reread the partition table with the command:
sfdisk -R /dev/sdb
Add a disk to the array
If partitions have been created on /dev/sdb
, you can add the disk to the array:
mdadm /dev/md0 -a /dev/sdb2
mdadm /dev/md1 -a /dev/sdb3
Once the disk is added to the array, synchronization should begin. The speed depends on the size and type of disk (ssd/hdd):
cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda3[1] sdb3[0]
975628288 blocks super 1.2 [2/1] [U_]
[============>........] recovery = 64.7% (632091968/975628288) finish=41.1min speed=139092K/sec
bitmap: 3/8 pages [12KB], 65536KB chunk
md0 : active raid1 sda2[2] sdb2[1]
999872 blocks super 1.2 [2/2] [UU]
unused devices: <none>
Install the bootloader
After adding a disk to the array, you need to install a bootloader on it.
If the server is loaded in normal mode or in infiltrate-root
, this is done with a single command:
grub-install /dev/sdb
If the server is booted to Recovery or Rescue (that is. from live cd), then to install the bootloader:
-
Mount the root file system to
/mnt
:mount /dev/md2 /mnt
-
Mount
boot
:mount /dev/md0 /mnt/boot
-
Mount
/dev
,/proc
, and/sys
:mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys -
Execute
chroot
to the mounted system:chroot /mnt
-
Set
grub
tosdb
:grub-install /dev/sdb
Then try booting into normal mode.
Replace the disk if it fails
A disk in the array can be conditionally made to fail with the --fail (-f)
key:
mdadm /dev/md0 --fail /dev/sda1
or
mdadm /dev/md0 -f /dev/sda1
You can remove a failed disk with the --remove (-r)
switch:
mdadm /dev/md0 --remove /dev/sda1
or
mdadm /dev/md0 -r /dev/sda1
You can add a new disk to the array by using the --add (-a)
and --re-add
keys:
mdadm /dev/md0 --add /dev/sda1
or
mdadm /dev/md0 -a /dev/sda1
Error when restoring bootloader after replacing disk in RAID1
The following error may occur when installing grub
:
root #grub-install --root-directory=/boot /dev/sda
Could not find device for /boot/boot: not found or not a block device
In that case, execute:
root #grep -v rootfs /proc/mounts > /etc/mtab