Replace a disk in RAID

1. Remove the disk from the array.
2. Define a disk partitioning scheme.
3. Copy the partitioning to the new disk.
4. Add the disk to a RAID array.
5. Install the bootloader.

1. Remove a disk from the array⁠

1. Connect to the server via SSH or via KVM console.

2. Output information about the partitioning of disks in the RAID array:

   cat /proc/mdstat

   The response will show information about the partitioning of the disks in the RAID array. For example:

   Personalities : [raid1]
   md1 : active raid1 sda3[0] sdb3[1]
       975628288 blocks super 1.2 [2/2] [UU]
       bitmap: 3/8 pages [12KB], 65536KB chunk
   
   md0 : active raid1 sda2[2] sdb2[1]
       999872 blocks super 1.2 [2/2] [UU]
   
   unused devices: <none>

   Here:

   • md0 — is the name of the RAID array that is assembled from the partitions sda2 и sdb2;
   • md1 — is the name of the RAID array that is assembled from the partitions sda3 и sdb3.

3. Print information about the partitions on the disks:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0     1M  0 part
   ├─sdb2             8:18   0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sdb3             8:19   0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /

   Here:

   • sda — name of the disk with working RAID arrays;
   • sdb — the name of the disk to be replaced;
   • sdb2 — disk partition name sdb which is a component of the first RAID array;
   • md0 — name of the first RAID array that is built from partitions sda2 и sdb2;
   • sdb3 — disk partition name sdb which is a component of the second RAID array;
   • md1 — name of the second RAID array, which is built from partitions sda3 и sdb3.

4. Mark the disk that needs to be replaced as faulty:

   mdadm /dev/<raid_array_1> -f /dev/<partition_1>
   mdadm /dev/<raid_array_2> -f /dev/<partition_2>

   Specify:

   • <raid_array_1> — the name of the first RAID you got in step 3, in the example it is md0;
   • <partition_1> — the partition name of the first RAID partition on the swap disk you got in step 3, in the example it is sdb2;
   • <raid_array_2> — the name of the second RAID you got in step 3, in the example it is md1;
   • <partition_2> — the partition name of the second RAID partition on the swap disk that you got in step 3, in the example it is sdb3.

5. Remove the disk that you marked as faulty in step 4 from all devices:

   mdadm /dev/<raid_array_1> --remove /dev/<partition_1>
   mdadm /dev/<raid_array_2> --remove /dev/<partition_2>

   Specify:

   • <raid_array_1> — the name of the first RAID you got in step 3, in the example it is md0;
   • <partition_1> — the partition name of the first RAID partition on the swap disk you got in step 3, in the example it is sdb2;
   • <raid_array_2> — the name of the second RAID you got in step 3, in the example it is md1;
   • <partition_2> — the partition name of the second RAID partition on the swap disk that you got in step 3, in the example it is sdb3.

2. Define the disk partitioning scheme⁠

1. Connect to the server via SSH or via KVM console.

2. Install a utility to work with the GPT partition table (GUID Partition Table) on disks — gdisk:

   apt-get install gdisk -y

3. Print information about the partitions on the disks:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk

   Here:

   • sda — the name of the disk with working RAID partitions;
   • sdb — the name of the new disk.

4. Define a disk partitioning scheme:

   gdisk -l /dev/<source_disk>

   Specify <source_disk> is the name of the working disk with RAID partitions that you got in step 3, in the example it is sda.

   The answer depends on the disk partitioning scheme:

   • GPT:

     Partition table scan:
     MBR: protective
     BSD: not present
     APM: not present
     GPT: present

   • MBR:

     Partition table scan:
     MBR: MBR only
     BSD: not present
     APM: not present
     GPT: not present

3. Copy the partitioning to a new disk⁠

The process of copying partitioning depends on the disk partitioning scheme — GPT or MBR.

carefully

When copying partitioning, it is important to specify the disks in the correct order in the command.If the order of disks is specified incorrectly, the correct disk will be overwritten and the data will be deleted.The data cannot be recovered.

GPT

1. Connect to the server via SSH or via KVM console.

2. Print information about the partitions on the disks:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk

   Here:

   • sda — the name of the disk with working RAID partitions;
   • sdb — the name of the new disk.

3. Copy the partitioning to the new disk.In the command, it is important to specify the disks in the correct order, otherwise the partitioning and data on the originally serviceable disk will be destroyed.

   sgdisk -R /dev/<new_disk> /dev/<source_disk>

   Specify:

   • <new_disk> — the name of the new disk to which the partitioning is copied, you got it in step 2, in the example it is sdb;
   • <source_disk> — the name of the disk with working RAID partitions from which the partitioning is copied, you got it in step 2, in the example it is sda.

4. Assign a random UUID to the new disk:

   sgdisk -G /dev/<new_disk>

   Specify <new_disk> is the name of the new disk to which the partitioning was copied, you got it in step 2, in the example it is sdb.

5. Make sure that the partitioning on the disks matches completely:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0     1M  0 part
   ├─sdb2             8:18   0   977M  0 part
   └─sdb3             8:19   0 930.6G  0 part

MBR

1. Connect to the server via SSH or via KVM console.

2. Print information about the partitions on the disks:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk

   Here:

   • sda — the name of the disk with working RAID partitions;
   • sdb — the name of the new disk.

3. Copy the partitioning to the new disk.In the command, it is important to specify the disks in the correct order, otherwise the partitioning and data on the originally serviceable disk will be destroyed.

   sfdisk -d /dev/<source_disk> | sfdisk /dev/<new_disk>

   Specify:

   • <source_disk> — name of the disk with working RAID partitions from which the partitioning is copied, you got it in step 2, in the example it is sda;
   • <new_disk> — the name of the new disk to which the partitioning is copied, you got it in step 2, in the example it is sdb.

4. If the partitions are not visible in the system, update the partition information:

   partprobe /dev/<new_disk>

   Specify <new_disk> is the name of the new disk to which the partitioning was copied, you got it in step 2, in the example it is sdb.

5. Make sure that the partitioning on the disks matches completely:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0     1M  0 part
   ├─sdb2             8:18   0   977M  0 part
   └─sdb3             8:19   0 930.6G  0 part

4. Add a disk to a RAID array⁠

1. Connect to the server via SSH or via KVM console.

2. Print information about the partitions on the disks:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0     1M  0 part
   ├─sdb2             8:18   0   977M  0 part
   └─sdb3             8:19   0 930.6G  0 part

   Here:

   • md0 — is the name of the first RAID array that is built from the sda2;
   • md1 — name of the second RAID array, which is built from the partition sda3;
   • sdb2 — name of the first partition on the new disk to be added to the RAID array;
   • sdb3 — name of the second partition on the new disk to be added to the RAID array.

3. Add the partitions of the new disk to the RAID array:

   mdadm /dev/<raid_array_1> -a /dev/<partition_1>
   mdadm /dev/<raid_array_2> -a /dev/<partition_2>

   Specify:

   • <raid_array_1> — the name of the first RAID array you got in step 2, in the example it is md0;
   • <partition_1> — the name of the first partition on the new disk you got in step 2, in the example it is sdb2;
   • <raid_array_2> — the name of the second RAID you got in step 2, in the example it is md1;
   • <partition_2> — the name of the second partition on the new disk that you got in step 2, in the example it is sdb3.

   After adding the disk to the RAID array, synchronization will start.The synchronization speed depends on the size and type of disk — SSD or HDD.

4. Wait for the synchronization to complete.

5. Make sure the new disk is added to the RAID array:

   lsblk

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0     1M  0 part
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sda3             8:3    0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0     1M  0 part
   ├─sdb2             8:18   0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1
   └─sdb3             8:19   0 930.6G  0 part
     └─md1            9:1    0 930.4G  0 raid1  /

5. Install the bootloader⁠

After adding the disk to the array, you need to install the operating system boot loader on it.

You can install the bootloader either in OS or in Rescue mode.

The server is loaded in the OS

1. Connect to the server via SSH or via KVM console.

2. Print information about the partitions on the disks:

   lsblk -o +FSTYPE

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT  FSTYPE
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0   500M  0 part              vfat
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1             ext4
   └─sda3             8:3    0 930.1G  0 part
     └─md1            9:1    0 929.9G  0 raid1  / 
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0   500M  0 part          
   ├─sdb2             8:18   0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1             ext4
   └─sdb3             8:19   0 930.1G  0 part
     └─md1            9:1    0 929.9G  0 raid1  /          ext4 

   Here:

   • sda — name of the disk with working RAID arrays;
   • sdb — the name of the new disk.

3. Install the GRUB boot loader:

   grub-install /dev/<new_disk>

   Specify <new_disk> is the name of the new disk you got in step 2, in the example it is sdb.

The server is booted in Rescue mode

1. Boot the server in Rescue recovery and diagnostic mode.

2. Connect to the server via SSH or via KVM console.

3. Output information about the OS boot mode:

   [ -d /sys/firmware/efi ] && echo "UEFI" || echo "BIOS"

   The response will show information about the OS boot mode — BIOS or UEFI.

4. Check if the system has detected RAID arrays, to do this, display the partition information on the disks:

   lsblk -o +FSTYPE

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT  FSTYPE
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0   500M  0 part              vfat
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1             ext4
   └─sda3             8:3    0 930.1G  0 part
     └─md1            9:1    0 929.9G  0 raid1  
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0   500M  0 part          
   ├─sdb2             8:18   0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1             ext4
   └─sdb3             8:19   0 930.1G  0 part
     └─md1            9:1    0 929.9G  0 raid1             ext4 

   Here md0, md1 are RAID names on disk partitions.

5. If there are RAIDs in the lsblk output at step 4, skip this step.If there are no RAIDs, run RAID build:

   mdadm --assemble --scan

6. Print information about the partitions on the available disks:

   lsblk -o +FSTYPE

   The response will show a list of disks with partitions.For example:

   NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT  FSTYPE
   loop0              7:0    0   985M  1 loop
   sda                8:0    0 931.5G  0 disk
   ├─sda1             8:1    0   500M  0 part              vfat
   ├─sda2             8:2    0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1             ext4
   └─sda3             8:3    0 930.1G  0 part
     └─md1            9:1    0 929.9G  0 raid1  
   sdb                8:16   0 931.5G  0 disk
   ├─sdb1             8:17   0   500M  0 part          
   ├─sdb2             8:18   0   977M  0 part
   │ └─md0            9:0    0 976.4M  0 raid1             ext4
   └─sdb3             8:19   0 930.1G  0 part
     └─md1            9:1    0 929.9G  0 raid1             ext4 

   Here:

   • sda — the name of the disk with working RAID partitions;
   • sdb — the name of the new disk;
   • md0, md1 — names of RAID arrays on disk partitions.

7. Determine the partitions you want to mount:

   • root partition / — usually located on the largest disk partition, in the example in step 6 this is md1;
   • boot partition /boot — usually has an ext4 file system and a size of 512 MB — 1 GB, in the example in step 6 this is md0;
   • EFI partition /boot/efi — partition with vfat file system, used when booting the OS in UEFI mode, in the example on step 6 these are partitions sda1 and sdb1.

8. Mount the root file system to the /mnt directory:

   mount /dev/<raid_array> /mnt

   Here <system_raid_array> is the name of the root partition you defined in step 7, in the example it is md1.

9. Mount the boot partition:

   mount /dev/<boot_partition> /mnt/boot

   Specify <boot_raid_array> is the name of the /boot boot partition you defined in step 7, in the example it is md0.

10. If you are using a UEFI OS loader, mount the EFI partition:

    mount /dev/<efi_partition> /mnt/boot/efi

    Specify <efi_partition> is the name of the EFI /boot/efi partition on the new disk that you defined in step 7, in the example it is sdb1.

11. Mount the service file systems:

    mount --bind /sys /mnt/sys
    mount --bind /proc /mnt/proc
    mount --bind /dev /mnt/dev
    mount -t devpts devpts /mnt/dev/pts

12. If you defined the OS boot mode as UEFI in step 3, mount the efivars file system:

mount -t efivarfs efivarfs /mnt/sys/firmware/efi/efivars

13. Connect to the environment:

    chroot /mnt /bin/bash

14. Export the PATH variable:

    export PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin

15. Install the GRUB boot loader.The command to install depends on the OS boot mode you defined in step 3:

    • UEFI:

    grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=GRUB

    • BIOS:

    grub-install /dev/<new_disk>

    Specify <new_disk> is the name of the new disk you got in step 6, in the example it is sdb.

16. Create a GRUB configuration file:

    /sbin/grub-mkconfig -o /boot/grub/grub.cfg

17. When the GRUB installation is complete, exit the environment:

    exit

18. Unmount the service file systems:

    mount -t devpts devpts /mnt/dev/pts
    mount --bind /dev /mnt/dev
    mount --bind /proc /mnt/proc
    mount --bind /sys /mnt/sys

19. Unmount the file system:

    umount /dev/<system_raid_array> /mnt

    Specify <system_raid_array> is the name of the root partition / that you mounted in step 8, in the example it is md1.

20. Return the server boot template to the previous one or reboot the server from the OS.When you reboot the server from the OS, the boot template will automatically change to the one that was set before the server was booted in Rescue mode.