Linux - Software RAID and Mirrors

From Public wiki of Kevin P. Inscoe
Jump to navigation Jump to search

Summary of RAID software in Linux

There are several options available to Linux: these are all software based solutions as opposed to hardware RAID we used to see in physical disk controllers or Storage Area Networks.

In Linux the three main commands for creating a software RAID between two or more volumes are: lvm, mdadm or dmraid. dmraid (Device Mapper) is limited to ATA type disks (commonly known as "fakeRAID") so we will not use that here. Mirroring with LVM2 (which ships with RHEL 7) is considered broken when it come to mirroring so we will focus here on mdadm.

Some notes:

https://jreypo.wordpress.com/tag/device-mapper/

http://stackoverflow.com/questions/23164384/what-is-the-difference-between-dm-and-md-in-linux-kernel

The Linux RAID FAQ: https://raid.wiki.kernel.org/index.php/Linux_Raid

RAID on AWS instance: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html

mdadm mirrors

You will likely need to install the madam softwware as it is not installed by default:

yum install mdadm

Notes: https://raid.wiki.kernel.org/index.php/Partitioning_RAID_/_LVM_on_RAID

An example of mirroring two AWS EBS volumes in an instance:

1. Create two same sized EBS volumes and attach to the instance as known devices say (/dev/xvde and /dev/xvds). Note if you are re-using a currently attached EBS volume drop to Step 4 below.

2. Verify the volumes are attached to your instance:

$ sudo lsblk  /dev/xvde
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvde 202:64   0  34G  0 disk 

3. You do not need to fdisk or partition these two volumes. This will be done through the mirror volume.

4. If this volume has been used before we need to erase it. This can take quite a long time. It is also known as "pre-warming". You may want to consier removing and creating a new EBS volume from sratch it would be quicker.

To erase an already connected EBS volume we will use output from fdisk to supply parameters to the dd command.

$ sudo fdisk -l /dev/xvdf

Disk /dev/xvdf: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

$ sudo dd if=/dev/zero of=/dev/xvdf bs=512
20428009+0 records in
20428009+0 records out
10459140608 bytes (10 GB) copied, 1356.81 s, 7.7 MB/s

real    22m36.817s
user    0m0.325s
sys     0m59.720s

As you can see above it took 23 minutes just to zero out 10GB.

5. Run this command to create mirrored volume of both EBS volumes. The resulting volume will be /dev/md/oracle0.

$ sudo mdadm --create --auto=mdp --verbose /dev/md/oracle0 --name=oracle0 --level=mirror --raid-devices=2 /dev/xvde /dev/xvds 
mdadm: /dev/xvde appears to be part of a raid array:
       level=raid0 devices=0 ctime=Wed Dec 31 19:00:00 1969
mdadm: partition table exists on /dev/xvde but will be lost or
       meaningless after creating array
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: /dev/xvds appears to be part of a raid array:
       level=raid0 devices=0 ctime=Wed Dec 31 19:00:00 1969
mdadm: partition table exists on /dev/xvds but will be lost or
       meaningless after creating array
mdadm: size set to 35618816K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/oracle0 started.

6. Partition your mirror device:

$ sudo fdisk /dev/md/oracle0
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x70a07f5e.

Command (m for help): p

Disk /dev/md/oracle0: 36.5 GB, 36473667584 bytes, 71237632 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x70a07f5e

           Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-71237631, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-71237631, default 71237631): 
Using default value 71237631
Partition 1 of type Linux and of size 34 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

$ sudo fdisk -l /dev/md/oracle0

Disk /dev/md/oracle0: 36.5 GB, 36473667584 bytes, 71237632 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x70a07f5e

           Device Boot      Start         End      Blocks   Id  System
/dev/md/oracle0p1            2048    71237631    35617792   83  Linux

7. Now format the filesystem (ext4):

$ sudo mkfs.ext4 -L /oracle /dev/md/oracle0p1

8. Check on the raid volume:

$ sudo mdadm --detail /dev/md/oracle0
/dev/md/oracle0:
        Version : 1.2
  Creation Time : Mon Feb 22 19:14:59 2016
     Raid Level : raid1
     Array Size : 35618816 (33.97 GiB 36.47 GB)
  Used Dev Size : 35618816 (33.97 GiB 36.47 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Mon Feb 22 19:24:56 2016
          State : active 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : awsmdmqld02.hmco.com:oracle0  (local to host awsmdmqld02.hmco.com)
           UUID : de418038:1019361b:90a24022:83f16131
         Events : 18

    Number   Major   Minor   RaidDevice State
       0     202       64        0      active sync   /dev/xvde
       1     202     4608        1      active sync   /dev/xvds

9. Update the madam.conf file for reboots.

$ sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm.conf

Monitoring the RAID

You can always take a look at /proc/mdstat. You can also use mdadm command to check the RAID array status.

mdadm --detail we looked at above.

This command will show spare and failed disks.

Notes of interest

Cheatsheet: http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/

HOWTO: http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html

http://edoceo.com/howto/mdadm-raid1