Linux - Software RAID and Mirrors
Summary of RAID software in Linux
There are several options available to Linux: these are all software based solutions as opposed to hardware RAID we used to see in physical disk controllers or Storage Area Networks.
In Linux the three main commands for creating a software RAID between two or more volumes are: lvm, mdadm or dmraid. dmraid (Device Mapper) is limited to ATA type disks (commonly known as "fakeRAID") so we will not use that here. Mirroring with LVM2 (which ships with RHEL 7) is considered broken when it come to mirroring so we will focus here on mdadm.
Some notes:
https://jreypo.wordpress.com/tag/device-mapper/
http://stackoverflow.com/questions/23164384/what-is-the-difference-between-dm-and-md-in-linux-kernel
The Linux RAID FAQ: https://raid.wiki.kernel.org/index.php/Linux_Raid
RAID on AWS instance: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/raid-config.html
mdadm mirrors
You will likely need to install the madam softwware as it is not installed by default:
yum install mdadm
Notes: https://raid.wiki.kernel.org/index.php/Partitioning_RAID_/_LVM_on_RAID
An example of mirroring two AWS EBS volumes in an instance:
1. Create two same sized EBS volumes and attach to the instance as known devices say (/dev/xvde and /dev/xvds). Note if you are re-using a currently attached EBS volume drop to Step 4 below.
2. Verify the volumes are attached to your instance:
$ sudo lsblk /dev/xvde NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvde 202:64 0 34G 0 disk
3. You do not need to fdisk or partition these two volumes. This will be done through the mirror volume.
4. If this volume has been used before we need to erase it. This can take quite a long time. It is also known as "pre-warming". You may want to consier removing and creating a new EBS volume from sratch it would be quicker.
To erase an already connected EBS volume we will use output from fdisk to supply parameters to the dd command.
$ sudo fdisk -l /dev/xvdf Disk /dev/xvdf: 34.4 GB, 34359738368 bytes, 67108864 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes $ sudo dd if=/dev/zero of=/dev/xvdf bs=512 20428009+0 records in 20428009+0 records out 10459140608 bytes (10 GB) copied, 1356.81 s, 7.7 MB/s real 22m36.817s user 0m0.325s sys 0m59.720s
As you can see above it took 23 minutes just to zero out 10GB.
5. Run this command to create mirrored volume of both EBS volumes. The resulting volume will be /dev/md/oracle0.
$ sudo mdadm --create --auto=mdp --verbose /dev/md/oracle0 --name=oracle0 --level=mirror --raid-devices=2 /dev/xvde /dev/xvds
mdadm: /dev/xvde appears to be part of a raid array:
level=raid0 devices=0 ctime=Wed Dec 31 19:00:00 1969
mdadm: partition table exists on /dev/xvde but will be lost or
meaningless after creating array
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: /dev/xvds appears to be part of a raid array:
level=raid0 devices=0 ctime=Wed Dec 31 19:00:00 1969
mdadm: partition table exists on /dev/xvds but will be lost or
meaningless after creating array
mdadm: size set to 35618816K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/oracle0 started.
6. Partition your mirror device:
$ sudo fdisk /dev/md/oracle0
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x70a07f5e.
Command (m for help): p
Disk /dev/md/oracle0: 36.5 GB, 36473667584 bytes, 71237632 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x70a07f5e
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-71237631, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-71237631, default 71237631):
Using default value 71237631
Partition 1 of type Linux and of size 34 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
$ sudo fdisk -l /dev/md/oracle0
Disk /dev/md/oracle0: 36.5 GB, 36473667584 bytes, 71237632 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x70a07f5e
Device Boot Start End Blocks Id System
/dev/md/oracle0p1 2048 71237631 35617792 83 Linux
7. Now format the filesystem (ext4):
$ sudo mkfs.ext4 -L /oracle /dev/md/oracle0p1
8. Check on the raid volume:
$ sudo mdadm --detail /dev/md/oracle0
/dev/md/oracle0:
Version : 1.2
Creation Time : Mon Feb 22 19:14:59 2016
Raid Level : raid1
Array Size : 35618816 (33.97 GiB 36.47 GB)
Used Dev Size : 35618816 (33.97 GiB 36.47 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Feb 22 19:24:56 2016
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : awsmdmqld02.hmco.com:oracle0 (local to host awsmdmqld02.hmco.com)
UUID : de418038:1019361b:90a24022:83f16131
Events : 18
Number Major Minor RaidDevice State
0 202 64 0 active sync /dev/xvde
1 202 4608 1 active sync /dev/xvds
9. Update the madam.conf file for reboots.
$ sudo mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm.conf
Monitoring the RAID
You can always take a look at /proc/mdstat. You can also use mdadm command to check the RAID array status.
mdadm --detail we looked at above.
This command will show spare and failed disks.
Notes of interest
Cheatsheet: http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/