Linux Disk and Storage Monitoring and Tuning
Jump to navigation
Jump to search
Tools
- iotop - http://guichaz.free.fr/iotop/
- Collectl
- iometer
iostat and sar -d
The output of sar -d will be dev minor and dev major numbers.
$ sudo sar -d 3 10 ... Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util Average: dev202-0 3.47 3.20 70.69 21.31 0.00 0.99 0.62 0.22 Average: dev202-20736 67.72 740.01 434.28 17.34 0.08 1.16 0.52 3.50 Average: dev202-20480 51.72 489.76 416.67 17.53 0.07 1.34 0.56 2.90 Average: dev202-20224 50.85 487.10 424.14 17.92 0.07 1.44 0.59 2.98 Average: dev202-19968 72.89 861.12 398.00 17.27 0.08 1.07 0.53 3.83 Average: dev202-27136 0.13 0.00 1.07 8.00 0.00 0.75 0.25 0.00 Average: dev202-26880 0.20 0.00 4.27 21.33 0.00 0.33 0.33 0.01 Average: dev202-5632 0.40 0.53 2.67 8.00 0.00 0.92 0.92 0.04 Average: dev202-208 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: dev202-112 116.41 12946.75 46.42 111.62 0.85 7.27 2.63 30.57 Average: dev202-32 49.08 0.53 744.48 15.18 0.04 0.78 0.78 3.82 Average: dev202-48 49.08 0.53 744.48 15.18 0.04 0.80 0.79 3.90 Average: dev202-160 8.94 0.00 169.26 18.94 0.54 60.80 8.40 7.51 Average: dev202-144 8.94 0.00 169.26 18.94 0.07 8.27 3.69 3.30 Average: dev202-26624 7.77 0.53 1242.01 159.93 0.01 1.65 0.71 0.55 Average: dev202-21248 46.68 436.41 401.73 17.95 0.06 1.35 0.57 2.64 Average: dev202-20992 49.75 476.43 379.86 17.21 0.07 1.36 0.65 3.25
Now let's say I want to see what device dev202-112 is. This system uses LVM so we have to trace back to logical volumes.
So first:
$ sudo pvs -all -o +lv_name PV VG Fmt Attr PSize PFree LV /dev/oraclevg07/oraclelv07 ---- 0 0 /dev/ram0 ---- 0 0 /dev/ram1 ---- 0 0 /dev/ram10 ---- 0 0 /dev/ram11 ---- 0 0 /dev/ram12 ---- 0 0 /dev/ram13 ---- 0 0 /dev/ram14 ---- 0 0 /dev/ram15 ---- 0 0 /dev/ram2 ---- 0 0 /dev/ram3 ---- 0 0 /dev/ram4 ---- 0 0 /dev/ram5 ---- 0 0 /dev/ram6 ---- 0 0 /dev/ram7 ---- 0 0 /dev/ram8 ---- 0 0 /dev/ram9 ---- 0 0 /dev/root ---- 0 0 /dev/swapvg01/swaplv01 ---- 0 0 /dev/xvdc ---- 0 0 /dev/xvdca ---- 0 0 /dev/xvdcb ---- 0 0 /dev/xvdcc ---- 0 0 /dev/xvdcd ---- 0 0 /dev/xvdce ---- 0 0 /dev/xvdcf ---- 0 0 /dev/xvdcg ---- 0 0 /dev/xvdch ---- 0 0 /dev/xvdd ---- 0 0 /dev/xvdda ---- 0 0 /dev/xvddb1 ---- 0 0 /dev/xvddc1 ---- 0 0 /dev/xvdh ---- 0 0 /dev/xvdi ---- 0 0 /dev/xvdj1 oraclevg01 lvm2 a--u 32.00g 652.00m [oraclelv01_mimage_1] /dev/xvdj1 oraclevg01 lvm2 a--u 32.00g 652.00m [oraclelv01_mlog_mimage_1] /dev/xvdj1 oraclevg01 lvm2 a--u 32.00g 652.00m /dev/xvdj2 oraclevg02 lvm2 a--u 32.00g 652.00m [oraclelv02_mimage_1] /dev/xvdj2 oraclevg02 lvm2 a--u 32.00g 652.00m [oraclelv02_mlog_mimage_1] /dev/xvdj2 oraclevg02 lvm2 a--u 32.00g 652.00m /dev/xvdj3 oraclevg04 lvm2 a--u 32.00g 652.00m [oraclelv04_mimage_1] /dev/xvdj3 oraclevg04 lvm2 a--u 32.00g 652.00m [oraclelv04_mlog_mimage_1] /dev/xvdj3 oraclevg04 lvm2 a--u 32.00g 652.00m /dev/xvdj5 oraclevg03 lvm2 a--u 32.00g 652.00m [oraclelv03_mimage_1] /dev/xvdj5 oraclevg03 lvm2 a--u 32.00g 652.00m [oraclelv03_mlog_mimage_1] /dev/xvdj5 oraclevg03 lvm2 a--u 32.00g 652.00m /dev/xvdj6 oraclevg05 lvm2 a--u 34.97g 716.00m [oraclelv05_mimage_1] /dev/xvdj6 oraclevg05 lvm2 a--u 34.97g 716.00m [oraclelv05_mlog_mimage_1] /dev/xvdj6 oraclevg05 lvm2 a--u 34.97g 716.00m /dev/xvdk1 oraclevg01 lvm2 a--u 32.00g 652.00m [oraclelv01_mimage_0] /dev/xvdk1 oraclevg01 lvm2 a--u 32.00g 652.00m [oraclelv01_mlog_mimage_0] /dev/xvdk1 oraclevg01 lvm2 a--u 32.00g 652.00m /dev/xvdk2 oraclevg02 lvm2 a--u 32.00g 652.00m [oraclelv02_mimage_0] /dev/xvdk2 oraclevg02 lvm2 a--u 32.00g 652.00m [oraclelv02_mlog_mimage_0] /dev/xvdk2 oraclevg02 lvm2 a--u 32.00g 652.00m /dev/xvdk3 oraclevg04 lvm2 a--u 32.00g 652.00m [oraclelv04_mimage_0] /dev/xvdk3 oraclevg04 lvm2 a--u 32.00g 652.00m [oraclelv04_mlog_mimage_0] /dev/xvdk3 oraclevg04 lvm2 a--u 32.00g 652.00m /dev/xvdk5 oraclevg03 lvm2 a--u 32.00g 652.00m [oraclelv03_mimage_0] /dev/xvdk5 oraclevg03 lvm2 a--u 32.00g 652.00m [oraclelv03_mlog_mimage_0] /dev/xvdk5 oraclevg03 lvm2 a--u 32.00g 652.00m /dev/xvdk6 oraclevg05 lvm2 a--u 34.97g 716.00m [oraclelv05_mimage_0] /dev/xvdk6 oraclevg05 lvm2 a--u 34.97g 716.00m [oraclelv05_mlog_mimage_0] /dev/xvdk6 oraclevg05 lvm2 a--u 34.97g 716.00m /dev/xvdn1 oraclevg07 lvm2 a--u 1023.99g 0 oraclelv07 /dev/xvdu ---- 0 0 /dev/xvdv ---- 0 0 /dev/xvdw ---- 0 0 /dev/xvdx1 swapvg01 lvm2 a--u 15.99g 0 swaplv01
Now to get minor and major numbers. Not this is a RHEL Linux system on AWS so I am only looking at /dev/sd and /dev/xv EBS volume devices plus the device mapper entries.
$ ls -lL /dev/mapper /dev/sd* /dev/xv* ls: cannot access /dev/sd*: No such file or directory brw-rw----. 1 root disk 202, 0 Mar 7 02:56 /dev/xvda brw-rw----. 1 root disk 202, 1 Aug 4 2018 /dev/xvda1 brw-rw----. 1 oracle dba 202, 32 Apr 5 16:08 /dev/xvdc brw-rw----. 1 oracle dba 202, 19968 Apr 5 16:08 /dev/xvdca brw-rw----. 1 oracle dba 202, 20224 Apr 5 16:08 /dev/xvdcb brw-rw----. 1 oracle dba 202, 20480 Apr 5 16:08 /dev/xvdcc brw-rw----. 1 oracle dba 202, 20736 Apr 5 16:08 /dev/xvdcd brw-rw----. 1 oracle dba 202, 20992 Apr 5 16:08 /dev/xvdce brw-rw----. 1 oracle dba 202, 21248 Apr 5 16:08 /dev/xvdcf brw-rw----. 1 oracle dba 202, 21504 Apr 5 16:08 /dev/xvdcg brw-rw----. 1 oracle dba 202, 21760 Apr 5 16:08 /dev/xvdch brw-rw----. 1 oracle dba 202, 48 Apr 5 16:08 /dev/xvdd brw-rw----. 1 oracle dba 202, 26624 Apr 5 16:08 /dev/xvdda brw-rw----. 1 root disk 202, 26880 Mar 7 02:56 /dev/xvddb brw-rw----. 1 root disk 202, 26881 Aug 4 2018 /dev/xvddb1 brw-rw----. 1 root disk 202, 27136 Mar 7 04:53 /dev/xvddc brw-rw----. 1 root disk 202, 27137 Mar 7 04:56 /dev/xvddc1 brw-rw----. 1 oracle dba 202, 112 Apr 5 16:08 /dev/xvdh brw-rw----. 1 oracle dba 202, 128 Apr 5 16:08 /dev/xvdi brw-rw----. 1 root disk 202, 144 Mar 7 02:56 /dev/xvdj brw-rw----. 1 root disk 202, 145 Aug 4 2018 /dev/xvdj1 brw-rw----. 1 root disk 202, 146 Aug 4 2018 /dev/xvdj2 brw-rw----. 1 root disk 202, 147 Aug 4 2018 /dev/xvdj3 brw-rw----. 1 root disk 202, 148 Aug 4 2018 /dev/xvdj4 brw-rw----. 1 root disk 202, 149 Aug 4 2018 /dev/xvdj5 brw-rw----. 1 root disk 202, 150 Aug 4 2018 /dev/xvdj6 brw-rw----. 1 root disk 202, 160 Mar 7 02:56 /dev/xvdk brw-rw----. 1 root disk 202, 161 Aug 4 2018 /dev/xvdk1 brw-rw----. 1 root disk 202, 162 Aug 4 2018 /dev/xvdk2 brw-rw----. 1 root disk 202, 163 Aug 4 2018 /dev/xvdk3 brw-rw----. 1 root disk 202, 164 Aug 4 2018 /dev/xvdk4 brw-rw----. 1 root disk 202, 165 Aug 4 2018 /dev/xvdk5 brw-rw----. 1 root disk 202, 166 Aug 4 2018 /dev/xvdk6 brw-rw----. 1 root disk 202, 208 Aug 4 2018 /dev/xvdn brw-rw----. 1 root disk 202, 209 Aug 4 2018 /dev/xvdn1 brw-rw----. 1 root disk 202, 5120 Mar 7 02:56 /dev/xvdu brw-rw----. 1 oracle dba 202, 5376 Apr 5 16:08 /dev/xvdv brw-rw----. 1 oracle dba 202, 5632 Apr 5 16:08 /dev/xvdw brw-rw----. 1 root disk 202, 5888 Mar 7 02:56 /dev/xvdx brw-rw----. 1 root disk 202, 5889 Aug 4 2018 /dev/xvdx1 /dev/mapper: total 0 crw-rw----. 1 root root 10, 58 Aug 4 2018 control brw-rw----. 1 root disk 253, 30 Mar 7 02:56 oraclevg01-oraclelv01 brw-rw----. 1 root disk 253, 28 Aug 4 2018 oraclevg01-oraclelv01_mimage_0 brw-rw----. 1 root disk 253, 29 Aug 4 2018 oraclevg01-oraclelv01_mimage_1 brw-rw----. 1 root disk 253, 27 Aug 4 2018 oraclevg01-oraclelv01_mlog brw-rw----. 1 root disk 253, 25 Aug 4 2018 oraclevg01-oraclelv01_mlog_mimage_0 brw-rw----. 1 root disk 253, 26 Aug 4 2018 oraclevg01-oraclelv01_mlog_mimage_1 brw-rw----. 1 root disk 253, 24 Mar 7 02:56 oraclevg02-oraclelv02 brw-rw----. 1 root disk 253, 22 Aug 4 2018 oraclevg02-oraclelv02_mimage_0 brw-rw----. 1 root disk 253, 23 Aug 4 2018 oraclevg02-oraclelv02_mimage_1 brw-rw----. 1 root disk 253, 21 Aug 4 2018 oraclevg02-oraclelv02_mlog brw-rw----. 1 root disk 253, 19 Aug 4 2018 oraclevg02-oraclelv02_mlog_mimage_0 brw-rw----. 1 root disk 253, 20 Aug 4 2018 oraclevg02-oraclelv02_mlog_mimage_1 brw-rw----. 1 root disk 253, 12 Mar 7 02:56 oraclevg03-oraclelv03 brw-rw----. 1 root disk 253, 10 Aug 4 2018 oraclevg03-oraclelv03_mimage_0 brw-rw----. 1 root disk 253, 11 Aug 4 2018 oraclevg03-oraclelv03_mimage_1 brw-rw----. 1 root disk 253, 9 Aug 4 2018 oraclevg03-oraclelv03_mlog brw-rw----. 1 root disk 253, 7 Aug 4 2018 oraclevg03-oraclelv03_mlog_mimage_0 brw-rw----. 1 root disk 253, 8 Aug 4 2018 oraclevg03-oraclelv03_mlog_mimage_1 brw-rw----. 1 root disk 253, 18 Mar 7 02:56 oraclevg04-oraclelv04 brw-rw----. 1 root disk 253, 16 Aug 4 2018 oraclevg04-oraclelv04_mimage_0 brw-rw----. 1 root disk 253, 17 Aug 4 2018 oraclevg04-oraclelv04_mimage_1 brw-rw----. 1 root disk 253, 15 Aug 4 2018 oraclevg04-oraclelv04_mlog brw-rw----. 1 root disk 253, 13 Aug 4 2018 oraclevg04-oraclelv04_mlog_mimage_0 brw-rw----. 1 root disk 253, 14 Aug 4 2018 oraclevg04-oraclelv04_mlog_mimage_1 brw-rw----. 1 root disk 253, 6 Mar 7 02:56 oraclevg05-oraclelv05 brw-rw----. 1 root disk 253, 4 Aug 4 2018 oraclevg05-oraclelv05_mimage_0 brw-rw----. 1 root disk 253, 5 Aug 4 2018 oraclevg05-oraclelv05_mimage_1 brw-rw----. 1 root disk 253, 3 Aug 4 2018 oraclevg05-oraclelv05_mlog brw-rw----. 1 root disk 253, 1 Aug 4 2018 oraclevg05-oraclelv05_mlog_mimage_0 brw-rw----. 1 root disk 253, 2 Aug 4 2018 oraclevg05-oraclelv05_mlog_mimage_1 brw-rw----. 1 root disk 253, 0 Mar 7 02:56 oraclevg07-oraclelv07 brw-rw----. 1 root disk 253, 31 Mar 7 02:56 swapvg01-swaplv01
So device dev202-112 is /dev/xvdh which turns out not to be under LVM control and is actually a raw volume used by Oracle.