Linux Disk and Storage Monitoring and Tuning

From Public wiki of Kevin P. Inscoe
Jump to navigation Jump to search

Tools

iostat and sar -d

The output of sar -d will be dev minor and dev major numbers.

$ sudo sar -d 3 10
...
Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:     dev202-0      3.47      3.20     70.69     21.31      0.00      0.99      0.62      0.22
Average:    dev202-20736     67.72    740.01    434.28     17.34      0.08      1.16      0.52      3.50
Average:    dev202-20480     51.72    489.76    416.67     17.53      0.07      1.34      0.56      2.90
Average:    dev202-20224     50.85    487.10    424.14     17.92      0.07      1.44      0.59      2.98
Average:    dev202-19968     72.89    861.12    398.00     17.27      0.08      1.07      0.53      3.83
Average:    dev202-27136      0.13      0.00      1.07      8.00      0.00      0.75      0.25      0.00
Average:    dev202-26880      0.20      0.00      4.27     21.33      0.00      0.33      0.33      0.01
Average:    dev202-5632      0.40      0.53      2.67      8.00      0.00      0.92      0.92      0.04
Average:    dev202-208      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    dev202-112    116.41  12946.75     46.42    111.62      0.85      7.27      2.63     30.57
Average:    dev202-32     49.08      0.53    744.48     15.18      0.04      0.78      0.78      3.82
Average:    dev202-48     49.08      0.53    744.48     15.18      0.04      0.80      0.79      3.90
Average:    dev202-160      8.94      0.00    169.26     18.94      0.54     60.80      8.40      7.51
Average:    dev202-144      8.94      0.00    169.26     18.94      0.07      8.27      3.69      3.30
Average:    dev202-26624      7.77      0.53   1242.01    159.93      0.01      1.65      0.71      0.55
Average:    dev202-21248     46.68    436.41    401.73     17.95      0.06      1.35      0.57      2.64
Average:    dev202-20992     49.75    476.43    379.86     17.21      0.07      1.36      0.65      3.25

Now let's say I want to see what device dev202-112 is. This system uses LVM so we have to trace back to logical volumes.

So first:

$ sudo pvs -all -o +lv_name            
  PV                         VG         Fmt  Attr PSize    PFree   LV                        
  /dev/oraclevg07/oraclelv07                 ----       0       0                            
  /dev/ram0                                  ----       0       0                            
  /dev/ram1                                  ----       0       0                            
  /dev/ram10                                 ----       0       0                            
  /dev/ram11                                 ----       0       0                            
  /dev/ram12                                 ----       0       0                            
  /dev/ram13                                 ----       0       0                            
  /dev/ram14                                 ----       0       0                            
  /dev/ram15                                 ----       0       0                            
  /dev/ram2                                  ----       0       0                            
  /dev/ram3                                  ----       0       0                            
  /dev/ram4                                  ----       0       0                            
  /dev/ram5                                  ----       0       0                            
  /dev/ram6                                  ----       0       0                            
  /dev/ram7                                  ----       0       0                            
  /dev/ram8                                  ----       0       0                            
  /dev/ram9                                  ----       0       0                            
  /dev/root                                  ----       0       0                            
  /dev/swapvg01/swaplv01                     ----       0       0                            
  /dev/xvdc                                  ----       0       0                            
  /dev/xvdca                                 ----       0       0                            
  /dev/xvdcb                                 ----       0       0                            
  /dev/xvdcc                                 ----       0       0                            
  /dev/xvdcd                                 ----       0       0                            
  /dev/xvdce                                 ----       0       0                            
  /dev/xvdcf                                 ----       0       0                            
  /dev/xvdcg                                 ----       0       0                            
  /dev/xvdch                                 ----       0       0                            
  /dev/xvdd                                  ----       0       0                            
  /dev/xvdda                                 ----       0       0                            
  /dev/xvddb1                                ----       0       0                            
  /dev/xvddc1                                ----       0       0                            
  /dev/xvdh                                  ----       0       0                            
  /dev/xvdi                                  ----       0       0                            
  /dev/xvdj1                 oraclevg01 lvm2 a--u   32.00g 652.00m [oraclelv01_mimage_1]     
  /dev/xvdj1                 oraclevg01 lvm2 a--u   32.00g 652.00m [oraclelv01_mlog_mimage_1]
  /dev/xvdj1                 oraclevg01 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdj2                 oraclevg02 lvm2 a--u   32.00g 652.00m [oraclelv02_mimage_1]     
  /dev/xvdj2                 oraclevg02 lvm2 a--u   32.00g 652.00m [oraclelv02_mlog_mimage_1]
  /dev/xvdj2                 oraclevg02 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdj3                 oraclevg04 lvm2 a--u   32.00g 652.00m [oraclelv04_mimage_1]     
  /dev/xvdj3                 oraclevg04 lvm2 a--u   32.00g 652.00m [oraclelv04_mlog_mimage_1]
  /dev/xvdj3                 oraclevg04 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdj5                 oraclevg03 lvm2 a--u   32.00g 652.00m [oraclelv03_mimage_1]     
  /dev/xvdj5                 oraclevg03 lvm2 a--u   32.00g 652.00m [oraclelv03_mlog_mimage_1]
  /dev/xvdj5                 oraclevg03 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdj6                 oraclevg05 lvm2 a--u   34.97g 716.00m [oraclelv05_mimage_1]     
  /dev/xvdj6                 oraclevg05 lvm2 a--u   34.97g 716.00m [oraclelv05_mlog_mimage_1]
  /dev/xvdj6                 oraclevg05 lvm2 a--u   34.97g 716.00m                           
  /dev/xvdk1                 oraclevg01 lvm2 a--u   32.00g 652.00m [oraclelv01_mimage_0]     
  /dev/xvdk1                 oraclevg01 lvm2 a--u   32.00g 652.00m [oraclelv01_mlog_mimage_0]
  /dev/xvdk1                 oraclevg01 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdk2                 oraclevg02 lvm2 a--u   32.00g 652.00m [oraclelv02_mimage_0]     
  /dev/xvdk2                 oraclevg02 lvm2 a--u   32.00g 652.00m [oraclelv02_mlog_mimage_0]
  /dev/xvdk2                 oraclevg02 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdk3                 oraclevg04 lvm2 a--u   32.00g 652.00m [oraclelv04_mimage_0]     
  /dev/xvdk3                 oraclevg04 lvm2 a--u   32.00g 652.00m [oraclelv04_mlog_mimage_0]
  /dev/xvdk3                 oraclevg04 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdk5                 oraclevg03 lvm2 a--u   32.00g 652.00m [oraclelv03_mimage_0]     
  /dev/xvdk5                 oraclevg03 lvm2 a--u   32.00g 652.00m [oraclelv03_mlog_mimage_0]
  /dev/xvdk5                 oraclevg03 lvm2 a--u   32.00g 652.00m                           
  /dev/xvdk6                 oraclevg05 lvm2 a--u   34.97g 716.00m [oraclelv05_mimage_0]     
  /dev/xvdk6                 oraclevg05 lvm2 a--u   34.97g 716.00m [oraclelv05_mlog_mimage_0]
  /dev/xvdk6                 oraclevg05 lvm2 a--u   34.97g 716.00m                           
  /dev/xvdn1                 oraclevg07 lvm2 a--u 1023.99g      0  oraclelv07                
  /dev/xvdu                                  ----       0       0                            
  /dev/xvdv                                  ----       0       0                            
  /dev/xvdw                                  ----       0       0                            
  /dev/xvdx1                 swapvg01   lvm2 a--u   15.99g      0  swaplv01             

Now to get minor and major numbers. Not this is a RHEL Linux system on AWS so I am only looking at /dev/sd and /dev/xv EBS volume devices plus the device mapper entries.

$ ls -lL /dev/mapper /dev/sd* /dev/xv*
ls: cannot access /dev/sd*: No such file or directory
brw-rw----. 1 root   disk 202,     0 Mar  7 02:56 /dev/xvda
brw-rw----. 1 root   disk 202,     1 Aug  4  2018 /dev/xvda1
brw-rw----. 1 oracle dba  202,    32 Apr  5 16:08 /dev/xvdc
brw-rw----. 1 oracle dba  202, 19968 Apr  5 16:08 /dev/xvdca
brw-rw----. 1 oracle dba  202, 20224 Apr  5 16:08 /dev/xvdcb
brw-rw----. 1 oracle dba  202, 20480 Apr  5 16:08 /dev/xvdcc
brw-rw----. 1 oracle dba  202, 20736 Apr  5 16:08 /dev/xvdcd
brw-rw----. 1 oracle dba  202, 20992 Apr  5 16:08 /dev/xvdce
brw-rw----. 1 oracle dba  202, 21248 Apr  5 16:08 /dev/xvdcf
brw-rw----. 1 oracle dba  202, 21504 Apr  5 16:08 /dev/xvdcg
brw-rw----. 1 oracle dba  202, 21760 Apr  5 16:08 /dev/xvdch
brw-rw----. 1 oracle dba  202,    48 Apr  5 16:08 /dev/xvdd
brw-rw----. 1 oracle dba  202, 26624 Apr  5 16:08 /dev/xvdda
brw-rw----. 1 root   disk 202, 26880 Mar  7 02:56 /dev/xvddb
brw-rw----. 1 root   disk 202, 26881 Aug  4  2018 /dev/xvddb1
brw-rw----. 1 root   disk 202, 27136 Mar  7 04:53 /dev/xvddc
brw-rw----. 1 root   disk 202, 27137 Mar  7 04:56 /dev/xvddc1
brw-rw----. 1 oracle dba  202,   112 Apr  5 16:08 /dev/xvdh
brw-rw----. 1 oracle dba  202,   128 Apr  5 16:08 /dev/xvdi
brw-rw----. 1 root   disk 202,   144 Mar  7 02:56 /dev/xvdj
brw-rw----. 1 root   disk 202,   145 Aug  4  2018 /dev/xvdj1
brw-rw----. 1 root   disk 202,   146 Aug  4  2018 /dev/xvdj2
brw-rw----. 1 root   disk 202,   147 Aug  4  2018 /dev/xvdj3
brw-rw----. 1 root   disk 202,   148 Aug  4  2018 /dev/xvdj4
brw-rw----. 1 root   disk 202,   149 Aug  4  2018 /dev/xvdj5
brw-rw----. 1 root   disk 202,   150 Aug  4  2018 /dev/xvdj6
brw-rw----. 1 root   disk 202,   160 Mar  7 02:56 /dev/xvdk
brw-rw----. 1 root   disk 202,   161 Aug  4  2018 /dev/xvdk1
brw-rw----. 1 root   disk 202,   162 Aug  4  2018 /dev/xvdk2
brw-rw----. 1 root   disk 202,   163 Aug  4  2018 /dev/xvdk3
brw-rw----. 1 root   disk 202,   164 Aug  4  2018 /dev/xvdk4
brw-rw----. 1 root   disk 202,   165 Aug  4  2018 /dev/xvdk5
brw-rw----. 1 root   disk 202,   166 Aug  4  2018 /dev/xvdk6
brw-rw----. 1 root   disk 202,   208 Aug  4  2018 /dev/xvdn
brw-rw----. 1 root   disk 202,   209 Aug  4  2018 /dev/xvdn1
brw-rw----. 1 root   disk 202,  5120 Mar  7 02:56 /dev/xvdu
brw-rw----. 1 oracle dba  202,  5376 Apr  5 16:08 /dev/xvdv
brw-rw----. 1 oracle dba  202,  5632 Apr  5 16:08 /dev/xvdw
brw-rw----. 1 root   disk 202,  5888 Mar  7 02:56 /dev/xvdx
brw-rw----. 1 root   disk 202,  5889 Aug  4  2018 /dev/xvdx1

/dev/mapper:
total 0
crw-rw----. 1 root root  10, 58 Aug  4  2018 control
brw-rw----. 1 root disk 253, 30 Mar  7 02:56 oraclevg01-oraclelv01
brw-rw----. 1 root disk 253, 28 Aug  4  2018 oraclevg01-oraclelv01_mimage_0
brw-rw----. 1 root disk 253, 29 Aug  4  2018 oraclevg01-oraclelv01_mimage_1
brw-rw----. 1 root disk 253, 27 Aug  4  2018 oraclevg01-oraclelv01_mlog
brw-rw----. 1 root disk 253, 25 Aug  4  2018 oraclevg01-oraclelv01_mlog_mimage_0
brw-rw----. 1 root disk 253, 26 Aug  4  2018 oraclevg01-oraclelv01_mlog_mimage_1
brw-rw----. 1 root disk 253, 24 Mar  7 02:56 oraclevg02-oraclelv02
brw-rw----. 1 root disk 253, 22 Aug  4  2018 oraclevg02-oraclelv02_mimage_0
brw-rw----. 1 root disk 253, 23 Aug  4  2018 oraclevg02-oraclelv02_mimage_1
brw-rw----. 1 root disk 253, 21 Aug  4  2018 oraclevg02-oraclelv02_mlog
brw-rw----. 1 root disk 253, 19 Aug  4  2018 oraclevg02-oraclelv02_mlog_mimage_0
brw-rw----. 1 root disk 253, 20 Aug  4  2018 oraclevg02-oraclelv02_mlog_mimage_1
brw-rw----. 1 root disk 253, 12 Mar  7 02:56 oraclevg03-oraclelv03
brw-rw----. 1 root disk 253, 10 Aug  4  2018 oraclevg03-oraclelv03_mimage_0
brw-rw----. 1 root disk 253, 11 Aug  4  2018 oraclevg03-oraclelv03_mimage_1
brw-rw----. 1 root disk 253,  9 Aug  4  2018 oraclevg03-oraclelv03_mlog
brw-rw----. 1 root disk 253,  7 Aug  4  2018 oraclevg03-oraclelv03_mlog_mimage_0
brw-rw----. 1 root disk 253,  8 Aug  4  2018 oraclevg03-oraclelv03_mlog_mimage_1
brw-rw----. 1 root disk 253, 18 Mar  7 02:56 oraclevg04-oraclelv04
brw-rw----. 1 root disk 253, 16 Aug  4  2018 oraclevg04-oraclelv04_mimage_0
brw-rw----. 1 root disk 253, 17 Aug  4  2018 oraclevg04-oraclelv04_mimage_1
brw-rw----. 1 root disk 253, 15 Aug  4  2018 oraclevg04-oraclelv04_mlog
brw-rw----. 1 root disk 253, 13 Aug  4  2018 oraclevg04-oraclelv04_mlog_mimage_0
brw-rw----. 1 root disk 253, 14 Aug  4  2018 oraclevg04-oraclelv04_mlog_mimage_1
brw-rw----. 1 root disk 253,  6 Mar  7 02:56 oraclevg05-oraclelv05
brw-rw----. 1 root disk 253,  4 Aug  4  2018 oraclevg05-oraclelv05_mimage_0
brw-rw----. 1 root disk 253,  5 Aug  4  2018 oraclevg05-oraclelv05_mimage_1
brw-rw----. 1 root disk 253,  3 Aug  4  2018 oraclevg05-oraclelv05_mlog
brw-rw----. 1 root disk 253,  1 Aug  4  2018 oraclevg05-oraclelv05_mlog_mimage_0
brw-rw----. 1 root disk 253,  2 Aug  4  2018 oraclevg05-oraclelv05_mlog_mimage_1
brw-rw----. 1 root disk 253,  0 Mar  7 02:56 oraclevg07-oraclelv07
brw-rw----. 1 root disk 253, 31 Mar  7 02:56 swapvg01-swaplv01

So device dev202-112 is /dev/xvdh which turns out not to be under LVM control and is actually a raw volume used by Oracle.