一、环境说明:

此次演示是dellr620 物理服务器,系统为centos7.6 x86_64位
硬件相关的信息如下:

[root@mysql-redis scripts]# sh hw_msg1.sh 
+--------------------------------------------------------------+
| This Machine's Hyper-Threading is Enabled(recommend disable) |
+--------------------------------------------------------------+
Systembit    :    64
MEM info     :    20*4096 MB
Disk_totle   :    
Pro_SN_name  :    Product Name: PowerEdge R620 Serial Number: 5RV2962
System name  :    \S
Board_SN name:    Product Name: Serial Number:
CPU model    :    32  Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
CPU_phy_num  :    2
CPU_core_num :    8
CPU_logic_num:    16
Netcard info :    Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
+++++++++++++++++++++++++++++++++++
<<raid informations>>
 Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 893.75 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
--
Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 223.0 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
+++++++++++++++++++++++++++++++++++
<<Virtual Drive Information>>
Device Id: 3
PD Type: SATA
Raw Size: 894.252 GB [0x6fc81ab0 Sectors]
Inquiry Data: 56AB609UKCDU        TOSHIBA Q300.                           SAFM12.3
Device Speed: 6.0Gb/s 
Drive Temperature : N/A
Device Id: 0
PD Type: SATA
Raw Size: 223.570 GB [0x1bf244b0 Sectors]
Inquiry Data: PHDV723500E8240AGN  INTEL SSDSC2BB240G7                     N2010112
Device Speed: 6.0Gb/s 
Drive Temperature :25C (77.00 F)

二、安装fio

1.redhat linux系统直接yum 安装
yum install fio.x86_64
centos7.6系统上直接yum 安装的话,默认的版本是fio.3.7

2.源码编译安装
为了测试异步io,需要安装libaio

yum install libaio-devel

wget  -P /data/soft http://github.com/axboe/fio/archive/fio-3.7.tar.gz
tar xf /data/soft/fio.3.7.tar.gz -C /usr/local/
cd /usr/local/fio.3.7
./configure
make && make install

三、重要参数介绍

Fio各个版本的参数不尽相同,这里只列出日常较为广发使用的重要参数
filename=/dev/sdb1 或者/data/soft/test1 : 指定测试硬盘名称或者测试文件名称(切忌选择系统设备测试,会导致系统奔溃的)
direct=1: 测试过程绕过机器自带的缓冲区,使得测试的结果更准确。
rw=randwrite: 随机写。 rw可取值:randwrite/randread/read/write/randrw/trim/randtrim/trimwrite ;分别是随机写,随机读,顺序读,顺序写,混合随机读写,
bs=16k : 单次I/O块大小为16KB,单次I/O大小对IOPS的影响比较大,一般来说要想得到最大的IOPS,bs越小越好。当然,这也和文件系统最小快的大小有关。一般linux下的文件分区最小快的大小为512B,1KB,2KB,4KB,8KB等可以根据文件系统类型进行选择。
size=5g : 测试文件大小为5GB
ioengine=libaio: I/O引擎使用libaio,libaio是异步的I/O引擎,一次提交一批I/O,然后等待这批I/O完成,这种方式减少了交互的次数,效率更好。FIO 至少支持13种不通的I/O引擎。sync,mmap,libaio,network等
iodepth=4 : I/O队列深度,主要是根据设备的并行度来调整。通常有两种I/O访问方式。也就是同步I/O,和异步I/O.同步I/O一次只能发送一个I/o请求,等待内核完成才可以返回。这对于单个线程来说iodepth总是小于1.想要iodepth大于1,可以通过多线程并发执行做到。异步I/O一次提交多个I/O请求。等I/O完成或者间隔一段时间收割一次。iodepth一般用在异步I/O模型(libaio),用于指定一次发起多少个I/O请求。Flash存储一般有多个并行的单元。支持多个I/O并发执行。较合适异步的I/O
-numjobs=4: 使用4个线程对硬盘设备进行压测 对于Raid设备,如raid1,raid5或者并行高的设备,可以考虑适当加大测试线程数-numjobs来测试。

runtime=1000: 测试时间为1000s.和参数size共同起作用,属于双限制,即达到任何一个限制都停止测试。

rwmixread=70: 在混合读写模式下,读占70%
group_reporting: 设置显示结果,指定group_reporting 将汇总每个进程的信息。
-name iops_randwrite: 改测试命名为 iops_randwrite
--output TestResult.log: 测试结果指定输出到文件TestResult.log 中

四、测试io性能

可以使用fio模拟测试各种不同类型的io请求,入随机写,顺序写,随机读,顺序读。混合随机读写

4.1随机写:

 time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite   --output TestResult.log

命令介绍:
采用4个线程对 /data/soft/test1文件做持续时间为60s,队列深度为1,块大小为4kb的direct y异步随机写(libaio)测试,该测试命名为iops_randwrite.输出结果不安4个jobs展示,而是按照group 组汇总 ,这样得出的在此压力下从此文件系统的随机写的IOPS。对于Raid设备,如raid1,raid5或者并行高的设备,可以考虑适当加大测试线程数-numjobs来测试。
--output TestResult.log 输出到指定的日志

下面是随机写具体执行过程:


 [root@mysql-redis soft]# time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite 
iops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 4 processes
iops_randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=68.2MiB/s][r=0,w=17.5k IOPS][eta 00m:00s]
iops_randwrite: (groupid=0, jobs=4): err= 0: pid=13505: Thu May 21 18:50:33 2020
  write: IOPS=39.2k, BW=153MiB/s (160MB/s)(9182MiB/60001msec)
    slat (usec): min=6, max=27813, avg=17.42, stdev=36.21
    clat (nsec): min=1177, max=123217k, avg=82183.17, stdev=323538.76
     lat (usec): min=53, max=123230, avg=99.84, stdev=325.69
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   51], 10.00th=[   57], 20.00th=[   62],
     | 30.00th=[   65], 40.00th=[   70], 50.00th=[   75], 60.00th=[   79],
     | 70.00th=[   84], 80.00th=[   91], 90.00th=[  103], 95.00th=[  117],
     | 99.00th=[  161], 99.50th=[  206], 99.90th=[  330], 99.95th=[  619],
     | 99.99th=[14615]
   bw (  KiB/s): min=17536, max=45304, per=25.12%, avg=39354.01, stdev=5145.44, samples=476
   iops        : min= 4384, max=11326, avg=9838.50, stdev=1286.36, samples=476
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.22%
  lat (usec)   : 100=84.07%, 250=11.50%, 500=0.12%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=4.33%, sys=19.34%, ctx=2824889, majf=0, minf=1185
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2350467,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9182MiB (9628MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=0/2349456, merge=0/0, ticks=0/167181, in_queue=166516, util=96.46%

real    1m0.706s
user    0m11.167s
sys 0m47.017s

4.2随机读:

time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 

关于命令参数说明:
命令介绍:采用2个线程对 /data/soft/test1文件做持续时间为60s,队列深度为1,块大小为4kb的direct y异步随机读(libaio)测试,该测试命名为iops_randread.输出结果不安2个jobs展示,而是按照group 组汇总 ,这样得出的在此压力下从此文件系统的随机读的IOPS

下面是随机读具体执行过程:

 [root@mysql-redis39 ~]# time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread 
iops_randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [r(2)][100.0%][r=36.7MiB/s,w=0KiB/s][r=9389,w=0 IOPS][eta 00m:00s]
iops_randread: (groupid=0, jobs=2): err= 0: pid=24566: Fri May 22 06:21:46 2020
   read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec)
    slat (usec): min=7, max=1480, avg=23.09, stdev= 8.09
    clat (usec): min=8, max=123984, avg=185.97, stdev=570.65
     lat (usec): min=136, max=123999, avg=209.50, stdev=570.70
    clat percentiles (usec):
     |  1.00th=[  139],  5.00th=[  141], 10.00th=[  145], 20.00th=[  169],
     | 30.00th=[  174], 40.00th=[  178], 50.00th=[  182], 60.00th=[  188],
     | 70.00th=[  198], 80.00th=[  204], 90.00th=[  212], 95.00th=[  219],
     | 99.00th=[  255], 99.50th=[  273], 99.90th=[  322], 99.95th=[  537],
     | 99.99th=[ 1549]
   bw (  KiB/s): min=13824, max=19616, per=49.99%, avg=18713.87, stdev=1060.66, samples=238
   iops        : min= 3456, max= 4904, avg=4678.43, stdev=265.16, samples=238
  lat (usec)   : 10=0.01%, 100=0.01%, 250=98.87%, 500=1.08%, 750=0.02%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.01%, 20=0.01%, 250=0.01%
  cpu          : usr=4.11%, sys=15.44%, ctx=561546, majf=0, minf=498
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=561503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=2193MiB (2300MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=560393/2, merge=0/0, ticks=102314/0, in_queue=101910, util=97.80%

real    1m0.706s
user    0m5.649s
sys 0m19.196s

以上需要关注的指标:
read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec) io=2193MiB (2300MB) 这些参数可以了解到iops和通途量
lat (usec): min=136, max=123999, avg=209.50, stdev=570.70 这些参数也要注意:io延迟平均在209.5微妙。

4.3顺序写:

time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write

说明:采用4个线程对/data/soft/test3文件做持续60s,队列深度为1的块大小为1M的direct异步顺序写(libaio)压力测试该测试命名为bw_write.输出结果不安4个job分别展示,而是按照group汇总展示。

[root@mysql-redis ~]# time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write  
bw_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 5 processes
bw_write: Laying out IO file (1 file / 5120MiB)
Jobs: 5 (f=5): [W(5)][100.0%][r=0KiB/s,w=184MiB/s][r=0,w=184 IOPS][eta 00m:00s]
bw_write: (groupid=0, jobs=5): err= 0: pid=26278: Fri May 22 06:44:38 2020
  write: IOPS=319, BW=320MiB/s (335MB/s)(18.7GiB/60009msec)
    slat (usec): min=56, max=303, avg=140.02, stdev=29.86
    clat (msec): min=3, max=1092, avg=15.49, stdev=67.68
     lat (msec): min=3, max=1093, avg=15.63, stdev=67.68

4.4顺序读:

time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read 
[root@mysql-redis ~]# time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read 
bw_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [R(2)][100.0%][r=259MiB/s,w=0KiB/s][r=259,w=0 IOPS][eta 00m:00s]
bw_read: (groupid=0, jobs=2): err= 0: pid=26789: Fri May 22 06:51:01 2020
   read: IOPS=255, BW=256MiB/s (268MB/s)(10.0GiB/40034msec)
    slat (usec): min=95, max=1149, avg=245.64, stdev=42.23
    clat (msec): min=4, max=130, avg= 7.57, stdev= 2.98
     lat (msec): min=4, max=130, avg= 7.81, stdev= 2.98

4.5混合读写

time fio  -rw=randrw  -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

命令介绍:
2个线程对/data/soft/test2 文件做持续时间为60s,队列深度为1,块大小为16kb的direct异步混合读写的(libaio)压力测试,读写比伪7:3,测试被命名为iops_randrw

rwmixwrite=30 意思是在混合读写的模式下,写占30%。
-rwmixread=70 意思是在混合读写的模式下,读占70%。

混合读写具体输出结果:


[root@mysql-redis ~]# time fio  -rw=randrw  -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2  -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

iops_randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
...
fio-3.7
Starting 2 processes
Jobs: 2 (f=2): [m(2)][100.0%][r=62.0MiB/s,w=27.2MiB/s][r=4030,w=1740 IOPS][eta 00m:00s]
iops_randrw: (groupid=0, jobs=2): err= 0: pid=27095: Fri May 22 06:55:08 2020
   read: IOPS=3931, BW=61.4MiB/s (64.4MB/s)(3686MiB/60001msec)
    slat (usec): min=8, max=6353, avg=36.20, stdev=19.77
    clat (nsec): min=1648, max=123466k, avg=346175.56, stdev=725094.33
     lat (usec): min=12, max=123516, avg=382.99, stdev=725.55
    clat percentiles (usec):
     |  1.00th=[  215],  5.00th=[  241], 10.00th=[  255], 20.00th=[  277],
     | 30.00th=[  293], 40.00th=[  310], 50.00th=[  322], 60.00th=[  334],
     | 70.00th=[  351], 80.00th=[  375], 90.00th=[  412], 95.00th=[  449],
     | 99.00th=[ 1221], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1827],
     | 99.99th=[16712]
   bw (  KiB/s): min=23328, max=33792, per=49.97%, avg=31431.39, stdev=1649.11, samples=238
   iops        : min= 1458, max= 2112, avg=1964.44, stdev=103.06, samples=238
  write: IOPS=1686, BW=26.4MiB/s (27.6MB/s)(1581MiB/60001msec)
    slat (usec): min=12, max=2005, avg=42.06, stdev=16.09
    clat (usec): min=3, max=123270, avg=225.73, stdev=432.84
     lat (usec): min=86, max=123320, avg=268.44, stdev=433.42
    clat percentiles (usec):
     |  1.00th=[   78],  5.00th=[   85], 10.00th=[  100], 20.00th=[  129],
     | 30.00th=[  149], 40.00th=[  174], 50.00th=[  202], 60.00th=[  233],
     | 70.00th=[  265], 80.00th=[  306], 90.00th=[  359], 95.00th=[  408],
     | 99.00th=[  510], 99.50th=[ 1090], 99.90th=[ 1549], 99.95th=[ 1614],
     | 99.99th=[ 5080]
   bw (  KiB/s): min= 9152, max=15329, per=49.97%, avg=13484.37, stdev=892.91, samples=238
   iops        : min=  572, max=  958, avg=842.76, stdev=55.80, samples=238
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=3.06%
  lat (usec)   : 250=22.22%, 500=72.91%, 750=0.76%, 1000=0.05%
  lat (msec)   : 2=0.96%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=4.64%, sys=15.91%, ctx=346233, majf=0, minf=266
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=235884,101198,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=61.4MiB/s (64.4MB/s), 61.4MiB/s-61.4MiB/s (64.4MB/s-64.4MB/s), io=3686MiB (3865MB), run=60001-60001msec
  WRITE: bw=26.4MiB/s (27.6MB/s), 26.4MiB/s-26.4MiB/s (27.6MB/s-27.6MB/s), io=1581MiB (1658MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=274477/101294, merge=0/0, ticks=88860/19119, in_queue=107576, util=95.82%

real    1m0.690s
user    0m6.300s
sys 0m19.718s

4.6SSD性能测试配置文件

配置文件内容如下:

[root@mysql-redis scripts]# cat /data/soft/ssd-test.fio 
##2020-05-24
[global]
bs=4k
ioengine=libaio
iodepth=4
numjobs=1
group_reporting
size=5g
direct=1
runtime=60
directory=/data/soft
filename=ssd.test.file
#inflate-log=/data/soft/TestResult.log
#output=/data/soft/TestResult.log
[seq-read]
name=iops_seqread
rw=read
stonewall

[rand-read]
name=iops_randread
rw=randread
stonewall

[seq-write]
name=iops_seqwrite
rw=write
stonewall

[rand-write]
name=iops_randwrite
rw=randwrite
stonewall

[randrw]
name=iops_randrw
rwmixread=70
rw=randrw
stonewall

下面是fio测试指定配置文件输出的内容:

[root@mysql-redis soft]# fio ssd-test.fio 
iops_seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randread: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_seqwrite: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randwrite: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
iops_randrw: (g=4): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.7
Starting 5 processes
iops_seqread: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [_(4),m(1)][90.5%][r=47.8MiB/s,w=20.7MiB/s][r=12.2k,w=5308 IOPS][eta 00m:21s]     
iops_seqread: (groupid=0, jobs=1): err= 0: pid=2142: Sun May 24 20:06:20 2020
   read: IOPS=57.5k, BW=225MiB/s (236MB/s)(5120MiB/22782msec)
    slat (usec): min=3, max=1943, avg= 5.13, stdev= 2.21
    clat (usec): min=25, max=12467, avg=63.13, stdev=202.65
     lat (usec): min=40, max=12472, avg=68.41, stdev=202.72
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   53], 10.00th=[   55], 20.00th=[   56],
     | 30.00th=[   56], 40.00th=[   56], 50.00th=[   57], 60.00th=[   57],
     | 70.00th=[   57], 80.00th=[   58], 90.00th=[   59], 95.00th=[   62],
     | 99.00th=[   69], 99.50th=[   79], 99.90th=[ 2073], 99.95th=[ 5080],
     | 99.99th=[10814]
   bw (  KiB/s): min=43579, max=193812, per=75.80%, avg=174446.82, stdev=47419.87, samples=45
   iops        : min=10894, max=48453, avg=43611.36, stdev=11855.06, samples=45
  lat (usec)   : 50=3.39%, 100=96.25%, 250=0.08%, 500=0.07%, 750=0.04%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.06%, 4=0.04%, 10=0.05%, 20=0.01%
  cpu          : usr=19.83%, sys=39.29%, ctx=1222141, majf=0, minf=114
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randread: (groupid=1, jobs=1): err= 0: pid=2170: Sun May 24 20:06:20 2020
   read: IOPS=21.7k, BW=84.9MiB/s (89.0MB/s)(5094MiB/60001msec)
    slat (usec): min=3, max=1068, avg=12.62, stdev= 6.14
    clat (usec): min=42, max=38568, avg=167.76, stdev=315.76
     lat (usec): min=62, max=38588, avg=180.75, stdev=316.07
    clat percentiles (usec):
     |  1.00th=[  130],  5.00th=[  135], 10.00th=[  137], 20.00th=[  141],
     | 30.00th=[  143], 40.00th=[  147], 50.00th=[  151], 60.00th=[  155],
     | 70.00th=[  161], 80.00th=[  172], 90.00th=[  192], 95.00th=[  206],
     | 99.00th=[  251], 99.50th=[  273], 99.90th=[ 3982], 99.95th=[ 7308],
     | 99.99th=[14091]
   bw (  KiB/s): min= 5397, max=74196, per=79.00%, avg=68682.74, stdev=15713.88, samples=119
   iops        : min= 1349, max=18549, avg=17170.32, stdev=3928.48, samples=119
  lat (usec)   : 50=0.01%, 100=0.02%, 250=98.94%, 500=0.82%, 750=0.02%
  lat (usec)   : 1000=0.02%
  lat (msec)   : 2=0.03%, 4=0.04%, 10=0.07%, 20=0.02%, 50=0.01%
  cpu          : usr=11.52%, sys=37.46%, ctx=668232, majf=0, minf=319
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1304064,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_seqwrite: (groupid=2, jobs=1): err= 0: pid=2231: Sun May 24 20:06:20 2020
  write: IOPS=47.1k, BW=184MiB/s (193MB/s)(5120MiB/27848msec)
    slat (usec): min=4, max=139, avg= 7.50, stdev= 2.96
    clat (usec): min=27, max=15374, avg=76.14, stdev=278.53
     lat (usec): min=49, max=15381, avg=83.78, stdev=279.26
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   59], 10.00th=[   60], 20.00th=[   61],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   71], 90.00th=[   73], 95.00th=[   74],
     | 99.00th=[   81], 99.50th=[   89], 99.90th=[ 3326], 99.95th=[ 3392],
     | 99.99th=[15139]
   bw (  KiB/s): min= 3769, max=175262, per=80.83%, avg=152166.87, stdev=43009.10, samples=55
   iops        : min=  942, max=43815, avg=38041.35, stdev=10752.22, samples=55
  lat (usec)   : 50=0.24%, 100=99.43%, 250=0.07%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.21%, 10=0.01%, 20=0.03%
  cpu          : usr=11.79%, sys=43.48%, ctx=652945, majf=0, minf=95
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randwrite: (groupid=3, jobs=1): err= 0: pid=2262: Sun May 24 20:06:20 2020
  write: IOPS=47.2k, BW=185MiB/s (194MB/s)(5120MiB/27743msec)
    slat (usec): min=4, max=415, avg= 7.49, stdev= 2.82
    clat (usec): min=26, max=21032, avg=75.63, stdev=276.91
     lat (usec): min=48, max=21058, avg=83.27, stdev=277.60
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   60], 10.00th=[   60], 20.00th=[   62],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   65], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   70], 90.00th=[   73], 95.00th=[   74],
     | 99.00th=[   80], 99.50th=[   86], 99.90th=[ 3326], 99.95th=[ 3392],
     | 99.99th=[15139]
   bw (  KiB/s): min= 3175, max=142274, per=68.21%, avg=128901.20, stdev=33671.41, samples=55
   iops        : min=  793, max=35568, avg=32224.95, stdev=8417.81, samples=55
  lat (usec)   : 50=0.21%, 100=99.50%, 250=0.04%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.20%, 10=0.01%, 20=0.02%, 50=0.01%
  cpu          : usr=12.97%, sys=43.05%, ctx=654916, majf=0, minf=258
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
iops_randrw: (groupid=4, jobs=1): err= 0: pid=2292: Sun May 24 20:06:20 2020
   read: IOPS=11.3k, BW=44.3MiB/s (46.5MB/s)(2659MiB/60001msec)
    slat (usec): min=4, max=594, avg=13.08, stdev= 6.88
    clat (usec): min=61, max=45073, avg=244.45, stdev=441.39
     lat (usec): min=134, max=45115, avg=257.84, stdev=441.61
    clat percentiles (usec):
     |  1.00th=[  137],  5.00th=[  143], 10.00th=[  151], 20.00th=[  176],
     | 30.00th=[  190], 40.00th=[  202], 50.00th=[  212], 60.00th=[  221],
     | 70.00th=[  233], 80.00th=[  258], 90.00th=[  330], 95.00th=[  359],
     | 99.00th=[  474], 99.50th=[ 1270], 99.90th=[ 6521], 99.95th=[10028],
     | 99.99th=[17695]
   bw (  KiB/s): min= 3072, max=51824, per=99.92%, avg=45336.55, stdev=11250.66, samples=119
   iops        : min=  768, max=12958, avg=11334.12, stdev=2812.66, samples=119
  write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(1139MiB/60001msec)
    slat (usec): min=6, max=874, avg=16.94, stdev= 7.42
    clat (usec): min=40, max=31265, avg=192.80, stdev=413.19
     lat (usec): min=56, max=31306, avg=210.09, stdev=413.32
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   58], 10.00th=[   62], 20.00th=[   78],
     | 30.00th=[  145], 40.00th=[  169], 50.00th=[  188], 60.00th=[  202],
     | 70.00th=[  212], 80.00th=[  227], 90.00th=[  251], 95.00th=[  277],
     | 99.00th=[  619], 99.50th=[ 1205], 99.90th=[ 5997], 99.95th=[ 9634],
     | 99.99th=[16581]
   bw (  KiB/s): min= 1184, max=22176, per=99.94%, avg=19428.64, stdev=4847.41, samples=119
   iops        : min=  296, max= 5544, avg=4857.13, stdev=1211.85, samples=119
  lat (usec)   : 50=0.12%, 100=7.22%, 250=74.36%, 500=17.22%, 750=0.24%
  lat (usec)   : 1000=0.10%
  lat (msec)   : 2=0.52%, 4=0.06%, 10=0.11%, 20=0.04%, 50=0.01%
  cpu          : usr=7.88%, sys=29.46%, ctx=340101, majf=0, minf=374
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=680630,291634,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=5120MiB (5369MB), run=22782-22782msec

Run status group 1 (all jobs):
   READ: bw=84.9MiB/s (89.0MB/s), 84.9MiB/s-84.9MiB/s (89.0MB/s-89.0MB/s), io=5094MiB (5341MB), run=60001-60001msec

Run status group 2 (all jobs):
  WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5120MiB (5369MB), run=27848-27848msec

Run status group 3 (all jobs):
  WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5120MiB (5369MB), run=27743-27743msec

Run status group 4 (all jobs):
   READ: bw=44.3MiB/s (46.5MB/s), 44.3MiB/s-44.3MiB/s (46.5MB/s-46.5MB/s), io=2659MiB (2788MB), run=60001-60001msec
  WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=1139MiB (1195MB), run=60001-60001msec

Disk stats (read/write):
  sda: ios=3294785/2912828, merge=0/3, ticks=430527/238028, in_queue=666582, util=99.09%
[root@mysql-redis soft]# 

五、总结

关于fio测试硬盘的建议:
一、建议使用顺序I/O和较大的blocksize 来测试设备的通吐量和延迟
二、建议使用随机I/O和较小的blocksize来测试设备的IOPS和延迟
三、在配置numjobs和iodepth 测试底层存储性能时,建议要深入了解应用到底采用的是同步的io还是异步的io(是多进程并发i/o请求还是一次提交一批的i/o的请求)
提示:对fio压测 磁盘性能感兴趣的话,可以参考下ezfio(直接在github.com上搜索ezfio)
重要提示:fio是 支持对裸设备直接进行I/O测试的,在测试裸设备时,一定要避开系统所在的裸设备,否则会导致系统奔溃。
例如:

[root@mysql-redis soft]# df -h / /boot 
文件系统        容量  已用  可用 已用% 挂载点
/dev/sdb5       207G  7.5G  200G    4% /
/dev/sdb1       297M  120M  177M   41% /boot

/dev/sdb /dev/sdb5 /dev/sdb1 这些设备是绝对禁止进行fio测试的,
同时在测试其他的设备裸设备时,一定要提前确定好此裸设备上是否有重要 的数据,一定要重要的设备提前做好备份。

以上就是我这边对fio测试硬盘的简单学习总结,仅供参考,如有网友直接复制应用造成损失,本博文不负担任何责任。
参考文档:
http://www.ssdfans.com/?p=6087
https://linux.die.net/man/1/fio
https://blog.csdn.net/don_chiang709/article/details/92628623