fio test

image.png

# sudo arc_summary -s arc

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Jul 18 14:10:53 2024
Linux 6.8.8-2-pve                                             2.2.4-pve1
Machine: pve (x86_64)                                         2.2.4-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    99.9 %    2.0 GiB
        Target size (adaptive):                       100.0 %    2.0 GiB
        Min size (hard limit):                         34.9 %  714.3 MiB
        Max size (high water):                            2:1    2.0 GiB
        Anonymous data size:                            0.0 %    0 Bytes
        Anonymous metadata size:                        0.0 %    0 Bytes
        MFU data target:                               37.5 %  765.4 MiB
        MFU data size:                                  2.5 %   51.1 MiB
        MFU ghost data size:                                     0 Bytes
        MFU metadata target:                           12.5 %  255.1 MiB
        MFU metadata size:                              0.4 %    7.6 MiB
        MFU ghost metadata size:                                 0 Bytes
        MRU data target:                               37.5 %  765.4 MiB
        MRU data size:                                 96.2 %    1.9 GiB
        MRU ghost data size:                                    38.2 MiB
        MRU metadata target:                           12.5 %  255.1 MiB
        MRU metadata size:                              0.9 %   17.8 MiB
        MRU ghost metadata size:                                 0 Bytes
        Uncached data size:                             0.0 %    0 Bytes
        Uncached metadata size:                         0.0 %    0 Bytes
        Bonus size:                                   < 0.1 %   31.6 KiB
        Dnode cache target:                            10.0 %  204.8 MiB
        Dnode cache size:                               0.1 %  252.7 KiB
        Dbuf size:                                    < 0.1 %  232.4 KiB
        Header size:                                    0.2 %    3.8 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                         < 0.1 %   22.5 KiB

ARC hash breakdown:
        Elements max:                                              41.1k
        Elements current:                              40.6 %      16.7k
        Collisions:                                                 3.5k
        Chain max:                                                     2
        Chains:                                                       41

ARC misc:
        Deleted:                                                    1.1M
        Mutex misses:                                                392
        Eviction skips:                                           520.1k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 128.7 GiB
        L2 eligible MFU evictions:                    < 0.1 %   10.4 MiB
        L2 eligible MRU evictions:                    100.0 %  128.7 GiB
        L2 ineligible evictions:                                 1.4 GiB

cd /hdd-st400/

下面是顺序写测试

fio --name=seqwrite --rw=write --direct=1 --ioengine=psync --bs=128k --iodepth=32 --numjobs=10 --size=10G --group_reporting

4M

# fio --name=seqwrite --rw=write --direct=1 --ioengine=psync --bs=4m --iodepth=32 --numjobs=5 --size=10G --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=32
...
fio-3.33
Starting 5 processes
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 2 (f=2): [W(1),_(2),W(1),_(1)][97.4%][eta 00m:08s]
seqwrite: (groupid=0, jobs=5): err= 0: pid=23831: Thu Jul 18 13:58:52 2024
  write: IOPS=42, BW=170MiB/s (178MB/s)(50.0GiB/301831msec); 0 zone resets
    clat (usec): min=450, max=3485.8k, avg=116870.67, stdev=577125.36
     lat (usec): min=495, max=3485.8k, avg=117138.97, stdev=577112.00
    clat percentiles (usec):
     |  1.00th=[    652],  5.00th=[    947], 10.00th=[   1106],
     | 20.00th=[   1221], 30.00th=[   1303], 40.00th=[   1385],
     | 50.00th=[   1483], 60.00th=[   1680], 70.00th=[   2147],
     | 80.00th=[   3818], 90.00th=[   6259], 95.00th=[  10683],
     | 99.00th=[3204449], 99.50th=[3271558], 99.90th=[3405775],
     | 99.95th=[3405775], 99.99th=[3472884]
   bw (  KiB/s): min=49103, max=2549616, per=100.00%, avg=926222.34, stdev=83890.32, samples=559
   iops        : min=   11, max=  622, avg=225.79, stdev=20.51, samples=559
  lat (usec)   : 500=0.13%, 750=1.84%, 1000=4.27%
  lat (msec)   : 2=61.92%, 4=12.63%, 10=13.78%, 20=1.38%, 50=0.19%
  lat (msec)   : 100=0.05%, >=2000=3.80%
  cpu          : usr=0.18%, sys=1.11%, ctx=34749, majf=0, minf=51
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,12800,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=50.0GiB (53.7GB), run=301831-301831msec

16K

# fio --name=seqwrite --rw=write --direct=1 --ioengine=psync --bs=16k --iodepth=32 --numjobs=5 --size=1G --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=32
...
fio-3.33
Starting 5 processes
seqwrite: Laying out IO file (1 file / 1024MiB)
seqwrite: Laying out IO file (1 file / 1024MiB)
seqwrite: Laying out IO file (1 file / 1024MiB)
seqwrite: Laying out IO file (1 file / 1024MiB)
seqwrite: Laying out IO file (1 file / 1024MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 3 (f=3): [W(3),_(1),E(1)][88.9%][w=751MiB/s][w=48.1k IOPS][eta 00m:01s]
seqwrite: (groupid=0, jobs=5): err= 0: pid=23380: Thu Jul 18 13:50:50 2024
  write: IOPS=47.0k, BW=735MiB/s (770MB/s)(5120MiB/6969msec); 0 zone resets
    clat (usec): min=2, max=965387, avg=102.46, stdev=8861.43
     lat (usec): min=2, max=965387, avg=102.54, stdev=8861.44
    clat percentiles (usec):
     |  1.00th=[     3],  5.00th=[     3], 10.00th=[     3], 20.00th=[     3],
     | 30.00th=[     3], 40.00th=[     4], 50.00th=[     4], 60.00th=[     4],
     | 70.00th=[     4], 80.00th=[     5], 90.00th=[    14], 95.00th=[    24],
     | 99.00th=[    35], 99.50th=[   123], 99.90th=[  2040], 99.95th=[  3556],
     | 99.99th=[708838]
   bw (  KiB/s): min=17120, max=2345824, per=100.00%, avg=868363.35, stdev=129742.76, samples=54
   iops        : min= 1070, max=146614, avg=54272.71, stdev=8108.92, samples=54
  lat (usec)   : 4=75.13%, 10=12.87%, 20=4.99%, 50=6.30%, 100=0.07%
  lat (usec)   : 250=0.25%, 500=0.08%, 750=0.04%, 1000=0.03%
  lat (msec)   : 2=0.13%, 4=0.07%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (msec)   : 750=0.01%, 1000=0.01%
  cpu          : usr=0.43%, sys=5.29%, ctx=15129, majf=0, minf=57
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,327680,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=735MiB/s (770MB/s), 735MiB/s-735MiB/s (770MB/s-770MB/s), io=5120MiB (5369MB), run=6969-6969msec

下面是顺序读测试

fio --name=seqread --rw=read --direct=1 --ioengine=psync --bs=128k --iodepth=32 --numjobs=10 --size=10G --group_reporting

16K

# fio --name=seqread --rw=read --direct=1 --ioengine=psync --bs=16k --iodepth=32 --numjobs=2 --size=10G --group_reporting
seqread: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=32
...
fio-3.33
Starting 2 processes
seqread: Laying out IO file (1 file / 10240MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 2 (f=2): [R(2)][100.0%][r=190MiB/s][r=12.2k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=2): err= 0: pid=25102: Thu Jul 18 14:04:59 2024
  read: IOPS=11.3k, BW=177MiB/s (186MB/s)(20.0GiB/115683msec)
    clat (nsec): min=1123, max=690700k, avg=175961.90, stdev=6918531.78
     lat (nsec): min=1138, max=690700k, avg=175979.81, stdev=6918531.72
    clat percentiles (nsec):
     |  1.00th=[     1160],  5.00th=[     1176], 10.00th=[     1176],
     | 20.00th=[     1192], 30.00th=[     1208], 40.00th=[     1224],
     | 50.00th=[     1256], 60.00th=[     1288], 70.00th=[     1384],
     | 80.00th=[     1576], 90.00th=[    23680], 95.00th=[    26240],
     | 99.00th=[  4751360], 99.50th=[  5406720], 99.90th=[  5996544],
     | 99.95th=[  6127616], 99.99th=[541065216]
   bw (  KiB/s): min= 2560, max=401408, per=100.00%, avg=188417.45, stdev=58829.57, samples=444
   iops        : min=  160, max=25088, avg=11776.07, stdev=3676.85, samples=444
  lat (usec)   : 2=84.80%, 4=1.26%, 10=1.41%, 20=0.46%, 50=10.45%
  lat (usec)   : 100=0.05%, 250=0.02%, 500=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=1.50%, 20=0.02%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=0.14%, sys=2.46%, ctx=21676, majf=0, minf=32
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=20.0GiB (21.5GB), run=115683-115683msec

disk io

4M

# fio --name=seqread --rw=read --direct=1 --ioengine=psync --bs=4m --iodepth=32 --numjobs=2 --size=10G --group_reporting
seqread: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=32
...
fio-3.33
Starting 2 processes
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 2 (f=2): [R(2)][100.0%][r=188MiB/s][r=47 IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=2): err= 0: pid=25630: Thu Jul 18 14:08:55 2024
  read: IOPS=45, BW=182MiB/s (191MB/s)(20.0GiB/112281msec)
    clat (usec): min=1168, max=898933, avg=43839.87, stdev=105450.26
     lat (usec): min=1168, max=898934, avg=43840.13, stdev=105450.26
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   20], 10.00th=[   20], 20.00th=[   21],
     | 30.00th=[   21], 40.00th=[   21], 50.00th=[   21], 60.00th=[   22],
     | 70.00th=[   23], 80.00th=[   23], 90.00th=[   26], 95.00th=[   67],
     | 99.00th=[  575], 99.50th=[  584], 99.90th=[  609], 99.95th=[  659],
     | 99.99th=[  902]
   bw (  KiB/s): min=16384, max=409600, per=100.00%, avg=197422.59, stdev=55594.06, samples=424
   iops        : min=    4, max=  100, avg=48.20, stdev=13.57, samples=424
  lat (msec)   : 2=0.37%, 20=16.78%, 50=77.01%, 100=1.64%, 250=0.12%
  lat (msec)   : 500=0.41%, 750=3.65%, 1000=0.02%
  cpu          : usr=0.01%, sys=2.20%, ctx=20665, majf=0, minf=2068
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=20.0GiB (21.5GB), run=112281-112281msec

image.png

随机读

16k 不带的buffer
fio --name=randread --rw=randread --direct=1 --ioengine=psync --bs=16k --iodepth=1 --numjobs=1 --size=10G --runtime=60 --group_reporting

fio --name=randread --rw=randread --direct=1 --ioengine=psync --bs=16k --iodepth=1 --numjobs=1 --size=10G --runtime=60  --group_reporting

randread: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
randread: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=1793KiB/s][r=112 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=27798: Thu Jul 18 14:24:22 2024
  read: IOPS=114, BW=1833KiB/s (1876kB/s)(107MiB/60001msec)
    clat (usec): min=2, max=606991, avg=8728.82, stdev=12293.03
     lat (usec): min=2, max=606991, avg=8729.02, stdev=12293.03
    clat percentiles (usec):
     |  1.00th=[    15],  5.00th=[    20], 10.00th=[    21], 20.00th=[  3949],
     | 30.00th=[  5866], 40.00th=[  7308], 50.00th=[  8717], 60.00th=[ 10159],
     | 70.00th=[ 11600], 80.00th=[ 12911], 90.00th=[ 14353], 95.00th=[ 15139],
     | 99.00th=[ 29492], 99.50th=[ 41157], 99.90th=[103285], 99.95th=[149947],
     | 99.99th=[608175]
   bw (  KiB/s): min=   32, max= 2368, per=100.00%, avg=1834.49, stdev=475.55, samples=119
   iops        : min=    2, max=  148, avg=114.66, stdev=29.72, samples=119
  lat (usec)   : 4=0.13%, 10=0.41%, 20=9.39%, 50=7.35%, 750=0.23%
  lat (usec)   : 1000=0.92%
  lat (msec)   : 2=0.01%, 4=1.73%, 10=38.31%, 20=39.81%, 50=1.56%
  lat (msec)   : 100=0.04%, 250=0.07%, 750=0.03%
  cpu          : usr=0.06%, sys=0.47%, ctx=5685, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1833KiB/s (1876kB/s), 1833KiB/s-1833KiB/s (1876kB/s-1876kB/s), io=107MiB (113MB), run=60001-60001msec

16K 带的buffer

# fio --name=randread --rw=randread  --ioengine=psync --bs=16k --iodepth=1 --numjobs=1 --size=10G --runtime=60  --group_reporting
randread: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1905KiB/s][r=119 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=28177: Thu Jul 18 14:27:05 2024
  read: IOPS=239, BW=3826KiB/s (3917kB/s)(224MiB/60008msec)
    clat (nsec): min=1824, max=22827k, avg=4180855.14, stdev=5360433.23
     lat (nsec): min=1839, max=22827k, avg=4181042.51, stdev=5360445.07
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   14], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   16], 40.00th=[   17], 50.00th=[   20], 60.00th=[ 4621],
     | 70.00th=[ 7570], 80.00th=[10159], 90.00th=[12911], 95.00th=[14222],
     | 99.00th=[15795], 99.50th=[16319], 99.90th=[17171], 99.95th=[17695],
     | 99.99th=[21365]
   bw (  KiB/s): min= 1600, max=221504, per=100.00%, avg=3844.57, stdev=20122.64, samples=119
   iops        : min=  100, max=13844, avg=240.29, stdev=1257.67, samples=119
  lat (usec)   : 2=0.01%, 4=0.22%, 10=0.20%, 20=51.94%, 50=5.16%
  lat (usec)   : 750=0.17%, 1000=0.46%
  lat (msec)   : 4=0.84%, 10=20.18%, 20=20.79%, 50=0.01%
  cpu          : usr=0.09%, sys=0.83%, ctx=6095, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=14348,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3826KiB/s (3917kB/s), 3826KiB/s-3826KiB/s (3917kB/s-3917kB/s), io=224MiB (235MB), run=60008-60008msec

16K 2 job 带的buffer

# fio --name=randread --rw=randread  --ioengine=psync --bs=16k --iodepth=1 --numjobs=2 --size=10G --runtime=60  --group_reporting
randread: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
fio-3.33
Starting 2 processes
randread: Laying out IO file (1 file / 10240MiB)
Jobs: 2 (f=2): [r(2)][0.5%][r=1953KiB/s][r=122 IOPS][eta 03h:40m:33s]
randread: (groupid=0, jobs=2): err= 0: pid=28496: Thu Jul 18 14:29:31 2024
  read: IOPS=189, BW=3033KiB/s (3106kB/s)(178MiB/60021msec)
    clat (usec): min=2, max=666725, avg=10546.56, stdev=17530.18
     lat (usec): min=2, max=666725, avg=10546.77, stdev=17530.19
    clat percentiles (usec):
     |  1.00th=[    13],  5.00th=[    15], 10.00th=[    15], 20.00th=[    16],
     | 30.00th=[    16], 40.00th=[    18], 50.00th=[    22], 60.00th=[ 16057],
     | 70.00th=[ 19268], 80.00th=[ 21890], 90.00th=[ 25035], 95.00th=[ 27132],
     | 99.00th=[ 30278], 99.50th=[ 32113], 99.90th=[177210], 99.95th=[189793],
     | 99.99th=[658506]
   bw (  KiB/s): min=  160, max=141408, per=100.00%, avg=3059.92, stdev=9017.09, samples=238
   iops        : min=   10, max= 8838, avg=191.21, stdev=563.57, samples=238
  lat (usec)   : 4=0.28%, 10=0.17%, 20=46.06%, 50=5.69%, 100=0.07%
  lat (msec)   : 10=0.25%, 20=20.04%, 50=27.12%, 100=0.09%, 250=0.21%
  lat (msec)   : 750=0.04%
  cpu          : usr=0.03%, sys=0.35%, ctx=5433, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=11379,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3033KiB/s (3106kB/s), 3033KiB/s-3033KiB/s (3106kB/s-3106kB/s), io=178MiB (186MB), run=60021-60021msec

---

随机写

16K 1 job 不带buffer

# fio --name=randread --rw=randwrite --direct=1 --ioengine=psync --bs=16k --iodepth=1 --numjobs=1 --size=10G --runtime=60  --group_reporting
randread: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=1809KiB/s][w=113 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=29806: Thu Jul 18 14:39:43 2024
  write: IOPS=242, BW=3881KiB/s (3974kB/s)(227MiB/60004msec); 0 zone resets
    clat (usec): min=2, max=2481.2k, avg=4121.94, stdev=26385.64
     lat (usec): min=2, max=2481.2k, avg=4122.10, stdev=26385.64
    clat percentiles (usec):
     |  1.00th=[     5],  5.00th=[    14], 10.00th=[    15], 20.00th=[    16],
     | 30.00th=[    17], 40.00th=[    19], 50.00th=[    22], 60.00th=[    40],
     | 70.00th=[  4621], 80.00th=[  8586], 90.00th=[ 12256], 95.00th=[ 14222],
     | 99.00th=[ 27657], 99.50th=[ 43254], 99.90th=[139461], 99.95th=[549454],
     | 99.99th=[641729]
   bw (  KiB/s): min=   32, max=192736, per=100.00%, avg=4071.58, stdev=19130.79, samples=114
   iops        : min=    2, max=12046, avg=254.47, stdev=1195.67, samples=114
  lat (usec)   : 4=0.54%, 10=1.14%, 20=43.73%, 50=17.37%, 100=2.69%
  lat (usec)   : 250=2.68%, 500=0.02%, 750=0.09%, 1000=0.36%
  lat (msec)   : 2=0.01%, 4=0.58%, 10=14.48%, 20=14.72%, 50=1.31%
  lat (msec)   : 100=0.16%, 250=0.03%, 500=0.01%, 750=0.07%, >=2000=0.01%
  cpu          : usr=0.04%, sys=0.74%, ctx=5329, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,14554,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3881KiB/s (3974kB/s), 3881KiB/s-3881KiB/s (3974kB/s-3974kB/s), io=227MiB (238MB), run=60004-60004msec

# fio --name=randread --rw=randwrite  --ioengine=psync --bs=16k --iodepth=1 --numjobs=1 --size=10G --runtime=60  --group_reporting
randread: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=1969KiB/s][w=123 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=30336: Thu Jul 18 14:43:35 2024
  write: IOPS=277, BW=4436KiB/s (4543kB/s)(260MiB/60013msec); 0 zone resets
    clat (usec): min=2, max=646309, avg=3605.52, stdev=14920.22
     lat (usec): min=2, max=646309, avg=3605.70, stdev=14920.23
    clat percentiles (usec):
     |  1.00th=[     6],  5.00th=[    14], 10.00th=[    16], 20.00th=[    18],
     | 30.00th=[    19], 40.00th=[    21], 50.00th=[    27], 60.00th=[   553],
     | 70.00th=[   889], 80.00th=[  7046], 90.00th=[ 11469], 95.00th=[ 13829],
     | 99.00th=[ 26870], 99.50th=[ 34866], 99.90th=[147850], 99.95th=[417334],
     | 99.99th=[641729]
   bw (  KiB/s): min=   64, max=68160, per=100.00%, avg=4459.56, stdev=7850.88, samples=119
   iops        : min=    4, max= 4260, avg=278.72, stdev=490.68, samples=119
  lat (usec)   : 4=0.52%, 10=1.17%, 20=36.78%, 50=13.89%, 100=0.22%
  lat (usec)   : 250=2.06%, 500=3.90%, 750=8.19%, 1000=3.79%
  lat (msec)   : 2=0.25%, 4=2.96%, 10=13.00%, 20=11.38%, 50=1.51%
  lat (msec)   : 100=0.20%, 250=0.10%, 500=0.04%, 750=0.03%
  cpu          : usr=0.05%, sys=0.97%, ctx=7931, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,16640,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4436KiB/s (4543kB/s), 4436KiB/s-4436KiB/s (4543kB/s-4543kB/s), io=260MiB (273MB), run=60013-60013msec

16K 3 job

# fio --name=randwrite --rw=randwrite --direct=1 --ioengine=psync --bs=16k --iodepth=1 --numjobs=3 --size=10G --runtime=60  --group_reporting
randwrite: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1
...
fio-3.33
Starting 3 processes
randwrite: Laying out IO file (1 file / 10240MiB)
randwrite: Laying out IO file (1 file / 10240MiB)
randwrite: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [w(3)][100.0%][w=17.3MiB/s][w=1109 IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=3): err= 0: pid=30909: Thu Jul 18 14:47:47 2024
  write: IOPS=2747, BW=42.9MiB/s (45.0MB/s)(2576MiB/60016msec); 0 zone resets
    clat (usec): min=2, max=917962, avg=1091.21, stdev=17215.38
     lat (usec): min=2, max=917962, avg=1091.36, stdev=17215.38
    clat percentiles (usec):
     |  1.00th=[     9],  5.00th=[    12], 10.00th=[    13], 20.00th=[    14],
     | 30.00th=[    15], 40.00th=[    17], 50.00th=[    20], 60.00th=[    22],
     | 70.00th=[    25], 80.00th=[    28], 90.00th=[    39], 95.00th=[    79],
     | 99.00th=[ 22414], 99.50th=[ 37487], 99.90th=[102237], 99.95th=[541066],
     | 99.99th=[725615]
   bw (  KiB/s): min=  384, max=345472, per=100.00%, avg=49161.27, stdev=21402.32, samples=321
   iops        : min=   24, max=21592, avg=3072.58, stdev=1337.65, samples=321
  lat (usec)   : 4=0.24%, 10=2.48%, 20=50.49%, 50=39.53%, 100=2.57%
  lat (usec)   : 250=0.33%, 500=0.32%, 750=0.34%, 1000=0.13%
  lat (msec)   : 2=0.31%, 4=0.48%, 10=0.75%, 20=0.86%, 50=0.85%
  lat (msec)   : 100=0.21%, 250=0.03%, 500=0.01%, 750=0.06%, 1000=0.01%
  cpu          : usr=0.06%, sys=1.99%, ctx=16937, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,164894,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=42.9MiB/s (45.0MB/s), 42.9MiB/s-42.9MiB/s (45.0MB/s-45.0MB/s), io=2576MiB (2702MB), run=60016-60016msec