spdk（bdev层） + fio测试nvme 设备的性能

1、背景

• spdk ：是一个基于dpdk的存储开发kit，这里主要利用了它提供的用户态nvme driver —— spdk链接
• fio: io测试工具，提供丰富的参数，可以构造复杂的io pattern

在spdk中，定义了一层虚拟的block device层，在这一层提供了若干的功能：

• A pluggable module API for implementing block devices that interface with different types of block storage devices.
• Driver modules for NVMe, malloc (ramdisk), Linux AIO, virtio-scsi, Ceph RBD, and more.
• An application API for enumerating and claiming SPDK block devices and then performing operations (read, write, unmap, etc.) on those devices.
• Facilities to stack block devices to create complex I/O pipelines, including logical volume management (lvol) and partition support (GPT).
• Configuration of block devices via JSON-RPC and a configuration file.
• Request queueing, timeout, and reset handling.
• Multiple, lockless queues for sending I/O to block devices.

spdk中提供了一个bdev的应用，将这一层的api打包成一个fio的ioengine，供fio进行测试

2、使用步骤

2.1下载dpdk、spdk、fio并解压

用最新的版本即可

2.2编译fio

cd fio_dir
./configure
make&&make install

2.3编译dpdk

vim <dpdk_dir>/config/defconfig_x86_64-native-linuxapp-gcc
#增加一行
EXTRA_CFLAGS=-fPIC
#回到<dpdk_dir>编译dpdk
make install T=x86_64-native-linuxapp-gcc DESTDIR=.

2.4编译spdk

cd <spdk_dir>
./configure --with-fio=/root/Downloads/fio-fio-3.3/ --with-dpdk=/root/Downloads/dpdk-17.11/x86_64-native-linuxapp-gcc
#修改<spdk_dir>/CONFIG文件
CONFIG_FIO_PLUGIN=y
FIO_SOURCE_DIR=fio的目录
#编译spdk
make DPDK_DIR=/root/Downloads/dpdk-17.11/x86_64-native-linuxapp-gcc

2.5unbind nvme driver替换为vfio

cd <spdk_dir>/scripts
sh setup.sh

2.6 配置bdev的conf文件

这个conf文件会设置一个虚拟块设备的各种参数，在spdk中<spdk_dir>/etc/spdk/中有几个例子，主要看nvme部分

[Nvme]
# NVMe Device Whitelist
# Users may specify which NVMe devices to claim by their transport id.
# See spdk_nvme_transport_id_parse() in spdk/nvme.h for the correct format.
# The second argument is the assigned name, which can be referenced from
# other sections in the configuration file. For NVMe devices, a namespace
# is automatically appended to each name in the format <YourName>nY, where
# Y is the NSID (starts at 1).
#将某个pcie总线上的设备“定义”成一个块设备Nvme0，这里的总线地址可以用lspci查询
TransportId "trtype:PCIe traddr:0000:00:00.0" Nvme0
TransportId "trtype:PCIe traddr:0000:01:00.0" Nvme1
# The number of attempts per I/O when an I/O fails. Do not include
# this key to get the default behavior.
RetryCount 4
# Timeout for each command, in seconds. If 0, don't track timeouts.
Timeout 0
# Action to take on command time out. Only valid when Timeout is greater
# than 0. This may be 'Reset' to reset the controller, 'Abort' to abort
# the command, or 'None' to just print a message but do nothing.
# Admin command timeouts will always result in a reset.
ActionOnTimeout None
# Set how often the admin queue is polled for asynchronous events.
# Units in microseconds.
AdminPollRate 100000
# Disable handling of hotplug (runtime insert and remove) events,
# users can set to Yes if want to enable it.
# Default: No
HotplugEnable No

2.7 编辑fio的jobfile，有几个特别的要求

• ioengine=spdk_bdev
• spdk_conf=path/to/bdev.conf 引入上一步中bdev设备的配置文件
• filename就是上一个配置文件中对应某个pci设备的那个nvme设备名

[global]
ioengine=spdk_bdev
spdk_conf=/root/Downloads/spdk-18.01/examples/bdev/fio_plugin/bdev.conf.in
thread=1
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=60
iodepth=128
rw=randread
bs=4k
[test]
numjobs=1
filename=Nvme0n1

2.8使用fio执行测试

LD_PRELOAD=/root/Downloads/spdk-18.01/examples/bdev/fio_plugin/fio_plugin /root/Downloads/fio-fio-3.3/fio example_config.fio

3、注意

• spdk做性能测试时，对每个namespace会绑定一个lcore，所以fio的thread只能等于1
• fio测试random的io时，需要设置norandommap=1 ，防止fio的random map影响性能