从上一节简单介绍完SPDK之后,这里我开始就想搭建一下SPDK,然后再去了解其细节。
测试环境
win10 + virtualbox 6.0.24 + vagrant 2.2.16
虚拟机操作系统是ubuntu1804
安装参考链接:
-在虚拟机上部署SPDK NVME测试案例 -Vagrant Development Environment
Introduction
安装spdk环境
1、准备软件、安装包
准备virtualbox和vagrant
我们下载virtualbox 6.0和vagrant 2.2(我这里是最新的)
下载地址:
- virtualbox:https://www.virtualbox.org/wiki/Download_Old_Builds_6_0
注: 据官网介绍:https://www.virtualbox.org/wiki/Downloads,virtualbox6.1不支持软件虚拟化,所以我这里下载的是6.0版本的virtualbox
- virtualbox extension Pack:https://www.virtualbox.org/wiki/Download_Old_Builds_6_0
- vagrant:https://www.vagrantup.com/
准备OS(ubuntu1804)和nvme
ubuntu文件:https://mirrors.tuna.tsinghua.edu.cn/ubuntu-cloud-images/bionic/current/bionic-server-cloudimg-amd64-vagrant.box
和nvme文件:https://pan.baidu.com/s/1ecOg5hP-ijEleeS_Y6YBUg(retrievecode is “gxce”),要的nvme文件在网盘中有
2、开始安装
添加box
# 讲离线下载好的box安装到vagrant中,注意后面的路径一定要是全部路径
$ vagrant box add ubuntu1804 D:\vagrant-workspace\downloads\bionic-server-cloudimg-amd64-vagrant.box
#或者直接网上下载,可以去查看create_vbox.sh文件支持哪些系统
$ ./spdk/scripts/vagrant/create_vbox.sh –n 4 –s 4096 ubuntu1804
其中 ubuntu1804 是自定义的box的名字,后面是本地box的路径
创建vagrantfile文件
这里我创建目录D:\vagrant-workspace\vagrantFile\ubuntu1804存放vagrantfile文件
cd D:\vagrant-workspace\vagrantFile\ubuntu1804
添加完后 vagrant init ubuntu1804 然后就会在目录下生成一个vagrantfile文件
然后打开这个文件,修改如下
启动镜像
配置好后,输入 vagrant up
启动成功,在上述配置好之后
框框中的信息是登录vagrant up后的系统用的
我这里登录用的是xshell-----下载一个xshell然后安装
打开xshell
按确定之后生成一个新的会话,然后登录用的用户名是vagrant,密码也是vagrant。
Xshell配置ssh免密码登录-密钥公钥(Public key)
上述配置完就可以用xshell登录
然后就进入那个box的系统里面了
配置Ubuntu环境
1、Ubuntu18.04更改apt-get的下载源
清华源:mirror.tuna.tsinghua.edu.cn/help/ubuntu/
可以用下面的语句清空
#清空文件内容 -s 指定文件大小
truncate -s 0 access.log
2、添加nvme包
打开virtual box,先将虚拟机停掉
2.1 右击当前虚拟机,选中setting
2.2 设置memory
因为后面测试blobfs 需要内存5个g,所以我这里给虚机分了8个g
2.3 设置cup
2.4 添加nvme controller
选中已有硬盘
添加
然后cloose
nvme已经被我们添加进来了
配置spdk环境
再启动虚机,用xshell连接
a、编译SPDK代码,命令行如下:
#之前安装19.07.x出现问题,现在换成21.04
vagrant@localhost:~$ git clone -b v21.04.x https://github.com/spdk/spdk.git
Cloning into 'spdk'...
remote: Enumerating objects: 136686, done.
remote: Counting objects: 100% (6009/6009), done.
remote: Compressing objects: 100% (1870/1870), done.
remote: Total 136686 (delta 4217), reused 5784 (delta 4103), pack-reused 130677
Receiving objects: 100% (136686/136686), 59.62 MiB | 4.13 MiB/s, done.
Resolving deltas: 100% (106903/106903), done.
vagrant@localhost:~$ cd spdk/
vagrant@localhost:~/spdk$ git branch
* v21.04.x
#下载相关子模块
vagrant@localhost:~/spdk$ git submodule update --init
Submodule 'dpdk' (https://github.com/spdk/dpdk.git) registered for path 'dpdk'
Submodule 'intel-ipsec-mb' (https://github.com/spdk/intel-ipsec-mb.git) registered for path 'intel-ipsec-mb'
Submodule 'isa-l' (https://github.com/spdk/isa-l.git) registered for path 'isa-l'
Submodule 'libvfio-user' (https://github.com/nutanix/libvfio-user.git) registered for path 'libvfio-user'
Submodule 'ocf' (https://github.com/Open-CAS/ocf.git) registered for path 'ocf'
Cloning into '/home/vagrant/spdk/dpdk'...
fatal: unable to access 'https://github.com/spdk/dpdk.git/': gnutls_handshake() failed: Error in the pull function.
fatal: clone of 'https://github.com/spdk/dpdk.git' into submodule path '/home/vagrant/spdk/dpdk' failed
Failed to clone 'dpdk'. Retry scheduled
Cloning into '/home/vagrant/spdk/intel-ipsec-mb'...
Cloning into '/home/vagrant/spdk/isa-l'...
Cloning into '/home/vagrant/spdk/libvfio-user'...
Cloning into '/home/vagrant/spdk/ocf'...
Cloning into '/home/vagrant/spdk/dpdk'...
Submodule path 'dpdk': checked out 'eb16786836e3a8380bb86fde67efa2ee0d9d3852'
Submodule path 'intel-ipsec-mb': checked out '93c2ddf877532f499a79e7111c9d9532cb44ceaf'
Submodule path 'isa-l': checked out '2df39cf5f1b9ccaa2973f6ef273857e4dc46f0cf'
Submodule path 'libvfio-user': checked out '3acb97477f261bf11c866b5764deabe693607bc3'
Submodule path 'ocf': checked out '05401cac64b6f8f4e7050ef404ad9c78353c7cdb'
Submodule path 'dpdk': checked out 'eb16786836e3a8380bb86fde67efa2ee0d9d3852'
b、验证虚拟NVMe设备是否存在,命令行如下
vagrant@localhost:~/spdk$ lspci | grep "Non-Volatile"
00:0e.0 Non-Volatile memory controller: InnoTek Systemberatung GmbH Device 4e56
c、编译spdk
vagrant@localhost:~/spdk$ sudo apt update
#安装spdk最小的环境
vagrant@localhost:~/spdk$ sudo scripts/pkgdep.sh
vagrant@localhost:~$ cd spdk/
vagrant@localhost:~/spdk$ ./configure
Notice: ISA-L, compression & crypto require NASM version 2.14 or newer. Turning off default ISA-L and crypto features.
Using default SPDK env in /home/vagrant/spdk/lib/env_dpdk
Using default DPDK in /home/vagrant/spdk/dpdk/build
Creating mk/config.mk...done.
Creating mk/cc.flags.mk...done.
Type 'make' to build.
vagrant@localhost:~/spdk$ make
/bin/sh: 1: meson: not found
Makefile:163: recipe for target '/home/vagrant/spdk/dpdk/build-tmp' failed
make[1]: *** [/home/vagrant/spdk/dpdk/build-tmp] Error 127
/home/vagrant/spdk/mk/spdk.subdirs.mk:44: recipe for target 'dpdkbuild' failed
make: *** [dpdkbuild] Error 2
错误提示是没有下载meson
下载meson
vagrant@localhost:~/spdk$ pip3 install meson -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
Collecting meson
Downloading http://mirrors.aliyun.com/pypi/packages/e6/e4/11fc433adb18567dcf90041c54e5e65b03e9451bc1a45f9cd2dac9e9bb05/meson-0.58.1.tar.gz (1.9MB)
100% |████████████████████████████████| 1.9MB 4.2MB/s
Building wheels for collected packages: meson
Running setup.py bdist_wheel for meson ... done
Stored in directory: /home/vagrant/.cache/pip/wheels/41/19/26/2ceb373a0a197deaf7d30c6580ca928c665f7c8f3ce808ef62
Successfully built meson
Installing collected packages: meson
Successfully installed meson-0.58.1
#查看meson版本
vagrant@localhost:~/spdk$ meson -v
Command 'meson' not found, but can be installed with:
sudo apt install meson
vagrant@localhost:~/spdk$ sudo apt install meson
。。。。。
vagrant@localhost:~/spdk$ meson -v
0.45.1
#这里发现meson不是0.58.1版本,发现是链接到 这目录下。
vagrant@localhost:~/spdk$ ll /usr/bin/meson
lrwxrwxrwx 1 root root 20 Feb 3 2020 /usr/bin/meson -> ../share/meson/meson*
#pip3 uninstall meson # 找到安装位置,不是要卸载,输入 ‘no’
vagrant@localhost:~/spdk$ pip3 uninstall meson
Uninstalling meson-0.58.1:
/home/vagrant/.local/bin/meson
/home/vagrant/.local/lib/python3.6/site-packages/meson-0.58.1.dist-info/DESCRIPTION.rst
#把链接设定为新安装的位置: /home/vagrant/.local/bin/meson
vagrant@localhost:~/spdk$ sudo rm /usr/bin/meson
vagrant@localhost:~/spdk$ sudo ln -sf /home/vagrant/.local/bin/meson /usr/bin/meson
vagrant@localhost:~/spdk$ meson -v
0.58.1
再次make
vagrant@localhost:~/spdk$ sudo git submodule update --init
vagrant@localhost:~/spdk$ ./configure
Notice: ISA-L, compression & crypto require NASM version 2.14 or newer. Turning off default ISA-L and crypto features.
Using default SPDK env in /home/vagrant/spdk/lib/env_dpdk
Using default DPDK in /home/vagrant/spdk/dpdk/build
Creating mk/config.mk...done.
Creating mk/cc.flags.mk...done.
Type 'make' to build.
vagrant@localhost:~/spdk$ make
运行hello_word 测试spdk环境是否成功
#绑定设备
vagrant@localhost:~/spdk$ sudo scripts/setup.sh
0000:00:0e.0 (80ee 4e56): nvme -> uio_pci_generic
vagrant@localhost:~/spdk$ sudo build/examples/hello_world
[2021-06-21 13:51:30.084504] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-06-21 13:51:30.084580] [ DPDK EAL parameters: [2021-06-21 13:51:30.084593] hello_world [2021-06-21 13:51:30.084603] -c 0x1 [2021-06-21 13:51:30.084607] --log-level=lib.eal:6 [2021-06-21 13:51:30.084611] --log-level=lib.cryptodev:5 [2021-06-21 13:51:30.084615] --log-level=user1:6 [2021-06-21 13:51:30.084622] --iova-mode=pa [2021-06-21 13:51:30.084626] --base-virtaddr=0x200000000000 [2021-06-21 13:51:30.084633] --match-allocations [2021-06-21 13:51:30.084637] --file-prefix=spdk0 [2021-06-21 13:51:30.084645] --proc-type=auto [2021-06-21 13:51:30.084649] ]
EAL: No legacy callbacks, legacy socket not created
Initializing NVMe Controllers
Attaching to 0000:00:0e.0
Attached to 0000:00:0e.0
Using controller ORCL-VBOX-NVME-VER12 (VB1234-56789 ) with 1 namespaces.
Namespace ID: 1 size: 1GB
Initialization complete.
[2021-06-21 13:51:30.207375] nvme_pcie.c: 462:nvme_pcie_ctrlr_map_io_cmb: *ERROR*: CMB is already in use for submission queues.
INFO: using host memory buffer for IO
Hello world!
到此我们的spdk环境就搭建好了,就可以去集成其他组件了,比如rocksdb;