工作需要配置多路径但是自己又没有搞过,所以搭建环境学习一下过程,这样以后操作感觉会踏实点。本着学习的心态,在虚拟环境下模拟,来做一个多路径的实验。
本文主要描述使用iscsi模拟存储划出的1个LUN,最终由两条链路连接到主机,显示为两个磁盘的场景。为后续配置multipath提供前置环境。

•1.模拟主机环境
•2.准备软件环境
•3.模拟存储加盘
•4.配置iSCSI服务端
•5.配置iSCSI客户端

1.模拟主机环境

首先虚拟出两个linux服务器,用作搭建iSCSI服务器跟客户端。每台服务器一共三块网卡。一块外网。两块内网网卡。两个内网网卡主要虚拟两条链路。架构如下、

服务端:
外网:10.0.0.205
内网1:172.16.1.5
内网2:172.16.2.5

客户端:
外网:10.0.0.206
内网1:172.16.1.6
内网2:172.16.2.6

内网主要用虚拟机的lan区段模拟,搭建完成实现互ping能通。

centos装完多路径软件后无法ssh centos iscsi 多路径_centos装完多路径软件后无法ssh

2.准备软件环境

scsi-target-utils:用來將 Linux 系統模擬成為 iSCSI target 的功能;
iscsi-initiator-utils:掛載來自 target 的磁碟到 Linux 本机上。

其中scsi-target-utils好像要用epel源下载,我用了阿里源的epql源。直接yum install 软件名字。

3.模拟存储加盘

我在虚拟机服务端填加了一个5G的盘,实际就是用来模拟存储新增实际的一块盘。 我这里新增加的盘显示为/dev/sdb,我将它创建成lvm,并先分出一个2g的lv,供后续使用。至于怎么分涉及到LVM知识,不在本文范围内,请学习lvm知识。

4.配置iSCSI服务端

iSCSI服务端主要配置文件:/etc/tgt/targets.conf

iSCSI 有一套自己分享 target 档名,基本上,藉由 iSCSI 分享出來的 target 檔名都是以 iqn 為開頭,意思是:『iSCSI Qualified Name (iSCSI 合格名稱)』的意思(註5)。那麼在 iqn 後面要接啥檔名呢?通常是這樣的:
iqn.yyyy-mm.

操作如下:
vi /etc/tgt/targets.conf

上面有一段显示不出来,所以我截图放在这.

centos装完多路径软件后无法ssh centos iscsi 多路径_运维_02

其中这一串就是lun的名字,下文客户端识别的就是这个名字: iqn.2017-07.com.cnblogs.jyzhao:alfreddisk

配置完成后,就启动服务和设置开机自启动:

[root@centos7 ~]# systemctl start tgtd
 [root@centos7 ~]# ps -ef |grep tgtd
 root 1597 1 0 23:38 ? 00:00:00 /usr/sbin/tgtd -f
 root 1657 1490 0 23:39 pts/0 00:00:00 grep --color=auto tgtd[root@centos7 ~]# systemctl status tgtd
 ● tgtd.service - tgtd iSCSI target daemon
 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)
 Active: active (running) since Fri 2020-07-24 23:38:45 CST; 1min 7s ago
 Process: 1629 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)
 Process: 1601 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)
 Process: 1600 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
 Process: 1598 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)
 Main PID: 1597 (tgtd)
 CGroup: /system.slice/tgtd.service
 └─1597 /usr/sbin/tgtd -fJul 24 23:38:40 centos7 systemd[1]: Starting tgtd iSCSI target daemon…
 Jul 24 23:38:40 centos7 tgtd[1597]: tgtd: iser_ib_init(3436) Failed to initialize RDMA; load kernel modules?
 Jul 24 23:38:40 centos7 tgtd[1597]: tgtd: work_timer_start(146) use timer_fd based scheduler
 Jul 24 23:38:40 centos7 tgtd[1597]: tgtd: bs_init_signalfd(267) could not open backing-store module directory /usr/lib64/tgt/backing-store
 Jul 24 23:38:40 centos7 tgtd[1597]: tgtd: bs_init(386) use signalfd notification
 Jul 24 23:38:45 centos7 tgtd[1597]: tgtd: device_mgmt(246) sz:29 params:path=/dev/vg_storage/lv_lun1
 Jul 24 23:38:45 centos7 tgtd[1597]: tgtd: bs_thread_open(408) 16
 Jul 24 23:38:45 centos7 systemd[1]: Started tgtd iSCSI target daemon.[root@centos7 ~]# systemctl enable tgtd
然后查询下相关的信息,比如占用的端口、LUN信息:
 [root@centos7 ~]# netstat -tlunp |grep tgt
 tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 1597/tgtd
 tcp6 0 0 :::3260 ::😗 LISTEN 1597/tgtdtgt-admin --show
 Target 1: iqn.2017-07.com.cnblogs.jyzhao:alfreddisk
 System information:
 Driver: iscsi
 State: ready
 I_T nexus information:
 LUN information:
 LUN: 0
 Type: controller
 SCSI ID: IET 00010000
 SCSI SN: beaf10
 Size: 0 MB, Block size: 1
 Online: Yes
 Removable media: No
 Prevent removal: No
 Readonly: No
 SWP: No
 Thin-provisioning: No
 Backing store type: null
 Backing store path: None
 Backing store flags:
 LUN: 1
 Type: disk
 SCSI ID: IET 00010001
 SCSI SN: beaf11
 Size: 2147 MB, Block size: 512
 Online: Yes
 Removable media: No
 Prevent removal: No
 Readonly: No
 SWP: No
 Thin-provisioning: No
 Backing store type: rdwr
 Backing store path: /dev/vg_storage/lv_lun1
 Backing store flags:
 Account information:
 ACL information:
 ALL

发现 LUN: 1就是我们刚刚配置的lun。已经成功。

5.配置iSCSI客户端

确认开机启动项设置开启:

systemctl start iscsid
[root@centos7 network-scripts]# systemctl status iscsid
 ● iscsid.service - Open-iSCSI
 Loaded: loaded (/usr/lib/systemd/system/iscsid.service; disabled; vendor preset: disabled)
 Active: active (running) since Fri 2020-07-24 23:48:39 CST; 5s ago
 Docs: man:iscsid(8)
 man:iscsiuio(8)
 man:iscsiadm(8)
 Main PID: 2381 (iscsid)
 Status: “Ready to process requests”
 CGroup: /system.slice/iscsid.service
 └─2381 /sbin/iscsid -fJul 24 23:48:39 centos7 systemd[1]: Starting Open-iSCSI…
 Jul 24 23:48:39 centos7 systemd[1]: Started Open-iSCSI.------------ping可以通
然后执行以下命令扫描链路:
 [root@centos7 network-scripts]# iscsiadm -m discovery -t sendtargets -p 172.16.1.5
 172.16.1.5:3260,1 iqn.2017-07.com.cnblogs.jyzhao:alfreddisk
 [root@centos7 network-scripts]#
 [root@centos7 network-scripts]# iscsiadm -m discovery -t sendtargets -p 172.16.2.5
 172.16.2.5:3260,1 iqn.2017-07.com.cnblogs.jyzhao:alfreddisk查看/var/lib/iscsi/nodes/下的文件:
 [root@centos7 network-scripts]# ll -R /var/lib/iscsi/nodes/
 /var/lib/iscsi/nodes/:
 total 0
 drw------- 4 root root 56 Jul 24 23:51 iqn.2017-07.com.cnblogs.jyzhao:alfreddisk/var/lib/iscsi/nodes/iqn.2017-07.com.cnblogs.jyzhao:alfreddisk:
 total 0
 drw------- 2 root root 21 Jul 24 23:50 172.16.1.5,3260,1
 drw------- 2 root root 21 Jul 24 23:51 172.16.2.5,3260,1/var/lib/iscsi/nodes/iqn.2017-07.com.cnblogs.jyzhao:alfreddisk/172.16.1.5,3260,1:
 total 4
 -rw------- 1 root root 2072 Jul 24 23:50 default/var/lib/iscsi/nodes/iqn.2017-07.com.cnblogs.jyzhao:alfreddisk/172.16.2.5,3260,1:
 total 4
 -rw------- 1 root root 2072 Jul 24 23:51 default[root@centos7 network-scripts]# iscsiadm -m node
 172.16.1.5:3260,1 iqn.2017-07.com.cnblogs.jyzhao:alfreddisk
 172.16.2.5:3260,1 iqn.2017-07.com.cnblogs.jyzhao:alfreddisk登陆登录Target
 [root@centos7 network-scripts]# iscsiadm -m node -T iqn.2017-07.com.cnblogs.jyzhao:alfreddisk --login
 Logging in to [iface: default, target: iqn.2017-07.com.cnblogs.jyzhao:alfreddisk, portal: 172.16.1.5,3260] (multiple)
 Logging in to [iface: default, target: iqn.2017-07.com.cnblogs.jyzhao:alfreddisk, portal: 172.16.2.5,3260] (multiple)
 Login to [iface: default, target: iqn.2017-07.com.cnblogs.jyzhao:alfreddisk, portal: 172.16.1.5,3260] successful.
 Login to [iface: default, target: iqn.2017-07.com.cnblogs.jyzhao:alfreddisk, portal: 172.16.2.5,3260] successful.target常用命令:
 1: # iscsiadm -m discovery -t sendtargets -p 192.168.1.1:3260
 此时找到并拥有了一个目标(target):192.168.1.1:3260,1 iqn.1997-05.com.test:raid 被发现的目标也叫做节点。

2 登入节点,以上面被发现的目标为例:

iscsiadm -m node –T iqn.1997-05.com.test:raid -p 192.168.1.1:3260 -l 其中iqn.1997-05.com.test:raid是目标名。

3:如何注销到target的连接?
如果要注销到某一个特定的Target的连接,可以使用下列的命令:
iscsiadm -m node -T iqn.2007-04.acme.com:h3c:200realm.rhel5 -p 200.200.10.200:3260 –u

4:如何从操作系统中删除一个target的信息?
iscsiadm -m node -o delete -T iqn.2005-03.com.max -p 192.168.0.4:3260
其中iqn.2005-03.com.max代表target的名称,192.168.0.4代表target的IP地址

5:如何查看就有哪些target记录在了Open-iSCSI数据库中?
使用iscsiadm -m node命令

最后查询 fdisk -l结果:
省略之前无关内容…这两个就是服务端映射过来的lun,已经成功识别、

Disk /dev/sdd: 2147 MB, 2147483648 bytes, 4194304 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sde: 2147 MB, 2147483648 bytes, 4194304 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes

下面是多路径(Multipath)配置

普通的电脑主机都是一个硬盘挂接到一个总线上,这里是一对一的关系。而到了有光纤组成的SAN环境,或者由iSCSI组成的IPSAN环境,由于主机和存储通过了光纤交换机或者多块网卡及IP来连接,这样的话,就构成了多对多的关系。也就是说,主机到存储可以有多条路径可以选择。主机到存储之间的IO由多条路径可以选择。每个主机到所对应的存储可以经过几条不同的路径,如果是同时使用的话,I/O流量如何分配?其中一条路径坏掉了,如何处理?还有在操作系统的角度来看,每条路径,操作系统会认为是一个实际存在的物理盘,但实际上只是通向同一个物理盘的不同路径而已,这样是在使用的时候,就给用户带来了困惑。多路径软件就是为了解决上面的问题应运而生的。
多路径的主要功能就是和存储设备一起配合实现如下功能:
故障的切换和恢复
IO流量的负载均衡
磁盘的虚拟化

1:安装:
yum install device-mapper-multipath
2.设置服务开机启动
systemctl enable multipathd
 systemctl start multipathd3.生成multipath配置文件
/sbin/mpathconf --enable
确认服务状态:
 [root@centos7 network-scripts]# systemctl status multipathd
 ● multipathd.service - Device-Mapper Multipath Device Controller
 Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
 Active: active (running) since Sat 2020-07-25 00:05:53 CST; 4s ago
 Process: 2647 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
 Process: 2645 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
 Process: 2643 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 2555 (multipathd)
 CGroup: /system.slice/multipathd.service
 ‣ 2555 multipathdJul 25 00:05:53 centos7 systemd[1]: Starting Device-Mapper Multipath Device Controller…
 Jul 25 00:05:53 centos7 systemd[1]: Started Device-Mapper Multipath Device Controller.4.multipath的常用命令
常用命令:
 –生成multipath配置文件
 /sbin/mpathconf --enable
 –显示多路径的布局
 multipath -ll
 –重新刷取
 multipath -v2
 –清空所有多路径
 multipath -F针对上述常用命令,实际操作的记录:
 查看多路径情况:
 [root@centos7 network-scripts]# multipath -ll
 mpatha (360000000000000000e00000000010001) dm-2 IET ,VIRTUAL-DISK
 size=2.0G features=‘0’ hwhandler=‘0’ wp=rw
 |-± policy=‘service-time 0’ prio=1 status=active
 | - 3:0:0:1 sdd 8:48 active ready running -± policy=‘service-time 0’ prio=1 status=enabled
 `- 4:0:0:1 sde 8:64 active ready running这里测试没有问题后,执行lsblk命令就可以看到多路径磁盘mpatha了
 [root@centos7 network-scripts]# lsblk
 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
 sda 8:0 0 20G 0 disk
 ├─sda1 8:1 0 1G 0 part /boot
 └─sda2 8:2 0 19G 0 part
 ├─centos-root 253:0 0 17G 0 lvm /
 └─centos-swap 253:1 0 2G 0 lvm [SWAP]
 sdb 8:16 0 1G 0 disk
 sdc 8:32 0 1G 0 disk
 sdd 8:48 0 2G 0 disk
 └─mpatha 253:2 0 2G 0 mpath
 sde 8:64 0 2G 0 disk
 └─mpatha 253:2 0 2G 0 mpath
 sr0 11:0 1 4.2G 0 rom可以对mpatha进行相应的磁盘操作了,以下是我的操作,仅供参考。
 cd /dev/mapper/
 [root@centos7 mapper]# ll
 total 0
 lrwxrwxrwx 1 root root 7 Jul 24 23:18 centos-root -> …/dm-0
 lrwxrwxrwx 1 root root 7 Jul 24 23:18 centos-swap -> …/dm-1
 crw------- 1 root root 10, 236 Jul 24 23:18 control
 lrwxrwxrwx 1 root root 7 Jul 25 00:03 mpatha -> …/dm-2[root@centos7 mapper]# fdisk -l
Disk /dev/sda: 21.5 GB, 21474836480 bytes, 41943040 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk label type: dos
 Disk identifier: 0x000c8d08Device Boot Start End Blocks Id System
 /dev/sda1 * 2048 2099199 1048576 83 Linux
 /dev/sda2 2099200 41943039 19921920 8e Linux LVMDisk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sdc: 1073 MB, 1073741824 bytes, 2097152 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/centos-root: 18.2 GB, 18249416704 bytes, 35643392 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sdd: 2147 MB, 2147483648 bytes, 4194304 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sde: 2147 MB, 2147483648 bytes, 4194304 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/mpatha: 2147 MB, 2147483648 bytes, 4194304 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes创建lvm
 [root@centos7 mapper]# pvcreate /dev/mapper/mpatha
 Physical volume “/dev/mapper/mpatha” successfully created.
 [root@centos7 mapper]# vgcreate vg01 /dev/mapper/mpatha
 Volume group “vg01” successfully created
 [root@centos7 mapper]# lvcreate -n lv01 -l 100%VG vg01
 Logical volume “lv01” created.[root@centos7 mapper]# lvs
 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
 root centos -wi-ao---- <17.00g
 swap centos -wi-ao---- 2.00g
 lv01 vg01 -wi-a----- <2.00g[root@centos7 mapper]# lvdisplay
 — Logical volume —
 LV Path /dev/vg01/lv01
 LV Name lv01
 VG Name vg01
 LV UUID WEUv92-jV8c-hxX7-xnTw-MdvQ-19m3-YDqoip
 LV Write Access read/write
 LV Creation host, time centos7, 2020-07-25 00:39:15 +0800
 LV Status availableopen 0
LV Size <2.00 GiB
 Current LE 511
 Segments 1
 Allocation inherit
 Read ahead sectors auto• currently set to 8192
 Block device 253:3— Logical volume —
 LV Path /dev/centos/swap
 LV Name swap
 VG Name centos
 LV UUID k4EpRI-Tojt-EK7k-k5rA-lwT0-jFYj-H4NIEc
 LV Write Access read/write
 LV Creation host, time localhost.localdomain, 2020-06-22 19:51:10 +0800
 LV Status availableopen 2
LV Size 2.00 GiB
 Current LE 512
 Segments 1
 Allocation inherit
 Read ahead sectors auto• currently set to 8192
 Block device 253:1— Logical volume —
 LV Path /dev/centos/root
 LV Name root
 VG Name centos
 LV UUID V5Na7e-PmMv-37YH-teg8-t5Sv-PPk7-DCA2bX
 LV Write Access read/write
 LV Creation host, time localhost.localdomain, 2020-06-22 19:51:10 +0800
 LV Status availableopen 1
LV Size <17.00 GiB
 Current LE 4351
 Segments 1
 Allocation inherit
 Read ahead sectors auto• currently set to 8192
 Block device 253:0格式化文件系统
 [root@centos7 mapper]# mkfs.ext4 /dev/vg01/lv01
 mke2fs 1.42.9 (28-Dec-2013)
 Filesystem label=
 OS type: Linux
 Block size=4096 (log=2)
 Fragment size=4096 (log=2)
 Stride=0 blocks, Stripe width=0 blocks
 130816 inodes, 523264 blocks
 26163 blocks (5.00%) reserved for the super user
 rst data block=0
 Maximum filesystem blocks=536870912
 16 block groups
 32768 blocks per group, 32768 fragments per group
 8176 inodes per group
 Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912Allocating group tables: done
 Writing inode tables: done
 Creating journal (8192 blocks): done
 Writing superblocks and filesystem accounting information: done挂载:
 mount /dev/vg01/lv01 /date/
 [root@centos7 mapper]# df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/mapper/centos-root 17G 1.6G 16G 10% /
 devtmpfs 2.0G 0 2.0G 0% /dev
 tmpfs 2.0G 0 2.0G 0% /dev/shm
 tmpfs 2.0G 8.7M 2.0G 1% /run
 tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
 /dev/sda1 1014M 125M 890M 13% /boot
 tmpfs 394M 0 394M 0% /run/user/0
 /dev/mapper/vg01-lv01 2.0G 6.0M 1.9G 1% /date

开机自动挂载。

完。。。