【ASM】多路径(multipath)简介及配置









使用OpenFiler来模拟存储配置RAC中ASM共享盘及多路径(multipath)的测试 请参考 : ​​http://blog.itpub.net/26736162/viewspace-2132858/​









普通的电脑主机都是一个硬盘挂接到一个总线上,这里是一对一的关系,而到了有光纤组成的SAN(Storage Area Network,存储网络)环境,由于主机和存储通过了光纤交换机连接,这样的话,就构成了多对多的关系。也就是说,主机到存储可以有多条路径可以选择,即主机到存储之间的I/O多条路径可以选择。

既然,每个主机到所对应的存储可以经过几条不同的路径,如果是同时使用的话,I/O流量如何分配?其中一条路径坏掉了,如何处理?还有在操作系统的角度来看,每条路径,操作系统会认为是一个实际存在的物理盘,但实际上只是通向同一个物理盘的不同路径而已,这样是在使用的时候,就给用户带来了困惑。多路径软件就是为了解决上面的问题应运而生的。多路径的主要功能就是和存储设备一起配合实现如下功能:

1.故障的切换和恢复

2.I/O流量的负载均衡

3.磁盘的虚拟化






由于多路径软件是需要和存储在一起配合使用的,不同的厂商基于不同的操作系统,都提供了不同的版本。并且有的厂商,软件和硬件也不是一起卖的,如果要使用多路径软件的话,可能还需要向厂商购买license才行。比如EMC公司基于linux下的多路径软件,就需要单独的购买license。

其中,EMC提供的就是PowerPath,HDS提供的就是HDLM。当然,使用系统自带的免费多路径软件包,同时也是一个比较通用的包,可以支持大多数存储厂商的设备,即使是一些不是出名的厂商,通过对配置文件进行稍作修改,也是可以支持并运行的很好的。

※ 请与IBM的RDAC、Qlogic的failover驱动区分开,它们都仅提供了Failover的功能,不支持Load Balance方式。但multipath根据选择的策略不同,可支持多种方式,如:Failover、Multipath等。





-------------------------- 多路径常用命令


rpm -ivh device-mapper-multipath-libs-0.4.9-72.el6.x86_64.rpm


rpm -ivh device-mapper-multipath-0.4.9-72.el6.x86_64.rpm




[root@raclhr-12cR1-N1 Packages]# rpm -qa|grep device-mapper


device-mapper-multipath-0.4.9-72.el6.x86_64


device-mapper-persistent-data-0.2.8-2.el6.x86_64


device-mapper-1.02.79-8.el6.x86_64


device-mapper-event-libs-1.02.79-8.el6.x86_64


device-mapper-event-1.02.79-8.el6.x86_64


device-mapper-multipath-libs-0.4.9-72.el6.x86_64


device-mapper-libs-1.02.79-8.el6.x86_64






rpm -qa|grep multipath


modprobe dm-multipath


modprobe dm-round-robin


lsmod |grep multipath


chkconfig  --level 2345 multipathd on


chkconfig  --list|grep multipathd


chkconfig --list multipathd


/sbin/mpathconf


service multipathd restart


#/etc/init.d/multipathd restart


ps -ef|grep multipathd


/sbin/mpathconf --enable --find_multipaths y --with_module y --with_chkconfig y


ll /etc/multipath.conf


multipath -F


multipath -v2


more /etc/multipath/wwids


multipath -ll


dmsetup ls|sort


ll /dev/dm-*


multipath -v3 -ll


multipathd -k


service multipathd reload






---------------获取wwid


--在RHEL 6中,可以通过如下方式获取磁盘wwid:


for i in `cat /proc/partitions | awk {'print $4'} |grep sd`; do echo "### $i: `scsi_id --whitelist /dev/$i`"; done


-- 在RHEL 5中,可以通过如下方式获取磁盘wwid:


for i in `cat /proc/partitions | awk {'print $4'} |grep sd`; do echo "### $i: `scsi_id -g -u -s /block/$i`"; done




编辑/etc/multipath.conf


for i in f g h i j k l m ;

do

echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted  --device=/dev/\$name\",RESULT==\"`scsi_id --whitelisted  --device=/dev/sd$i`\",NAME=\"asm-disk$i\",OWNER=\"grid\",GROUP=\"asmadmin\",MODE=\"0660\""

done



配置udev规则

脚本如下所示:


for i in f g h i j k l m ;

do

echo "KERNEL==\"dm-*\", BUS==\"block\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\",RESULT==\"`scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\",NAME=\"asm-disk$i\",OWNER=\"grid\",GROUP=\"asmadmin\",MODE=\"0660\"" >> /etc/udev/rules.d/99-oracleasm.rules

done










​Linux MultiPath多路径软件实施说明​




原文地址:​​Linux MultiPath多路径软件实施说明​​​ 作者:​​kuqlan​






Multipath的工作原理

当multipath启动的时候,它通过系统命令scsi_id -eg -s /block/sdX得到proc/partitions 里面所有块设备的 UUID(universally unique identify),然后把所有具有同一个UUID的块设备组成一个Group,在/dev/mapper 生产一个对应的单独的设备。当设备生成后就可以使用fdisk或者parted进行分区,分区可以使用kpartx命令进行注册,然后就可以是用系统命令进行创建文件系统和mount 。


一般在系列安装过程中都会同时安装device-mapper软件包,使用linux自带命令rpm查询一下是否已经安装次软件包。使用命令 rpm –ivh 进行安装,安装以下rpm包:

device-mapper-*

device-mapper-multipath-*

device-mapper-1*


启动多路径及配置多路径

启动多路径,通过mpathconf命令创建默认模板。创建默认配置,启动和激活multipathd进程,可以使用以下命令:

mpathconf --enable --with_multipathd y

配置完成后建议重新启动多路径软件:

/etc/init.d/multipathd restart


备注:启动完成后会在/etc/下生成multipath.conf 文件以及在/etc/multipath下生成bindings 及wwid,其中wwid记录了系统中所有挂载盘的盘唯一ID,Bindings记录了ID对应的映射盘一般绑定完后会在/dev/mapper下产生类似于mpathx等文件


备注-----------------------------------------------------------------------------------------------------

如果没有没有生成可以按照Deploying Oracle RAC 11g R2 Database on Red Hat EnterpriseLinux 6

的如下方式来进行操作:

1. As the root user, install the device-mapper-multipath package using the yum package

manager.

# yum install device-mapper-multipath

2. Copy the multipath.conf file found within /usr/share/doc/device-mapper-multipath-0.4.9/

to /etc/

# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/

3. Capture the scsi id of the local disk(s) on the system.

# scsi_id --whitelisted --replace-whitespace –-device=/dev/sda

3600508b1001030353434363646301200

4. Uncomment and modify the blacklist section within the /etc/multipath.conf file to include

the scsi id of the local disk on the system. Once complete, save the changes made to

the multipath.conf file.

blacklist {

wwid 3600508b1001030353434363646301200

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

devnode "^hd[a-z]"

}

5. Start the multipath daemon.

# service multipathd start

Starting multipathd daemon: [ OK ]

6. Enable the multipath daemon to ensure it is started upon boot time.

# chkconfig multipathd on

7. Identify the dm- device, size, and WWID of each device mapper volume for Oracle

OCR and voting disks, data disks and recovery disks. In this example, volume mpathb

is identified via the following command:

# multipath -ll


Figure  Multipath Device (mpathb)


8. Uncomment the defaults section found within the /etc/multipath.conf file.

defaults {

udev_dir /dev

polling_interval 10

path_selector "round-robin 0"

path_grouping_policy multibus

getuid_callout "/lib/udev/scsi_id --whitelisted

--device=/dev/%n"

prio alua

path_checker readsector0

rr_min_io 100

max_fds 8192

rr_weight priorities

failback immediate

no_path_retry fail

user_friendly_names yes

}

NOTE: The standard options can be customized to better fit your storage array's

capabilities. Check with your storage vendor for details.


9. Un-comment the multipath section found within the /etc/multipath.conf file and create

an alias for each device mapper volume in order to enable persistent naming of those

volumes. Once complete, save the changes made to the multipath.conf file. The output

should resemble the example below. For reference, refer the Oracle data volumes

created for the reference environment as displayed in Table 2.4.3: Oracle OCR,

Voting Disk, & Data File Sizes.

/etc/multipath.conf

multipaths {

multipath {

wwid 3600c0ff000d7e7a854a0f65101000000

alias db1

}

multipath {

wwid 3600c0ff000dabfe562a0f65101000000

alias db2

}

multipath {

wwid 3600c0ff000d7e7a874a0f65101000000

alias fra

}

multipath {

wwid 3600c0ff000dabfe585a0f65101000000

alias redo

}

multipath {

wwid 3600c0ff000dabfe596a0f65101000000

alias ocrvote1

}

multipath {

wwid 3600c0ff000dabfe5a2a0f65101000000

alias ocrvote2

}

multipath {

wwid 3600c0ff000dabfe5b4a0f65101000000

alias ocrvote3

}

}

10.Restart the device mapper multipath daemon.

# service multipathd restart

ok

Stopping multipathd daemon: [ OK ]

Starting multipathd daemon: [ OK ]

11.Verify the device mapper paths and aliases are displayed properly. Below is an

example of one device mapper device labeled fra.

# multipath -ll

fra (3600c0ff000d7e7a89e85ac5101000000) dm-10 HP,MSA2324fc

size=186G features='1 queue_if_no_path' hwhandler='0' wp=rw

|-+- policy='round-robin 0' prio=130 status=active

| |- 3:0:0:3 sdd 8:48 active ready running

| |- 3:0:1:3 sdh 8:112 active ready running

| |- 4:0:0:3 sdt 65:48 active ready running

| `- 4:0:1:3 sdx 65:112 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

|- 3:0:2:3 sdl 8:176 active ready running

|- 3:0:3:3 sdp 8:240 active ready running

|- 4:0:2:3 sdab 65:176 active ready running

`- 4:0:3:3 sdaf 65:240 active ready running


----------------------------------------------------------------------------------------------



配置文件


          默认情况下,DM-Multipath 为大多数多路径的使用提供配置值。另外,DM-Multipath 支持大多数常见的、支持 DM-Multipath 的存储阵列。默认配置值和支持的设备请参考 /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults 文件。

您可以编辑 /etc/multipath.conf 覆盖 DM-Multipath 的默认配置。如果有必要,您还可以默认配置文件不支持的存储阵列。


多路径配置文件可分为以下几个部分:

blacklist

不被视为多路径的具体设备列表。

blacklist_exceptions

根据 blacklist 部分中的参数列出不在黑名单中的多路径设备。

defaults

DM-Multipath 的常规默认设置。

multipaths

各个独立多路径设备的特性设置。这些数值覆盖了在配置文件的 defaults 和 devices 部分中指定的数值。

devices

各个存储控制器的设置。这些数值覆盖了在配置文件的 defaults 部分指定的数值。如果您要使用不是默认支持的存储阵列,您可能需要为您的阵列创建 devices 子部分。

当系统决定多路径设备的属性时,首先它会检查多路径设置,然后是每个设备的设置,然后才是多路径系统默认设置。



以上配置完成后,启动multipathd 进程,进程启动后会自动scan设备,在/dev/mapper目录下可以看到 生成的设备 如:mapth4

  1. 使用命令multipath 可以查看路径状态是否正常,

[root@dbserv1 mapper]# multipath -d -l


  1. 使用系统fdisk 命令分区后,使用命令  kpartx l   /dev/mapper/mpath4查看分区信息


  1. 如果未发现可以使用kpartx d /dev/mapper/mpath4命令进行注册分区
  2. 如果下使用ASM磁盘管理,则下一步就是通过udev进行配置分区权限和别名,以便让Oracle在ASM配置过程中能够看到这些分区。关于udev的具体配置过程在下篇文章中会进行说明。
  3. 当然也可以使用Oracle ASMLib来配置ASM磁盘,这个相对udev方式来时稍微麻烦,但是也有相应的好处,即,在ASM里多余出来空间通过配置能够让OS文件系统识别到。


  1. 如果不用Rac,在单节点环境,则使用mke2fs –j /dev/mapper/mapth4p1 在相对应的分区上创建文件系统
  2. 使用mount的命令挂载文件系统就可以用。


多路径基本维护命令

1.启停多路径:

/etc/init.d/multipathd stop/start/reatart

2.查看当前多路径链路状态及盘情况:

multipath –ll


3.删除多路径设备:

multipath –F

4.重新扫描设备:

multipath –v3


5. 当在 DM-Multipath中添加新设备时,这些新设备会位于 /dev目录的两个不同位置:

/dev/mapper/mpathn

/dev/dm-n

/dev/mapper中的设备是在引导过程中生成的。可使用这些设备访问多路径设备,例如在生成逻辑卷时

所有 /dev/dm-n格式的设备都只能是作为内部使用,请不要使用它们






  ​​Linux平台的多路径软件multipath的使用案例​




原文地址:​​Linux平台的多路径软件multipath的使用案例​​​ 作者:​​尛样儿​






        在Linux和AIX平台都有一部分存储产品使用操作系统自带的多路径软件,包括最常见的HP和IBM的部分存储产品,在Linux自带的多路径软件叫做multipath,这篇文章以IBM N系列存储在Linux平台的使用为例,讨论Linux平台multipath的使用。


1.确保安装以下的包:



device-mapper             



device-mapper-multipath   







2.编辑配置文件/etc/multipath.conf



[root@rac01 ~]# vi /etc/multipath.conf







defaults



{



       user_friendly_names       yes



       max_fds                   max



       queue_without_daemon      no



       flush_on_last_del         yes



}







## Blacklist non-SAN devices



#sample:



#devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"



#devnode "^hd[a-z]"



#devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"







blacklist



{



       wwid    3600605b002b6b890163d47661e8d4fbe



}







# FCP configuration, for a NetApp FAS3xxx / IBM Nxx00



# Vendor is "NETAPP  ", spaces are important



# Product is "LUN", though "*" will do







devices



{



       device



       {



              vendor                     "NETAPP"



              product                    "LUN"



              getuid_callout             "/sbin/scsi_id -g -u -s /block/%n"



              prio_callout               "/sbin/mpath_prio_ontap /dev/%n"



              features                   "1 queue_if_no_path"



              hardware_handler           "0"



              path_grouping_policy       group_by_prio



              failback                   immediate



              rr_weight                  uniform



              rr_min_io                  128



              path_checker               directio



              }



}



#下面的注释配置不是必须的,配置之后可以固定系统设备的名称,这在服务器重启设备文件对应的磁盘设备发生变化的情况下使用。



#multipaths 



#{



#              multipath {



#                                 wwid 360a9800065344e6f465a6d5773745558



#                                 alias mocr1



#                              }



#              multipath {



#                                 wwid 360a9800065344e6e536f6d586d306630



#                                 alias mocr2



#                              }



#}











3.启动multipathd服务,及设置其自动启动。



执行以下的命令:



#service multipathd restart



#chkconfig --level 345 multipathd on



#chkconfig --list | grep multipathd







4.检查multipath聚合后的设备名,以及设备对应的链路情况。



[root@rac2 ~]# multipath -ll



mpath2 (360a9800065344e6f465a6d5773747545) dm-3 NETAPP,LUN



size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:5  sdf 8:80  active ready  running



  `- 2:0:0:5  sdp 8:240 active ready  running



mpath1 (360a9800065344e6f465a6d5773746878) dm-1 NETAPP,LUN



size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:3  sdd 8:48  active ready  running



  `- 2:0:0:3  sdn 8:208 active ready  running



mpath0 (360a9800065344e6f465a6d5773745558) dm-0 NETAPP,LUN



size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:1  sdb 8:16  active ready  running



  `- 2:0:0:1  sdl 8:176 active ready  running



mpath9 (360a9800065344e6e536f6d586d33666e) dm-9 NETAPP,LUN



size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:10 sdk 8:160 active ready  running



  `- 2:0:0:10 sdu 65:64 active ready  running



mpath8 (360a9800065344e6e536f6d586d32766b) dm-7 NETAPP,LUN



size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:8  sdi 8:128 active ready  running



  `- 2:0:0:8  sds 65:32 active ready  running



mpath7 (360a9800065344e6e536f6d586d313851) dm-6 NETAPP,LUN



size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:6  sdg 8:96  active ready  running



  `- 2:0:0:6  sdq 65:0  active ready  running



mpath6 (360a9800065344e6e536f6d586d307464) dm-4 NETAPP,LUN



size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:4  sde 8:64  active ready  running



  `- 2:0:0:4  sdo 8:224 active ready  running



mpath5 (360a9800065344e6e536f6d586d306630) dm-2 NETAPP,LUN



size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:2  sdc 8:32  active ready  running



  `- 2:0:0:2  sdm 8:192 active ready  running



mpath4 (360a9800065344e6f465a6d577376304f) dm-8 NETAPP,LUN



size=3.9T features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:9  sdj 8:144 active ready  running



  `- 2:0:0:9  sdt 65:48 active ready  running



mpath3 (360a9800065344e6f465a6d5773766c58) dm-5 NETAPP,LUN



size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw



`-+- policy='round-robin 0' prio=1 status=active



  |- 1:0:0:7  sdh 8:112 active ready  running



  `- 2:0:0:7  sdr 65:16 active ready  running




        从以上的结果可以看出,每个磁盘设备对应的是两条链路,根据不存的存储以及不同的配置会有所不同。每个设备都有两个multipath设备名,分别是mpath[x]和dm-[x],mpath位于/dev/mpath目录下,dm-[x]位于/dev目录下。通常推荐使用/dev/mpath目录下的设备名,这对于我们来说更加的方便。







5.下面我们重点关注一下fdisk -l执行的结果。



[root@rac2 ~]# fdisk -l







Disk /dev/sda: 145.9 GB, 145999527936 bytes



255 heads, 63 sectors/track, 17750 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sda1   *           1          64      514048+  83  Linux



/dev/sda2              65        6438    51199155   83  Linux



/dev/sda3            6439        7743    10482412+  82  Linux swap / Solaris



/dev/sda4            7744       17750    80381227+   5  Extended



/dev/sda5            7744       17750    80381196   83  Linux







Disk /dev/sdb: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdb1               1         261     2096451   83  Linux







Disk /dev/sdd: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdd1               1         261     2096451   83  Linux







Disk /dev/sdf: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdf1               1         261     2096451   83  Linux







Disk /dev/sdc: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdc1               1         261     2096451   83  Linux







Disk /dev/sde: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sde1               1         261     2096451   83  Linux







Disk /dev/sdh: 161.0 GB, 161061273600 bytes



255 heads, 63 sectors/track, 19581 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdh1               1       19581   157284351   83  Linux







Disk /dev/sdg: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdg1               1         261     2096451   83  Linux







Disk /dev/sdi: 161.0 GB, 161061273600 bytes



255 heads, 63 sectors/track, 19581 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdi1               1       19581   157284351   83  Linux







WARNING: GPT (GUID Partition Table) detected on '/dev/sdj'! The util fdisk doesn't support GPT. Use GNU Parted.











WARNING: The size of this disk is 4.2 TB (4241280204800 bytes).



DOS partition table format can not be used on drives for volumes



larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 



partition table format (GPT).











Disk /dev/sdj: 4241.2 GB, 4241280204800 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdj1               1      267350  2147483647+  ee  EFI GPT







WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util fdisk doesn't support GPT. Use GNU Parted.











WARNING: The size of this disk is 4.2 TB (4241280204800 bytes).



DOS partition table format can not be used on drives for volumes



larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 



partition table format (GPT).











Disk /dev/sdk: 4241.2 GB, 4241280204800 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdk1               1      267350  2147483647+  ee  EFI GPT







Disk /dev/sdl: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdl1               1         261     2096451   83  Linux







Disk /dev/sdm: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdm1               1         261     2096451   83  Linux







Disk /dev/sdn: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdn1               1         261     2096451   83  Linux







Disk /dev/sdp: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdp1               1         261     2096451   83  Linux







Disk /dev/sdo: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdo1               1         261     2096451   83  Linux







Disk /dev/sdr: 161.0 GB, 161061273600 bytes



255 heads, 63 sectors/track, 19581 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdr1               1       19581   157284351   83  Linux







Disk /dev/sdq: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdq1               1         261     2096451   83  Linux







Disk /dev/sds: 161.0 GB, 161061273600 bytes



255 heads, 63 sectors/track, 19581 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sds1               1       19581   157284351   83  Linux







WARNING: GPT (GUID Partition Table) detected on '/dev/sdt'! The util fdisk doesn't support GPT. Use GNU Parted.











WARNING: The size of this disk is 4.2 TB (4241280204800 bytes).



DOS partition table format can not be used on drives for volumes



larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 



partition table format (GPT).











Disk /dev/sdt: 4241.2 GB, 4241280204800 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdt1               1      267350  2147483647+  ee  EFI GPT







WARNING: GPT (GUID Partition Table) detected on '/dev/sdu'! The util fdisk doesn't support GPT. Use GNU Parted.











WARNING: The size of this disk is 4.2 TB (4241280204800 bytes).



DOS partition table format can not be used on drives for volumes



larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 



partition table format (GPT).











Disk /dev/sdu: 4241.2 GB, 4241280204800 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







   Device Boot      Start         End      Blocks   Id  System



/dev/sdu1               1      267350  2147483647+  ee  EFI GPT







        这之前的设备是/dev/sd[n]的设备名,这部分设备是没有聚合的设备,根据上面显示的两条链路,那么相同的LUN应该对应两个sd[n]设备,这部分不是使用的设备,可以不关注。







Disk /dev/dm-0: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-0p1               1         261     2096451   83  Linux







Disk /dev/dm-1: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-1p1               1         261     2096451   83  Linux







Disk /dev/dm-2: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-2p1               1         261     2096451   83  Linux







Disk /dev/dm-3: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-3p1               1         261     2096451   83  Linux







Disk /dev/dm-4: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-4p1               1         261     2096451   83  Linux







Disk /dev/dm-5: 161.0 GB, 161061273600 bytes



255 heads, 63 sectors/track, 19581 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-5p1               1       19581   157284351   83  Linux







Disk /dev/dm-6: 2147 MB, 2147483648 bytes



255 heads, 63 sectors/track, 261 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-6p1               1         261     2096451   83  Linux







Disk /dev/dm-7: 161.0 GB, 161061273600 bytes



255 heads, 63 sectors/track, 19581 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-7p1               1       19581   157284351   83  Linux







WARNING: GPT (GUID Partition Table) detected on '/dev/dm-8'! The util fdisk doesn't support GPT. Use GNU Parted.











WARNING: The size of this disk is 4.2 TB (4241280204800 bytes).



DOS partition table format can not be used on drives for volumes



larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 



partition table format (GPT).











Disk /dev/dm-8: 4241.2 GB, 4241280204800 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-8p1               1      267350  2147483647+  ee  EFI GPT







WARNING: GPT (GUID Partition Table) detected on '/dev/dm-9'! The util fdisk doesn't support GPT. Use GNU Parted.











WARNING: The size of this disk is 4.2 TB (4241280204800 bytes).



DOS partition table format can not be used on drives for volumes



larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID 



partition table format (GPT).











Disk /dev/dm-9: 4241.2 GB, 4241280204800 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







     Device Boot      Start         End      Blocks   Id  System



/dev/dm-9p1               1      267350  2147483647+  ee  EFI GPT







        这部分设备分别是/dev/dm-[n]以及它们的分区/dev/dm-[n]p1,这部分是安装好multipath多路径软件及针对每个磁盘设备分区后产生的,但实际在/dev/目录下不存在dm-[n]p1这部分设备文件,只需要了解即可。







Disk /dev/dm-10: 2146 MB, 2146765824 bytes



255 heads, 63 sectors/track, 260 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-10 doesn't contain a valid partition table







Disk /dev/dm-11: 2146 MB, 2146765824 bytes



255 heads, 63 sectors/track, 260 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-11 doesn't contain a valid partition table







Disk /dev/dm-12: 2146 MB, 2146765824 bytes



255 heads, 63 sectors/track, 260 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-12 doesn't contain a valid partition table







Disk /dev/dm-13: 2146 MB, 2146765824 bytes



255 heads, 63 sectors/track, 260 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-13 doesn't contain a valid partition table







Disk /dev/dm-14: 161.0 GB, 161059175424 bytes



255 heads, 63 sectors/track, 19580 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-14 doesn't contain a valid partition table







Disk /dev/dm-15: 2146 MB, 2146765824 bytes



255 heads, 63 sectors/track, 260 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-15 doesn't contain a valid partition table







Disk /dev/dm-16: 161.0 GB, 161059175424 bytes



255 heads, 63 sectors/track, 19580 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-16 doesn't contain a valid partition table







Disk /dev/dm-17: 4241.2 GB, 4241280170496 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-17 doesn't contain a valid partition table







Disk /dev/dm-18: 4241.2 GB, 4241280170496 bytes



255 heads, 63 sectors/track, 515639 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-18 doesn't contain a valid partition table







Disk /dev/dm-19: 2146 MB, 2146765824 bytes



255 heads, 63 sectors/track, 260 cylinders



Units = cylinders of 16065 * 512 = 8225280 bytes







Disk /dev/dm-19 doesn't contain a valid partition table







        这部分设备名是/dev/dm-[n],这部分设备并不是multipath多路径软件聚合后的磁盘设备,而是做完分区的分区设备文件,再刚对前面的/dev/dm-0,dev/dm-1做完分区之后执行fdisk -l是看不到这部分设备信息的,只有重启之后才看到这些设备信息并产生了相应的设备文件。例如,/dev/dm-10对应的是/dev/dm-0p1,/dev/dm-11对应的是/dev/dm-1p1,以此类推,这部分设备文件可以被使用,但由于这些设备文件对应multipath聚合的磁盘设备比较麻烦,所以并不推荐使用这种方式。


下面讨论/dev/mpath目录下的设备文件:



        安装完multipath软件并配置完成之后会在/dev/mpath目录下生成聚合后的磁盘设备文件,实际是/dev/dm-[n]的链接文件:



  [root@rac2 mpath]# ls -al



total 0



drwxr-xr-x  2 root root  440 Sep 21 02:05 .



drwxr-xr-x 13 root root 7060 Sep 21 02:12 ..



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath0 -> ../dm-0



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath0p1 -> ../dm-19



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath1 -> ../dm-1



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath1p1 -> ../dm-10



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath2 -> ../dm-2



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath2p1 -> ../dm-17



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath3 -> ../dm-5



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath3p1 -> ../dm-12



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath4 -> ../dm-7



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath4p1 -> ../dm-14



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath5 -> ../dm-3



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath5p1 -> ../dm-18



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath6 -> ../dm-4



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath6p1 -> ../dm-11



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath7 -> ../dm-6



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath7p1 -> ../dm-13



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath8 -> ../dm-9



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath8p1 -> ../dm-16



lrwxrwxrwx  1 root root    7 Sep 21 02:05 mpath9 -> ../dm-8



lrwxrwxrwx  1 root root    8 Sep 21 02:05 mpath9p1 -> ../dm-15




        从上面的输出结果我们可以非常容易的了解到磁盘分区设备文件与磁盘设备文件之间的对应关系,所以使用/dev/mpath目录下的文件比使用/dev/目录下磁盘设备和磁盘分区设备文件都以dm-[n]方式表示的文件更加方便。之后我们在使用oracleasm创建ASM磁盘的时候,使用fdisk -l /dev/mpath/mpath[n]p1查看磁盘信息,直接使用/dev/mpath/mpath[n]p1作为ASM磁盘,非常的方便。







还需要注意如下几点:



1).经过multipath多路径软件聚合后的设备文件(/dev/mpath/mpath[n])不会随着服务器的重启,对应的磁盘设备发生变化,而/dev/sd[n]对应的磁盘LUN重启后是可能发生变化的。




2).如果单个磁盘大于2TB,那么在分区的时候不要使用fdisk工具(传统的MBR),应该使用parted(gpt),使用parted工具创建gpt分区的步骤是:



#parted



PARTED> select /dev/mpath/mpath0



PARTED> mklabel gpt



PARTED> mkpart primary 0 100%




3).如果我们使用ASM作为数据库数据文件的存储方式,那么单个ASM磁盘的大小不能超过2TB,所以在系统分区的时候就应该做好更小的分区,例如,/dev/mpath/mpath0磁盘的大小为3.8TB,那么在分区的时候应该做如下操作:



#parted



PARTED> select /dev/mpath/mpath0



PARTED> mklabel gpt



PARTED> mkpart primary 0 50%



PARTED> mkpart primary 50% 100%



        完成以上分区操作之后,重启服务器,在/dev/mpath/目录下就会看到mpath0p1和mpath0p2两个设备文件,将这两个设备文件作为ASM磁盘即可。







--end--










About Me


...............................................................................................................................

● QQ群:230161599  

...............................................................................................................................

拿起手机使用微信客户端扫描下边的左边图片来关注小麦苗的微信公众号:xiaomaimiaolhr,扫描右边的二维码加入小麦苗的QQ群,学习最实用的数据库技术。

【ASM】多路径(multipath)简介及配置_linux


​ ​

【ASM】多路径(multipath)简介及配置_linux_02

【ASM】多路径(multipath)简介及配置_oracle_03

【ASM】多路径(multipath)简介及配置_linux_04