这里有一套VMware的测试环境，上面部署了一套oracle 12c RAC。由于esxi主机上的空间有限，导致rac不能扩容。刚好用一台闲置机做了一个NAS服务器，使用ISCSI的方式共享给rac节点，再通过Linux的dd命令对现有的rac共享存储进行替换来实现迁移。
1、配置ISCSI服务端
这个是在oracle linux 7上使用targetcli来实现，具体配置过程参考我的另外一篇博文：Configure iSCSI Target on RHEL7。配置完后的信息如下图：
2、RAC节点配置ISCSI客户端
确保各个节点安装了iscsi-initiator-utils软件包，如果没有通过yum进行安装。然后在各个RAC节点分别执行下面的操作：
- 设置客户端InitiatorName 编辑initiatorname.iscsi文件，设置友好的名称，如下：
[root@odb02 ~]# vi /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2019-07.com.12c:odb02
[root@odb03 ~]# vi /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2019-07.com.12c:odb03
设置完成后，重启iscsid服务，如下：
[root@odb02 ~]# for i in enable restart status;do systemctl $i iscsid;done
[root@odb03 ~]# for i in enable restart status;do systemctl $i iscsid;done
- 设置客户端访问ISCSI服务端
[root@odb02 ~]# iscsiadm -m discovery -t sendtargets -p 192.168.120.9:3260
[root@odb02 ~]# iscsiadm -m node -T iqn.2019-07.com.12c:lun1 -p 192.168.120.9:3260 -l
[root@odb02 ~]# lsscsi -g
[2:0:0:0] disk VMware Virtual disk 1.0 /dev/sda /dev/sg0
[3:0:0:0] disk VMware Virtual disk 1.0 /dev/sdb /dev/sg1
[3:0:1:0] disk VMware Virtual disk 1.0 /dev/sdc /dev/sg2
[8:0:0:0] disk LIO-ORG sharedisk01 4.0 /dev/sdd /dev/sg3
[8:0:0:1] disk LIO-ORG sharedisk02 4.0 /dev/sde /dev/sg4
[root@odb03 ~]# lsscsi -g
[0:0:0:0] disk VMware Virtual disk 1.0 /dev/sda /dev/sg0
[3:0:0:0] disk VMware Virtual disk 1.0 /dev/sdb /dev/sg1
[3:0:1:0] disk VMware Virtual disk 1.0 /dev/sdc /dev/sg2
[4:0:0:0] disk LIO-ORG sharedisk01 4.0 /dev/sdd /dev/sg3
[4:0:0:1] disk LIO-ORG sharedisk02 4.0 /dev/sde /dev/sg4
3、停止ORACLE RAC服务
在任意一个rac节点以root身份停止rac所有服务：
[root@odb02 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'odb03'
CRS-2673: Attempting to stop 'ora.crsd' on 'odb02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'odb03'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'odb03'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'odb02'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'odb02'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'odb02'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'odb02'
CRS-2677: Stop of 'ora.racdb.db' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'odb03'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'odb03'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'odb03'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.odb03.vip' on 'odb03'
CRS-2677: Stop of 'ora.racdb.db' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'odb02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'odb02'
CRS-2673: Attempting to stop 'ora.cvu' on 'odb02'
CRS-2677: Stop of 'ora.DATA.dg' on 'odb03' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'odb02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'odb03'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.odb02.vip' on 'odb02'
CRS-2677: Stop of 'ora.asm' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'odb03'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'odb02'
CRS-2677: Stop of 'ora.odb03.vip' on 'odb03' succeeded
CRS-2677: Stop of 'ora.cvu' on 'odb02' succeeded
CRS-2677: Stop of 'ora.odb02.vip' on 'odb02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'odb02' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'odb02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'odb02'
CRS-2673: Attempting to stop 'ora.ons' on 'odb03'
CRS-2677: Stop of 'ora.ons' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'odb03'
CRS-2677: Stop of 'ora.net1.network' on 'odb03' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'odb03' has completed
CRS-2677: Stop of 'ora.FRA.dg' on 'odb02' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'odb02'
CRS-2677: Stop of 'ora.asm' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'odb02'
CRS-2677: Stop of 'ora.crsd' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'odb03'
CRS-2673: Attempting to stop 'ora.evmd' on 'odb03'
CRS-2673: Attempting to stop 'ora.storage' on 'odb03'
CRS-2677: Stop of 'ora.storage' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'odb03'
CRS-2677: Stop of 'ora.ctssd' on 'odb03' succeeded
CRS-2677: Stop of 'ora.evmd' on 'odb03' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'odb02' succeeded
CRS-2677: Stop of 'ora.asm' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'odb03'
CRS-2677: Stop of 'ora.qosmserver' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'odb02'
CRS-2677: Stop of 'ora.ons' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'odb02'
CRS-2677: Stop of 'ora.net1.network' on 'odb02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'odb02' has completed
CRS-2677: Stop of 'ora.crsd' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'odb02'
CRS-2673: Attempting to stop 'ora.evmd' on 'odb02'
CRS-2673: Attempting to stop 'ora.storage' on 'odb02'
CRS-2677: Stop of 'ora.storage' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'odb02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'odb03' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'odb03'
CRS-2677: Stop of 'ora.ctssd' on 'odb02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'odb02' succeeded
CRS-2677: Stop of 'ora.cssd' on 'odb03' succeeded
CRS-2677: Stop of 'ora.asm' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'odb02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'odb02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'odb02'
CRS-2677: Stop of 'ora.cssd' on 'odb02' succeeded
4、同步旧盘数据至ISCSI盘
由于是两块磁盘以及两个rac节点，所以这里在两个节点分别执行同步操作，如下：
[root@odb02 ~]# dd if=/dev/sdc of=/dev/sde bs=4096k
25600+0 records in
25600+0 records out
107374182400 bytes (107 GB) copied, 8172.08 s, 13.1 MB/s
[root@odb03 ~]# dd if=/dev/sdb of=/dev/sdd bs=4096k
51200+0 records in
51200+0 records out
214748364800 bytes (215 GB) copied, 12859.5 s, 16.7 MB/s
总共耗时3.5小时。
5、删除旧盘并识别新盘
首先使用操作系统oracleasm命令删除，然后执行扫描操作识别新的磁盘。
[root@odb02 ~]# oracleasm deletedisk data01
[root@odb02 ~]# oracleasm deletedisk fra001
[root@odb02 ~]# oracleasm scandisks
[root@odb02 ~]# oracleasm listdisks
DATA01
FRA001
[root@odb03 ~]# oracleasm scandisks
[root@odb03 ~]# oracleasm listdisks
DATA01
FRA001
6、启动Oracle RAC服务
[root@odb02 ~]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.evmd' on 'odb03'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'odb03'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'odb02'
CRS-2672: Attempting to start 'ora.evmd' on 'odb02'
CRS-2676: Start of 'ora.cssdmonitor' on 'odb03' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'odb03'
CRS-2672: Attempting to start 'ora.diskmon' on 'odb03'
CRS-2676: Start of 'ora.cssdmonitor' on 'odb02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'odb02'
CRS-2672: Attempting to start 'ora.diskmon' on 'odb02'
CRS-2676: Start of 'ora.diskmon' on 'odb02' succeeded
CRS-2676: Start of 'ora.diskmon' on 'odb03' succeeded
CRS-2676: Start of 'ora.evmd' on 'odb02' succeeded
CRS-2676: Start of 'ora.evmd' on 'odb03' succeeded
CRS-2676: Start of 'ora.cssd' on 'odb03' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'odb03'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'odb03'
CRS-2676: Start of 'ora.cssd' on 'odb02' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'odb02'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'odb02'
CRS-2676: Start of 'ora.ctssd' on 'odb03' succeeded
CRS-2676: Start of 'ora.ctssd' on 'odb02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'odb03' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'odb03'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'odb02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'odb02'
CRS-2676: Start of 'ora.asm' on 'odb02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'odb02'
CRS-2676: Start of 'ora.asm' on 'odb03' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'odb03'
CRS-2676: Start of 'ora.storage' on 'odb03' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'odb03'
CRS-2676: Start of 'ora.crsd' on 'odb03' succeeded
CRS-2676: Start of 'ora.storage' on 'odb02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'odb02'
CRS-2676: Start of 'ora.crsd' on 'odb02' succeeded
确保rac顺利启动后，然后关机移除虚拟机上的旧盘即可。
7、设置oracleasm.service服务
在rhel 7操作系统上，如果使用了iscsi服务，当系统重启了之后，iscsi共享盘是可以在操作系统上识别到，但oracleasm并不会去扫描磁盘，这就导致了grid发现不了磁盘，从而无法启动RAC服务。解决的办法很简单，只需编辑oracleasm.service文件，加入以下内容：
[root@odb02 ~]# vi /usr/lib/systemd/system/oracleasm.service
[Unit]
Description=Load oracleasm Modules
---以下两行为新增内容
Requires=multipathd.service iscsi.service
After=multipathd.service iscsi.service
.......
最后重启操作系统进行验证。
