在node1节点发现如下问题
[grid@node1 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
而在node2节点，运行正常。
[grid@node2 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE node2
ora.FLASH.dg ora....up.type ONLINE ONLINE node2
ora.GRIDDG.dg ora....up.type ONLINE ONLINE node2
ora....ER.lsnr ora....er.type ONLINE ONLINE node2
ora....N1.lsnr ora....er.type ONLINE ONLINE node2
ora.asm ora.asm.type ONLINE ONLINE node2
ora.devdb.db ora....se.type ONLINE ONLINE node2
ora.eons ora.eons.type ONLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node2
ora.node1.vip ora....t1.type ONLINE ONLINE node2
ora....SM2.asm application ONLINE ONLINE node2
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application OFFLINE OFFLINE
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE node2
ora....ry.acfs ora....fs.type ONLINE ONLINE node2
ora.scan1.vip ora....ip.type ONLINE ONLINE node2
在node1节点检查crs服务
[grid@node1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
查看crsd.log日志，发现了如下错误/u01/app/11.2.0/grid/log/node1/crsd/crsd.log
2019-11-28 18:39:58.616: [ OCRASM][319107664]proprasmo: Error in open/create file in dg [GRIDDG]
[ OCRASM][319107664]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
2019-11-28 18:39:58.620: [ OCRASM][319107664]proprasmo: kgfoCheckMount returned [7]
2019-11-28 18:39:58.620: [ OCRASM][319107664]proprasmo: The ASM instance is down
2019-11-28 18:39:58.620: [ OCRRAW][319107664]proprioo: Failed to open [+GRIDDG]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2019-11-28 18:39:58.620: [ OCRRAW][319107664]proprioo: No OCR/OLR devices are usable
2019-11-28 18:39:58.620: [ OCRASM][319107664]proprasmcl: asmhandle is NULL
2019-11-28 18:39:58.620: [ OCRRAW][319107664]proprinit: Could not open raw device
2019-11-28 18:39:58.620: [ OCRASM][319107664]proprasmcl: asmhandle is NULL
2019-11-28 18:39:58.621: [ OCRAPI][319107664]a_init:16!: Backend init unsuccessful : [26]
2019-11-28 18:39:58.622: [ CRSOCR][319107664] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2019-11-28 18:39:58.622: [ CRSD][319107664][PANIC] CRSD exiting: Could not init OCR, code: 26
2019-11-28 18:39:58.622: [ CRSD][319107664] Done.
解决方法
清理节点1的配置信息，打算重新执行root.sh
[root@node1 ~]# /u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
2019-12-02 11:13:42: Checking for super user privileges
2019-12-02 11:13:42: User has super user privileges
2019-12-02 11:13:42: Parsing the host name
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
CRS resources for listeners are still configured
CRS-2613: Could not find resource 'ora.cssd'.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2613: Could not find resource 'ora.cssd'.
CRS-4000: Command Delete failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.GRIDDG.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.devdb.db' on 'node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node1'
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2677: Stop of 'ora.node2.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'node1'
CRS-2677: Stop of 'ora.scan1.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.GRIDDG.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.devdb.db' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'node1'
CRS-2677: Stop of 'ora.FLASH.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2673: Attempting to stop 'ora.eons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2677: Stop of 'ora.eons' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2673: Attempting to stop 'ora.diskmon' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
ACFS-9200: Supported
Successfully deconfigured Oracle Restart stack
[root@node1 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2019-12-02 11:15:49: Parsing the host name
2019-12-02 11:15:49: Checking for super user privileges
2019-12-02 11:15:49: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node2, number 2, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'node1'
CRS-2676: Start of 'ora.mdnsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node1'
CRS-2676: Start of 'ora.gipcd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node1'
CRS-2676: Start of 'ora.gpnpd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node1'
CRS-2676: Start of 'ora.cssdmonitor' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node1'
CRS-2672: Attempting to start 'ora.diskmon' on 'node1'
CRS-2676: Start of 'ora.diskmon' on 'node1' succeeded
CRS-2676: Start of 'ora.cssd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node1'
CRS-2676: Start of 'ora.ctssd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node1'
CRS-2676: Start of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node1'
CRS-2676: Start of 'ora.asm' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node1'
CRS-2676: Start of 'ora.crsd' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'node1'
CRS-2676: Start of 'ora.evmd' on 'node1' succeeded
PRKO-2190 : VIP exists for node node1, VIP name node1-vip
add vip -n node1 -k 1 -A node1-vip/255.255.255.0/eth0 on node=node1 ... failed
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... failed
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3498 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# su - grid
[grid@node1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE node1
ora.FLASH.dg ora....up.type ONLINE ONLINE node1
ora.GRIDDG.dg ora....up.type ONLINE ONLINE node1
ora....N1.lsnr ora....er.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.devdb.db ora....se.type ONLINE ONLINE node1
ora.eons ora.eons.type ONLINE ONLINE node1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node1
ora....SM1.asm application ONLINE ONLINE node1
ora.node1.gsd application OFFLINE OFFLINE
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip ora....t1.type ONLINE ONLINE node1
ora.node2.vip ora....t1.type ONLINE ONLINE node1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE node1
ora....ry.acfs ora....fs.type ONLINE ONLINE node1
ora.scan1.vip ora....ip.type ONLINE ONLINE node1
