一、先关闭GI，命令不再贴出来了，参见：
二、模拟OLR损坏
[root@hostb cdata]# ls -lrt
total 2884
drwxr-xr-x 2 grid oinstall 4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan 8 13:38 hostb.olr
[root@hostb cdata]# mv hostb.olr back_hostb.olr -----------> mv掉OLR
[root@hostb cdata]# ls -lrt
total 2884
drwxr-xr-x 2 grid oinstall 4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan 8 13:38 back_hostb.olr
[root@hostb cdata]# /u01/app/11.2.0/grid/bin/crsctl start crs --------->起不来。
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
[root@hostb cdata]# pwd
/u01/app/11.2.0/grid/cdata
[root@hostb cdata]#
[oracle@hostb ohasd]$ pwd
/u01/app/11.2.0/grid/log/hostb/ohasd
[oracle@hostb ohasd]$ ls -lrt
total 95720
-rw-r--r-- 1 root root 10540929 Nov 4 03:51 ohasd.l09
-rw-r--r-- 1 root root 10556558 Nov 8 16:28 ohasd.l08
-rw-r--r-- 1 root root 10558423 Nov 13 12:39 ohasd.l07
-rw-r--r-- 1 root root 10558697 Nov 18 05:44 ohasd.l06
-rw-r--r-- 1 root root 10557602 Nov 22 18:15 ohasd.l05
-rw-r--r-- 1 root root 10532670 Nov 27 08:24 ohasd.l04
-rw-r--r-- 1 root root 10556869 Dec 2 01:30 ohasd.l03
-rw-r--r-- 1 root root 10557883 Dec 6 23:49 ohasd.l02
-rw-r--r-- 1 root root 10542530 Dec 21 11:49 ohasd.l01
-rw-r--r-- 1 root root 6700 Jan 8 13:42 ohasdOUT.log
-rw-r--r-- 1 root root 2986628 Jan 8 13:42 ohasd.log
[oracle@hostb ohasd]$ vi ohasd.log
2016-01-08 13:42:59.562: [ default][3149834016] Initializing OLR
2016-01-08 13:42:59.564: [ OCROSD][3149834016]utopen:6m': failed in stat OCR file/disk /u01/app/11.2.0/grid/cdata/hostb.olr, errno=2, os err string=No such file or directory
2016-01-08 13:42:59.564: [ OCROSD][3149834016]utopen:7: failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2016-01-08 13:42:59.564: [ OCRRAW][3149834016]proprinit: Could not open raw device
2016-01-08 13:42:59.564: [ OCRAPI][3149834016]a_init:16!: Backend init unsuccessful : [26]
2016-01-08 13:42:59.564: [ CRSOCR][3149834016] OCR context init failure. Error: PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
2016-01-08 13:42:59.565: [ default][3149834016] Created alert : (:OHAS00106:) : OLR initialization failed, error: PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
2016-01-08 13:42:59.565: [ default][3149834016][PANIC] OHASD exiting; Could not init OLR
2016-01-08 13:42:59.565: [ default][3149834016] Done.
开始对本地节点（主机名为hostb）进行OLR的recreate,参考文章：
Top 5 Grid Infrastructure Startup Issues (文档 ID 1368382.1) ，搜索关键字为"OLR"
[root@hostb bin]# ./crsctl stop crs
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
第一步：# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force
[root@hostb install]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle clusterware stack on this node
[root@hostb install]#
此时的/u01/app/11.2.0/grid/cdata目录情况：
[grid@hostb cdata]$ ls -lrt
total 13124
drwxr-xr-x 2 grid oinstall 4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan 8 13:38 back_hostb.olr
-rw-r--r-- 1 root root 10485760 Jan 12 18:08 hostb.olr
[grid@hostb cdata]$
第二步：# /u01/app/11.2.0/grid/root.sh
[root@hostb install]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'hostb'
CRS-2676: Start of 'ora.mdnsd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hostb'
CRS-2676: Start of 'ora.gpnpd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hostb'
CRS-2672: Attempting to start 'ora.gipcd' on 'hostb'
CRS-2676: Start of 'ora.cssdmonitor' on 'hostb' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hostb'
CRS-2672: Attempting to start 'ora.diskmon' on 'hostb'
CRS-2676: Start of 'ora.diskmon' on 'hostb' succeeded
CRS-2676: Start of 'ora.cssd' on 'hostb' succeeded
PRKO-2190 : VIP exists for node hostb, VIP name hostb-vip
PRCS-1037 : Single Client Access Name VIPs already exist
PRCS-1028 : Single Client Access Name listeners already exist
OC4J could not be created as it already exists
PRCR-1086 : resource ora.oc4j is already registered
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'hostb'
PRCC-1014 : LISTENER_SCAN1 was already running
PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'hostb'
PRCC-1014 : oc4j was already running
PRCR-1004 : Resource ora.oc4j is already running
PRCR-1079 : Failed to start resource ora.oc4j
CRS-5702: Resource 'ora.oc4j' is already running on 'hostb'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'hostb'
CRS-2676: Start of 'ora.registry.acfs' on 'hostb' succeeded
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'hostb'
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@hostb install]#
完成后的/u01/app/11.2.0/grid/cdata目录情况：
[grid@hostb cdata]$ ls -lrt
total 5612
drwxr-xr-x 2 grid oinstall 4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall 4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan 8 13:38 back_hostb.olr
drwxr-xr-x 2 grid oinstall 4096 Jan 12 18:19 hostb
-rw------- 1 root oinstall 272756736 Jan 12 18:19 hostb.olr
[grid@hostb cdata]$
[grid@hostb cdata]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE hostb
ora.DATAFILE.dg
ONLINE ONLINE hostb
ora.FRA.dg
ONLINE ONLINE hostb
ora.LISTENER.lsnr
ONLINE ONLINE hostb
ora.asm
ONLINE ONLINE hostb Started
ora.gsd
OFFLINE OFFLINE hostb
ora.net1.network
ONLINE ONLINE hostb
ora.ons
ONLINE ONLINE hostb
ora.registry.acfs
ONLINE ONLINE hostb
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE hostb
ora.cvu
1 ONLINE ONLINE hostb
ora.orcl.db
1 ONLINE OFFLINE
2 ONLINE ONLINE hostb Open
ora.orcl.orclxa.svc
1 ONLINE ONLINE hostb
ora.orcla.vip
1 ONLINE INTERMEDIATE hostb FAILED OVER
ora.hostb.vip
1 ONLINE ONLINE hostb
ora.oc4j
1 ONLINE ONLINE hostb
ora.scan1.vip
1 ONLINE ONLINE hostb
[grid@hostb cdata]$
[grid@hostb cdata]$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 12-JAN-2016 18:21:34
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 12-JAN-2016 18:18:21
Uptime 0 days 0 hr. 3 min. 13 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File /u01/app/grid/diag/tnslsnr/hostb/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.182)(PORT=1528)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.184)(PORT=1528)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "orcl" has 1 instance(s).
Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclXDB" has 1 instance(s).
Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclxa" has 1 instance(s).
Instance "orcl2", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@hostb cdata]$
[oracle@hostb ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Tue Jan 12 18:29:52 2016
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ WRITE
