节点1:

Wed Feb 13 16:08:06 2019
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb1/trace/testdb1_dia0_9267.trc (incident=1248083):
ORA-32701: Possible hangs up to hang ID=4 detected
Incident details in: /u01/app/oracle/diag/rdbms/testdb/testdb1/incident/incdir_1248083/testdb1_dia0_9267_i1248083.trc
DIA0 requesting termination of session sid:5190 with serial # 42237 (ospid:180727) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=4.
Hang Resolution Reason: Although the number of affected sessions did not
justify automatic hang resolution initially, this previously ignored
hang was automatically resolved.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=4.
Wed Feb 13 16:08:08 2019
Sweep [inc][1248083]: completed
Sweep [inc2][1248083]: completed
Wed Feb 13 16:09:41 2019
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb1/trace/testdb1_dia0_9267.trc (incident=1248084):
ORA-32701: Possible hangs up to hang ID=4 detected
Incident details in: /u01/app/oracle/diag/rdbms/testdb/testdb1/incident/incdir_1248084/testdb1_dia0_9267_i1248084.trc
DIA0 requesting termination of process sid:5190 with serial # 42237 (ospid:180727) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=4.
Previous SESSION termination failed.
Hang Resolution Reason: Although the number of affected sessions did not
justify automatic hang resolution initially, this previously ignored
hang was automatically resolved.
DIA0: Examine the alert log on instance 2 for process termination status of hang with ID=4.

[oracle@testdb1 trace]$ more /u01/app/oracle/diag/rdbms/testdb/testdb1/trace/testdb1_dia0_9267.trc
Trace file /u01/app/oracle/diag/rdbms/testdb/testdb1/trace/testdb1_dia0_9267.trc

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

Incident 1248083 created, dump file: /u01/app/oracle/diag/rdbms/testdb/testdb1/incident/incdir_1248083/testdb1_dia0_9267_i1248083.trc
ORA-32701: Possible hangs up to hang ID=4 detected

Incident 1248084 created, dump file: /u01/app/oracle/diag/rdbms/testdb/testdb1/incident/incdir_1248084/testdb1_dia0_9267_i1248084.trc
ORA-32701: Possible hangs up to hang ID=4 detected

节点2:

Wed Feb 13 16:09:41 2019
Errors in file /u01/app/oracle/diag/rdbms/testdb/testdb2/trace/testdb2_dia0_7404.trc (incident=1008107):
ORA-32701: Possible hangs up to hang ID=4 detected
Incident details in: /u01/app/oracle/diag/rdbms/testdb/testdb2/incident/incdir_1008107/testdb2_dia0_7404_i1008107.trc
DIA0 terminating blocker (ospid: 180727 sid: 5190 ser#: 42237) of hang with ID = 4

 

[oracle@testdb2 trace]$ more /u01/app/oracle/diag/rdbms/testdb/testdb2/trace/testdb2_dia0_7404.trc
Trace file /u01/app/oracle/diag/rdbms/testdb/testdb2/trace/testdb2_dia0_7404.trc

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

Incident 1008106 created, dump file: /u01/app/oracle/diag/rdbms/testdb/testdb2/incident/incdir_1008106/testdb2_dia0_7404_i1008106.trc
ORA-32701: Possible hangs up to hang ID=4 detected

Incident 1008107 created, dump file: /u01/app/oracle/diag/rdbms/testdb/testdb2/incident/incdir_1008107/testdb2_dia0_7404_i1008107.trc
ORA-32701: Possible hangs up to hang ID=4 detected

 

等待事件

inst# SessId Ser# OSPID PrcNm Event
----- ------ ----- --------- ----- -----
1 6746 23425 37352 M000 enq: WF - contention
2 5190 42237 180727 M000 not in wait

inst# SessId Ser# OSPID PrcNm Event
----- ------ ----- --------- ----- -----
1 6746 23425 37352 M000 enq: WF - contention
2 5190 42237 180727 M000 not in wait

 

---解决办法

-----MOS 上文章 2226216.1
1. Collect statistics on following fixed table:

SQL> exec dbms_stats.gather_table_stats('SYS', 'X$KEWRATTRNEW');

PL/SQL procedure successfully completed.

SQL> exec dbms_stats.gather_table_stats('SYS', 'X$KEWRSQLIDTAB');

PL/SQL procedure successfully completed.

Or

2. Restarting the database will release of X$KQLFBC table data

Or

3. Flush shared_pool on a regular basis

*** 2019-02-14 06:25:08.352
current sql: insert into wrh$_sql_bind_metadata (snap_id, dbid, sql_id, name, position, dup_position, datatype, datatype_string, characte
r_sid, precision, scale, max_length) SELECT /*+ ordered use_nl(bnd) index(bnd sql_id) */ :lah_snap_id, :dbid, bnd.sql_id, name, position,
dup_position, datatype, dataty

---处理方法

exec dbms_stats.gather_table_stats('SYS', 'X$KEWRATTRNEW');

exec dbms_stats.gather_table_stats('SYS', 'X$KEWRSQLIDTAB');

--立马生效

exec dbms_stats.gather_table_stats(ownname => 'SYS',tabname => 'X$KEWRATTRNEW',no_invalidate => FALSE);
exec dbms_stats.gather_table_stats(ownname => 'SYS',tabname => 'X$KEWRSQLIDTAB',no_invalidate => FALSE);

 

---定时任务

# flush shared_pool 每个月执行一次
33 02 15 * * /bin/sh /home/oracle/flush_shared_pool/flush_shared_pool.sh &> /dev/null

[oracle@testdb2 ~]$ cat /home/oracle/flush_shared_pool/flush_shared_pool.sh
#!/bin/bash
source /home/oracle/.bash_profile

sqlplus / as sysdba >> /home/oracle/flush_shared_pool/exec_shared_pool.log <<EOF
set timing on;
@/home/oracle/flush_shared_pool/flush_shared_pool.sql
EOF

[oracle@testdb2 ~]$ cat /home/oracle/flush_shared_pool/flush_shared_pool.sql
alter system flush shared_pool;

---还有可能是死锁引起的故障

1.业务查询程序死锁问题
2.执行刷新shared_pool--两个节点都需要执行

EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;
EXEC DBMS_STATS.GATHER_SCHEMA_STATS ('SYS');

alter system flush shared_pool;

 

--查询2个节点基表信息

select count(*) from x$ksmsp;

exec dbms_stats.gather_table_stats(ownname => 'SYS',tabname => 'X$KEWRATTRNEW',no_invalidate => FALSE);
exec dbms_stats.gather_table_stats(ownname => 'SYS',tabname => 'X$KEWRSQLIDTAB',no_invalidate => FALSE);

--绝招

alter system set "_awr_disabled_flush_tables" = 'wrh$_sql_bind_metadata';

 

日积月累