晚上接到用户电话,DG备机报错无法正常同步。ORA-00600: internal error code, arguments: [2619], [84747],了解到之前出现磁盘空间满,已经人为删除处理过;这是一个典型的DG空间满后的问题,官方文档ORA-600[2619] During Physical Standby Recovery (Doc ID 1138913.1)对此有说明;总结理解一下,问题原因为磁盘空间满,导致归档日志未完全写入而出现损失,DG备机的MRP进程无法使用此归档日志。了解了原理后,处理就比较简单了。方法有两种:

方法1.mos文章上的方法,释放磁盘空间,从主库复制报错的归档日志到备机,再进行恢复。--->>>这种方式其实在操作的多时略烦琐; 

方法2:使用rman进行归档删除(delete noprompt archivelog until time 'sysdate-2';),再使用crosscheck archivelog all;将损坏的归档日志标记EXPIRED,使用delete noprompt expired archivelog all;命令删除损坏的归档日志;此时通常会自动重新传之前错误的归档日志,后续有缺少日志时在GAP机制下也可以自动传。方法2在顺利情况下就是会3条命令刷一下,然后就等着自动恢复即可。

本次问题使用这方法处理,三条命令刷上,问题顺利解决。相关日志如下:

1.DG备机alert日志中的RA-00600: internal error code, arguments: [2619], [84747]报错信息

Clearing online redo logfile 11 complete
Media Recovery Log /oracle/arch/1_84747_958258329.dbf
Errors in file /oracle/app/oracle/diag/rdbms/bysdbdg/bysdbdg/trace/bysdbdg_pr00_11664.trc (incident=32211):
ORA-00600: internal error code, arguments: [2619], [84747], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/oracle/diag/rdbms/bysdbdg/bysdbdg/incident/incdir_32211/bysdbdg_pr00_11664_i32211.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors with log /oracle/arch/1_84747_958258329.dbf
MRP0: Background Media Recovery terminated with error 600
Errors in file /oracle/app/oracle/diag/rdbms/bysdbdg/bysdbdg/trace/bysdbdg_pr00_11664.trc:
ORA-00600: internal error code, arguments: [2619], [84747], [], [], [], [], [], [], [], [], [], []
Managed Standby Recovery not using Real Time Apply

2.对于日志操作的判断

at Apr 10 22:16:20 2021   =====>>>>报84747号日志文件操作
archivelog header validation failure for file /oracle/arch/1_84747_958258329.dbf
***
Corrupt block seq: 84747 blocknum=1.
Bad header found during deleting archived log
Data in bad block - flag:1. format:34. bno:1. seq:84747
beg:0 cks:7858
calculated check value: 7858
archivelog header validation failure for file /oracle/arch/1_84747_958258329.dbf
Reread of seq=84747, blocknum=1, file=/oracle/arch/1_84747_958258329.dbf, found same corrupt data

3.RMAN命令进行校验与删除EXPIRED状态日志

校验结果:
validation succeeded for archived log
archived log file name=/oracle/arch/1_84745_958258329.dbf RECID=27692 STAMP=1069534982
validation succeeded for archived log
archived log file name=/oracle/arch/1_84746_958258329.dbf RECID=27685 STAMP=1069534731
validation failed for archived log =====>>>>>
archived log file name=/oracle/arch/1_84747_958258329.dbf RECID=27686 STAMP=1069534732
validation failed for archived log
archived log file name=/oracle/arch/1_84748_958258329.dbf RECID=27687 STAMP=1069534733
validation succeeded for archived log

删除情况
archived log file name=/oracle/arch/1_84747_958258329.dbf RECID=27686 STAMP=1069534732
deleted archived log
archived log file name=/oracle/arch/1_84748_958258329.dbf RECID=27687 STAMP=1069534733
deleted archived log
archived log file name=/oracle/arch/1_84760_958258329.dbf RECID=27691 STAMP=1069534766
Deleted 920 EXPIRED objects

4.删除后自动重新传输归档日志

Sat Apr 10 22:17:07 2021
RFS[4]: Assigned to RFS process 11741
RFS[4]: Opened log for thread 1 sequence 84747 dbid -1999453415 branch 958258329
Archived Log entry 27784 added for thread 1 sequence 84748 rlc 958258329 ID 0x88d24019 dest 2:
Archived Log entry 27785 added for thread 1 sequence 84747 rlc 958258329 ID 0x88d24019 dest 2:

ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Clearing online redo logfile 11 complete
Media Recovery Waiting for thread 1 sequence 84753
Fetching gap sequence in thread 1, gap sequence 84753-84753
Sat Apr 10 22:18:01 2021
RFS[6]: Assigned to RFS process 11799
RFS[6]: Allowing overwrite of partial archivelog for thread 1 sequence 84753
RFS[6]: Opened log for thread 1 sequence 84753 dbid -1999453415 branch 958258329
Archived Log entry 27787 added for thread 1 sequence 84753 rlc 958258329 ID 0x88d24019 dest 2:
Sat Apr 10 22:18:11 2021
Media Recovery Log /oracle/arch/1_84753_958258329.dbf
Media Recovery Log /oracle/arch/1_84754_958258329.dbf

5.检查DG同步进程恢复正常

SQL>  select process,status,THREAD#,SEQUENCE#,BLOCK# from v$managed_standby;

PROCESS STATUS THREAD# SEQUENCE# BLOCK#
--------- ------------ ---------- ---------- ----------
ARCH CONNECTED 0 0 0
ARCH CONNECTED 0 0 0
ARCH CONNECTED 0 0 0
ARCH CONNECTED 0 0 0
RFS IDLE 0 0 0
RFS IDLE 0 0 0
RFS IDLE 1 84844 4353
RFS IDLE 0 0 0
RFS IDLE 0 0 0
MRP0 WAIT_FOR_GAP 1 84753 0

10 rows selected.

SQL> /

PROCESS STATUS THREAD# SEQUENCE# BLOCK#
--------- ------------ ---------- ---------- ----------
ARCH CONNECTED 0 0 0
ARCH CONNECTED 0 0 0
ARCH CONNECTED 0 0 0
ARCH CONNECTED 0 0 0
RFS IDLE 0 0 0
RFS IDLE 0 0 0
RFS IDLE 1 84846 4
RFS IDLE 0 0 0
RFS IDLE 0 0 0
MRP0 WAIT_FOR_LOG 1 84846 0

10 rows selected.