oracle 11gR2 启动报错：
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Process ID: 21644
Session ID: 1426 Serial number: 3
++++alter信息如下：
Fri Dec 27 14:28:59 2013
SMON: enabling cache recovery
Errors in file /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc (incident=112161):
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/incident/incdir_112161/coredb_ora_21644_i112161.trc
Fri Dec 27 14:29:06 2013
Errors in file /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Errors in file /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 21644): terminating the instance due to error 704
Instance terminated by USER, pid = 21644
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (21644) as a result of ORA-1092
Fri Dec 27 14:29:08 2013
ORA-1092 : opitsk aborting process
++++dbv检测：
[oracle@dg_136 ~]$ dbv file=/tol/oradata/datafile/coredb/system01.dbf
DBVERIFY: Release 11.2.0.1.0 - Production on Fri Dec 27 14:58:21 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
DBVERIFY - Verification starting : FILE = /tol/oradata/datafile/coredb/system01.dbf
DBVERIFY - Verification complete
Total Pages Examined : 264960
Total Pages Processed (Data) : 218039
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 22126
Total Pages Failing (Index): 0
Total Pages Processed (Other): 6165
Total Pages Processed (Seg) : 1
Total Pages Failing (Seg) : 0
Total Pages Empty : 18630
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Total Pages Encrypted : 0
Highest block SCN : 496559453 (2.496559453)
+++++查看coredb_ora_21644.trc
** ACTION NAME:() 2013-12-27 14:28:41.140
parallel recovery setup failed: using serial mode
Thread 1 checkpoint: logseq 46403, block 2, scn 9086475510
cache-low rba: logseq 46403, block 40010
on-disk rba: logseq 46403, block 43232, scn 9086494047
start recovery at logseq 46403, block 40010, scn 0
*** 2013-12-27 14:28:41.327
Started writing zeroblks thread 1 seq 46403 blocks 43232-43239
*** 2013-12-27 14:28:41.345
Completed writing zeroblks thread 1 seq 46403
*** 2013-12-27 14:28:41.360
==== Redo read statistics for thread 1 ====
Total physical reads (from disk and memory): 4096Kb
-- Redo read_disk statistics --
Read rate (ASYNC): 1611Kb in 0.22s => 7.15 Mb/sec
Longest record: 2Kb, moves: 0/6706 (0%)
Longest LWN: 109Kb, moves: 0/67 (0%), moved: 0Mb
Last redo scn: 0x0002.1d98e55e (9086494046)
----------------------------------------------
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 2
Average hash chain = 1833/1831 = 1.0
Max compares per lookup = 1
Avg compares per lookup = 9869/11700 = 0.8
----------------------------------------------
*** 2013-12-27 14:28:41.400
KCRA: start recovery claims for 1833 data blocks
*** 2013-12-27 14:28:42.430
KCRA: blocks processed = 1833/1833, claimed = 1833, eliminated = 0
*** 2013-12-27 14:28:42.448
Recovery of Online Redo Log: Thread 1 Group 5 Seq 46403 Reading mem 0
*** 2013-12-27 14:28:42.556
Completed redo application of 1.29MB
*** 2013-12-27 14:28:55.112
Completed recovery checkpoint
IR RIA: redo_size 1649664 bytes, time_taken 1390 ms
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 2
Average hash chain = 1833/1831 = 1.0
Max compares per lookup = 2
Avg compares per lookup = 11704/11700 = 1.0
----------------------------------------------
Recovery sets nab of thread 1 seq 46403 to 43232 with 8 zeroblks
*** 2013-12-27 14:29:00.278
Incident 112161 created, dump file: /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/incident/incdir_112161/coredb_ora_21644_i112161.trc
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
*** 2013-12-27 14:29:06.871
USER (ospid: 21644): terminating the instance due to error 704
++++coredb_ora_21644_i112161.trc内容如下：
Dump continued from file: /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
========= Dump for incident 112161 (ORA 600 [4000]) ========
*** 2013-12-27 14:29:00.292
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=6apq2rjyxmxpj) -----
select line#, sql_text from bootstrap$ where obj# != :1
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
*** 2013-12-27 14:29:00.423
skdstdst()+36 call kgdsdst() 000000000 ? 000000000 ?
7FFFFDDEA2C8 ? 000000001 ?
7FFFFDDEE7C8 ? 000000000 ?
ksedst1()+98 call skdstdst() 000000000 ? 000000000 ?
。。。。
============
Plan Table
============
---------------------------------------+-----------------------------------+
| Id | Operation | Name | Rows | Bytes | Cost | Time |
---------------------------------------+-----------------------------------+
| 0 | SELECT STATEMENT | | | | | |
| 1 | TABLE ACCESS FULL | BOOTSTRAP$| | | | |
---------------------------------------+-----------------------------------+
Content of other_xml column
===========================
db_version : 11.2.0.1
parse_schema : SYS
plan_hash : 867914364
plan_hash_2 : 382997521
Outline Data:
/*+
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE('11.2.0.1')
DB_VERSION('11.2.0.1')
RBO_OUTLINE
OUTLINE_LEAF(@"SEL$1")
FULL(@"SEL$1" "BOOTSTRAP$"@"SEL$1")
END_OUTLINE_DATA
*/
。。。。。。
PINNED BUFFER HISTORY (oldest pin first)
---------------------
BH (0xa6faef68) file#: 1 rdba: 0x00400208 (1/520) class: 4 ba: 0xa67b0000
set: 31 pool 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 0,0
dbwrid: 0 obj: 59 objn: -1 tsn: 0 afn: 1 hint: f
hash: [0xd69fee08,0xd69fee08] lru: [0xa6faedf0,0xa6faf180]
ckptq: [NULL] fileq: [NULL] objq: [0xc8e5b670,0xc8e5b670]
st: XCURRENT md: NULL tch: 1
flags:
LRBA: [0x0.0.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [65535]
cr pin refcnt: 0 sh pin refcnt: 0
buffer tsn: 0 rdba: 0x00400208 (1/520)
scn: 0x0002.1d98d857 seq: 0x01 flg: 0x04 tail: 0xd8571001
frmt: 0x02 chkval: 0x7485 type: 0x10=DATA SEGMENT HEADER - UNLIMITED
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x00000000A67B0000 to 0x00000000A67B2000
0A67B0000 0000A210 00400208 1D98D857 04010002 [......@.W.......]
0A67B0010 00007485 00000000 00000000 00000000 [.t..............]
0A67B0020 00000000 00000001 00000007 00001020 [............ ...]
0A67B0030 00000000 00000003 00000007 0040020C [..............@.]
0A67B0040 00000000 00000000 00000001 00000003 [................]
0A67B0050 00000001 007F8D23 00000001 00000001 [....#...........]
0A67B0060 00000000 0000003B 40000000 00400209 [....;......@..@.]
0A67B0070 00000007 00000000 00000000 00000000 [................]
0A67B0080 00000000 00000000 00000000 00000000 [................]
Repeat 250 times
0A67B1030 00000000 00010000 00010001 00000001 [................]
0A67B1040 00000000 00000003 0040020B 0040020B [..........@...@.]
0A67B1050 00000000 00000000 00000000 00000000 [................]
Repeat 249 times
0A67B1FF0 00000000 00000000 00000000 D8571001 [..............W.]
Extent Control Header
-----------------------------------------------------------------
Extent Header:: spare1: 0 spare2: 0 #extents: 1 #blocks: 7
last map 0x00000000 #maps: 0 offset: 4128
Highwater:: 0x0040020c ext#: 0 blk#: 3 ext size: 7
#blocks in seg. hdr's freelists: 1
#blocks below: 3
mapblk 0x00000000 offset: 0
Disk Lock:: Locked by xid: 0x0001.000.007f8d23
Map Header:: next 0x00000000 #extents: 1 obj#: 59 flag: 0x40000000
Extent Map
-----------------------------------------------------------------
0x00400209 length: 7
nfl = 1, nfb = 1 typ = 1 nxf = 0 ccnt = 3
SEG LST:: flg: USED lhd: 0x0040020b ltl: 0x0040020b
The buffer with tsn: 0 rdba: 0x00400208 has already been dumped
END OF PINNED BUFFER HISTORY
++++仔细观察上面trace文件的内容，有用的信息如下
select line#, sql_text from bootstrap$ where obj# != :1
rdba: 0x00400208 (1/520)
Disk Lock:: Locked by xid: 0x0001.000.007f8d23
obj#: 59
正常来说，bootstrap$ 的内容是固定不变的，不会在这个对象上出现事务
++++那我们查一下file 1 block 520 这个块
BBED> set file 1 block 520
FILE# 1
BLOCK# 520
BBED> p locker_ktech
struct locker_ktech, 8 bytes @80
ub2 kxidusn @80 0x0001
ub2 kxidslt @82 0x0000
ub4 kxidsqn @84 0x007f8d23
BBED> p ktech.flag_ktech
ub4 flag_ktech @88 0x00000001 (NONE)
+++发现这个块确实有事务，跟xid对应的上
+++接下来我们手工把这个锁清除了看看
BBED> m /x 00 offset 88
File: /tol/oradata/datafile/coredb/system01.dbf (1)
Block: 520 Offsets: 88 to 599 Dba:0x00400208
------------------------------------------------------------------------
00000000 01000000 00000000 3b000000 00000040 09024000 07000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
BBED> sum apply;
Check value for File 1, Block 520:
current = 0x7484, required = 0x7484
BBED> p offset 88
ktech.flag_ktech
----------------
ub4 flag_ktech @88 0x00000000 (NONE)
++++再次open，已经可以正常启动
SQL> alter database open;
Database altered.
