最近在生产库上碰到这个错误。从错误号可以辨别是因为存储空间的问题。
由于生产环境是RAC,数据文件存于存储上面。于是查看存储空间情况。发现存储上面的空间是足够的。而且看了所有表空间状态都是对的,没有异常。无从查起,于是看节点的alert日志。
终于在其中一个节点上看到如下信息:
- ORA-01114: <BD><AB><BF><E9>写<C8><EB><CE>募<FE> 时<B3><F6><CF><D6> IO <B4><ED><CE><F3> (<BF><E9> # )
- Tue Apr 16 18:34:36 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_ora_10993.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
- Could not open audit file: /u01/app/oracle/admin/center/adump/center2_ora_10993_1.aud
- Retry Iteration No: 1 OS Error: 28
- Retry Iteration No: 2 OS Error: 28
- Retry Iteration No: 3 OS Error: 28
- Retry Iteration No: 4 OS Error: 28
- Retry Iteration No: 5 OS Error: 28
- OS Audit file could not be created; failing after 5 retries
- Tue Apr 16 18:35:14 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_ora_11034.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
- Could not open audit file: /u01/app/oracle/admin/center/adump/center2_ora_11034_1.aud
- Retry Iteration No: 1 OS Error: 28
- Retry Iteration No: 2 OS Error: 28
- Retry Iteration No: 3 OS Error: 28
- Retry Iteration No: 4 OS Error: 28
- Retry Iteration No: 5 OS Error: 28
- OS Audit file could not be created; failing after 5 retries
- Tue Apr 16 18:35:26 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_gcr0_10269.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
- Tue Apr 16 18:37:31 2013
- Non critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/center/center2/trace/
- center2_ora_3430.trc"
- Error message: Linux-x86_64 Error: 28: No space left on device
- Additional information: 1
- Writing to the above trace file is disabled for now on...
从上面可以看到很重要的字眼: No space left on device 。于是用df查看磁盘空间,发现报以上错误的这个节点上面的根目录空间已用完。
于是开始清理磁盘空间,发现是由于备份文件存放太多,没有定期清理而把空间用完。清理空间后问题得以解决。