现象描述

  • 业务侧收到告警，数据库归档空间使用率接近100%，但是数据库主机上已经部署计划任务，会定期备份归档并清理。因此推测crontab出现问题，需要进行排查

排查步骤

  • 1）查看磁盘组归档日志当前的保留情况
[grid@node1 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  4194304   4194304   537268                0          537268              0             N  ARCHDG/
MOUNTED  EXTERN  N         512   4096  4194304   8388608   184672                0          184672              0             N  DATADG01/
MOUNTED  EXTERN  N         512   4096  4194304   8388608   921352                0          921352              0             N  DATADG02/
MOUNTED  HIGH    N         512   4096  1048576     10240     8984             4096            1629              0             Y  OCRVOTDG/
ASMCMD> cd ARCHDG/bidbdg/archivelog
ASMCMD> ls
2023_03_22/
2023_03_23/
2023_03_24/
2023_03_25/
2023_03_26/
2023_03_27/
2023_03_28/
2023_03_29/
  • 2）查看oracle用户的计划任务，发现由于pam配置无法查看
[oracle@node1 ~]$ crontab -l
You (oracle) are not allowed to access to (crontab) because of pam configuration.
  • 3）查看 /var/log/secure 日志，显示oracle用户密码过期
[root@node1 ~]# more /var/log/secure-20230324

Mar 19 08:15:01 node1 crond[46729]: pam_unix(crond:account): password for user oracle will expire in 0 days
....
Mar 20 08:15:01 node1 crond[83242]: pam_unix(crond:account): expired password for user oracle (password aged)
  • 4）查看计划任务日志，发现自3月20日开始，任务没有启动
[root@node1 ~]# more /var/log/cron-20230324

Mar 19 08:15:01 node1 CROND[46730]: (oracle) CMD (/home/oracle/rman/rman-arch-backup.sh)
Mar 19 12:00:01 node1 CROND[162459]: (oracle) CMD (/home/oracle/rman/rman-db-fullbackup.sh)
Mar 19 20:15:01 node1 CROND[70964]: (oracle) CMD (/home/oracle/rman/rman-arch-backup.sh)


Mar 20 08:15:01 node1 crond[83242]: (oracle) PAM ERROR (鉴定令牌不再有效；需要新的鉴定令牌)
Mar 20 08:15:01 node1 crond[83242]: (oracle) FAILED to authorize user with PAM (鉴定令牌不再有效；需要新的鉴定令牌)
  • 5）查看 oracle 用户的账户信息
[root@node1 ~]# chage -l oracle
Last password change                                : Dec 19, 2022
Password expires                                    : Mar 19, 2023
Password inactive                                   : never
Account expires                                     : never
Minimum number of days between password change      : 0
Maximum number of days between password change      : 90
Number of days of warning before password expires   : 7
  • 6）设置密码永不过期
[root@node1 ~]# chage -M 99999 oracle
[root@node1 ~]# chage -l oracle
Last password change                                : Dec 19, 2022
Password expires                                    : never
Password inactive                                   : never
Account expires                                     : never
Minimum number of days between password change      : 0
Maximum number of days between password change      : 99999
Number of days of warning before password expires   : 7
  • 7）再次查看计划任务信息
[oracle@node1 ~]$ crontab -l
15 8,20 * * *  /home/oracle/rman/rman-arch-backup.sh
00 12 * * 2  /home/oracle/rman/rman-db-fullbackup.sh