本帖最后由 whutabs 于 2013-11-20 13:21 编辑

单位2012年买的ODA一体机，双节点RAC，生产库。

操作系统：Linux 2.6.32-300.11.1.el5uek #1 SMP x86_64 x86_64 x86_64 GNU/Linux

数据库：Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

昨晚0点左右，实例重启，造成关键业务中断，求刘大帮忙分析，谢谢！

问题1：此ODA是ORACLE厂家安装11.2.0.2，我升级至11.2.0.3，一切正常，未见错误。

但正常运行几个月后，总会莫名其妙出现： CHECK TIMED OUT  ，重启正常后隔几个月又出现。

不光是LISTENER这个资源，有时还有其他资源，貌似业务正常，很奇怪。

例如：crsctl status res -t

ora.LISTENER.lsnr

ONLINE  INTERMEDIATE oda1                     CHECK TIMED OUT

ONLINE  ONLINE           oda2

ora.LISTENER_SCAN1.lsnr

1        ONLINE  ONLINE          oda2

ora.LISTENER_SCAN2.lsnr

1        ONLINE  INTERMEDIATE oda1                     CHECK TIMED OUT

问题2，查看这次数据库重启日志，发现并没有生成对应的错误trc文件，而该时间点却有其他trc文件，很奇怪

例如：

Tue Nov 19 00:02:17 2013

NOTE: ASMB terminating

Errors in file /u01/app/oracle/diag/rdbms/whut/whut1/trace/whut1_asmb_17970.trc:

ORA-15064: communication failure with ASM instance

ORA-03113: end-of-file on communication channel

ls -al whut1_asmb*

-rw-r----- 1 oracle asmadmin 865 Nov 19 00:02 whut1_asmb_6787.trc

-rw-r----- 1 oracle asmadmin  60 Nov 19 00:02 whut1_asmb_6787.trm

问题3，此次重启，alert日志说是asmb终止了实例，发生重启，而asm的日志说，asmb进程僵死，无法通讯。

问题处在哪里呢?

alert_whut1.log:

Tue Nov 19 00:02:17 2013

NOTE: ASMB terminating

Errors in file /u01/app/oracle/diag/rdbms/whut/whut1/trace/whut1_asmb_17970.trc:

ORA-15064: communication failure with ASM instance

ORA-03113: end-of-file on communication channel

Process ID:

Session ID: 587 Serial number: 7

Errors in file /u01/app/oracle/diag/rdbms/whut/whut1/trace/whut1_asmb_17970.trc:

ORA-15064: communication failure with ASM instance

ORA-03113: end-of-file on communication channel

Process ID:

Session ID: 587 Serial number: 7

ASMB (ospid: 17970): terminating the instance due to error 15064

Tue Nov 19 00:02:17 2013

System state dump requested by (instance=1, osid=17970 (ASMB)), summary=[abnormal instance termination].

System State dumped to trace file /u01/app/oracle/diag/rdbms/whut/whut1/trace/whut1_diag_17926.trc

alert_+ASM1.log:

Mon Nov 18 23:59:29 2013

WARNING: client [whut1:whut] not responsive for 200s; state=0x1. killing pid 17978

Tue Nov 19 00:02:01 2013

WARNING: ASM waited 355 secs for ASMB process in whut1:whut

Tue Nov 19 00:02:50 2013

NOTE: client whut1:whut registered, osid 6791, mbr 0x2

[grid@oda1 trace]$

请各位大侠指点迷津，谢谢！

我会上传相关日志。