客户10.2.0.4 RAC环境出现ORA-600[13310]错误。
错误信息如下：
Sun Aug 1 04:08:24 2010 Errors IN file /oracle/admin/ORCL/udump/orcl1_ora_14964.trc: ORA-00603: ORACLE server SESSION TERMINATED BY fatal error ORA-27544: Failed TO map memory region FOR export ORA-27300: OS system dependent operation:socket failed WITH STATUS: 23 ORA-27301: OS failure message: File TABLE overflow ORA-27302: failure occurred at: sskgxpcre1 Sun Aug 1 04:08:24 2010 Trace dumping IS performing id=[cdmp_20100801040824] Sun Aug 1 16:08:25 2010 Trace dumping IS performing id=[cdmp_20100801160825] Sun Aug 1 16:08:27 2010 Errors IN file /oracle/admin/ORCL/bdump/orcl1_diag_17090.trc: ORA-27050: FUNCTION called WITH invalid FIB/IOV STRUCTURE Additional information: 2 ORA-27041: unable TO OPEN file HPUX-ia64 Error: 23: File TABLE overflow Additional information: 3 Sun Aug 1 04:08:42 2010 Errors IN file /oracle/admin/ORCL/udump/orcl1_ora_15065.trc: ORA-00603: Message 603 NOT found; No message file FOR product=RDBMS, facility=ORA ORA-27544: Message 27544 NOT found; No message file FOR product=RDBMS, facility=ORA ORA-27300: Message 27300 NOT found; No message file FOR product=RDBMS, facility=ORA; arguments: [socket] [23] ORA-27301: Message 27301 NOT found; No message file FOR product=RDBMS, facility=ORA; arguments: [File TABLE overflow] ORA-27302: Message 27302 NOT found; No message file FOR product=RDBMS, facility=ORA; arguments: [sskgxpcre1] Sun Aug 1 04:08:48 2010 Trace dumping IS performing id=[cdmp_20100801160843] Sun Aug 1 04:08:48 2010 Trace dumping IS performing id=[cdmp_20100801160843] Sun Aug 1 04:08:56 2010 ospid 17096: network interface WITH IP address 192.168.1.81 no longer operational Sun Aug 1 04:08:58 2010 Errors IN file /oracle/admin/ORCL/bdump/orcl1_ora_15148.trc: ORA-00600: internal error code, arguments: [13310], [], [], [], [], [], [], [] Sun Aug 1 04:08:58 2010 Errors IN file /oracle/admin/ORCL/bdump/orcl1_ora_15148.trc: ORA-00210: cannot OPEN the specified control file ORA-00202: control file: '/dev/vgpmcs11/rr01G1001' ORA-27041: unable TO OPEN file HPUX-ia64 Error: 23: File TABLE overflow Additional information: 3 ORA-00600: internal error code, arguments: [13310], [], [], [], [], [], [], [] Sun Aug 1 04:09:03 2010 Trace dumping IS performing id=[cdmp_20100801040859]
根据MOS文档ORA-600 [13310] Reported When Obtaining An OS Unique Identifier [ID 1234083.1]，ORA-600[13310]错误出现的原因是无法获取操作系统上的唯一标识信息，而Oracle文档给出的解决方案是释放或者增加系统资源，比如内存或SWAP空间。
由于出现600错误之前出现了ORA-2730N的错误，说明确实出现了操作系统层面的问题。根据ORA-27544错误以及File table overflow信息进行定位，确认是HP-UX上10.2.0.4的bug：hp-ux: File handles not released after upgrade to 10.2.0.3 CRS Bundle#2 or 10.2.0.4 [ID 739557.1]。
这个问题是10.2.0.3的CRS Bundle补丁引入，由mmap引发的错误，影响版本包括10.2.0.3、10.2.0.4和11.1.0.6。Oracle在10.2.0.5、11.1.0.7中解决了这个问题，在10.2.0.4 CRS Bundle Patch #2以及10.2.0.4最新的CRS PSU中同样修正了该错误。