一套Power AIX上的9.2.0.1系统在数据库打开过程中遇到ORA-00600:[2667]内部错误，详细日志如下:
ORA-00600: 内部错误代码, 参数:[2667]
Wed Mar 9 19:03:38 2011 RESETLOGS is being done without consistancy checks. This may result in a corrupted database. The database should be recreated. RESETLOGS after incomplete recovery UNTIL CHANGE 117699122479 Resetting resetlogs activation ID 2197911857 (0x83017931) Wed Mar 9 19:03:47 2011 LGWR: Primary database is in CLUSTER CONSISTENT mode Assigning activation ID 2284878888 (0x88307c28) Thread 1 opened at log sequence 1 Current log# 3 seq# 1 mem# 0: /s01/maclean/oradata/PROD/redo03.log Successful open of redo thread 1. Wed Mar 9 19:03:47 2011 SMON: enabling cache recovery Wed Mar 9 19:03:47 2011 Errors in file /s01/maclean/admin/PROD/udump/PROD_ora_585914.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] Wed Mar 9 19:03:48 2011 Errors in file /s01/maclean/admin/PROD/bdump/PROD_pmon_602286.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] Wed Mar 9 19:03:50 2011 Errors in file /s01/maclean/admin/PROD/bdump/PROD_lgwr_548884.trc: ORA-00600: internal error code, arguments: [2667], [1], [1], [3], [1739273391], [1739273391], [1739273391], [7267] LGWR: terminating instance due to error 600 相关的初始化参数 fast_start_mttr_target = 300 _allow_resetlogs_corruption= TRUE undo_management = AUTO undo_tablespace = UNDOTBS1
以上可以看到lgwr关键进程在数据库open后几秒后遭遇了ORA-00600:[2667]内部错误后终止了实例。
该数据库在之前因为丢失当前日志文件进行了已经实施了一系列的非常规恢复操作，包括设置一系列的underscore参数:
Before I provide the steps to fix the ora-00600 error, I want to tell you that this database is opened with the unsupported parameter "allow_resetlogs_corruption". ************************************************************************* * By forcing open the database using this parameter, there is a strong * * likelihood of logical corruption, possibly affecting the data * * dictionary. Oracle does not guarantee that all of the data will be * * accessible nor will it support a database that has been opened by * * this method and that the database users will be allowed to continue * * work. All this does is provide a way to get at the contents of the * * database for extraction, usually by export. It is up to you to * * determine the amount of lost data and to correct any logical * * corruption issues. * * * ************************************************************************* 2) The steps to get rid of the ora-00600 are as follows: + Change UNDO_MANAGEMENT=AUTO to UNDO_MANAGEMENT=MANUAL + Remove or comment out UNDO_TABLESPACE and UNDO_RETENTION. + Add _CORRUPTED_ROLLBACK_SEGMENTS =(comma separated list of Automatic Undo segments) Example: _CORRUPTED_ROLLBACK_SEGMENTS = (_SYSSMU1$, _SYSSMU2$, _SYSSMU3$, _SYSSMU4$, _SYSSMU5$, _SYSSMU6$, _SYSSMU7$, _SYSSMU8$, _SYSSMU9$, _SYSSMU10$) Note, sometimes the alert log will tell you what Automatic Undo segments are in use. Search the alert log for SYSS. If the alert log does not contain that information then use _SYSSMU1$ through _SYSSMU10$ as shown in the example above. In UNIX you can issue this command to get the undo segment names: $ strings system01.dbf | grep _SYSSMU | cut -d $ -f 1 | sort -u From the output of the strings command above, add a $ to end of each _SYSSMU undo segment name. ++ Startup mount the database as follows: SQL > startup mount SQL > recover database; SQl > alter database open; *._corrupted_rollback_segments= (_SYSSMU730$, _SYSSMU731$, _SYSSMU732$, _SYSSMU733$, _SYSSMU734$, _SYSSMU735$, _SYSSMU736$, _SYSSMU737$, _SYSSMU738$, _SYSSMU739$, _SYSSMU744$, _SYSSMU740$, _SYSSMU741$, _SYSSMU742$, _SYSSMU743$, _SYSSMU744$, _SYSSMU745$, _SYSSMU746$, _SYSSMU747$, _SYSSMU748$, _SYSSMU749$, _SYSSMU74t$, _SYSSMU75$, _SYSSMU750$, _SYSSMU751$, _SYSSMU752$, _SYSSMU753$, _SYSSMU754$, _SYSSMU755$, _SYSSMU756$, _SYSSMU757$, _SYSSMU758$, _SYSSMU759$, _SYSSMU76$, _SYSSMU760$, _SYSSMU761$, _SYSSMU762$, _SYSSMU763$, _SYSSMU764$, _SYSSMU765$, _SYSSMU766$, _SYSSMU767$, _SYSSMU768$)
但在完成以上设置后仍不能避免ora-00600[2667]的发生，下位errpt日志中磁盘阵列损坏信息:
errpt -a --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR6 IDENTIFIER: B9735AF4 Date/Time: Thu Mar 10 19:01:51 THAIST 2011 Sequence Number: 106800 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: PERM Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description SUBSYSTEM COMPONENT FAILURE Probable Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Failure Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600 0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D544 0000 0000 0000 0000 0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020 2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5B8C 32C1 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR6 IDENTIFIER: B9735AF4 Date/Time: Thu Mar 10 19:01:51 THAIST 2011 Sequence Number: 106799 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: PERM Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description SUBSYSTEM COMPONENT FAILURE Probable Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Failure Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600 0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D524 0000 0000 0000 0000 0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020 2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5B8C 32C0 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR10 IDENTIFIER: C86ACB7E Date/Time: Thu Mar 10 19:01:51 THAIST 2011 Sequence Number: 106798 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: INFO Resource Name: dac0 Resource Class: array Resource Type: ibm-dac-V4 Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21 VPD: Manufacturer................IBM Machine Type and Model......1814 FAStT Part Number.................24288-00 ROS Level and ID............0916 Description ARRAY CONFIGURATION CHANGED Probable Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Failure Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Recommended Actions NO ACTION NECESSARY Detail Data SENSE DATA 0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600 0000 0098 0000 0000 9502 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0008 1800 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020 2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5B8C 32BF 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR6 IDENTIFIER: B9735AF4 Date/Time: Thu Mar 10 19:01:47 THAIST 2011 Sequence Number: 106797 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: PERM Resource Name: hdisk4 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L0 Description SUBSYSTEM COMPONENT FAILURE Probable Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Failure Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600 0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D544 0000 0000 0000 0000 0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020 2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5B8C 3215 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR6 IDENTIFIER: B9735AF4 Date/Time: Thu Mar 10 19:01:47 THAIST 2011 Sequence Number: 106796 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: PERM Resource Name: hdisk4 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L0 Description SUBSYSTEM COMPONENT FAILURE Probable Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Failure Causes ARRAY DASD MEDIA POWER OR FAN COMPONENT Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600 0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D524 0000 0000 0000 0000 0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020 2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5B8C 3214 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR10 IDENTIFIER: C86ACB7E Date/Time: Thu Mar 10 19:01:47 THAIST 2011 Sequence Number: 106795 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: INFO Resource Name: dac0 Resource Class: array Resource Type: ibm-dac-V4 Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21 VPD: Manufacturer................IBM Machine Type and Model......1814 FAStT Part Number.................24288-00 ROS Level and ID............0916 Description ARRAY CONFIGURATION CHANGED Probable Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Failure Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Recommended Actions NO ACTION NECESSARY Detail Data SENSE DATA 0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600 0000 0098 0000 0000 9502 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0008 1800 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020 2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5B8C 3213 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106794 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2800 4644 2F00 0003 0004 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106793 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 132E C948 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106792 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 1335 55F0 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106791 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 136C 0F98 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106790 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 13D7 29B0 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106789 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 1337 FD88 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106788 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 136C 1F08 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106787 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2A00 2578 E838 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106786 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2800 4644 3300 0001 0004 0000 0000 0000 0000 0007 7497 0200 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR4 IDENTIFIER: D5385D18 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106785 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: hdisk5 Resource Class: disk Resource Type: array Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000 Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD MEDIA ARRAY DASD DEVICE Failure Causes DASD MEDIA DISK DRIVE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2800 4643 BA00 0004 0004 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR8 IDENTIFIER: 483C9D10 Date/Time: Thu Mar 10 18:49:23 THAIST 2011 Sequence Number: 106784 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: INFO Resource Name: dac0 Resource Class: array Resource Type: ibm-dac-V4 Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21 VPD: Manufacturer................IBM Machine Type and Model......1814 FAStT Part Number.................24288-00 ROS Level and ID............0916 Description ARRAY ACTIVE CONTROLLER SWITCH Probable Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Failure Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Recommended Actions NO ACTION NECESSARY Detail Data SENSE DATA 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0400 00EE 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0002 0000 0001 0000 0000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR3 IDENTIFIER: D9770360 Date/Time: Thu Mar 10 18:48:47 THAIST 2011 Sequence Number: 106783 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: PERM Resource Name: dac0utm Resource Class: NONE Resource Type: NONE Location: Description ARRAY OPERATION ERROR Probable Causes ARRAY DASD DEVICE STORAGE DEVICE CABLE EB4F Failure Causes DISK DRIVE DISK DRIVE ELECTRONICS STORAGE DEVICE CABLE ARRAY CONTROLLER Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 2800 0000 0200 0000 0104 0000 0000 0000 0000 0000 0013 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 1103 8000 F205 3703 0000 0200 0000 0000 0000 0000 0040 0002 0000 0000 0000 0000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7 Date/Time: Thu Mar 10 18:48:47 THAIST 2011 Sequence Number: 106782 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: fscsi0 Resource Class: driver Resource Type: efscsi Location: U787B.001.DNWGK9Y-P1-C4-T1 Description ADAPTER ERROR Probable Causes ADAPTER HARDWARE OR CABLE ADAPTER MICROCODE FIBRE CHANNEL SWITCH OR FC-AL HUB Failure Causes ADAPTER CABLES AND CONNECTIONS DEVICE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES CHECK CABLES AND THEIR CONNECTIONS VERIFY DEVICE CONFIGURATION Detail Data SENSE DATA 0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0500 0000 0000 0001 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 422F 0000 0812 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2005 00A0 B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 1105 C000 --------------------------------------------------------------------------- LABEL: FCP_ARRAY_ERR9 IDENTIFIER: 8B79A4BD Date/Time: Thu Mar 10 18:48:45 THAIST 2011 Sequence Number: 106781 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: PERM Resource Name: dac1 Resource Class: array Resource Type: ibm-dac-V4 Location: U787B.001.DNWGK9Y-P1-C1-T1-W200400A0B8484C21 VPD: Manufacturer................IBM Machine Type and Model......1814 FAStT Part Number.................24288-00 ROS Level and ID............0916 Description ARRAY CONTROLLER SWITCH FAILURE Probable Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Failure Causes ARRAY CONTROLLER CABLES AND CONNECTIONS Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES Detail Data SENSE DATA 0A00 5A00 2C01 0000 0002 0004 0000 0000 0000 0000 0000 0000 0200 0B00 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 526D 6000 F205 3703 0000 0000 0000 0010 0000 0000 0040 0002 0000 0000 0000 0000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7 Date/Time: Thu Mar 10 18:48:45 THAIST 2011 Sequence Number: 106780 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: fscsi1 Resource Class: driver Resource Type: efscsi Location: U787B.001.DNWGK9Y-P1-C1-T1 Description ADAPTER ERROR Probable Causes ADAPTER HARDWARE OR CABLE ADAPTER MICROCODE FIBRE CHANNEL SWITCH OR FC-AL HUB Failure Causes ADAPTER CABLES AND CONNECTIONS DEVICE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES CHECK CABLES AND THEIR CONNECTIONS VERIFY DEVICE CONFIGURATION Detail Data SENSE DATA 0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 422F 0000 1212 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0 B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0FFF A000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7 Date/Time: Thu Mar 10 18:48:43 THAIST 2011 Sequence Number: 106779 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: fscsi1 Resource Class: driver Resource Type: efscsi Location: U787B.001.DNWGK9Y-P1-C1-T1 Description ADAPTER ERROR Probable Causes ADAPTER HARDWARE OR CABLE ADAPTER MICROCODE FIBRE CHANNEL SWITCH OR FC-AL HUB Failure Causes ADAPTER CABLES AND CONNECTIONS DEVICE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES CHECK CABLES AND THEIR CONNECTIONS VERIFY DEVICE CONFIGURATION Detail Data SENSE DATA 0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 422F 0000 1112 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0 B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0FFF A000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7 Date/Time: Thu Mar 10 18:48:42 THAIST 2011 Sequence Number: 106778 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: fscsi0 Resource Class: driver Resource Type: efscsi Location: U787B.001.DNWGK9Y-P1-C4-T1 Description ADAPTER ERROR Probable Causes ADAPTER HARDWARE OR CABLE ADAPTER MICROCODE FIBRE CHANNEL SWITCH OR FC-AL HUB Failure Causes ADAPTER CABLES AND CONNECTIONS DEVICE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES CHECK CABLES AND THEIR CONNECTIONS VERIFY DEVICE CONFIGURATION Detail Data SENSE DATA 0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0500 0000 0000 0001 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 422F 0000 0712 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2005 00A0 B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 1105 C000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7 Date/Time: Thu Mar 10 18:48:41 THAIST 2011 Sequence Number: 106777 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: fscsi1 Resource Class: driver Resource Type: efscsi Location: U787B.001.DNWGK9Y-P1-C1-T1 Description ADAPTER ERROR Probable Causes ADAPTER HARDWARE OR CABLE ADAPTER MICROCODE FIBRE CHANNEL SWITCH OR FC-AL HUB Failure Causes ADAPTER CABLES AND CONNECTIONS DEVICE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES CHECK CABLES AND THEIR CONNECTIONS VERIFY DEVICE CONFIGURATION Detail Data SENSE DATA 0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 422F 0000 1012 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0 B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0FFF A000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7 Date/Time: Thu Mar 10 18:48:39 THAIST 2011 Sequence Number: 106776 Machine Id: 000786F9D600 Node Id: p550su Class: H Type: TEMP Resource Name: fscsi1 Resource Class: driver Resource Type: efscsi Location: U787B.001.DNWGK9Y-P1-C1-T1 Description ADAPTER ERROR Probable Causes ADAPTER HARDWARE OR CABLE ADAPTER MICROCODE FIBRE CHANNEL SWITCH OR FC-AL HUB Failure Causes ADAPTER CABLES AND CONNECTIONS DEVICE Recommended Actions PERFORM PROBLEM DETERMINATION PROCEDURES CHECK CABLES AND THEIR CONNECTIONS VERIFY DEVICE CONFIGURATION Detail Data SENSE DATA 0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 422F 0000 0F12 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0 B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0FFF A000 --------------------------------------------------------------------------- LABEL: FSCSI_ERR4 IDENTIFIER: 3074FEB7
因为lgwr的crash是发生在database open后，所以实际上我们是可以在这段时间内操作数据库的，这个case最终通过新建undo tablespace代替老的问题回滚表空间解决了。
We have been repair the disk controller, but ora-600[2667] still occur when we try to open database. After recreating the undo tablespace we were able to open the database.