今天一个客户的存储系统出现故障导致Oracle RAC 11G 11.2.0.3.0异常退出。
存储回复后，启动数据库（startup）的时候报错，如题所示。
具体的细节如下：
[vcdgrid@vcenter-server ~]$ env | grep SID ORACLE_SID=+ASM1 [vcdgrid@vcenter-server ~]$ asmcmd ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 307200 246921 0 246921 0 N DATA/ MOUNTED NORMAL N 512 4096 1048576 122880 15110 0 7555 0 N LOG/ MOUNTED NORMAL N 512 4096 1048576 30720 29796 10240 9778 0 Y OCR/ ASMCMD> exit [vcdgrid@vcenter-server ~]$ [vcdoracle@vcenter-server ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Fri Feb 10 19:26:17 2017 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup; ORACLE instance started. Total System Global Area 1.2594E+10 bytes Fixed Size 2240056 bytes Variable Size 8254390728 bytes Database Buffers 4328521728 bytes Redo Buffers 8384512 bytes Database mounted. ORA-10873: file 1 needs to be either taken out of backup mode or media recovered ORA-01110: data file 1: '+DATA/vclouddb/datafile/system.256.781109793' SQL> exit Disconnected from Oracle Database 11g Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options [vcdoracle@vcenter-server ~]$
解决方法：
[vcdoracle@vcenter-server ~]$ ps -ef | grep pmon
502 345 1 0 19:26 ? 00:00:00 ora_pmon_vclouddb1
502 2767 32283 0 19:27 pts/21 00:00:00 grep pmon
vcdgrid 28899 1 0 19:19 ? 00:00:00 asm_pmon_+ASM1
[vcdoracle@vcenter-server ~]$
[vcdoracle@vcenter-server ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Fri Feb 10 19:30:19 2017
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL>
SQL> startup mount;
ORACLE instance started.
Total System Global Area 1.2594E+10 bytes
Fixed Size 2240056 bytes
Variable Size 8254390728 bytes
Database Buffers 4328521728 bytes
Redo Buffers 8384512 bytes
Database mounted.
SQL>
SQL> show parameter background
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
background_core_dump string partial
background_dump_dest string /u01/app/oracle/diag/rdbms/vcl
ouddb/vclouddb1/trace
SQL>
SQL>
SQL>
SQL> select * from v$backup;
FILE# STATUS CHANGE# TIME
---------- ------------------ ---------- ---------
1 ACTIVE 5295644458 10-FEB-17
2 ACTIVE 5295644458 10-FEB-17
3 ACTIVE 5295644458 10-FEB-17
4 ACTIVE 5295644458 10-FEB-17
5 ACTIVE 5295644458 10-FEB-17
6 ACTIVE 5295644458 10-FEB-17
7 ACTIVE 5295644458 10-FEB-17
8 ACTIVE 5295644458 10-FEB-17
9 ACTIVE 5295644458 10-FEB-17
10 ACTIVE 5295644458 10-FEB-17
10 rows selected.
SQL>
SQL> alter database end backup;
Database altered.
SQL>
SQL> alter database open;
Database altered.
SQL>
SQL>
SQL> select name,database_role,open_mode,current_scn from v$database;
NAME DATABASE_ROLE OPEN_MODE CURRENT_SCN
--------- ---------------- -------------------- -----------
VCLOUDDB PRIMARY READ WRITE 5295682168
SQL> select instance_name,status from v$instance;
INSTANCE_NAME STATUS
---------------- ------------
vclouddb1 OPEN
SQL>
SQL> exit
Disconnected from Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[vcdoracle@vcenter-server ~]$
这样就修复完成了。
这时候你在查看v$backup的时候，其实跟上面是不一样的：
SQL> select * from v$backup;
FILE# STATUS CHANGE# TIME
---------- ------------------ ---------- ---------
1 NOT ACTIVE 5295704341 10-FEB-17
2 NOT ACTIVE 5295704341 10-FEB-17
3 NOT ACTIVE 5295704341 10-FEB-17
4 NOT ACTIVE 5295704341 10-FEB-17
5 NOT ACTIVE 5295704341 10-FEB-17
6 NOT ACTIVE 5295704341 10-FEB-17
7 NOT ACTIVE 5295704341 10-FEB-17
8 NOT ACTIVE 5295704341 10-FEB-17
9 NOT ACTIVE 5295704341 10-FEB-17
10 NOT ACTIVE 5295704341 10-FEB-17
10 rows selected.
SQL>
SQL>
以上过程中的数据库日志：
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.157.0.71)(PORT=1521))))' SCOPE=MEMORY SID='vclouddb1'; NOTE: Loaded library: System SUCCESS: diskgroup LOG was mounted NOTE: dependency between database vclouddb and diskgroup resource ora.LOG.dg is established Successful mount of redo thread 1, with mount id 275350761 Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) Lost write protection disabled Completed: ALTER DATABASE MOUNT SUCCESS: diskgroup DATA was mounted NOTE: dependency between database vclouddb and diskgroup resource ora.DATA.dg is established Fri Feb 10 19:32:58 2017 alter database end backup Completed: alter database end backup alter database open This instance was first to open Beginning crash recovery of 2 threads Started redo scan Fri Feb 10 19:33:08 2017 Completed redo scan read 37368 KB redo, 3964 data blocks need recovery Fri Feb 10 19:35:13 2017 Started redo application at Thread 1: logseq 74401, block 57277 Thread 2: logseq 68456, block 10229 Recovery of Online Redo Log: Thread 1 Group 1 Seq 74401 Reading mem 0 Mem# 0: +LOG/vclouddb/onlinelog/group_1.257.781109871 Recovery of Online Redo Log: Thread 2 Group 4 Seq 68456 Reading mem 0 Mem# 0: +LOG/vclouddb/onlinelog/group_4.260.781109981 Recovery of Online Redo Log: Thread 1 Group 2 Seq 74402 Reading mem 0 Mem# 0: +LOG/vclouddb/onlinelog/group_2.258.781109871 Completed redo application of 6.01MB Completed crash recovery at Thread 1: logseq 74402, block 27387, scn 5295680828 Thread 2: logseq 68456, block 15686, scn 5295681845 3964 data blocks read, 3755 data blocks written, 37368 redo k-bytes read ARCH: STARTING ARCH PROCESSES Fri Feb 10 19:35:15 2017 ARC0 started with pid=60, OS id=14729 ARC0: Archival started ARCH: STARTING ARCH PROCESSES COMPLETE ARC0: STARTING ARCH PROCESSES Fri Feb 10 19:35:16 2017 ARC1 started with pid=59, OS id=14747 Fri Feb 10 19:35:16 2017 ARC2 started with pid=61, OS id=14753 ARC1: Archival started Fri Feb 10 19:35:16 2017 ARC3 started with pid=62, OS id=14757 ARC2: Archival started ARC2: Becoming the 'no FAL' ARCH ARC2: Becoming the 'no SRL' ARCH ARC1: Becoming the heartbeat ARCH Thread 2 advanced to log sequence 68457 (thread recovery) ARC3: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE Picked broadcast on commit scheme to generate SCNs Fri Feb 10 19:35:29 2017 Archived Log entry 142852 added for thread 1 sequence 74401 ID 0x7334fea dest 1: Fri Feb 10 19:35:29 2017 Thread 1 advanced to log sequence 74403 (thread open) Thread 1 opened at log sequence 74403 Current log# 1 seq# 74403 mem# 0: +LOG/vclouddb/onlinelog/group_1.257.781109871 Successful open of redo thread 1 Fri Feb 10 19:35:29 2017 SMON: enabling cache recovery Fri Feb 10 19:35:32 2017 Archived Log entry 142853 added for thread 1 sequence 74402 ID 0x7334fea dest 1: Fri Feb 10 19:35:35 2017 minact-scn: Inst 1 is now the master inc#:8 mmon proc-id:9935 status:0x7 minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000 minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:8 errcnt:0 Fri Feb 10 19:35:36 2017 [10313] Successfully onlined Undo Tablespace 2. Undo initialization finished serial:0 start:2968151842 end:2968154542 diff:2700 (27 seconds) Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed SMON: enabling tx recovery Database Characterset is AL32UTF8 No Resource Manager plan active Starting background process GTX0 Fri Feb 10 19:35:43 2017 GTX0 started with pid=66, OS id=15025 replication_dependency_tracking turned off (no async multimaster replication found) Fri Feb 10 19:35:57 2017 Starting background process QMNC Fri Feb 10 19:35:57 2017 QMNC started with pid=53, OS id=15336 Fri Feb 10 19:36:12 2017 Completed: alter database open Fri Feb 10 19:36:22 2017 Starting background process CJQ0 Fri Feb 10 19:36:22 2017 CJQ0 started with pid=72, OS id=15442 Fri Feb 10 19:38:29 2017 Reconfiguration started (old inc 8, new inc 10) List of instances: 1 (myinst: 1) Global Resource Directory frozen * dead instance detected - domain 0 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Feb 10 19:38:29 2017 LMS 0: 37 GCS shadows cancelled, 1 closed, 0 Xw survived Fri Feb 10 19:38:29 2017 LMS 1: 32 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Feb 10 19:38:29 2017 Redo thread 2 internally disabled at seq 68457 (CKPT) Fri Feb 10 19:38:29 2017 minact-scn: master found reconf/inst-rec before recscn scan old-inc#:10 new-inc#:10 Fri Feb 10 19:38:30 2017 Instance recovery: looking for dead threads Instance recovery: lock domain invalid but no dead threads Fri Feb 10 19:38:31 2017 Archived Log entry 142854 added for thread 2 sequence 68456 ID 0x7334fea dest 1: Fri Feb 10 19:38:31 2017 ARC3: Archiving disabled thread 2 sequence 68457 Archived Log entry 142855 added for thread 2 sequence 68457 ID 0x7334fea dest 1: minact-scn: master continuing after IR Fri Feb 10 19:38:43 2017 Starting background process SMCO Fri Feb 10 19:38:43 2017 SMCO started with pid=85, OS id=16329 Fri Feb 10 19:38:54 2017 Reconfiguration started (old inc 10, new inc 12) List of instances: 1 2 (myinst: 1) Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Feb 10 19:38:54 2017 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Fri Feb 10 19:38:54 2017 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Feb 10 19:43:10 2017 Immediate Kill Session#: 10, Serial#: 5 Immediate Kill Session: sess: 0x3505c5478 OS pid: 17172 Immediate Kill Session#: 120, Serial#: 5 Immediate Kill Session: sess: 0x3465b59f8 OS pid: 17176 Immediate Kill Session#: 229, Serial#: 5 Immediate Kill Session: sess: 0x346623d40 OS pid: 17180 Fri Feb 10 20:00:01 2017 alter database begin backup Completed: alter database begin backup Fri Feb 10 20:01:00 2017 Thread 1 advanced to log sequence 74404 (LGWR switch) Current log# 2 seq# 74404 mem# 0: +LOG/vclouddb/onlinelog/group_2.258.781109871 Fri Feb 10 20:01:11 2017 Archived Log entry 142856 added for thread 1 sequence 74403 ID 0x7334fea dest 1: Fri Feb 10 20:01:54 2017 alter database end backup Completed: alter database end backup Fri Feb 10 20:03:32 2017 Immediate Kill Session#: 10, Serial#: 7 Immediate Kill Session: sess: 0x3505c5478 OS pid: 17711
——————————————
Done。