RAC：CRS-4535: Cannot communicate with Cluster Ready Services
Adamhuan发表于 2017-04-07 01:14:02
今天，查看自己的测试环境的运行状态的时候，发现前几天搭建的RAC出现问题了：
登陆系统后，发现确实有问题，并遇到了如题所示的错误：
[root@oracle12c1 ~]# /u01/app/12/grid/bin/crs_stat -t -v CRS-0184: Cannot communicate with the CRS daemon. [root@oracle12c1 ~]# [root@oracle12c1 ~]# /u01/app/12/grid/bin/crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager [root@oracle12c1 ~]#
查看ORACLE ASM的情况：
[root@oracle12c1 ~]# oracleasm listdisks [root@oracle12c1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 66G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 65.5G 0 part ├─ol-root 252:0 0 39.6G 0 lvm / ├─ol-swap 252:1 0 6.6G 0 lvm [SWAP] └─ol-home 252:2 0 19.3G 0 lvm /home sdb 8:16 0 43.9G 0 disk ├─sdb1 8:17 0 4G 0 part └─sdb2 8:18 0 39G 0 part sdc 8:32 0 286.1G 0 disk ├─sdc1 8:33 0 183G 0 part └─sdc2 8:34 0 103.1G 0 part sr0 11:0 1 3.5G 0 rom [root@oracle12c1 ~]#
可以看到，oracleasm都不到盘了？
具体看看磁盘设备的情况：
[root@oracle12c1 ~]# ls -ltr --time-style="+|%Y-%m-%d|%H:%M:%S|" /dev/sd* brw-rw---- 1 root disk 8, 0 |2017-04-05|23:40:29| /dev/sda brw-rw---- 1 root disk 8, 2 |2017-04-05|23:40:29| /dev/sda2 brw-rw---- 1 root disk 8, 1 |2017-04-05|23:40:29| /dev/sda1 brw-rw---- 1 root disk 8, 32 |2017-04-05|23:41:00| /dev/sdc brw-rw---- 1 root disk 8, 33 |2017-04-05|23:41:00| /dev/sdc1 brw-rw---- 1 root disk 8, 16 |2017-04-05|23:41:00| /dev/sdb brw-rw---- 1 root disk 8, 18 |2017-04-05|23:41:01| /dev/sdb2 brw-rw---- 1 root disk 8, 34 |2017-04-05|23:41:02| /dev/sdc2 brw-rw---- 1 root disk 8, 17 |2017-04-05|23:41:02| /dev/sdb1 [root@oracle12c1 ~]# [root@oracle12c1 ~]# [root@oracle12c1 ~]# last | grep boot reboot system boot 3.8.13-55.1.6.el Wed Apr 5 23:40 - 09:58 (10:18) reboot system boot 3.8.13-55.1.6.el Mon Apr 3 19:22 - 09:58 (2+14:36) reboot system boot 3.8.13-55.1.6.el Fri Mar 31 17:30 - 19:21 (3+01:51) reboot system boot 3.8.13-55.1.6.el Fri Mar 31 11:43 - 19:21 (3+07:38) reboot system boot 3.8.13-55.1.6.el Fri Mar 31 11:34 - 19:21 (3+07:47) reboot system boot 3.8.13-55.1.6.el Thu Mar 30 18:05 - 19:21 (4+01:16) reboot system boot 3.8.13-55.1.6.el Thu Mar 30 17:15 - 17:56 (00:40) reboot system boot 3.8.13-55.1.6.el Thu Mar 30 16:02 - 17:14 (01:12) reboot system boot 3.8.13-55.1.6.el Thu Mar 30 14:53 - 16:01 (01:08) [root@oracle12c1 ~]#
近期发生了一次重启，并且，Oracle ASM没有自动的扫盘。
修复：
1. 重新扫盘：
节点一：
[root@oracle12c1 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "OCR" Instantiating disk "GIMR" Instantiating disk "MGMT" Instantiating disk "DATA01" [root@oracle12c1 ~]# [root@oracle12c1 ~]# oracleasm listdisks DATA01 GIMR MGMT OCR [root@oracle12c1 ~]#
节点二：
[root@oracle12c2 ~]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "MGMT" Instantiating disk "DATA01" Instantiating disk "OCR" Instantiating disk "GIMR" [root@oracle12c2 ~]# [root@oracle12c2 ~]# oracleasm listdisks DATA01 GIMR MGMT OCR [root@oracle12c2 ~]#
2. 集群
节点一：
[root@oracle12c1 ~]# /u01/app/12/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE OFFLINE STABLE
ora.crf
1 ONLINE OFFLINE STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE OFFLINE STABLE
ora.cssdmonitor
1 ONLINE ONLINE oracle12c1 STABLE
ora.ctssd
1 ONLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE oracle12c1 STABLE
ora.evmd
1 ONLINE INTERMEDIATE oracle12c1 STABLE
ora.gipcd
1 ONLINE ONLINE oracle12c1 STABLE
ora.gpnpd
1 ONLINE ONLINE oracle12c1 STABLE
ora.mdnsd
1 ONLINE ONLINE oracle12c1 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
[root@oracle12c1 ~]#
[root@oracle12c1 ~]# /u01/app/12/grid/bin/crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crf' on 'oracle12c1'
CRS-2672: Attempting to start 'ora.cssd' on 'oracle12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'oracle12c1'
CRS-2676: Start of 'ora.diskmon' on 'oracle12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'oracle12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'oracle12c1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oracle12c1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oracle12c1'
CRS-2676: Start of 'ora.ctssd' on 'oracle12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oracle12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oracle12c1'
CRS-2676: Start of 'ora.asm' on 'oracle12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oracle12c1'
CRS-2676: Start of 'ora.storage' on 'oracle12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oracle12c1'
CRS-2676: Start of 'ora.crsd' on 'oracle12c1' succeeded
[root@oracle12c1 ~]#
[root@oracle12c1 ~]# /u01/app/12/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE oracle12c1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE oracle12c1 STABLE
ora.crf
1 ONLINE ONLINE oracle12c1 STABLE
ora.crsd
1 ONLINE ONLINE oracle12c1 STABLE
ora.cssd
1 ONLINE ONLINE oracle12c1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE oracle12c1 STABLE
ora.ctssd
1 ONLINE ONLINE oracle12c1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE oracle12c1 STABLE
ora.evmd
1 ONLINE ONLINE oracle12c1 STABLE
ora.gipcd
1 ONLINE ONLINE oracle12c1 STABLE
ora.gpnpd
1 ONLINE ONLINE oracle12c1 STABLE
ora.mdnsd
1 ONLINE ONLINE oracle12c1 STABLE
ora.storage
1 ONLINE ONLINE oracle12c1 STABLE
--------------------------------------------------------------------------------
[root@oracle12c1 ~]#
节点二：
[root@oracle12c2 ~]# /u01/app/12/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE STABLE
ora.cluster_interconnect.haip
1 ONLINE OFFLINE STABLE
ora.crf
1 ONLINE OFFLINE STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE OFFLINE STABLE
ora.cssdmonitor
1 ONLINE ONLINE oracle12c2 STABLE
ora.ctssd
1 ONLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE oracle12c2 STABLE
ora.evmd
1 ONLINE INTERMEDIATE oracle12c2 STABLE
ora.gipcd
1 ONLINE ONLINE oracle12c2 STABLE
ora.gpnpd
1 ONLINE ONLINE oracle12c2 STABLE
ora.mdnsd
1 ONLINE ONLINE oracle12c2 STABLE
ora.storage
1 ONLINE OFFLINE STABLE
--------------------------------------------------------------------------------
[root@oracle12c2 ~]#
[root@oracle12c2 ~]# /u01/app/12/grid/bin/crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crf' on 'oracle12c2'
CRS-2672: Attempting to start 'ora.cssd' on 'oracle12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oracle12c2'
CRS-2676: Start of 'ora.diskmon' on 'oracle12c2' succeeded
CRS-2676: Start of 'ora.crf' on 'oracle12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oracle12c2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'oracle12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oracle12c2'
CRS-2676: Start of 'ora.ctssd' on 'oracle12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oracle12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oracle12c2'
CRS-2676: Start of 'ora.asm' on 'oracle12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oracle12c2'
CRS-2676: Start of 'ora.storage' on 'oracle12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oracle12c2'
CRS-2676: Start of 'ora.crsd' on 'oracle12c2' succeeded
[root@oracle12c2 ~]#
[root@oracle12c2 ~]# /u01/app/12/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE oracle12c2 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE oracle12c2 STABLE
ora.crf
1 ONLINE ONLINE oracle12c2 STABLE
ora.crsd
1 ONLINE ONLINE oracle12c2 STABLE
ora.cssd
1 ONLINE ONLINE oracle12c2 STABLE
ora.cssdmonitor
1 ONLINE ONLINE oracle12c2 STABLE
ora.ctssd
1 ONLINE ONLINE oracle12c2 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE oracle12c2 STABLE
ora.evmd
1 ONLINE ONLINE oracle12c2 STABLE
ora.gipcd
1 ONLINE ONLINE oracle12c2 STABLE
ora.gpnpd
1 ONLINE ONLINE oracle12c2 STABLE
ora.mdnsd
1 ONLINE ONLINE oracle12c2 STABLE
ora.storage
1 ONLINE ONLINE oracle12c2 STABLE
--------------------------------------------------------------------------------
[root@oracle12c2 ~]#
这时候，CRS的状态就恢复了：
节点一：
[root@oracle12c1 ~]# /u01/app/12/grid/bin/crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@oracle12c1 ~]# [grid@oracle12c1 ~]$ crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora....SM.lsnr ora....er.type 0/5 0/ ONLINE ONLINE oracle12c1 ora....SM.lsnr ora....er.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.DATA01.dg ora....up.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.GIMR.dg ora....up.type 0/5 0/ ONLINE ONLINE oracle12c1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE oracle12c1 ora....AF.lsnr ora....er.type 0/5 0/ OFFLINE OFFLINE ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE oracle12c1 ora.MGMTLSNR ora....nr.type 0/0 0/0 ONLINE ONLINE oracle12c1 ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.asm ora.asm.type 0/5 0/0 ONLINE ONLINE oracle12c1 ora.chad ora.chad.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE oracle12c1 ora.lenka.db ora....se.type 0/2 0/1 ONLINE ONLINE oracle12c1 ora.mgmtdb ora....db.type 0/2 0/1 ONLINE ONLINE oracle12c1 ora....network ora....rk.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE oracle12c1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE oracle12c1 ora....2c1.ons application 0/3 0/0 ONLINE ONLINE oracle12c1 ora....2c1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE oracle12c1 ora....2c2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE oracle12c1 ora.proxy_advm ora....vm.type 0/5 0/ OFFLINE OFFLINE ora.qosmserver ora....er.type 0/1 0/2 ONLINE ONLINE oracle12c1 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE oracle12c1 [grid@oracle12c1 ~]$ [root@oracle12c1 ~]# [root@oracle12c1 ~]# ps -ef | grep pmon root 314 25738 0 09:11 pts/0 00:00:00 grep --color=auto pmon grid 4178 1 0 Apr06 ? 00:00:03 asm_pmon_+ASM1 grid 5175 1 0 Apr06 ? 00:00:03 mdb_pmon_-MGMTDB oracle 6643 1 0 Apr06 ? 00:00:03 ora_pmon_lenka_2 [root@oracle12c1 ~]#
节点二：
[root@oracle12c2 ~]# /u01/app/12/grid/bin/crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@oracle12c2 ~]# [root@oracle12c2 ~]# /u01/app/12/grid/bin/crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora....SM.lsnr ora....er.type 0/5 0/ ONLINE ONLINE oracle12c1 ora....SM.lsnr ora....er.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.DATA01.dg ora....up.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.GIMR.dg ora....up.type 0/5 0/ ONLINE ONLINE oracle12c1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE oracle12c1 ora....AF.lsnr ora....er.type 0/5 0/ OFFLINE OFFLINE ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE oracle12c1 ora.MGMTLSNR ora....nr.type 0/0 0/0 ONLINE ONLINE oracle12c1 ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.asm ora.asm.type 0/5 0/0 ONLINE ONLINE oracle12c1 ora.chad ora.chad.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE oracle12c1 ora.lenka.db ora....se.type 0/2 0/1 ONLINE ONLINE oracle12c1 ora.mgmtdb ora....db.type 0/2 0/1 ONLINE ONLINE oracle12c1 ora....network ora....rk.type 0/5 0/ ONLINE ONLINE oracle12c1 ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE oracle12c1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE oracle12c1 ora....2c1.ons application 0/3 0/0 ONLINE ONLINE oracle12c1 ora....2c1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE oracle12c1 ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE oracle12c2 ora....2c2.ons application 0/3 0/0 ONLINE ONLINE oracle12c2 ora....2c2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE oracle12c2 ora.proxy_advm ora....vm.type 0/5 0/ OFFLINE OFFLINE ora.qosmserver ora....er.type 0/1 0/2 ONLINE ONLINE oracle12c1 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE oracle12c1 [root@oracle12c2 ~]# [root@oracle12c2 ~]# ps -ef | grep pmon grid 13224 1 0 09:09 ? 00:00:00 asm_pmon_+ASM2 oracle 13804 1 0 09:09 ? 00:00:00 ora_pmon_lenka_1 root 14358 8363 10 09:12 pts/0 00:00:00 grep --color=auto pmon [root@oracle12c2 ~]#
恢复后，EM CC的状态：
————————————————————————
Done。