怎么修kingsql中RAC节点故障

这篇文章主要讲解了“怎么修kingsql中RAC节点故障”，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着小编的思路慢慢深入，一起来研究和学习“怎么修kingsql中RAC节点故障”吧！

很久之前安装的虚拟机，今天开机之后，rac2和rac3集群服务无法启动

[grid@rac2 cssd]$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@rac3 cssd]$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@rac1 ~]$ srvctl status database -d kingsql

Instance kingsql1 is running on node rac1

Instance kingsql2 is not running on node rac2

Instance kingsql3 is not running on node rac3

rac2的cssd日志发现问题

2016-04-19 17:00:47.205: [    CSSD][3666859776]clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 356447081, wrtcnt, 294277, LATS 4294697710, lastSeqNo 294274, uniqueness 1461034467, timestamp 1461056437/21174804

2016-04-19 17:00:47.205: [    CSSD][3666859776]clssnmvDHBValidateNCopy: node 3, rac3, has a disk HB, but no network HB, DHB has rcfg 356447080, wrtcnt, 41563, LATS 4294697710, lastSeqNo 41560, uniqueness 1463643843, timestamp 1463643914/18290394

[grid@rac1 ~]$ crs_stop -all

CRS-2500: Cannot stop resource ora.gsd as it is not running

Attempting to stop `ora.oc4j` on member `rac1`

Attempting to stop `ora.CRS.dg` on member `rac1`

Attempting to stop `ora.DATA.dg` on member `rac1`

Attempting to stop `ora.kingsql.db` on member `rac1`

Attempting to stop `ora.ons` on member `rac1`

CRS-2789: Cannot stop resource ora.gsd as it is not running on server rac1

Stop of `ora.ons` on member `rac1` succeeded.

Attempting to stop `ora.cvu` on member `rac1`

Attempting to stop `ora.rac2.vip` on member `rac1`

Attempting to stop `ora.rac3.vip` on member `rac1`

Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`

Stop of `ora.rac3.vip` on member `rac1` succeeded.

Stop of `ora.rac2.vip` on member `rac1` succeeded.

Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.

Attempting to stop `ora.scan1.vip` on member `rac1`

Stop of `ora.scan1.vip` on member `rac1` succeeded.

Stop of `ora.oc4j` on member `rac1` succeeded.

Stop of `ora.cvu` on member `rac1` succeeded.

Attempting to stop `ora-1-work` on member `rac1`

Stop of `ora-1-work` on member `rac1` succeeded.

Stop of `ora.kingsql.db` on member `rac1` succeeded.

Stop of `ora.DATA.dg` on member `rac1` succeeded.

Stop of `ora.CRS.dg` on member `rac1` succeeded.

Attempting to stop `ora.asm` on member `rac1`

Stop of `ora.asm` on member `rac1` succeeded.

Attempting to stop `ora.asm` on member `rac1`

--卡在这不动了，于是执行crsctl stop crs

CRS-0184: Cannot communicate with the CRS daemon.

[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac1

CRS-2673: Attempting to stop ora.mdnsd on rac1

CRS-2673: Attempting to stop ora.ctssd on rac1

CRS-2673: Attempting to stop ora.evmd on rac1

CRS-2673: Attempting to stop ora.cluster_interconnect.haip on rac1

CRS-2677: Stop of ora.cluster_interconnect.haip on rac1 succeeded

CRS-2677: Stop of ora.mdnsd on rac1 succeeded

CRS-2677: Stop of ora.evmd on rac1 succeeded

CRS-2677: Stop of ora.ctssd on rac1 succeeded

CRS-2673: Attempting to stop ora.cssd on rac1

CRS-2677: Stop of ora.cssd on rac1 succeeded

CRS-2673: Attempting to stop ora.crf on rac1

CRS-2677: Stop of ora.crf on rac1 succeeded

CRS-2673: Attempting to stop ora.gipcd on rac1

CRS-2677: Stop of ora.gipcd on rac1 succeeded

CRS-2673: Attempting to stop ora.gpnpd on rac1

CRS-2677: Stop of ora.gpnpd on rac1 succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac1 has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-4687: Shutdown command has completed with errors.

CRS-4000: Command Stop failed, or completed with errors.

[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac3

CRS-2673: Attempting to stop ora.crsd on rac3

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on rac3

CRS-2673: Attempting to stop ora.CRS.dg on rac3

CRS-2673: Attempting to stop ora.DATA.dg on rac3

CRS-2677: Stop of ora.DATA.dg on rac3 succeeded

CRS-2677: Stop of ora.CRS.dg on rac3 succeeded

CRS-2673: Attempting to stop ora.asm on rac3

CRS-2677: Stop of ora.asm on rac3 succeeded

CRS-2673: Attempting to stop ora.ons on rac3

CRS-2677: Stop of ora.ons on rac3 succeeded

CRS-2673: Attempting to stop ora-1-work on rac3

CRS-2677: Stop of ora-1-work on rac3 succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on rac3 has completed

CRS-2677: Stop of ora.crsd on rac3 succeeded

CRS-2673: Attempting to stop ora.ctssd on rac3

CRS-2673: Attempting to stop ora.evmd on rac3

CRS-2673: Attempting to stop ora.asm on rac3

CRS-2673: Attempting to stop ora.mdnsd on rac3

CRS-2677: Stop of ora.evmd on rac3 succeeded

CRS-2677: Stop of ora.mdnsd on rac3 succeeded

CRS-2677: Stop of ora.ctssd on rac3 succeeded

CRS-2677: Stop of ora.asm on rac3 succeeded

CRS-2673: Attempting to stop ora.cluster_interconnect.haip on rac3

CRS-2677: Stop of ora.cluster_interconnect.haip on rac3 succeeded

CRS-2673: Attempting to stop ora.cssd on rac3

CRS-2677: Stop of ora.cssd on rac3 succeeded

CRS-2673: Attempting to stop ora.crf on rac3

CRS-2677: Stop of ora.crf on rac3 succeeded

CRS-2673: Attempting to stop ora.gipcd on rac3

CRS-2677: Stop of ora.gipcd on rac3 succeeded

CRS-2673: Attempting to stop ora.gpnpd on rac3

CRS-2677: Stop of ora.gpnpd on rac3 succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac3 has completed

CRS-4133: Oracle High Availability Services has been stopped.

启动

[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

--还是有问题

[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[grid@rac1 ~]$ crs_start -all

[grid@rac1 ~]$ crs_stat -t

Name           Type           Target    State     Host        

------------------------------------------------------------

ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        

ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1

ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        

ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac3

ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        

ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac3

ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               

ora.kingsql.db ora....se.type ONLINE    ONLINE    rac3

ora...-work ora....rk.type ONLINE    ONLINE    rac1

ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac3        

ora.ons        ora.ons.type   ONLINE    ONLINE    rac1

ora....SM1.asm application    ONLINE    ONLINE    rac1        

ora....C1.lsnr application    ONLINE    ONLINE    rac1

ora.rac1.gsd   application    OFFLINE   OFFLINE               

ora.rac1.ons   application    ONLINE    ONLINE    rac1

ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        

ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac1

ora....SM3.asm application    ONLINE    ONLINE    rac3        

ora....C3.lsnr application    ONLINE    ONLINE    rac3

ora.rac3.gsd   application    OFFLINE   OFFLINE               

ora.rac3.ons   application    ONLINE    ONLINE    rac3

ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3        

ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac3

重启rac2操作系统

[root@rac2 ~]# reboot

[root@rac2 ~]#

Broadcast message from root@rac2

        (/dev/pts/0) at 17:40 ...

The system is going down for reboot NOW!

[grid@rac1 ~]$ crs_stat -t

Name           Type           Target    State     Host        

------------------------------------------------------------

ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        

ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1

ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        

ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac3

ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        

ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac3

ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               

ora.kingsql.db ora....se.type ONLINE    ONLINE    rac3

ora...-work ora....rk.type ONLINE    ONLINE    rac1

ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac3        

ora.ons        ora.ons.type   ONLINE    ONLINE    rac1

ora....SM1.asm application    ONLINE    ONLINE    rac1        

ora....C1.lsnr application    ONLINE    ONLINE    rac1

ora.rac1.gsd   application    OFFLINE   OFFLINE               

ora.rac1.ons   application    ONLINE    ONLINE    rac1

ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        

ora....SM2.asm application    ONLINE    ONLINE    rac2

ora....C2.lsnr application    ONLINE    ONLINE    rac2        

ora.rac2.gsd   application    OFFLINE   OFFLINE

ora.rac2.ons   application    ONLINE    ONLINE    rac2        

ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2

ora....SM3.asm application    ONLINE    ONLINE    rac3        

ora....C3.lsnr application    ONLINE    ONLINE    rac3

ora.rac3.gsd   application    OFFLINE   OFFLINE               

ora.rac3.ons   application    ONLINE    ONLINE    rac3

ora.rac3.vip   ora....t1.type ONLINE    ONLINE    rac3

ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac3     

[grid@rac1 ~]$ crsctl status res -t

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.CRS.dg

               ONLINE  ONLINE       rac1                                         

               ONLINE  ONLINE       rac2                                         

ONLINE  ONLINE       rac3

ora.DATA.dg

               ONLINE  ONLINE       rac1                                         

               ONLINE  ONLINE       rac2                                         

               ONLINE  ONLINE       rac3                                         

ora.LISTENER.lsnr

               ONLINE  ONLINE       rac1                                         

               ONLINE  ONLINE       rac2                                         

               ONLINE  ONLINE       rac3                                         

ora.asm

ONLINE  ONLINE       rac1                     Started

               ONLINE  ONLINE       rac2                     Started             

ONLINE  ONLINE       rac3                     Started

ora.gsd

               OFFLINE OFFLINE      rac1                                         

OFFLINE OFFLINE      rac2

               OFFLINE OFFLINE      rac3                                         

ora-1-work

               ONLINE  ONLINE       rac1                                         

               ONLINE  ONLINE       rac2                                         

ONLINE  ONLINE       rac3

ora.ons

               ONLINE  ONLINE       rac1                                         

               ONLINE  ONLINE       rac2                                         

ONLINE  ONLINE       rac3

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       rac3                                         

ora.cvu

      1        ONLINE  ONLINE       rac3                                         

ora.kingsql.db

1        ONLINE  ONLINE       rac3                     Open

2        ONLINE  ONLINE       rac2                     Open

      3        ONLINE  ONLINE       rac1                     Open                

ora.oc4j

1        ONLINE  ONLINE       rac3

ora.rac1.vip

      1        ONLINE  ONLINE       rac1                                         

ora.rac2.vip

1        ONLINE  ONLINE       rac2

ora.rac3.vip

      1        ONLINE  ONLINE       rac3                                         

ora.scan1.vip

      1        ONLINE  ONLINE       rac3                                        

感谢各位的阅读，以上就是“怎么修kingsql中RAC节点故障”的内容了，经过本文的学习后，相信大家对怎么修kingsql中RAC节点故障这一问题有了更深刻的体会，具体使用情况还需要大家实践验证。这里是，小编将为大家推送更多相关知识点的文章，欢迎关注！