测试环境的一套两节点RAC，一个节点出现故障，启动不了实例

查看节点1的grid日志，发现找不到表决盘

tail -100 /opt/ora11grid/log/vgerndpud853/alertvgerndpud853.log

..

[cssd(3824)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /opt/ora11grid/log/vgerndpud853/cssd/ocssd.log

2016-10-17 07:56:11.768

[cssd(3824)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /opt/ora11grid/log/vgerndpud853/cssd/ocssd.log

2016-10-17 07:56:26.781

[cssd(3824)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /opt/ora11grid/log/vgerndpud853/cssd/ocssd.log

2016-10-17 07:56:41.794

[cssd(3824)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /opt/ora11grid/log/vgerndpud853/cssd/ocssd.log

2016-10-17 07:56:55.095

[/opt/ora11grid/bin/cssdagent(3812)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:34:3} in /opt/ora11grid/log/vgerndpud853/agent/ohasd/oracssdagent_root/oracssdagent_root.log.

2016-10-17 07:56:55.096

[cssd(3824)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /opt/ora11grid/log/vgerndpud853/cssd/ocssd.log

查看grid日志中提到的详细日志，进一步验证了表决盘丢失

tail -100 /opt/ora11grid/log/vgerndpud853/cssd/ocssd.log

..

222499 2016-10-17 08:06:58.959: [   SKGFD][257890048]Discovery with str:/voting/vot1/vot1.data,/voting/vot2/vot2.data,/voting/vot3/vot3.data:

222500

222501 2016-10-17 08:06:58.959: [   SKGFD][257890048]UFS discovery with :/voting/vot1/vot1.data:

222502

222503 2016-10-17 08:06:58.959: [   SKGFD][257890048]OSS discovery with :/voting/vot1/vot1.data:

222504

222505 2016-10-17 08:06:58.959: [   SKGFD][257890048]Discovery advancing to nxt string :/voting/vot2/vot2.data:

222506

222507 2016-10-17 08:06:58.959: [   SKGFD][257890048]UFS discovery with :/voting/vot2/vot2.data:

222508

222509 2016-10-17 08:06:58.959: [   SKGFD][257890048]OSS discovery with :/voting/vot2/vot2.data:

222510

222511 2016-10-17 08:06:58.959: [   SKGFD][257890048]Discovery advancing to nxt string :/voting/vot3/vot3.data:

222512

222513 2016-10-17 08:06:58.959: [   SKGFD][257890048]UFS discovery with :/voting/vot3/vot3.data:

222514

222515 2016-10-17 08:06:58.959: [   SKGFD][257890048]OSS discovery with :/voting/vot3/vot3.data:

222516

222517 2016-10-17 08:06:58.959: [    CSSD][257890048]clssnmvDiskVerify: Successful discovery of 0 disks

222518 2016-10-17 08:06:58.959: [    CSSD][257890048]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery

登陆节点2

查看集群状态，发现节点2的状态正常

crs_stat -t -v

查看表决盘和crs盘的挂载情况，正常

vgerndpud852: /opt/ora11grid/bin # mount | grep vot

/dev/vx/dsk/dg_orabe/v_vot2 on /voting/vot2 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)

/dev/vx/dsk/dg_orabe/v_vot1 on /voting/vot1 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)

/dev/vx/dsk/dg_orabe/v_vot3 on /voting/vot3 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)

vgerndpud852: /opt/ora11grid/bin # mount | grep ocr

/dev/vx/dsk/dg_orabe/v_ocr2 on /ocr/ocr2 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)

/dev/vx/dsk/dg_orabe/v_ocr3 on /ocr/ocr3 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)

/dev/vx/dsk/dg_orabe/v_ocr1 on /ocr/ocr1 type vxfs (rw,mntlock=VCS,cluster,crw,delaylog,largefiles,ioerror=mdisable)

在节点1，查看表决盘和crs盘的挂载情况，发现盘没有挂载上

/opt/ora11grid[FRWK]:mount | grep vot

/opt/ora11grid[FRWK]:mount | grep ocr

重新挂载盘，重启集群，RAC恢复正常