CAS半数主机管理网中断触发虚拟机HA迁移

CAS集群使用两个刀箱，每刀箱9个节点。

一个刀箱升级固件，网络模块重启，此时未升级固件刀箱中的主机由于与一半的主机心跳网络不通，并且编号最小的节点也是不通的节点之一，根据HA半数原则判断机制，触发主机fence，虚拟机HA迁移。

1.通过主机syslog日志分析

如下主机是由于管理网HA心跳与半数主机心跳不通，并且与编号最小的节点不通，导致触发fence，详细的日志记录如下：

Nov 16 00:06:42 cvknode-171 kernel: [332263.151942] (kworker/9:1,366,9):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 9 (dlm 9), lowest: 1 (unreachable)

Nov 16 00:06:42 cvknode-171 kernel: [332263.151944] (kworker/9:1,366,9):o2quo_make_decision:483 ERROR: fencing this node because it is connected to a half-quorum of 9 out of 18 nodes which doesn't include the lowest active node 1//超过一半的主机心跳不通并且与节点号最小的节点不通，所以触发fence ，虚拟机HA。

Nov 16 00:06:42 cvknode-171 kernel: [332263.151953] (kworker/9:1,366,9):o2quo_insert_fault_type:202 File system(AC6B8B57F42E446A9974ED07E7DC5B98) is suffering fault(0:TCP fault leading to being fenced)//管理网心跳超时导致触发fence

Nov 16 00:06:42 cvknode-173 kernel: [332264.159033] (kworker/12:1,376,12):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 9 (dlm 9), lowest: 1 (unreachable)

Nov 16 00:06:42 cvknode-173 kernel: [332264.159035] (kworker/12:1,376,12):o2quo_make_decision:483 ERROR: fencing this node because it is connected to a half-quorum of 9 out of 18 nodes which doesn't include the lowest active node 1

Nov 16 00:06:42 cvknode-173 kernel: [332264.159044] (kworker/12:1,376,12):o2quo_insert_fault_type:202 File system(28189F552C6248718B5B71B7E2E80106) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:34 cvknode-174 kernel: [332182.668952] (kworker/30:1,395,30):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 9 (dlm 9), lowest: 1 (unreachable)

Nov 16 00:05:34 cvknode-174 kernel: [332182.668955] (kworker/30:1,395,30):o2quo_make_decision:483 ERROR: fencing this node because it is connected to a half-quorum of 9 out of 18 nodes which doesn't include the lowest active node 1

Nov 16 00:05:34 cvknode-174 kernel: [332182.668968] (kworker/30:1,395,30):o2quo_insert_fault_type:202 File system(FCB7CA2B4BCC4A6CB2A6B7E18BDF8F8F) is suffering fault(0:TCP fault leading to being fenced)

2.如下节点此时间点与其他主机心跳全部断开触发fence，详细日志记录如下：

Nov 16 00:05:43 cvknode-154 kernel: [ 9853.786416] (kworker/7:1,364,7):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:43 cvknode-154 kernel: [ 9853.786418] (kworker/7:1,364,7):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:43 cvknode-154 kernel: [ 9853.786426] (kworker/7:1,364,7):o2quo_insert_fault_type:202 File system(AC6B8B57F42E446A9974ED07E7DC5B98) is suffering fault(0:TCP fault leading to being fenced) //管理网心跳超时导致触发fence

Nov 16 00:05:42 cvknode-155 kernel: [ 9857.854288] (kworker/42:1,406,42):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:42 cvknode-155 kernel: [ 9857.854290] (kworker/42:1,406,42):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:42 cvknode-155 kernel: [ 9857.854299] (kworker/42:1,406,42):o2quo_insert_fault_type:202 File system(AC6B8B57F42E446A9974ED07E7DC5B98) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:43 cvknode-175 kernel: [ 9962.437417] (kworker/6:1,363,6):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:43 cvknode-175 kernel: [ 9962.437418] (kworker/6:1,363,6):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:43 cvknode-175 kernel: [ 9962.437427] (kworker/6:1,363,6):o2quo_insert_fault_type:202 File system(AC6B8B57F42E446A9974ED07E7DC5B98) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:41 cvknode-176 kernel: [ 9966.439724] (kworker/0:5,1021,0):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:41 cvknode-176 kernel: [ 9966.439725] (kworker/0:5,1021,0):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:41 cvknode-176 kernel: [ 9966.439734] (kworker/0:5,1021,0):o2quo_insert_fault_type:202 File system(AC6B8B57F42E446A9974ED07E7DC5B98) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:41 cvknode-177 kernel: [ 9954.174021] (kworker/0:4,1004,0):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:41 cvknode-177 kernel: [ 9954.174023] (kworker/0:4,1004,0):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:41 cvknode-177 kernel: [ 9954.174034] (kworker/0:4,1004,0):o2quo_insert_fault_type:202 File system(FCB7CA2B4BCC4A6CB2A6B7E18BDF8F8F) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:43 cvknode-178 kernel: [ 9896.748707] (kworker/0:0,3,0):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (reachable)

Nov 16 00:05:43 cvknode-178 kernel: [ 9896.748709] (kworker/0:0,3,0):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:43 cvknode-178 kernel: [ 9896.748717] (kworker/0:0,3,0):o2quo_insert_fault_type:202 File system(6DFEDA05B77446A2A0F8BE7483C160A6) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:38 cvknode-179 kernel: [ 9894.769940] (kworker/10:1,367,10):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:38 cvknode-179 kernel: [ 9894.769942] (kworker/10:1,367,10):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:38 cvknode-179 kernel: [ 9894.769951] (kworker/10:1,367,10):o2quo_insert_fault_type:202 File system(E436ED0C55F34684BCECF16F4A2C652B) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:42 cvknode-189 kernel: [ 9917.321935] (kworker/22:1,483,22):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:42 cvknode-189 kernel: [ 9917.321936] (kworker/22:1,483,22):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:42 cvknode-189 kernel: [ 9917.321944] (kworker/22:1,483,22):o2quo_insert_fault_type:202 File system(6DFEDA05B77446A2A0F8BE7483C160A6) is suffering fault(0:TCP fault leading to being fenced)

Nov 16 00:05:41 cvknode-190 kernel: [ 9862.097434] (kworker/59:1,519,59):o2quo_make_decision:445 heartbeating: 18 (dlm 18), connected: 1 (dlm 1), lowest: 1 (unreachable)

Nov 16 00:05:41 cvknode-190 kernel: [ 9862.097437] (kworker/59:1,519,59):o2quo_make_decision:474 ERROR: fencing this node because it is only connected to 1 nodes and 9 is needed to make a quorum out of 18 heartbeating nodes

Nov 16 00:05:41 cvknode-190 kernel: [ 9862.097445] (kworker/59:1,519,59):o2quo_insert_fault_type:202 File system(AC6B8B57F42E446A9974ED07E7DC5B98) is suffering fault(0:TCP fault leading to being fenced)

1. 保证网络稳定性，避免出现两个刀箱中间网络中断的情况。

2. 建议新增节点，独立于两个刀箱外，对于出现整个箱故障的情况，可以保证剩余的节点数量大于半数。

3. 后期对主机维护时将主机进入维护模式从集群中隔离，此时将不会对集群造成影响。