某局点 S6800 BGP同时震荡案例分析

两台S6800（CE）通过运营商链路和对面两台PE分别建立BGP邻居

两台S6800（CE）通过运营商链路和对面两台PE建立BGP邻居，近期持续出现BGP邻居震荡问题。

CE01：

%@162636%Mar 31 10:08:14:659 2022 CE01 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.1/10.0.0.3, LD/RD:259/53176, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: UP->DOWN, Diag: 1 (Control Detection Time Expired)

%@162637%Mar 31 10:08:14:662 2022 CE01 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.3 state has changed from ESTABLISHED to IDLE for session down event received from BFD.

%@162638%Mar 31 10:08:43:498 2022 CE01 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.3 State is changed from OPENCONFIRM to ESTABLISHED.

%@162639%Mar 31 10:08:43:500 2022 CE01 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.1/10.0.0.3, LD/RD:261/53178, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: DOWN->INIT, Diag: 0 (No Diagnostic)

%@162640%Mar 31 10:08:43:502 2022 CE01 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.1/10.0.0.3, LD/RD:261/53178, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: INIT->UP, Diag: 0 (No Diagnostic)

%@162645%Mar 31 10:09:02:848 2022 CE01 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.1/10.0.0.3, LD/RD:261/53178, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: UP->DOWN, Diag: 1 (Control Detection Time Expired)

%@162646%Mar 31 10:09:02:851 2022 CE01 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.3 state has changed from ESTABLISHED to IDLE for session down event received from BFD.

%@162647%Mar 31 10:09:33:498 2022 CE01 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.3 State is changed from OPENCONFIRM to ESTABLISHED.

%@162648%Mar 31 10:09:33:499 2022 CE01 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.1/10.0.0.3, LD/RD:261/53178, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: DOWN->INIT, Diag: 0 (No Diagnostic)

%@162649%Mar 31 10:09:33:501 2022 CE01 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.1/10.0.0.3, LD/RD:261/53178, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: INIT->UP, Diag: 0 (No Diagnostic)

CE02：

%@208562%Mar 31 10:08:17:222 2022 CE02 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.2/10.0.0.4, LD/RD:259/53171, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: UP->DOWN, Diag: 1 (Control Detection Time Expired)

%@208563%Mar 31 10:08:17:224 2022 CE02 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.4 state has changed from ESTABLISHED to IDLE for session down event received from BFD.

%@208564%Mar 31 10:08:47:952 2022 CE02 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.4 State is changed from OPENCONFIRM to ESTABLISHED.

%@208565%Mar 31 10:08:47:954 2022 CE02 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.2/10.0.0.4, LD/RD:259/53173, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: DOWN->INIT, Diag: 0 (No Diagnostic)

%@208566%Mar 31 10:08:47:955 2022 CE02 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.2/10.0.0.4, LD/RD:259/53173, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: INIT->UP, Diag: 0 (No Diagnostic)

%@208579%Mar 31 10:09:05:414 2022 CE02 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.2/10.0.0.4, LD/RD:259/53173, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: UP->DOWN, Diag: 1 (Control Detection Time Expired)

%@208580%Mar 31 10:09:05:416 2022 CE02 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.4 state has changed from ESTABLISHED to IDLE for session down event received from BFD.

%@208581%Mar 31 10:09:34:698 2022 CE02 BGP/5/BGP_STATE_CHANGED:

 BGP.: 10.0.0.4 State is changed from OPENCONFIRM to ESTABLISHED.

%@208582%Mar 31 10:09:34:700 2022 CE02 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.2/10.0.0.4, LD/RD:259/53173, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: DOWN->INIT, Diag: 0 (No Diagnostic)

%@208583%Mar 31 10:09:34:701 2022 CE02 BFD/5/BFD_CHANGE_FSM: Sess[10.0.128.2/10.0.0.4, LD/RD:259/53173, Interface:N/A, SessType:Ctrl, LinkType:INET], Ver:1, Sta: INIT->UP, Diag: 0 (No Diagnostic)agnostic)

1、查看两台S6800建立BGP的接口底层信息，发现在对应时间点有STP阻塞记录（底层信息需要加8小时），每次阻塞时间大约30秒。

debug l2 slot 1 c 0 stp/rec/show/44

debug l2 slot 1 c 0 stp/rec/show/45

 CE01

115: 2022/3/31 2:8.12'848387 MSTP set stg 1 block.

116: 2022/3/31 2:8.43'64552 MSTP set stg 1 forward.

117: 2022/3/31 2:9.1'121339 MSTP set stg 1 block.

118: 2022/3/31 2:9.32'64479 MSTP set stg 1 forward.

CE02

44: 2022/3/31 2:8.15'498593 MSTP set stg 1 block.

45: 2022/3/31 2:8.45'719417 MSTP set stg 1 forward.

46: 2022/3/31 2:9.3'499974 MSTP set stg 1 block.

47: 2022/3/31 2:9.33'718702 MSTP set stg 1 forward.

2、在CE02的诊断，可以看到如下stp根桥变化记录，根桥会在1/0/10和1/0/13下面的28b4-4887-8d40和bc22-4778-1000设备之间切换。

Port Ten-GigabitEthernet1/0/10           

   Role change         : DESI->ROOT            

   Time                : 2022/03/31 10:09:01

   Port priority       : 32768.28b4-4887-8d40 38 32768.4873-9729-7b46 0

                         32768.b845-f462-e8db 128.28 128.10

   Designated priority : 32768.28b4-4887-8d40 58 32768.4873-9729-7b46 0

                         32768.4873-9729-7b46 128.10 128.10

 Port Ten-GigabitEthernet1/0/13           

   Role change         : DESI->ROOT            

   Time                : 2022/03/31 10:08:50

   Port priority       : 0.bc22-4778-1000 0 32768.4873-9729-7b46 0

                         0.bc22-4778-1000 128.27 128.13

   Designated priority : 0.bc22-4778-1000 2 32768.4873-9729-7b46 0

                         32768.4873-9729-7b46 128.13 128.13 

3、查看CE02的log发现，根桥切换前1/0/13口有up/down。正常情况下，现场的根桥是接在1/0/13下的，但是当13口震荡后，根桥就会切换到1/0/10口下的设备。

%@208550%Mar 31 10:08:05:261 2022 CE02 IFNET/3/PHY_UPDOWN: Physical state on the interface Ten-GigabitEthernet1/0/13 changed to up.

%@208551%Mar 31 10:08:05:267 2022 CE02 IFNET/5/LINK_UPDOWN: Line protocol state on the interface Ten-GigabitEthernet1/0/13 changed to up.

%@208558%Mar 31 10:08:13:059 2022 CE02 IFNET/3/PHY_UPDOWN: Physical state on the interface Ten-GigabitEthernet1/0/13 changed to down.

%@208559%Mar 31 10:08:13:059 2022 CE02 IFNET/5/LINK_UPDOWN: Line protocol state on the interface Ten-GigabitEthernet1/0/13 changed to down.

%@208567%Mar 31 10:08:48:637 2022 CE02 IFNET/3/PHY_UPDOWN: Physical state on the interface Ten-GigabitEthernet1/0/13 changed to up.

%@208568%Mar 31 10:08:48:637 2022 CE02 IFNET/5/LINK_UPDOWN: Line protocol state on the interface Ten-GigabitEthernet1/0/13 changed to up.

%@208575%Mar 31 10:09:01:379 2022 CE02 IFNET/3/PHY_UPDOWN: Physical state on the interface Ten-GigabitEthernet1/0/13 changed to down.

%@208576%Mar 31 10:09:01:379 2022 CE02 IFNET/5/LINK_UPDOWN: Line protocol state on the interface Ten-GigabitEthernet1/0/13 changed to down.

4、由于现场建立BGP的接口1/0/44和45口没有配置为边缘端口，且对端是运营商链路，没有运行BPDU，导致在网络中根桥切换后，这两个端口无法快速收敛，需要阻塞2个forwarding delay即30秒后才能正常转发。

本次故障原因是： CE02连接根桥的接口有up/down，导致stp根桥切换，网络重新收敛。但建立BGP邻居的接口没有配置边缘端口，使接口阻塞最终引发BGP震荡。

需要连接运营商链路的接口配置边缘端口或者关闭stp规避。