OSPF的邻居不定时从FULL状态变成DOWN状态

核心交换机S10510X堆叠，核心跟所有的汇聚交换机跑了OSPF。

部分的OSPF邻居会经常从full变成down状态，并且10秒-30秒左右会重新成功建立ospf邻居，故障大概几十分钟复现一次。

OSPF报错信息：显示计时器超时

 [S10510X]%Feb 28 16:06:32:663 2024  S10510X IFNET/4/IF_EGRESS_DROP_RECOVER: -MDC=1-Chassis=1-Slot=2; Packet loss recovers in queue 2 of Ten-GigabitEthernet1/2/0/15.

%Feb 28 16:06:54:519 2024  S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.

%Feb 28 16:06:55:308 2024  S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.

%Feb 28 16:06:57:169 2024  S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from LOADING to FULL.

 [S10510X]%Feb 28 16:07:32:729 2024  S10510X IFNET/4/IF_EGRESS_DROP: -MDC=1-Chassis=1-Slot=2; Packet loss occurs in queue 2 of Ten-GigabitEthernet1/2/0/15.

%Feb 28 16:08:32:832 2024  S10510X IFNET/4/IF_EGRESS_DROP_RECOVER: -MDC=1-Chassis=1-Slot=2; Packet loss recovers in queue 2 of Ten-GigabitEthernet1/2/0/15.

%Feb 28 16:10:12:308 2024  S10510X OSPF/5/OSPF_NBR_CHG_REASON: -MDC=1; OSPF 100 Area 0.0.0.0 Router 172.16.254.1(Vlan1007)

CPU usage: 1%, IfMTU: 1500, Neighbor address: 172.16.251.26, NbrID:172.16.254.8 changed from Full to DOWN because the dead timer expired at 2024-02-28 16:10:12:308.

Last 4 hello packets received at:

  2024-02-28 16:08:07:142

  2024-02-28 16:08:17:141

  2024-02-28 16:08:27:141

  2024-02-28 16:08:37:142

Last 4 hello packets sent at:

  2024-02-28 16:09:37:308

  2024-02-28 16:09:47:308

  2024-02-28 16:09:57:308

  2024-02-28 16:10:07:308

%Feb 28 16:10:12:308 2024  S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from FULL to DOWN.

%Feb 28 16:10:37:182 2024  S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from LOADING to FULL.

%Feb 28 16:14:12:524 2024  S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.

%Feb 28 16:14:12:741 2024  S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.

%Feb 28 16:17:32:160 2024  S10510X DRVPLAT/4/SOFTCAR DROP: -MDC=1-Chassis=1-Slot=2;

 Cos=14, Drop at Stage=5, StageCnt=47997, TotalCnt=18120089, possible protocol IPV6_ND_PASS/VSI_IPV6_ND_PASS/IPV6_ND_DEST/IPV6_PING/IPV6_ND_DAI/VSI IPV6 NA/VSI IPV6 NS/VSI_IPV6_PROXY_ND_PASS/VSI_IPV6_ND_DEST/MAC_MOVE

%Feb 28 16:17:33:417 2024  S10510X IFNET/4/IF_EGRESS_DROP: -MDC=1-Chassis=1-Slot=2; Packet loss occurs in queue 2 of Ten-GigabitEthernet1/2/0/15.

%Feb 28 16:31:46:486 2024  S10510X DRVPLAT/4/SOFTCAR DROP: -MDC=1-Chassis=1-Slot=2;

 PktType= IPV4_TTL , srcMAC=000d-489a-201d, Drop From Interface=Ten-GigabitEthernet1/2/0/48 at Stage=8, StageCnt=4627, TotalCnt=395874

%Feb 28 16:35:17:660 2024  S10510X DRVPLAT/4/SOFTCAR DROP: -MDC=1-Chassis=2-Slot=2;

 PktType= IPV4_TTL , srcMAC=000d-489a-201d, Drop From Interface=Ten-GigabitEthernet2/2/0/48 at Stage=34, StageCnt=12714, TotalCnt=390740

%Feb 28 16:44:09:309 2024  S10510X OSPF/5/OSPF_NBR_CHG_REASON: -MDC=1; OSPF 100 Area 0.0.0.0 Router 172.16.254.1(Vlan1010)

CPU usage: 1%, IfMTU: 1500, Neighbor address: 172.16.251.38, NbrID:172.16.251.38 changed from Full to DOWN because the dead timer expired at 2024-02-28 16:44:09:308.

Last 4 hello packets received at:

  2024-02-28 16:42:47:795

  2024-02-28 16:42:57:635

  2024-02-28 16:43:07:479

  2024-02-28 16:43:17:327

Last 4 hello packets sent at:

  2024-02-28 16:43:39:308

  2024-02-28 16:43:49:308

  2024-02-28 16:43:59:308

  2024-02-28 16:44:09:308

%Feb 28 16:44:09:309 2024  S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.38(Vlan-interface1010) changed from FULL to DOWN.

%Feb 28 16:44:19:459 2024  S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.38(Vlan-interface1010) changed from LOADING to FULL.

%Feb 28 16:48:32:308 2024  S10510X OSPF/5/OSPF_NBR_CHG_REASON: -MDC=1; OSPF 100 Area 0.0.0.0 Router 172.16.254.1(Vlan1007)

CPU usage: 1%, IfMTU: 1500, Neighbor address: 172.16.251.26, NbrID:172.16.254.8 changed from Full to DOWN because the dead timer expired at 2024-02-28 16:48:32:308.

Last 4 hello packets received at:

  2024-02-28 16:46:37:202

  2024-02-28 16:46:47:202

  2024-02-28 16:46:57:202

  2024-02-28 16:47:07:203

Last 4 hello packets sent at:

  2024-02-28 16:47:57:308

  2024-02-28 16:48:07:308

  2024-02-28 16:48:17:308

  2024-02-28 16:48:27:308

%Feb 28 16:48:32:308 2024  S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from FULL to DOWN.

1. 检查链路问题，替换线和接口模块故障依旧。

2. 检查过router-id，没有冲突的情况。

3. 查看ACL资源是足够的。

4. 查看设备软件丢包，发现OSPF报文超限速。

 ====debug rxtx softcar show chassis 1 slot 2====  

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

0   ROOT                0      16442445   5418       1500 S   On  SMAC 0

1   ISIS                0      0          0          800  D   On  SMAC 8

2   ESIS                0      0          0          100  S   On  SMAC 8

3   CLNP                0      0          0          100  S   On  SMAC 8

4   VRRP                0      0          0          1500 S   On  SMAC 8

5   UNKNOWN_IPV4MC      0      0          0          300  S   On  SMAC 8

6   UNKNOWN_IPV6MC      0      0          0          300  S   On  SMAC 8

7   IPV4_MC_RIP         0      15         0          150  S   On  SMAC 8

8   IPV4_BC_RIP         0      0          0          150  S   On  SMAC 8

9   MCAST_NTP           0      0          0          100  S   On  SMAC 8

10  BCAST_NTP           0      0          0          100  S   On  SMAC 8

11  IPV4_MC_OSPF_5      1      338578346  187931503  400  S   On  SMAC 8

12  IPV4_MC_OSPF_6      0      0          0          400  S   On  SMAC 8

13  IPV4_UC_OSPF        0      0          0          800  S   On  SMAC 8

14  IPV4_MC_PIM         0      0          0          500  D   On  SMAC 8

15  IPV4_UC_PIM         0      0          0          500  D   On  SMAC 8

16  IPV4_IGMP           0      197582789  0          2500 S   On  SMAC 8

5. 设备logbuffer里面有软件丢包的日志

11  IPV4_MC_OSPF_5      1      338578346  187931503  400  S   On  SMAC 8

%Feb 28 17:35:46:251 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;

 PktType= IPV4_MC_OSPF_5 , srcMAC=c407-7814-d801, Drop From Interface=Ten-GigabitEthernet1/2/0/18 at Stage=46, StageCnt=42677, TotalCnt=7882099

%Feb 28 18:22:34:795 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;

 PktType= IPV6_ND_PASS , Drop at Stage=0, StageCnt=308422, TotalCnt=308422, Max Rate Interface=Ten-GigabitEthernet1/2/0/18

%Feb 28 18:36:03:687 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;

 PktType= IPV4_MC_OSPF_5 , srcMAC=c407-7814-d801, Drop From Interface=Ten-GigabitEthernet1/2/0/18 at Stage=8, StageCnt=10762, TotalCnt=7892861

%Feb 28 18:50:31:786 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;

 PktType= IPV6_ND_PASS , srcMAC=c407-7814-d801, Drop From Interface=Ten-GigabitEthernet1/2/0/18 at Stage=63, StageCnt=13238764, TotalCnt=497883366

查看Ten-GigabitEthernet1/2/0/18接口配置，发现设备Ten-GigabitEthernet1/2/0/18口配置成了报文反射口：

#

interface Ten-GigabitEthernet1/2/0/18

port link-mode bridge

mirroring-group 1 reflector-port

#

interface Ten-GigabitEthernet1/2/0/15

port link-mode bridge

port link-type trunk

undo port trunk permit vlan 1

port trunk permit vlan 7 131 to 132 306 402 1322 2001 to 2002 3005

mirroring-group 1 mirroring-port both

port link-aggregation group 22

#

现场镜像的ospf报文反射口仍会处理OSPF报文导致OSPF大量超限速，把镜像先取消观察下现象，镜像取消之后，OSPF邻居恢复正常，问题解决。