核心交换机S10510X堆叠,核心跟所有的汇聚交换机跑了OSPF。
部分的OSPF邻居会经常从full变成down状态,并且10秒-30秒左右会重新成功建立ospf邻居,故障大概几十分钟复现一次。
OSPF报错信息:显示计时器超时
[S10510X]%Feb 28 16:06:32:663 2024 S10510X IFNET/4/IF_EGRESS_DROP_RECOVER: -MDC=1-Chassis=1-Slot=2; Packet loss recovers in queue 2 of Ten-GigabitEthernet1/2/0/15.
%Feb 28 16:06:54:519 2024 S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.
%Feb 28 16:06:55:308 2024 S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.
%Feb 28 16:06:57:169 2024 S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from LOADING to FULL.
[S10510X]%Feb 28 16:07:32:729 2024 S10510X IFNET/4/IF_EGRESS_DROP: -MDC=1-Chassis=1-Slot=2; Packet loss occurs in queue 2 of Ten-GigabitEthernet1/2/0/15.
%Feb 28 16:08:32:832 2024 S10510X IFNET/4/IF_EGRESS_DROP_RECOVER: -MDC=1-Chassis=1-Slot=2; Packet loss recovers in queue 2 of Ten-GigabitEthernet1/2/0/15.
%Feb 28 16:10:12:308 2024 S10510X OSPF/5/OSPF_NBR_CHG_REASON: -MDC=1; OSPF 100 Area 0.0.0.0 Router 172.16.254.1(Vlan1007)
CPU usage: 1%, IfMTU: 1500, Neighbor address: 172.16.251.26, NbrID:172.16.254.8 changed from Full to DOWN because the dead timer expired at 2024-02-28 16:10:12:308.
Last 4 hello packets received at:
2024-02-28 16:08:07:142
2024-02-28 16:08:17:141
2024-02-28 16:08:27:141
2024-02-28 16:08:37:142
Last 4 hello packets sent at:
2024-02-28 16:09:37:308
2024-02-28 16:09:47:308
2024-02-28 16:09:57:308
2024-02-28 16:10:07:308
%Feb 28 16:10:12:308 2024 S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from FULL to DOWN.
%Feb 28 16:10:37:182 2024 S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from LOADING to FULL.
%Feb 28 16:14:12:524 2024 S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.
%Feb 28 16:14:12:741 2024 S10510X STP/6/STP_NOTIFIED_TC: -MDC=1; Instance 0's port Bridge-Aggregation6 was notified a topology change.
%Feb 28 16:17:32:160 2024 S10510X DRVPLAT/4/SOFTCAR DROP: -MDC=1-Chassis=1-Slot=2;
Cos=14, Drop at Stage=5, StageCnt=47997, TotalCnt=18120089, possible protocol IPV6_ND_PASS/VSI_IPV6_ND_PASS/IPV6_ND_DEST/IPV6_PING/IPV6_ND_DAI/VSI IPV6 NA/VSI IPV6 NS/VSI_IPV6_PROXY_ND_PASS/VSI_IPV6_ND_DEST/MAC_MOVE
%Feb 28 16:17:33:417 2024 S10510X IFNET/4/IF_EGRESS_DROP: -MDC=1-Chassis=1-Slot=2; Packet loss occurs in queue 2 of Ten-GigabitEthernet1/2/0/15.
%Feb 28 16:31:46:486 2024 S10510X DRVPLAT/4/SOFTCAR DROP: -MDC=1-Chassis=1-Slot=2;
PktType= IPV4_TTL , srcMAC=000d-489a-201d, Drop From Interface=Ten-GigabitEthernet1/2/0/48 at Stage=8, StageCnt=4627, TotalCnt=395874
%Feb 28 16:35:17:660 2024 S10510X DRVPLAT/4/SOFTCAR DROP: -MDC=1-Chassis=2-Slot=2;
PktType= IPV4_TTL , srcMAC=000d-489a-201d, Drop From Interface=Ten-GigabitEthernet2/2/0/48 at Stage=34, StageCnt=12714, TotalCnt=390740
%Feb 28 16:44:09:309 2024 S10510X OSPF/5/OSPF_NBR_CHG_REASON: -MDC=1; OSPF 100 Area 0.0.0.0 Router 172.16.254.1(Vlan1010)
CPU usage: 1%, IfMTU: 1500, Neighbor address: 172.16.251.38, NbrID:172.16.251.38 changed from Full to DOWN because the dead timer expired at 2024-02-28 16:44:09:308.
Last 4 hello packets received at:
2024-02-28 16:42:47:795
2024-02-28 16:42:57:635
2024-02-28 16:43:07:479
2024-02-28 16:43:17:327
Last 4 hello packets sent at:
2024-02-28 16:43:39:308
2024-02-28 16:43:49:308
2024-02-28 16:43:59:308
2024-02-28 16:44:09:308
%Feb 28 16:44:09:309 2024 S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.38(Vlan-interface1010) changed from FULL to DOWN.
%Feb 28 16:44:19:459 2024 S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.38(Vlan-interface1010) changed from LOADING to FULL.
%Feb 28 16:48:32:308 2024 S10510X OSPF/5/OSPF_NBR_CHG_REASON: -MDC=1; OSPF 100 Area 0.0.0.0 Router 172.16.254.1(Vlan1007)
CPU usage: 1%, IfMTU: 1500, Neighbor address: 172.16.251.26, NbrID:172.16.254.8 changed from Full to DOWN because the dead timer expired at 2024-02-28 16:48:32:308.
Last 4 hello packets received at:
2024-02-28 16:46:37:202
2024-02-28 16:46:47:202
2024-02-28 16:46:57:202
2024-02-28 16:47:07:203
Last 4 hello packets sent at:
2024-02-28 16:47:57:308
2024-02-28 16:48:07:308
2024-02-28 16:48:17:308
2024-02-28 16:48:27:308
%Feb 28 16:48:32:308 2024 S10510X OSPF/5/OSPF_NBR_CHG: -MDC=1; OSPF 100 Neighbor 172.16.251.26(Vlan-interface1007) changed from FULL to DOWN.
1. 检查链路问题,替换线和接口模块故障依旧。
2. 检查过router-id,没有冲突的情况。
3. 查看ACL资源是足够的。
4. 查看设备软件丢包,发现OSPF报文超限速。
====debug rxtx softcar show chassis 1 slot 2====
ID Type RcvPps Rcv_All DisPkt_All Pps Dyn Swi Hash ACLmax
0 ROOT 0 16442445 5418 1500 S On SMAC 0
1 ISIS 0 0 0 800 D On SMAC 8
2 ESIS 0 0 0 100 S On SMAC 8
3 CLNP 0 0 0 100 S On SMAC 8
4 VRRP 0 0 0 1500 S On SMAC 8
5 UNKNOWN_IPV4MC 0 0 0 300 S On SMAC 8
6 UNKNOWN_IPV6MC 0 0 0 300 S On SMAC 8
7 IPV4_MC_RIP 0 15 0 150 S On SMAC 8
8 IPV4_BC_RIP 0 0 0 150 S On SMAC 8
9 MCAST_NTP 0 0 0 100 S On SMAC 8
10 BCAST_NTP 0 0 0 100 S On SMAC 8
11 IPV4_MC_OSPF_5 1 338578346 187931503 400 S On SMAC 8
12 IPV4_MC_OSPF_6 0 0 0 400 S On SMAC 8
13 IPV4_UC_OSPF 0 0 0 800 S On SMAC 8
14 IPV4_MC_PIM 0 0 0 500 D On SMAC 8
15 IPV4_UC_PIM 0 0 0 500 D On SMAC 8
16 IPV4_IGMP 0 197582789 0 2500 S On SMAC 8
5. 设备logbuffer里面有软件丢包的日志
11 IPV4_MC_OSPF_5 1 338578346 187931503 400 S On SMAC 8
%Feb 28 17:35:46:251 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;
PktType= IPV4_MC_OSPF_5 , srcMAC=c407-7814-d801, Drop From Interface=Ten-GigabitEthernet1/2/0/18 at Stage=46, StageCnt=42677, TotalCnt=7882099
%Feb 28 18:22:34:795 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;
PktType= IPV6_ND_PASS , Drop at Stage=0, StageCnt=308422, TotalCnt=308422, Max Rate Interface=Ten-GigabitEthernet1/2/0/18
%Feb 28 18:36:03:687 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;
PktType= IPV4_MC_OSPF_5 , srcMAC=c407-7814-d801, Drop From Interface=Ten-GigabitEthernet1/2/0/18 at Stage=8, StageCnt=10762, TotalCnt=7892861
%Feb 28 18:50:31:786 2024 YNPESZ-CORE-S10510X DRVPLAT/4/SOFTCAR DROP: -Chassis=1-Slot=2;
PktType= IPV6_ND_PASS , srcMAC=c407-7814-d801, Drop From Interface=Ten-GigabitEthernet1/2/0/18 at Stage=63, StageCnt=13238764, TotalCnt=497883366
查看Ten-GigabitEthernet1/2/0/18接口配置,发现设备Ten-GigabitEthernet1/2/0/18口配置成了报文反射口:
#
interface Ten-GigabitEthernet1/2/0/18
port link-mode bridge
mirroring-group 1 reflector-port
#
interface Ten-GigabitEthernet1/2/0/15
port link-mode bridge
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan 7 131 to 132 306 402 1322 2001 to 2002 3005
mirroring-group 1 mirroring-port both
port link-aggregation group 22
#
现场镜像的ospf报文反射口仍会处理OSPF报文导致OSPF大量超限速,把镜像先取消观察下现象,镜像取消之后,OSPF邻居恢复正常,问题解决。
该案例暂时没有网友评论
✖
案例意见反馈
亲~登录后才可以操作哦!
确定你的邮箱还未认证,请认证邮箱或绑定手机后进行当前操作