组网及说明
MSR 3620DP配置两块SIC-2E1-F插卡，各出一个serial接口与对端直连
问题描述
两个Serial接口物理协议均up，接口下无任何错包，ping对端直连地址丢包
现象1：两个接口都UP，S1/1丢包，S2/1正常
现象2：shutdown S1/1，S2/1开始丢包；undo shutdown S1/1，S2/1逐渐恢复不丢包，S1/1继续丢包
现象3：shutdown S2/1，S1/1依旧丢包；undo shutdown S2/1，S1/1继续丢包，S2/1不丢包
过程分析
1、通过S1/1和S2/1下发QOS流量统计，初步判定包不是丢在路由器上的
#
acl advanced 3999
rule 0 permit icmp source X.X.X.117 0 destination X.X.X.118 0
rule 5 permit icmp source X.X.X.118 0 destination X.X.X.117 0
#
traffic classifier lt operator and
if-match acl 3999
#
traffic behavior lt
#
qos policy lt
classifier lt behavior lt
#
#
interface Serial1/1
qos apply policy lt inbound
qos apply policy lt outbound
#
<H3C>ping -c 20 X.X.X.117
20 packet(s) transmitted, 13 packet(s) received, 35.0% packet loss
<H3C>
<H3C>dis qos policy interface s1/1
Interface: Serial1/1
Direction: Inbound
Policy: lt
Classifier: lt
Matched : 13 (Packets) 1144 (Bytes)
5-minute statistics:
Forwarded: 0/0 (pps/bps)
Dropped : 0/0 (pps/bps)
Operator: AND
Rule(s) :
If-match acl 3999
Behavior: lt
-none-
Interface: Serial1/1
Direction: Outbound
Policy: lt
Classifier: lt
Matched : 20 (Packets) 1760 (Bytes)
5-minute statistics:
Forwarded: 0/0 (pps/bps)
Dropped : 0/0 (pps/bps)
Operator: AND
Rule(s) :
If-match acl 3999
Behavior: lt
-none-
<H3C>
2、对于以上解释，用户表示可以接受，但是仍觉得现象诡异，要求再深入分析
<H3C>debugging physical packet all interface Serial 1/1 key ffffffffffffffff offset 50 length 16
//查看硬件收发包
<H3C>t d
<H3C>t m
<H3C>ping -c 10 -p ffffffff -s 160 X.X.X.117
PATTERN: 0xffffffff
Ping X.X.X.117 (X.X.X.117): 160 data bytes, press CTRL+C to break
Request time out
160 bytes from X.X.X.117: icmp_seq=1 ttl=255 time=49.937 ms
*Dec 18 00:18:16:429 2021 H3C DRVDBG/7/debugging:
(Serial1/1)PHY/PKT:
Packet Output, Packet Len =192,Partial data as follows:
FF 03 00 21 45 00 00 BC 18 35 00 00 FF 01 5A 1B
17 03 0D 76 17 03 0D 75 08 00 7D D0 06 B3 00 01
61 BD 28 C8 00 05 E8 F0 FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
*Dec 18 00:18:16:436 2021 H3C DRVDBG/7/debugging:
(Serial1/1)PHY/PKT:
Packet Input, Packet Len = 192, Partial data as follows:
FF 03 00 21 45 00 00 BC 13 C9 00 00 FF 01 5E 87
17 03 0D 75 17 03 0D 76 00 00 85 D0 06 B3 00 01
61 BD 28 C8 00 05 E8 F0 FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
Request time out
Request time out
Request time out
160 bytes from X.X.X.117: icmp_seq=5 ttl=255 time=49.907 ms
*Dec 18 00:18:23:282 2021 H3C DRVDBG/7/debugging:
(Serial1/1)PHY/PKT:
Packet Output, Packet Len =192,Partial data as follows:
FF 03 00 21 45 00 00 BC 18 4A 00 00 FF 01 5A 06
17 03 0D 76 17 03 0D 75 08 00 BC 0F 06 B3 00 05
61 BD 28 CF 00 03 AA A8 FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
*Dec 18 00:18:23:290 2021 H3C DRVDBG/7/debugging:
(Serial1/1)PHY/PKT:
Packet Input, Packet Len = 192, Partial data as follows:
FF 03 00 21 45 00 00 BC 13 CA 00 00 FF 01 5E 86
17 03 0D 75 17 03 0D 76 00 00 C4 0F 06 B3 00 05
61 BD 28 CF 00 03 AA A8 FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
Request time out
Request time out
Request time out
160 bytes from X.X.X.117: icmp_seq=9 ttl=255 time=57.101 ms
*Dec 18 00:18:30:142 2021 H3C DRVDBG/7/debugging:
(Serial1/1)PHY/PKT:
Packet Output, Packet Len =192,Partial data as follows:
FF 03 00 21 45 00 00 BC 18 61 00 00 FF 01 59 EF
17 03 0D 76 17 03 0D 75 08 00 F9 1F 06 B3 00 09
61 BD 28 D6 00 01 6D 8F FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
*Dec 18 00:18:30:150 2021 H3C DRVDBG/7/debugging:
(Serial1/1)PHY/PKT:
Packet Input, Packet Len = 192, Partial data as follows:
FF 03 00 21 45 00 00 BC 13 CB 00 00 FF 01 5E 85
17 03 0D 75 17 03 0D 76 00 00 01 20 06 B3 00 09
61 BD 28 D6 00 01 6D 8F FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
--- Ping statistics for X.X.X.117 ---
10 packet(s) transmitted, 3 packet(s) received, 70.0% packet loss
round-trip min/avg/max/std-dev = 49.907/52.315/57.101/3.384 ms
3、以上信息又可以说明，此次ping10个包，丢7个包，实际上是路由器只发了3个包出去，因此判定问题确实存在于路由器上
4、查看路由器接口广播报文情况
[H3C]dis int | in Description|broadcasts
Description: GigabitEthernet0/1 Interface
- unicasts, - broadcasts, - multicasts, - pauses
1 unicasts, 50685821 broadcasts, 179 multicasts, 0 pauses
- unicasts, - broadcasts, - multicasts, - pauses
57 unicasts, 50 broadcasts, 0 multicasts, 2107 pauses
[H3C]
[H3C]dis int | in Description|broadcasts
Description: GigabitEthernet0/1 Interface
- unicasts, - broadcasts, - multicasts, - pauses
2 unicasts, 60159430 broadcasts, 212 multicasts, 0 pauses
- unicasts, - broadcasts, - multicasts, - pauses
69 unicasts, 62 broadcasts, 0 multicasts, 2107 pauses
[H3C]
[H3C]dis int g0/1
GigabitEthernet0/1
Input (normal): 63853971 packets, - bytes
2 unicasts, 63853744 broadcasts, 225 multicasts, 0 pauses
Output (normal): 4352 packets, - bytes
73 unicasts, 65 broadcasts, 0 multicasts, 2107 pauses
5、以上信息可以看出G0/1口的广播报文异常多，查看CPU转发核占用率
[H3C]dis process cpu | in kdrv
148 0.0% 0.0% 0.0% [kdrvmac]
149 0.0% 0.0% 0.0% [kdrvlswtmr]
150 0.0% 0.0% 0.0% [kdrvfwdd1]
151 12.7% 0.2% 12.0% [kdrvfwdd2]
152 1.4% 0.0% 1.5% [kdrvfwdd3]
153 0.0% 0.0% 0.0% [kdrvfwdd4]
154 0.0% 0.0% 0.0% [kdrvfwdd5]
155 0.0% 0.0% 0.0% [kdrvfwdd6]
156 0.0% 0.0% 0.0% [kdrvfwdd7]
[H3C]
[H3C]dis process cpu | in kdrv
148 0.0% 0.0% 0.0% [kdrvmac]
149 0.0% 0.0% 0.0% [kdrvlswtmr]
150 0.0% 0.0% 0.0% [kdrvfwdd1]
151 12.7% 0.2% 12.0% [kdrvfwdd2]
152 0.9% 0.0% 1.5% [kdrvfwdd3]
153 0.0% 0.0% 0.0% [kdrvfwdd4]
154 0.0% 0.0% 0.0% [kdrvfwdd5]
155 0.0% 0.0% 0.0% [kdrvfwdd6]
156
0.0% 0.0% 0.0%
[kdrvfwdd7
6、可以看到，当前一共8个转发核，分摊下来每个转发核在12.5%就已经达到极限，也就是说目前存在单核打满的情况
7、与现场协商，可以shutdown g0/1口
[H3C]inter GigabitEthernet 0/1
[H3C-GigabitEthernet0/1]shutdown
[H3C-GigabitEthernet0/1]
[H3C-GigabitEthernet0/1]dis process cpu | in kdrv
148 0.0% 0.0% 0.0% [kdrvmac]
149 0.0% 0.0% 0.0% [kdrvlswtmr]
150 0.0% 0.0% 0.0% [kdrvfwdd1]
151 3.3% 11.9% 2.4% [kdrvfwdd2]
152 0.3% 1.0% 0.2% [kdrvfwdd3]
153 0.0% 0.0% 0.0% [kdrvfwdd4]
154 0.0% 0.0% 0.0% [kdrvfwdd5]
155 0.0% 0.0% 0.0% [kdrvfwdd6]
156 0.0% 0.0% 0.0% [kdrvfwdd7]
[H3C-GigabitEthernet0/1]
[H3C-GigabitEthernet0/1]dis process cpu | in kdrv
148 0.0% 0.0% 0.0% [kdrvmac]
149 0.0% 0.0% 0.0% [kdrvlswtmr]
150 0.0% 0.0% 0.0% [kdrvfwdd1]
151 0.0% 11.9% 2.4% [kdrvfwdd2]
152 0.0% 1.0% 0.2% [kdrvfwdd3]
153 0.0% 0.0% 0.0% [kdrvfwdd4]
154 0.0% 0.0% 0.0% [kdrvfwdd5]
155 0.0% 0.0% 0.0% [kdrvfwdd6]
156 0.0% 0.0% 0.0% [kdrvfwdd7]
[H3C-GigabitEthernet0/1]qu
[H3C]qu
<H3C>ping -c 30 X.X.X.117
30 packet(s) transmitted, 30 packet(s) received, 0.0% packet loss
<H3C>
<H3C>ping -c 30 X.X.X.113
30 packet(s) transmitted, 30 packet(s) received, 0.0% packet loss
8、可以看到，g0/1口down掉后，转发核降下来了，丢包现象也消失
解决方法
路由器G0/1口收到大量广播包，怀疑下端交换机有环路，登陆后查看交换机确实存在大量MAC漂移，现场排除环路后问题最终解决
12:26:08.056 UTC Wed 01/02/2013
MAC address VLAN Current port Source port Last time Times
XXXX-XXXX-XXXX 200 GE1/0/2 GE1/0/3 2013-01-02 12:25:50 6639968
……
XXXX-XXXX-XXXX 200 GE1/0/3 GE1/0/2 2013-01-02 12:26:24 6656640
该案例暂时没有网友评论
编辑评论
✖
案例意见反馈
亲~登录后才可以操作哦!
确定你的邮箱还未认证，请认证邮箱或绑定手机后进行当前操作