• 全部
  • 经验案例
  • 典型配置
  • 技术公告
  • FAQ
  • 漏洞说明
  • 全部
  • 全部
  • 大数据引擎
  • 知了引擎
产品线
搜索
取消
案例类型
发布者
是否解决
是否官方
时间
搜索引擎
匹配模式
高级搜索

某局点高端防火墙OSPF邻居震荡问题排查

  • 0关注
  • 1收藏 432浏览
粉丝:35人 关注:3人

组网及说明

不涉及

 


告警信息

不涉及


问题描述

防火墙对接友商设备某一个业务接口OSPF以及OSPFv3邻居不定时震荡,日志如下:

07:37:10:756%Jan  6 04:31:35:216 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from FULL to INIT.
07:37:10:756%Jan  6 04:31:35:225 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from LOADING to FULL.
07:37:10:756%Jan  6 04:38:22:018 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from FULL to INIT.
07:37:10:756%Jan  6 04:38:46:664 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from LOADING to FULL.
07:37:10:756%Jan  6 04:54:18:250 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from FULL to INIT.
07:37:10:756%Jan  6 04:54:25:756 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from LOADING to FULL.
07:37:10:756%Jan  6 05:01:55:351 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from FULL to INIT.
07:37:10:756%Jan  6 05:02:00:673 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from LOADING to FULL.
07:37:10:756%Jan  6 05:04:08:777 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGIN: ITCuser logged in from 10.136.128.23.
07:37:10:756%Jan  6 05:04:09:992 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/4/SHELL_CMD_MATCHFAIL: -User=ITCuser-IPAddr=10.136.128.23; Command echo $LANG in view shell failed to be matched.
07:37:10:756%Jan  6 05:09:42:835 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGOUT: ITCuser logged out from 10.136.128.23.
07:37:10:756%Jan  6 05:12:28:235 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPFV3/5/OSPFv3_NBR_CHG: OSPFv3 100 Neighbor 10.139.0.2(Route-Aggregation2) received 1-Way and its state from FULL to INIT.
07:37:10:756%Jan  6 05:13:07:218 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPFV3/5/OSPFv3_NBR_CHG: OSPFv3 100 Neighbor 10.139.0.2(Route-Aggregation2) received LoadingDone and its state from LOADING to FULL.
07:37:10:756%Jan  6 05:15:22:826 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGIN: ITCuser logged in from 10.136.128.23.
07:37:10:756%Jan  6 05:15:24:037 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/4/SHELL_CMD_MATCHFAIL: -User=ITCuser-IPAddr=10.136.128.23; Command echo $LANG in view shell failed to be matched.
07:37:10:756%Jan  6 05:23:21:837 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGOUT: ITCuser logged out from 10.136.128.23.
07:37:10:756%Jan  6 05:30:28:215 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPFV3/5/OSPFv3_NBR_CHG: OSPFv3 100 Neighbor 10.139.0.2(Route-Aggregation2) received 1-Way and its state from FULL to INIT.
07:37:10:756%Jan  6 05:30:37:218 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPFV3/5/OSPFv3_NBR_CHG: OSPFv3 100 Neighbor 10.139.0.2(Route-Aggregation2) received LoadingDone and its state from LOADING to FULL.
07:37:10:756%Jan  6 05:41:14:135 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGIN: ITCuser logged in from 10.136.128.23.
07:37:10:756%Jan  6 05:41:15:351 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/4/SHELL_CMD_MATCHFAIL: -User=ITCuser-IPAddr=10.136.128.23; Command echo $LANG in view shell failed to be matched.
07:37:10:756%Jan  6 05:42:26:304 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/4/SHELL_CMD_MATCHFAIL: -User=ITCuser-IPAddr=10.136.128.23; Command dis ospf event-log  in view shell failed to be matched.
07:37:10:756%Jan  6 05:42:43:884 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from FULL to INIT.
07:37:10:756%Jan  6 05:43:00:673 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from LOADING to FULL.
07:37:10:756%Jan  6 05:51:34:984 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/4/SHELL_CMD_MATCHFAIL: -User=ITCuser-IPAddr=10.136.128.23; Command dis interface  HundredGigE 0/0/26 in view shell failed to be matched.
07:37:10:756%Jan  6 05:58:11:204 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGOUT: ITCuser logged out from 10.136.128.23.
07:37:10:756%Jan  6 05:58:12:936 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from FULL to INIT.
07:37:10:756%Jan  6 05:58:30:672 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPF/5/OSPF_NBR_CHG: OSPF 10 Neighbor 10.139.1.121(Route-Aggregation2) changed from LOADING to FULL.
07:37:10:756%Jan  6 06:00:13:308 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGIN: ITCuser logged in from 10.121.4.149.
07:37:10:756%Jan  6 06:00:14:568 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGOUT: ITCuser logged out from 10.121.4.149.
07:37:10:756%Jan  6 06:01:00:262 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGIN: ITCuser logged in from 10.136.128.23.
07:37:10:756%Jan  6 06:01:01:476 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/4/SHELL_CMD_MATCHFAIL: -User=ITCuser-IPAddr=10.136.128.23; Command echo $LANG in view shell failed to be matched.
07:37:10:756%Jan  6 06:13:18:165 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPFV3/5/OSPFv3_NBR_CHG: OSPFv3 100 Neighbor 10.139.0.2(Route-Aggregation2) received 1-Way and its state from FULL to INIT.
07:37:10:756%Jan  6 06:13:35:352 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 SHELL/5/SHELL_LOGOUT: ITCuser logged out from 10.136.128.23.
07:37:10:756%Jan  6 06:13:47:217 2024 JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11 OSPFV3/5/OSPFv3_NBR_CHG: OSPFv3 100 Neighbor 10.139.0.2(Route-Aggregation2) received LoadingDone and its state from LOADING to FULL.

现场要求分析邻居异常中断原因。


过程分析

收集display ospf event-log peer信息如下:

08:49:14:308Date       Time     Local Address   Remote Address  Router ID       Reason
08:49:14:3082024-01-06 08:34:49 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3252024-01-06 08:19:38 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3252024-01-06 07:52:50 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3252024-01-06 07:36:47 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3412024-01-06 06:28:23 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3412024-01-06 06:23:50 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3412024-01-06 05:58:12 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3412024-01-06 05:42:43 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3612024-01-06 05:01:55 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3612024-01-06 04:54:18 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
08:49:14:3782024-01-06 04:38:22 10.139.1.122    10.139.1.121    10.139.0.2      1-Way
 
08:49:14:3782024-01-06 04:31:35 10.139.1.122    10.139.1.121    10.139.0.2      1-Way

 

可以发现邻居震荡是因为收到对端的1-Way报文导致本端邻居震荡。需要对端分析为何发送1-Way报文。

本端周期发送报文,如下:

08:48:15:708
08:48:15:708         OSPF Process 10 with Router ID 10.139.0.3
08:48:15:708                 Hello Log
08:48:15:724
08:48:15:724
08:48:15:724Interface RAGG2
08:48:15:724Neighbor address: 10.139.1.121, NbrID: 10.139.0.2
08:48:15:724First 4 hello packets sent:
08:48:15:724  2023-09-06 02:00:21:577, succeeded
08:48:15:724  2023-09-06 02:00:35:239, succeeded
08:48:15:724  2023-09-06 02:00:45:239, succeeded
08:48:15:755  2023-09-06 02:00:55:239, succeeded
08:48:15:758Last 4 hello packets sent before Full->Init at 2024-01-06 08:34:49:504
08:48:15:758  2024-01-06 08:34:10:664, succeeded
08:48:15:758  2024-01-06 08:34:20:664, succeeded
08:48:15:758  2024-01-06 08:34:30:664, succeeded
08:48:15:758  2024-01-06 08:34:40:664, succeeded
 
08:48:04:670RBM_P<JNSC-P-PUB-CMNET-H3C-M9016V-1-ITC11>dis ospf event-log hello received
08:48:04:701
08:48:04:733         OSPF Process 10 with Router ID 10.139.0.3
08:48:04:733                 Hello Log
08:48:04:733
08:48:04:733
08:48:04:733Interface RAGG2
08:48:04:733Neighbor address: 10.139.1.121, NbrID: 10.139.0.2
08:48:04:733First 4 hello packets received:
08:48:04:733  2023-09-07 14:41:52:642
08:48:04:733  2023-09-07 14:41:52:674
08:48:04:749  2023-09-07 14:42:00:142
08:48:04:749  2023-09-07 14:42:10:142
08:48:04:749Last 4 hello packets received before Full->Init at 2024-01-06 08:34:49:504
08:48:04:749  2024-01-06 08:34:26:203
08:48:04:749  2024-01-06 08:34:36:202
08:48:04:749  2024-01-06 08:34:43:702
08:48:04:765  2024-01-06 08:34:49:504


将以上分析同步现场后,协调对端一起查看,对端显示Reason:DeadTimerExpire。

 

于是建议现场两边同步抓包对比,本端抓包如下:


可以看出10:28:10之前,防火墙hello报文都是10s周期发包,对端10.139.1.121设备给我们发了个DB报文重新主从选举。

分析到现在,问题很明显和对端有关系。但是对端坚持说他们自己的hello报文也是规律发送,未收到华三发送的hello报文。

如果双方均显示正常的话,一般会在中间串一个交换机镜像抓包做仲裁,看到底是谁的问题。可惜客户不同意。

 

本人不信邪,查看了下对端的日志。疑点有两处:

1. 对端的hello包发送的并不规律,并非10 S一次。

2. 本端发送的Hello报文很有规律,10S一次。


对端答复他们机制就是这样,把我整的很无语。


解决方法

对端做了接口替换发现接口卡存在故障,更换故障接口卡之后解决。

 


该案例对您是否有帮助:

您的评价:1

若您有关于案例的建议,请反馈:

0 个评论

该案例暂时没有网友评论

编辑评论

举报

×

侵犯我的权益 >
对根叔知了社区有害的内容 >
辱骂、歧视、挑衅等(不友善)

侵犯我的权益

×

泄露了我的隐私 >
侵犯了我企业的权益 >
抄袭了我的内容 >
诽谤我 >
辱骂、歧视、挑衅等(不友善)
骚扰我

泄露了我的隐私

×

您好,当您发现根叔知了上有泄漏您隐私的内容时,您可以向根叔知了进行举报。 请您把以下内容通过邮件发送到zhiliao@h3c.com 邮箱,我们会尽快处理。
  • 1. 您认为哪些内容泄露了您的隐私?(请在邮件中列出您举报的内容、链接地址,并给出简短的说明)
  • 2. 您是谁?(身份证明材料,可以是身份证或护照等证件)

侵犯了我企业的权益

×

您好,当您发现根叔知了上有关于您企业的造谣与诽谤、商业侵权等内容时,您可以向根叔知了进行举报。 请您把以下内容通过邮件发送到 zhiliao@h3c.com 邮箱,我们会在审核后尽快给您答复。
  • 1. 您举报的内容是什么?(请在邮件中列出您举报的内容和链接地址)
  • 2. 您是谁?(身份证明材料,可以是身份证或护照等证件)
  • 3. 是哪家企业?(营业执照,单位登记证明等证件)
  • 4. 您与该企业的关系是?(您是企业法人或被授权人,需提供企业委托授权书)
我们认为知名企业应该坦然接受公众讨论,对于答案中不准确的部分,我们欢迎您以正式或非正式身份在根叔知了上进行澄清。

抄袭了我的内容

×

原文链接或出处

诽谤我

×

您好,当您发现根叔知了上有诽谤您的内容时,您可以向根叔知了进行举报。 请您把以下内容通过邮件发送到zhiliao@h3c.com 邮箱,我们会尽快处理。
  • 1. 您举报的内容以及侵犯了您什么权益?(请在邮件中列出您举报的内容、链接地址,并给出简短的说明)
  • 2. 您是谁?(身份证明材料,可以是身份证或护照等证件)
我们认为知名企业应该坦然接受公众讨论,对于答案中不准确的部分,我们欢迎您以正式或非正式身份在根叔知了上进行澄清。

对根叔知了社区有害的内容

×

垃圾广告信息
色情、暴力、血腥等违反法律法规的内容
政治敏感
不规范转载 >
辱骂、歧视、挑衅等(不友善)
骚扰我
诱导投票

不规范转载

×

举报说明

提出建议

    +

亲~登录后才可以操作哦!

确定

亲~检测到您登陆的账号未在http://hclhub.h3c.com进行注册

注册后可访问此模块

跳转hclhub

你的邮箱还未认证,请认证邮箱或绑定手机后进行当前操作