设备及版本:
S6805-54HF Feature 6619
现场ADDC组网,设备堆叠应用场景,为leaf角色,流量从AC进,通过PBR指向防火墙,防火墙出外网
%Jun 24 05:21:58:502 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:22:20:098 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.
%Jun 24 05:22:49:805 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.
%Jun 24 05:39:44:856 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:40:00:777 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:41:18:855 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.
%Jun 24 05:41:23:138 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.
现场ADDC 组网中,早上6点左右,所有访问ADDC的业务出现不通,把该service leaf所连接的服务链防火墙端口关了后,业务恢复正常,故障点交换机有大量pbr资源不足告警生成。
%Jun 24 05:21:58:502 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:22:20:098 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.
%Jun 24 05:22:49:805 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.
%Jun 24 05:39:44:856 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:40:00:777 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:41:18:855 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.
%Jun 24 05:41:23:138 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.
%Jun 24 05:41:31:995 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
......
1.从反馈的日志信息看,生成资源不足告警的时间点之前, 没有任何给设备进行配置变更的操作。
%Jun 24 00:57:55:837 2022 6AH+1_云平台_SDN_接入交换机 FTP/6/AUTH: User N/A@10.100.117.29 for connection.
%Jun 24 03:32:50:676 2022 6AH+1_云平台_SDN_接入交换机 FTP/6/AUTH: User N/A@10.100.117.30 for connection.
%Jun 24 03:33:01:679 2022 6AH+1_云平台_SDN_接入交换机 FTP/6/AUTH: User N/A@10.100.117.30 for connection.
%Jun 24 05:21:58:502 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.
%Jun 24 05:22:20:098 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.
%Jun 24 05:22:49:805 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.
2.控制器侧排查反馈故障点前没有给设备进行下发配置的操作。
3.查看底层acl资源,从剩余资源来看应该不是由于资源不足导致。
===============display qos-acl resource===============
Interfaces: XGE1/0/1 to XGE1/0/24, HGE1/0/25 to HGE1/0/30
XGE1/0/31 to XGE1/0/54 (slot 1)
---------------------------------------------------------------------
Type Total Reserved Configured Remaining Usage
---------------------------------------------------------------------
VFP ACL 1024 256 0 768 25%
IFP ACL 18432 6144 606 11682 36%
IFP Meter 3072 384 0 2688 12%
IFP Counter 18432 6144 17 12271 33%
EFP ACL 2048 0 0 2048 0%
EFP Meter 1024 0 0 1024 0%
EFP Counter 1024 0 0 1024 0%
Interfaces: XGE2/0/1 to XGE2/0/24, HGE2/0/25 to HGE2/0/30
XGE2/0/31 to XGE2/0/54 (slot 2)
---------------------------------------------------------------------
Type Total Reserved Configured Remaining Usage
---------------------------------------------------------------------
VFP ACL 1024 256 0 768 25%
IFP ACL 18432 6144 606 11682 36%
IFP Meter 3072 384 0 2688 12%
IFP Counter 18432 6144 17 12271 33%
EFP ACL 2048 0 0 2048 0%
EFP Meter 1024 0 0 1024 0%
EFP Counter 1024 0 0 1024 0%
4.经研发进一步分析确认,从当前收集的信息来看,现场此次故障现象的触发原因、日志报错、底层报错信息都和下述已知问题吻合,判断涉及该已知问题导致。
已知问题:
【S5800 B70D022SP1205】【补丁单】基于R2612P02H27补丁合入:台州联通政务云S6800交换机东西向引流报错。
问题触发原因:从备机vsi虚接口上收到的arp报文,1s中会同步到主设备一次,用于刷新时戳。这个同步的报文,在主设备上判断表项是否一致时,存在问题,导致认为表项发生了变化,刷新了底层pbr,概率导致pbr资源一直被刷新占用,生成资源不足告警。
查看底层有相关报错:
Slot01 2022/06/24 06:31:26:170589 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:26:297575 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:26:419604 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:26:540629 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:26:664111 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:26:788098 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:26:912674 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:27:35500 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot01 2022/06/24 06:31:38:76156 [LINE:480-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:group Aset
Slot01 2022/06/24 06:31:38:76254 [LINE:483-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:entry Qset
Slot01 2022/06/24 06:31:38:76395 [LINE:486-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:entry Aset
Slot01 2022/06/24 06:31:38:76717 [LINE:271-TASK:pbrd-FUNC:Drv_Qacl_Soc_Dump_Qset]:OuterVlanId-[1]L4SrcPort-[1]L4DstPort-[1]EtherType-[1]IpProtocol-[1]HiGig-[1]Stage-[1]InterfaceClassPort-[1]IngressStpState-[1]
Slot01 2022/06/24 06:31:38:77244 [LINE:309-TASK:pbrd-FUNC:Drv_Qacl_Soc_Dump_Aset]:RedirectCancel-[1]DropCancel-[1]L3Switch-[1]RpSwitchToCpuReinstate-[1]YpSwitchToCpuReinstate-[1]GpSwitchToCpuReinstate-[1]
Slot01 2022/06/24 06:31:38:77519 [LINE:3866-TASK:pbrd-FUNC:Drv_Qacl_Bcm_Group_Entry_Install]:bcm_field_entry_prio_set failed,entry 1074180690,pstEntry->uiRtnPrio_Sub 1,ulRet -4
lot01 2022/06/24 06:31:38:77602 [LINE:470-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:Group_Entry_Install failed ,ulRet -14, enPipe=1
Slot02 2022/06/24 06:31:21:661313 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:21:779641 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:21:897748 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:22:20236 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:22:140638 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:22:260726 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:22:379123 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
Slot02 2022/06/24 06:31:22:499117 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b
综上所述,现场现象为当前版本触发已知问题导致,该问题在R6616及之后版本解决,建议升级至R6635版本。
解决方案:
该问题在R6616及之后版本解决,建议升级至R6635版本解决。
该案例暂时没有网友评论
✖
案例意见反馈
亲~登录后才可以操作哦!
确定你的邮箱还未认证,请认证邮箱或绑定手机后进行当前操作