某局点 ADDC环境 leaf S6805 异常PBR资源不足告警

设备及版本： S6805-54HF       Feature 6619

现场ADDC组网，设备堆叠应用场景，为leaf角色，流量从AC进，通过PBR指向防火墙，防火墙出外网

%Jun 24 05:21:58:502 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:22:20:098 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.

%Jun 24 05:22:49:805 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.

%Jun 24 05:39:44:856 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:40:00:777 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:41:18:855 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.

%Jun 24 05:41:23:138 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.

现场ADDC 组网中，早上6点左右，所有访问ADDC的业务出现不通，把该service leaf所连接的服务链防火墙端口关了后，业务恢复正常，故障点交换机有大量pbr资源不足告警生成。

%Jun 24 05:21:58:502 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:22:20:098 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.

%Jun 24 05:22:49:805 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.

%Jun 24 05:39:44:856 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:40:00:777 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:41:18:855 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.

%Jun 24 05:41:23:138 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_1033 because of insufficient hardware resources.

%Jun 24 05:41:31:995 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: -Slot=2; Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

......

1．从反馈的日志信息看，生成资源不足告警的时间点之前， 没有任何给设备进行配置变更的操作。

%Jun 24 00:57:55:837 2022 6AH+1_云平台_SDN_接入交换机 FTP/6/AUTH: User N/A@10.100.117.29 for connection.

%Jun 24 03:32:50:676 2022 6AH+1_云平台_SDN_接入交换机 FTP/6/AUTH: User N/A@10.100.117.30 for connection.

%Jun 24 03:33:01:679 2022 6AH+1_云平台_SDN_接入交换机 FTP/6/AUTH: User N/A@10.100.117.30 for connection.

%Jun 24 05:21:58:502 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70002 because of insufficient hardware resources.

%Jun 24 05:22:20:098 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.

%Jun 24 05:22:49:805 2022 6AH+1_云平台_SDN_接入交换机 PBR4/4/PBR_HARDWARE_ERROR: Failed to update the policy SDN_SC_L3_70001 because of insufficient hardware resources.

2．控制器侧排查反馈故障点前没有给设备进行下发配置的操作。

3．查看底层acl资源，从剩余资源来看应该不是由于资源不足导致。

  ===============display qos-acl resource=============== 

Interfaces: XGE1/0/1 to XGE1/0/24, HGE1/0/25 to HGE1/0/30

            XGE1/0/31 to XGE1/0/54 (slot 1)

---------------------------------------------------------------------

Type             Total      Reserved   Configured Remaining  Usage

---------------------------------------------------------------------

VFP ACL          1024       256        0          768        25%

IFP ACL          18432      6144       606        11682      36%

IFP Meter        3072       384        0          2688       12%

IFP Counter      18432      6144       17         12271      33%

EFP ACL          2048       0          0          2048       0%

EFP Meter        1024       0          0          1024       0%

EFP Counter      1024       0          0          1024       0%

Interfaces: XGE2/0/1 to XGE2/0/24, HGE2/0/25 to HGE2/0/30

            XGE2/0/31 to XGE2/0/54 (slot 2)

---------------------------------------------------------------------

Type             Total      Reserved   Configured Remaining  Usage

---------------------------------------------------------------------

VFP ACL          1024       256        0          768        25%

IFP ACL          18432      6144       606        11682      36%

IFP Meter        3072       384        0          2688       12%

IFP Counter      18432      6144       17         12271      33%

EFP ACL          2048       0          0          2048       0%

EFP Meter        1024       0          0          1024       0%

EFP Counter      1024       0          0          1024       0%

4．经研发进一步分析确认，从当前收集的信息来看，现场此次故障现象的触发原因、日志报错、底层报错信息都和下述已知问题吻合，判断涉及该已知问题导致。

已知问题：

【S5800 B70D022SP1205】【补丁单】基于R2612P02H27补丁合入：台州联通政务云S6800交换机东西向引流报错。

问题触发原因：从备机vsi虚接口上收到的arp报文，1s中会同步到主设备一次，用于刷新时戳。这个同步的报文，在主设备上判断表项是否一致时，存在问题，导致认为表项发生了变化，刷新了底层pbr，概率导致pbr资源一直被刷新占用，生成资源不足告警。

查看底层有相关报错：

Slot01 2022/06/24 06:31:26:170589 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:26:297575 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:26:419604 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:26:540629 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:26:664111 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:26:788098 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:26:912674 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:27:35500 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot01 2022/06/24 06:31:38:76156 [LINE:480-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:group Aset 

Slot01 2022/06/24 06:31:38:76254 [LINE:483-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:entry Qset 

Slot01 2022/06/24 06:31:38:76395 [LINE:486-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:entry Aset 

Slot01 2022/06/24 06:31:38:76717 [LINE:271-TASK:pbrd-FUNC:Drv_Qacl_Soc_Dump_Qset]:OuterVlanId-[1]L4SrcPort-[1]L4DstPort-[1]EtherType-[1]IpProtocol-[1]HiGig-[1]Stage-[1]InterfaceClassPort-[1]IngressStpState-[1]

Slot01 2022/06/24 06:31:38:77244 [LINE:309-TASK:pbrd-FUNC:Drv_Qacl_Soc_Dump_Aset]:RedirectCancel-[1]DropCancel-[1]L3Switch-[1]RpSwitchToCpuReinstate-[1]YpSwitchToCpuReinstate-[1]GpSwitchToCpuReinstate-[1]

Slot01 2022/06/24 06:31:38:77519 [LINE:3866-TASK:pbrd-FUNC:Drv_Qacl_Bcm_Group_Entry_Install]:bcm_field_entry_prio_set failed,entry 1074180690,pstEntry->uiRtnPrio_Sub 1,ulRet -4

lot01 2022/06/24 06:31:38:77602 [LINE:470-TASK:pbrd-FUNC:Drv_Qacl_Soc_Entry_Install]:Group_Entry_Install failed ,ulRet -14, enPipe=1

Slot02 2022/06/24 06:31:21:661313 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:21:779641 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:21:897748 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:22:20236 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:22:140638 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:22:260726 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:22:379123 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

Slot02 2022/06/24 06:31:22:499117 [LINE:624-TASK:pbrd-FUNC:drv_pbr_mod_add_if_policy]:call DRV_QACL_PBR_Control failed. ifIndex = 0x882, ulRet = 0x4001000b

综上所述，现场现象为当前版本触发已知问题导致，该问题在R6616及之后版本解决，建议升级至R6635版本。

解决方案：

该问题在R6616及之后版本解决，建议升级至R6635版本解决。