某局点 S12516F-AF CAR限速不生效问题

组网说明：两台S125F-AF的聚合2和聚合3接口都是上行对外的出接口，组网运行ISIS路由协议。S125F-AF的下行口HundredGigE2/0/11--- HundredGigE2/0/14连接的是用户1，对这些接口进行了入方向的CAR限速，限速47G。下行口XGE4/0/2---XGE4/0/25连接的是用户2，未做限速。

从设备诊断中连接两个用户的下行接口、上行聚合口的成员口近5s的出入流量总和来看，限速了的用户1 inbound占239 G、未限速的用户2 inbound占195.6 G、上行聚合口2/聚合口3 outbound总和为430 G，排除部分设备自身的协议报文及可能存在的未被限速的流量来看，转发的被限速用户报文达430-195.6=234.4G，但是理论上接口下限速值为47 G，应该是不会超过47*4=188G，所以怀疑CAR限速未生效。

通过对设备进行排查：

1．接口入方向限速47G时，以接口HundredGigE 2/0/11为例，通过dis qos policy interface HundredGigE 2/0/11多次查看，发现接口下Red packets计数是有不断增长的， 说明从接口进来的报文是有被CAR限速染色丢弃的。

 [FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: youyun_limit_speed

   Classifier: server

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: host

      Committed Access Rate:

        CIR 47000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 1503360605537 (Packets)

        Red packets   : 35928332805 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: youyun_limit_speed

   Classifier: server

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: host

      Committed Access Rate:

        CIR 47000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 1503364842057 (Packets)

        Red packets   : 35928417527 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: youyun_limit_speed

   Classifier: server

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: host

      Committed Access Rate:

        CIR 47000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 1503368293192 (Packets)

        Red packets   : 35928481373 (Packets)

2.接口限速47G时，进一步查看限速的底层acl信息，发现acl成功下发，并且限速的四个接口都是有限速流量命中的，说明限速是生效了。

  ====debug port mapping chassis 1 slot 2==== 

[Interface]       [Unit] [Port] [Name] [Combo?] [Active?] [IfIndex] [MID] [Link]

===============================================================================

HGE2/0/11        1       5     ce5     no        no      0x1ff     2     up

HGE2/0/12        1       1     ce1     no        no      0x200     2     up

HGE2/0/13        2       21    ce21    no        no      0x201     5     up

HGE2/0/14        2       17    ce17    no        no      0x202     5     up

[FJXIM-PS-IDC-SW01-H3CS12516F-probe]debug qacl show slot 2 c 2  verbose 0 acl-type 120

========

Acl-Type MQC L3 Port, Stage IFP, SinglePort, Installed, Active

L2 PROGRAM : Prio Mjr/Sub 4/15, Group 7 [0], enRtnHealth 1, Entry 306, Mdc 1

PolicyID 100, CBMapID 0, ClassID 100, BehaviorID 100 [IN]

ACL GroupNo : 3021, RuleID : 1000 V4v6 1

Rule Match --------

        In Port: 21

        Lookup: STP forwarding, 0x18, 0x18  Arad Only Set Stp and L3Route

        EtherType: 0x800, 0xffff

        Number-of-tags: 0x0, 0xf

Actions --------

        CAR cir 0x2cd29c0, cbs 0x7a, pir 0x0, pbs 0x7a, pps 0, kbps 1, mode srTCM color blind

        Account mode  packets,  green and red

        Grn Permit

        Red Deny

        Yel Permit

Stat L2 PROGRAM : 0x9000001f

Accounting: Green 2137431230, NoGreen 362895796

MeterPolicerId: 120

L2 PROGRAM : Bank 8 Location 0 HIT(read-clear): YES

========

Acl-Type MQC L3 Port, Stage IFP, SinglePort, Installed, Active

L2 PROGRAM : Prio Mjr/Sub 4/15, Group 7 [0], enRtnHealth 1, Entry 284, Mdc 1

PolicyID 100, CBMapID 0, ClassID 100, BehaviorID 100 [IN]

ACL GroupNo : 3021, RuleID : 1000 V4v6 1

Rule Match --------

        In Port: 17

        Lookup: STP forwarding, 0x18, 0x18  Arad Only Set Stp and L3Route

        EtherType: 0x800, 0xffff

        Number-of-tags: 0x0, 0xf

Actions --------

        CAR cir 0x2cd29c0, cbs 0x7a, pir 0x0, pbs 0x7a, pps 0, kbps 1, mode srTCM color blind

        Account mode  packets,  green and red

        Grn Permit

        Red Deny

        Yel Permit

Stat L2 PROGRAM : 0x9000005b

Accounting: Green 1187297687, NoGreen 1035645483

MeterPolicerId: 121

L2 PROGRAM : Bank 8 Location 1 HIT(read-clear): YES

[FJXIM-PS-IDC-SW01-H3CS12516F-probe]debug qacl show slot 2 c 1  verbose 0 acl-type 120

========

Acl-Type MQC L3 Port, Stage IFP, SinglePort, Installed, Active

L2 PROGRAM : Prio Mjr/Sub 4/15, Group 7 [0], enRtnHealth 1, Entry 274, Mdc 1

PolicyID 100, CBMapID 0, ClassID 100, BehaviorID 100 [IN]

ACL GroupNo : 3021, RuleID : 1000 V4v6 1

Rule Match --------

        In Port: 5

        Lookup: STP forwarding, 0x18, 0x18  Arad Only Set Stp and L3Route

        EtherType: 0x800, 0xffff

        Number-of-tags: 0x0, 0xf

Actions --------

        CAR cir 0x2cd29c0, cbs 0x7a, pir 0x0, pbs 0x7a, pps 0, kbps 1, mode srTCM color blind

        Account mode  packets,  green and red

        Grn Permit

        Red Deny

        Yel Permit

Stat L2 PROGRAM : 0x9000001f

Accounting: Green 769576384, NoGreen 1348298398

MeterPolicerId: 79

L2 PROGRAM : Bank 8 Location 0 HIT(read-clear): YES

========

Acl-Type MQC L3 Port, Stage IFP, SinglePort, Installed, Active

L2 PROGRAM : Prio Mjr/Sub 4/15, Group 7 [0], enRtnHealth 1, Entry 295, Mdc 1

PolicyID 100, CBMapID 0, ClassID 100, BehaviorID 100 [IN]

ACL GroupNo : 3021, RuleID : 1000 V4v6 1

Rule Match --------

        In Port: 1

        Lookup: STP forwarding, 0x18, 0x18  Arad Only Set Stp and L3Route

        EtherType: 0x800, 0xffff

        Number-of-tags: 0x0, 0xf

Actions --------

        CAR cir 0x2cd29c0, cbs 0x7a, pir 0x0, pbs 0x7a, pps 0, kbps 1, mode srTCM color blind

        Account mode  packets,  green and red

        Grn Permit

        Red Deny

        Yel Permit

Stat L2 PROGRAM : 0x90000055

Accounting: Green 2107624429, NoGreen 1730619348

MeterPolicerId: 80

L2 PROGRAM : Bank 8 Location 1 HIT(read-clear): YES

3.通过调整接口CAR限速值的大小，分析发现：

首先查看接口流量带宽近5s占用率达近50%

<FJXIM-PS-IDC-SW01-H3CS12516F>DIS INT HundredGigE 2/0/11

HundredGigE2/0/11

Current state: UP

Line protocol state: UP

Description: To_NS403_Mod7-5-37U_KAOPUYUN_HGE1/0/1

Bandwidth: 100000000 kbps

Maximum transmission unit: 1500

Allow jumbo frames to pass

Broadcast max-ratio: 100%

Unknown-multicast max-ratio: 100%

Unicast max-ratio: 100%

Internet address: 192.168.40.1/30 (primary)

IP packet frame type: Ethernet II, hardware address: 7057-bff4-b201

IPv6 packet frame type: Ethernet II, hardware address: 7057-bff4-b201

Media type is optical fiber, port hardware type is 100G_BASE_LR4_QSFP28

Ethernet port mode: LAN

Port priority: 0

100Gbps-speed mode, full-duplex mode

Link speed type is autonegotiation, link duplex type is autonegotiation

Flow-control is not enabled

The maximum frame length is 12288

Last link flapping: 25 weeks 0 days 6 hours 43 minutes

Last clearing of counters: Never

 Peak input rate: 11109142836 bytes/sec, at 2021-10-08 18:35:38

 Peak output rate: 2343764421 bytes/sec, at 2021-10-27 21:34:33

 Last 5 second input: 4600932 packets/sec 6206166279 bytes/sec 50%

 Last 5 second output: 2283975 packets/sec 689643067 bytes/sec 5%

 Input (total):  60176570802678 packets, 81445277394635951 bytes

         60176570286286 unicasts, 8 broadcasts, 516384 multicasts, 0 pauses

 Input (normal):  60176570791996 packets, - bytes

         60176570284988 unicasts, 8 broadcasts, 516384 multicasts, 0 pauses

 Input:  0 input errors, 0 runts, 0 giants, 0 throttles

         0 CRC, 0 frame, - overruns, 0 aborts

         - ignored, - parity errors

 Output (total): 28297218271752 packets, 10378954204366743 bytes

         28297216023619 unicasts, 18 broadcasts, 2248115 multicasts, 0 pauses

 Output (normal): 28297218270627 packets, - bytes

         28297216023176 unicasts, 18 broadcasts, 2248115 multicasts, 0 pauses

 Output: 0 output errors, - underruns, 0 buffer failures

         0 aborts, 0 deferred, 0 collisions, 0 late collisions

         0 lost carrier, - no carrier

接口限速47G时（小于50%），多次连续查看接口计数发现Red packets不断增长，且丢弃的报文数增长速度近五位数增加：

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: youyun_limit_speed

   Classifier: server

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: host

      Committed Access Rate:

        CIR 47000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 1503360605537 (Packets)

        Red packets   : 35928332805 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: youyun_limit_speed

   Classifier: server

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: host

      Committed Access Rate:

        CIR 47000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 1503364842057 (Packets)

        Red packets   : 35928417527 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: youyun_limit_speed

   Classifier: server

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: host

      Committed Access Rate:

        CIR 47000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 1503368293192 (Packets)

        Red packets   : 35928481373 (Packets)

接口限速增大至70G时（大于50%），多次连续查看接口计数发现Red packets仍有不断增长，但是相比于限速47G时，增长明显缓慢，丢弃的报文数增长速度近三位数增加：

 [FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]dis qos policy  interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 70000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 65475024 (Packets)

        Red packets   : 13203 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]

[FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]

[FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]dis qos policy  interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 70000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 71054640 (Packets)

        Red packets   : 13941 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]

[FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]

[FJXIM-PS-IDC-SW01-H3CS12516F-HundredGigE2/0/11]dis qos policy  interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 70000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 74382393 (Packets)

        Red packets   : 14605 (Packets)

接口限速进一步增大至80G时（大于50%），再次多次连续查看接口计数发现Red packets仍在增长，但是但是相比于限速70G时，增长再次变缓慢，丢弃的报文数增长速度近两位数增加：

[FJXIM-PS-IDC-SW01-H3CS12516F-behavior-test]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 80000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 34403843 (Packets)

        Red packets   : 171 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F-behavior-test]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 80000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 40511749 (Packets)

        Red packets   : 206 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F-behavior-test]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 80000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 46898104 (Packets)

        Red packets   : 229 (Packets)

接口限速增大至90G时（大于50%），再次多次连续查看接口计数发现Red packets基本不在增长，但是可以发现还会存在突然丢弃个位数的情况：

 [FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 151357187 (Packets)

        Red packets   : 0 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 155224154 (Packets)

        Red packets   : 0 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 159853575 (Packets)

        Red packets   : 0 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 165019592 (Packets)

        Red packets   : 0 (Packets)

 [FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 219680665 (Packets)

        Red packets   : 7 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 239890595 (Packets)

        Red packets   : 7 (Packets)

[FJXIM-PS-IDC-SW01-H3CS12516F]dis qos policy interface HundredGigE 2/0/11

Interface: HundredGigE2/0/11

  Direction: Inbound

  Policy: test

   Classifier: test

     Operator: AND

     Rule(s) :

      If-match acl 3021

     Behavior: test

      Committed Access Rate:

        CIR 90000000 (kbps), CBS 15360 (Bytes), EBS 0 (Bytes)

        Green action  : pass

        Yellow action : pass

        Red action    : discard

        Green packets : 246020057 (Packets)

        Red packets   : 7 (Packets)

经过上述测试结果，可以发现现网流量不稳定，存在锯齿形的突发流量（burst流量）。

综上所述，排查发现接口CAR限速是生效的。但是由于客户现场流量存在锯齿形burst流量，设备是基于RFC标准实现的，即在RFC实现标准下，burst流量是会导致出向流量大于限速流量的情况，要想达到稳定的限速结果只有测试仪打稳定的流量才可以达到。

另外建议如果没有特殊要求，CBS值（桶深）不需要自己配置，取设备默认值即可。