SR88聚合口流量负载不均衡问题经典案例

某客户组网如图所示，PPPOE用户网关在BRAS上，BRAS进行QINQ终结，PPPOE用户的流量经过BRAS后，再流向SR88到Internet；流量回来的路径相反，在使用过程中，用户发现交换机到SR88方向聚合口的流量负载时均衡的，SR88跟BRAS之间的流量负载也是很均衡的，但是从SR88回来的流量，SR88聚合口的成员接口G0/1的output方向占用带宽59%，G0/2的output方向占用带宽8%，流量负载悬殊很大。

现场接口流量如下：

dis int GigabitEthernet 0/1

GigabitEthernet9/0/7 current state: UP

IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 3822-d674-4a00

Description: To WengAn-5800

Loopback is not set

Media type is optical fiber, Port hardware type is 1000_BASE_LX_SFP

1000Mbps-speed mode, full-duplex mode

Link speed type is force link, link duplex type is force link

Flow-control is not enabled

The Maximum Frame Length is 1552

Broadcast MAX-ratio: 100%

Unicast MAX-ratio: 100%

Multicast MAX-ratio: 100%

PVID: 1

Mdi type: auto

Link delay is 1(sec)

Port link-type: trunk

  VLAN passing  : 22, 51, 80, 98, 540-579, 732-747, 916-947, 1502-1503, 1507-1513, 3999, 4081

  VLAN permitted: 22, 51, 80, 98, 540-579, 732-747, 916-947, 1502-1503, 1507-1513, 3999, 4081

  Trunk port encapsulation: IEEE 802.1q

Port priority: 0

Peak value of input: 100536560 bytes/sec, at 2016-02-17 16:50:34

 Peak value of output: 108970080 bytes/sec, at 2016-02-15 12:55:19

 Last 300 seconds input:  28280 packets/sec 5251220 bytes/sec 5%

Last 300 seconds output:  57950 packets/sec 72895890 bytes/sec 59%

Input (total):  781673740178 packets, 171227564059126 bytes

         - unicasts, - broadcasts, - multicasts, - pauses

Input (normal):  781673719853 packets, 171227555119645 bytes

         762537050317 unicasts, 1654698593 broadcasts, 17481970943 multicasts, 0 pauses

Input:  20325 input errors, 0 runts, 1012 giants, 0 throttles

         2 CRC, 19311 frame, 0 overruns, - aborts

         - ignored, - parity errors

Output (total): 977457443439 packets, 1089097796266960 bytes

         - unicasts, - broadcasts, - multicasts, - pauses

Output (normal): 977457443439 packets, 1089097796266960 bytes

         961811630721 unicasts, 7204914884 broadcasts, 8440897834 multicasts, 0 pauses

Output: 0 output errors, - underruns, - buffer failures

         0 aborts, 0 deferred, 0 collisions, 0 late collisions

         - lost carrier, - no carrier

dis int GigabitEthernet 0/2

GigabitEthernet9/0/12 current state: UP

IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 3822-d674-4a00

Description: To WengAn-5800

Loopback is not set

Media type is optical fiber, Port hardware type is 1000_BASE_LX_SFP

1000Mbps-speed mode, full-duplex mode

Link speed type is force link, link duplex type is force link

Flow-control is not enabled

The Maximum Frame Length is 1552

Broadcast MAX-ratio: 100%

Unicast MAX-ratio: 100%

Multicast MAX-ratio: 100%

PVID: 1

Mdi type: auto

Link delay is 1(sec)

Port link-type: trunk

  VLAN passing  : 22, 51, 80, 98, 540-579, 732-747, 916-947, 1502-1503, 1507-1513, 3999, 4081

  VLAN permitted: 22, 51, 80, 98, 540-579, 732-747, 916-947, 1502-1503, 1507-1513, 3999, 4081

  Trunk port encapsulation: IEEE 802.1q

Port priority: 0

Peak value of input: 6147570 bytes/sec, at 2016-02-18 22:31:43

 Peak value of output: 12977830 bytes/sec, at 2016-02-19 13:31:23

 Last 300 seconds input:  27690 packets/sec 4248090 bytes/sec 4%

Last 300 seconds output:  7660 packets/sec 9270250 bytes/sec 8%

Input (total):  2012022214 packets, 348067891188 bytes

         - unicasts, - broadcasts, - multicasts, - pauses

Input (normal):  2012022214 packets, 348067891188 bytes

         2009638926 unicasts, 1467026 broadcasts, 916262 multicasts, 0 pauses

Input:  0 input errors, 0 runts, 0 giants, 0 throttles

         0 CRC, 0 frame, 0 overruns, - aborts

         - ignored, - parity errors

Output (total): 580317190 packets, 674377758834 bytes

         - unicasts, - broadcasts, - multicasts, - pauses

Output (normal): 580317190 packets, 674377758834 bytes

         578647473 unicasts, 1388766 broadcasts, 280951 multicasts, 0 pauses

Output: 0 output errors, - underruns, - buffer failures

         0 aborts, 0 deferred, 0 collisions, 0 late collisions

         - lost carrier, - no carrier

链路聚合所使用的单板为SPC单板，查看该SPC单板的链路聚合负载分担方式：

dis link-aggregation  load-sharing  mode 

Link-Aggregation Load-Sharing Mode:

Layer 2 traffic: destination-mac address, source-mac address      

Layer 3 traffic: destination-mac address, source-mac address,     

                 destination-ip address,  source-ip address       

Layer 4 traffic: destination-mac address, source-mac address,     

                 destination-ip address,  source-ip address,      

                 destination-port,        source-port             

MPLS traffic   : destination-mac address, source-mac address,     

                 mpls-label1,             mpls-label2,            

                 mpls-label3             

正常情况下该单板是可以正确负载分担的。

然后，现在需要的就是了解到该网络的各个流量模型和组网，经过了解，从PPPOE用户到SR88设备都走的是二层转发，根据DMAC和SMAC算法hash出来。SR88和BRAS之间走的是三层转发，根据DMAC + SMAC + DIP + SIP算法hash出来。所以目前只能抓包了解AGG1两个物理接口的流量模型，分析负载不均衡的原因。经抓包，分析得到，SR88的G0/1的output方向几乎全为PPPOE业务，G0/2的output方向一半为PPPOE业务，一半为IP数据。并且发现DMAC为奇数的都hash到G0/1口，DMAC为偶数的都hash到G0/2口，通过用户了解到，PPPOE用户采用的机顶盒为同一厂家烧制的，并且该用户采购的机顶盒大部分为MAC为奇数的机顶盒，这样造成了聚合组两个物理接口流量悬殊很大的情况。

现场可以为该聚合口再增加一个端口，二层报文就是取同一条流的SMAC和DMAC的0～5bit进行异或，异或后mod取余，2个端口就是mod2（余数0，1），3个端口就是mod3（余数0，1，2），这样的话，流量可以hash得开。

建议在流量负载不均的时候，在流量没有按理负载的时候，最好抓包查看流量的特征。