某局点S7606镜像流量在高峰期流量减少问题

某局点反馈核心设备S7606版本是Release 7182在业务高峰期时镜像目的端口流量比镜像源端口流量少5%到6%，其他时段镜像流量是正常的。

下图红色框中是镜像源端口的高峰期流量

 

下图红色框中是镜像目的1口的流量

 

下图红色框中是镜像目的2口的流量

很明显可以看出镜像流量少了6G多

设备镜像相关配置如下

#

 mirroring-group 2 local

 mirroring-group 4 local

#

interface Bridge-Aggregation910

 port link-type trunk

 undo port trunk permit vlan 1

 mirroring-group 4 monitor-port

#

interface Bridge-Aggregation915

 port link-type trunk

 undo port trunk permit vlan 1

 mirroring-group 2 monitor-port

 #

Mirroring group 2:

Mirroring port:

        Ten-GigabitEthernet1/6/0/1  Outbound

        Ten-GigabitEthernet1/6/0/8  Outbound

        Ten-GigabitEthernet1/7/0/8  Outbound

        Ten-GigabitEthernet2/5/0/1  Outbound

        Ten-GigabitEthernet2/6/0/1  Outbound

        Ten-GigabitEthernet2/6/0/8  Outbound

        Ten-GigabitEthernet2/7/0/8  Outbound

Monitor port: Bridge-Aggregation915

Aggregate Interface: Bridge-Aggregation915

Aggregation Mode: Static

Loadsharing Type: Shar

Management VLAN : None

  Port             Status  Priority Oper-Key

-------------------------------------------------------------------------

  XGE1/5/0/2       S       32768    18       

  XGE1/5/0/3       S       32768    18       

  XGE1/5/0/4       S       32768    18       

  XGE2/5/0/2       S       32768    18       

  XGE2/5/0/3       S       32768    18       

  XGE2/5/0/4       S       32768    18

Mirroring group 4:

Mirroring port:

        Ten-GigabitEthernet1/2/0/1  Outbound

        Ten-GigabitEthernet1/3/0/1  Outbound

        Ten-GigabitEthernet1/4/0/8  Outbound

        Ten-GigabitEthernet1/5/0/1  Outbound

        Ten-GigabitEthernet2/2/0/1  Outbound

        Ten-GigabitEthernet2/3/0/1  Outbound

        Ten-GigabitEthernet2/4/0/8  Outbound

Monitor port: Bridge-Aggregation910

Aggregate Interface: Bridge-Aggregation910

Aggregation Mode: Static

Loadsharing Type: Shar

Management VLAN : None

  Port             Status  Priority Oper-Key

-------------------------------------------------------------------------

  XGE1/2/0/2       S       32768    11       

  XGE1/3/0/2       S       32768    11       

  XGE1/6/0/4       S       32768    11       

  XGE2/2/0/2       S       32768    11       

  XGE2/3/0/2       S       32768    11       

  XGE2/6/0/4       S       32768    11

排查过程如下

第一步

分别计算高峰期镜像源端口和目的端口300秒和5秒流量统计如下，这里只列出5秒的

Mirroring group 2

镜像源端口流量如下，总的流量是5.6*2+5.2*2+5.1+5.3+5.4=37.4G

Ten-GigabitEthernet1/6/0/1

Last 5 second output: 555070 packets/sec 690467196 bytes/sec 56%

Ten-GigabitEthernet1/6/0/8

Last 5 second output: 525493 packets/sec 648377524 bytes/sec 52%

Ten-GigabitEthernet1/7/0/8

Last 5 second output: 521915 packets/sec 643415663 bytes/sec 52%

Ten-GigabitEthernet2/5/0/1

Last 5 second output: 513485 packets/sec 632245698 bytes/sec 51%

Ten-GigabitEthernet2/6/0/1

Last 5 second output: 535041 packets/sec 659945402 bytes/sec 53%

Ten-GigabitEthernet2/6/0/8 

Last 5 second output: 556199 packets/sec 692836471 bytes/sec 56%

Ten-GigabitEthernet2/7/0/8

Last 5 second output: 535851 packets/sec 666425049 bytes/sec 54%

镜像目的接口流量如下5.9*3+5.6*3=34.5

Ten-GigabitEthernet1/5/0/2

Last 5 second output: 565851 packets/sec 694025774 bytes/sec 56%

Ten-GigabitEthernet1/5/0/3

Last 5 second output: 593157 packets/sec 734567498 bytes/sec 59%

Ten-GigabitEthernet1/5/0/4

Last 5 second output: 592102 packets/sec 733389734 bytes/sec 59%

Ten-GigabitEthernet2/5/0/2

Last 5 second output: 581707 packets/sec 729468029 bytes/sec 59%

Ten-GigabitEthernet2/5/0/3

Last 5 second output: 561833 packets/sec 691812747 bytes/sec 56%

Ten-GigabitEthernet2/5/0/4

Last 5 second output: 560908 packets/sec 692152731 bytes/sec 56%

Mirroring group 4:

镜像源端口流量如下，总的流量是5.6*2+5.2*2+5.7+5.5+5.4=38.2

Ten-GigabitEthernet1/2/0/1

Last 5 second output: 563058 packets/sec 691543523 bytes/sec 56%

Ten-GigabitEthernet1/3/0/1

Last 5 second output: 543876 packets/sec 664868661 bytes/sec 54%

Ten-GigabitEthernet1/4/0/8

Last 5 second output: 559758 packets/sec 689571856 bytes/sec 56%

Ten-GigabitEthernet1/5/0/1

Last 5 second output: 534715 packets/sec 651687428 bytes/sec 52%

Ten-GigabitEthernet2/2/0/1

Last 5 second output: 557809 packets/sec 684019689 bytes/sec 55%

Ten-GigabitEthernet2/3/0/1

Last 5 second output: 527180 packets/sec 647957778 bytes/sec 52%

Ten-GigabitEthernet2/4/0/8

Last 5 second output: 566328 packets/sec 709580162 bytes/sec 57%     

镜像目的接口流量如下5.9*2+6*2+6.2+5.7=35.7G            

Ten-GigabitEthernet1/2/0/2

Last 5 second output: 622197 packets/sec 768059281 bytes/sec 62%

Ten-GigabitEthernet1/3/0/2

Last 5 second output: 601957 packets/sec 740746492 bytes/sec 60%

Ten-GigabitEthernet1/6/0/4

Last 5 second output: 604492 packets/sec 745027035 bytes/sec 60%

Ten-GigabitEthernet2/2/0/2

Last 5 second output: 595072 packets/sec 731934928 bytes/sec 59%

Ten-GigabitEthernet2/3/0/2

Last 5 second output: 580979 packets/sec 711983609 bytes/sec 57%

Ten-GigabitEthernet2/6/0/4

Last 5 second output: 592967 packets/sec 725937944 bytes/sec 59%

通过上述数据对比发现确实如软件侧统计的，镜像目的端口流量减少了

第二步

分析镜像相关配置发现，现场镜像流量存在跨框情况，因此想到排查IRF物理链路流量情况

设备IRF相关配置如下

#

irf-port 1/2

 port group mdc 1 interface Ten-GigabitEthernet1/3/0/9 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/3/0/10 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/3/0/11 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/3/0/12 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/4/0/9 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/4/0/10 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/4/0/11 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet1/4/0/12 mode enhanced

#

irf-port 2/1

 port group mdc 1 interface Ten-GigabitEthernet2/3/0/9 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/3/0/10 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/3/0/11 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/3/0/12 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/4/0/9 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/4/0/10 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/4/0/11 mode enhanced

 port group mdc 1 interface Ten-GigabitEthernet2/4/0/12 mode enhanced

#

IRF物理接口虽然有80G带宽，但是实际UP的只有40G，查看IRF物理接口流量统计，发现接口流量分布很不均匀，且部分接口有打满的情况

XGE1/3/0/9           DOWN auto    A      --   --               IRF-5-预留

XGE1/3/0/10          DOWN auto    A      --   --               IRF-6-预留

XGE1/3/0/11          UP   10G(a)  F(a)   --   --        IRF-1

XGE1/3/0/12          UP   10G(a)  F(a)   --   --        IRF-2

XGE1/4/0/9           DOWN auto    A      --   --               IRF-7-预留

XGE1/4/0/10          DOWN auto    A      --   --               IRF-8-预留

XGE1/4/0/11          UP   10G(a)  F(a)   --   --        IRF-3

XGE1/4/0/12          UP   10G(a)  F(a)   --   --        IRF-4

XGE2/3/0/9           DOWN auto    A      --   --               IRF-5-预留

XGE2/3/0/10          DOWN auto    A      --   --               IRF-6-预留

XGE2/3/0/11          UP   10G(a)  F(a)   --   --        IRF-1

XGE2/3/0/12          UP   10G(a)  F(a)   --   --        IRF-2

XGE2/4/0/9           DOWN auto    A      --   --               IRF-7-预留

XGE2/4/0/10          DOWN auto    A      --   --               IRF-8-预留

XGE2/4/0/11          UP   10G(a)  F(a)   --   --        IRF-3

XGE2/4/0/12          UP   10G(a)  F(a)   --   --        IRF-4

Ten-GigabitEthernet1/3/0/11

Last 300 second input: 1618795 packets/sec 1235262497 bytes/sec 100%

Last 300 second output: 459266 packets/sec 134246126 bytes/sec 11%

Ten-GigabitEthernet1/3/0/12

Last 300 second input: 157978 packets/sec 46478844 bytes/sec 3%

Last 300 second output: 1009635 packets/sec 754211816 bytes/sec 61%

Ten-GigabitEthernet1/4/0/11

Last 300 second input: 43 packets/sec 10083 bytes/sec 0%

Last 300 second output: 41 packets/sec 13799 bytes/sec 0%

Ten-GigabitEthernet1/4/0/12

Last 300 second input: 1707391 packets/sec 1182926426 bytes/sec 97%

Last 300 second output: 1950434 packets/sec 1214277259 bytes/sec 100%

Ten-GigabitEthernet2/3/0/11

Last 300 second input: 204119 packets/sec 52731423 bytes/sec 4%

Last 300 second output: 908095 packets/sec 710731997 bytes/sec 58%

Ten-GigabitEthernet2/3/0/12

Last 300 second input: 1010847 packets/sec 754972070 bytes/sec 62%

Last 300 second output: 157982 packets/sec 46543348 bytes/sec 3%

Ten-GigabitEthernet2/4/0/11

Last 300 second input: 37 packets/sec 13105 bytes/sec 0%

Last 300 second output: 43 packets/sec 9955 bytes/sec 0%

Ten-GigabitEthernet2/4/0/12

Last 300 second input: 944132 packets/sec 608854439 bytes/sec 50%

Last 300 second output: 804049 packets/sec 565898897 bytes/sec 46%

故障分析

根据上述分析过程，此问题原因应该是40GIRF带宽不够导致的，实验室复现现场问题，结果显示当前设备版本的跨框镜像到一个跨框的聚合端口并不是本框优先镜像转发的，而是hash的，当网络流量较大时，IRF链路在跑业务流量的同时，还要承载相当一部分的镜像流量；从而造成了拥塞的丢包，于是源目端口的流量不一致；

建议现场将镜像组和聚合组一分为二，不要跨框镜像；若当前的版本的镜像组数量（4个）不够用，可以升级到升级到7557P03 ，新版本可以支持创建16 个镜像组；现场目前没有升级的打算，因此将预留的IRF物理接口启用，之后观察几天发现镜像少流量问题已经解决。