6800以及10506聚合成员口部分接口突然变为未选中问题

6800堆叠（聚合口）---------（聚合口）105堆叠

    6800和10506之间的聚合口突然出现很多聚合口的成员口突然有接口变为未选中，且对应的成员口物理层为up，但是协议层为down。6800侧和10506侧日志均提示是对方聚合配置不正确导致，但是实际上两端均未做过配置更改。6800 侧为聚合 64, 10506侧为聚合4。

    6800侧日志如下：

    %Jul  8 12:57:29:790 2019 GB-ZWY-GGQ-H6H7-S-SW-S6800-2C-5&6 LAGG/6/LAGG_INACTIVE_PARTNER: Member port FGE1/2/25 of aggregation group BAGG64 changed to the inactive state, because the aggregation configuration of its peer port is incorrect.

    对应聚合口

#

Aggregate Interface: Bridge-Aggregation64

Aggregation Mode: Dynamic

Loadsharing Type: Shar

Management VLANs: None

System ID: 0x8000, 7c1e-062c-cd8a

Local:

  Port                Status   Priority Index    Oper-Key               Flag

  FGE1/2/25           U        32768    1        1                      {ACD}

  FGE1/2/26           S        32768    2        1                      {ACDEF}

  FGE2/2/25           U        32768    18       1                      {ACD}

  FGE2/2/26           S        32768    34       1                      {ACDEF}

Remote:

  Actor               Priority Index    Oper-Key SystemID               Flag  

  FGE1/2/25(R)        32768    4        4        0x8000, 3cf5-cc6e-6c00 {ACG}

  FGE1/2/26           32768    15       4        0x8000, 3cf5-cc6e-6c00 {ACDEF}

  FGE2/2/25           32768    8        4        0x8000, 3cf5-cc6e-6c00 {ACG}

  FGE2/2/26           32768    16       4        0x8000, 3cf5-cc6e-6c00 {ACDEF}

    10506侧日志如下：

     %Jul  8 13:06:09:881 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/6/LAGG_INACTIVE_PARTNER: Member port FGE1/0/0/7 of aggregation group BAGG4 changed to the inactive state, because the aggregation configuration of its peer port is incorrect。

     相关聚合接口

#

Aggregate Interface: Bridge-Aggregation4

Creation Mode: Manual

Aggregation Mode: Dynamic

Loadsharing Type: Shar

Management VLANs: None

System ID: 0x8000, 3cf5-cc6e-6c00

Local:

  Port                Status   Priority Index    Oper-Key               Flag

  FGE1/0/0/7          U        32768    4        4                      {ACG}

  FGE1/0/0/8          U        32768    8        4                      {ACG}

  FGE2/0/0/7(R)       S        32768    15       4                      {ACDEF}

  FGE2/0/0/8          S        32768    16       4                      {ACDEF}

Remote:

  Actor               Priority Index    Oper-Key SystemID               Flag  

  FGE1/0/0/7          32768    0        0        0x8000, 0000-0000-0000 {DEF}

  FGE1/0/0/8          32768    0        0        0x8000, 0000-0000-0000 {DEF}

  FGE2/0/0/7          32768    2        1        0x8000, 7c1e-062c-cd8a {ACDEF}

  FGE2/0/0/8          32768    34       1        0x8000, 7c1e-062c-cd8a {ACDEF}

        从现象看，好几组6800侧的聚合口都存在聚合成员口突然非选中的情况，且slot 1和slot 2均有分布。同时查看105突然变为非选中接口的统计，发现有问题的接口都在1框0槽位，具体如下。

        所以问题的关注点聚焦为10506的一框0槽位。

        分析1框0槽位单板。以1/0/0/7为例：

        打开debugging开关，105侧 1/0/0/7口 lacp报文只有发送，没有接收的打印。2/0/0/7有收有发

        #1/0/0/7只有发送，没有接收

<GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2>debugging link-aggregation lacp packet all int f 1/0/0/7

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0x4, state=0x45

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=0000-0000-0000, key=0x0, pri=0x8000, port-index=0x0, state=0x38

Collector: type=3, len=16, col-max-delay=0x0

Terminator: type=0, len=0

*Jul  8 19:41:09:779 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/7/Packet: -MDC=1-Chassis=1-Slot=0; PACKET.FortyGigE1/0/0/7.send.

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0x4, state=0x45

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=0000-0000-0000, key=0x0, pri=0x8000, port-index=0x0, state=0x38

Collector: type=3, len=16, col-max-delay=0x0

Terminator: type=0, len=0

*Jul  8 19:41:39:779 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/7/Packet: -MDC=1-Chassis=1-Slot=0; PACKET.FortyGigE1/0/0/7.send.

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0x4, state=0x45

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=0000-0000-0000, key=0x0, pri=0x8000, port-index=0x0, state=0x38

Collector: type=3, len=16, col-max-delay=0x0

Terminator: type=0, len=0

        2/0/0/7 有收有发

<GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2>debugging link-aggregation lacp packet all int f 2/0/0/7

*Jul  8 19:38:28:131 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/7/Packet: -MDC=1-Chassis=2-Slot=0; PACKET.FortyGigE2/0/0/7.receive.

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=7c1e-062c-cd8a, key=0x1, pri=0x8000, port-index=0x2, state=0x3d

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0xf, state=0x3d

Collector: type=3, len=16, col-max-delay=0x0

Terminator: type=0, len=0

*Jul  8 19:38:39:779 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/7/Packet: -MDC=1-Chassis=1-Slot=0; PACKET.FortyGigE1/0/0/7.send.

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0x4, state=0x45

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=0000-0000-0000, key=0x0, pri=0x8000, port-index=0x0, state=0x38

Collector: type=3, len=16, col-max-delay=0x0

Terminator: type=0, len=0

*Jul  8 19:38:54:561 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/7/Packet: -MDC=1-Chassis=2-Slot=0; PACKET.FortyGigE2/0/0/7.send.

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0xf, state=0x3d

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=7c1e-062c-cd8a, key=0x1, pri=0x8000, port-index=0x2, state=0x3d

Collector: type=3, len=16, col-max-delay=0x0

Terminator: type=0, len=0

*Jul  8 19:38:58:130 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 LAGG/7/Packet: -MDC=1-Chassis=2-Slot=0; PACKET.FortyGigE2/0/0/7.receive.

size=110, subtype=1, version=1

Actor: type=1, len=20, sys-pri=0x8000, sys-mac=7c1e-062c-cd8a, key=0x1, pri=0x8000, port-index=0x2, state=0x3d

Partner: type=2, len=20, sys-pri=0x8000, sys-mac=3cf5-cc6e-6c00, key=0x4, pri=0x8000, port-index=0xf, state=0x3d

Collector: type=3, len=16, col-max-delay=0x0

        打印驱动上cpu报文开关，1框0号槽只有发送的打印，没有接收打印（ 0180-c200-0002为lacp报文的目的mac）

        要先打开 t  d和 t  m

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe] display rxtx dest_mac 0180-c200-0002 chassis 1 slot 0   

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx pkt c 1 s 0

 Debug RxTx packet is on!

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]*Jul  8 20:11:39:779 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 DRVPLAT/7/RxTxDebug: -MDC=1-Chassis=1-Slot=0;

From board 0: transmit packet from chip0,port1,Priority=3,len=124

-----------------------------------------------------

0000  01 80 c2 00 00 02 00 0f e2 07 f2 e0 88 09 01 01

 0010  01 14 80 00 3c f5 cc 6e 6c 00 00 02 80 00 00 02

 0020  45 00 00 00 02 14 80 00 00 00 00 00 00 00 00 00

 0030  80 00 00 00 38 00 00 00 03 10 00 00 00 00 00 00

 -----------------------------------------------------

*Jul  8 20:11:39:780 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 DRVPLAT/7/RxTxDebug: -MDC=1-Chassis=1-Slot=0;

From board 0: transmit packet from chip0,port9,Priority=3,len=124

-----------------------------------------------------

0000  01 80 c2 00 00 02 3c f5 cc 6e 6c 01 88 09 01 01

 0010  01 14 80 00 3c f5 cc 6e 6c 00 00 01 80 00 00 01

 0020  45 00 00 00 02 14 80 00 00 00 00 00 00 00 00 00

 0030  80 00 00 00 38 00 00 00 03 10 00 00 00 00 00 00

 -----------------------------------------------------

        2框0号槽，有收发的打印

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]display rxtx dest_mac 0180-c200-0002 chassis 2 slot 0

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx pkt c 2 s 0

 Debug RxTx packet is on!

*Jul  8 20:09:46:256 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 DRVPLAT/7/RxTxDebug: -MDC=1-Chassis=2-Slot=0;

From board 18: received packet from chip0,port21,reason=0x1000,cos=42,sMod=32,sPort=21,len=128,Matched=40,time=2,src_vp=-1

*Jul  8 20:09:46:256 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 DRVPLAT/7/RxTxDebug: -MDC=1-Chassis=2-Slot=0;

 -----------------------------------------------------

0000  01 80 c2 00 00 02 90 e7 10 0f 4f 8c 81 00 00 01

 0010  88 09 01 01 01 14 80 00 90 e7 10 0f 4e 41 00 01

 0020  80 00 00 22 3d 00 00 00 02 14 80 00 3c f5 cc 6e

 0030  6c 00 00 03 80 00 00 0d 3d 00 00 00 03 10 00 00

 -----------------------------------------------------

*Jul  8 20:09:54:560 2019 GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2 DRVPLAT/7/RxTxDebug: -MDC=1-Chassis=2-Slot=0;

From board 18: transmit packet from chip0,port5,Priority=3,len=124

-----------------------------------------------------

0000  01 80 c2 00 00 02 00 0f e2 07 f2 e0 88 09 01 01

 0010  01 14 80 00 3c f5 cc 6e 6c 00 00 02 80 00 00 0c

 0020  3d 00 00 00 02 14 80 00 3c f5 cc 91 5e f7 00 01

 0030  80 00 00 56 3d 00 00 00 03 10 00 00 00 00 00 00

 -----------------------------------------------------

        对比了下 上CPU报文计数，从收集诊断的时间点，到现在，1框0号槽上CPU报文计数没有增加。 2框有计数增加。

1框多次查看结果，计数没有增长

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 1 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3326588    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 1 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3326588    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 1 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3326588    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 1 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3326588    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]dis clo

19:45:21.104 BJ Mon 07/08/2019

Time Zone : BJ add 08:00:00

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]dis clodiws clodebug rxtx softcar show c 1 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

39  STP                 0      148        0          100  S   On  SMAC 8

40  LACP                0      3326588    0          100  S   On  SMAC 8

2框多次查看结果，计数有增加

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 2 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3334506    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 2 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                1      3334507    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 2 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3334507    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 2 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                0      3334508    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 2 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                3      3334510    0          100  S   On  SMAC 8

[GB-ZWY-GGQ-I1I2-S-SW-S10506-1&2-probe]debug rxtx softcar show c 2 s 0    40

ID  Type                RcvPps Rcv_All    DisPkt_All Pps  Dyn Swi Hash ACLmax

40  LACP                3      3334510    0          100  S   On  SMAC 8

通过如上分析，问题是由于1框0号槽单板，芯片收到报文后，无法上送CPU导致，该单板芯片硬件故障了