SR66SR66-X系列路由器FIP-110线卡性能不足导致直连丢包问题经验案例

一、   组网：

某客户出口为一台SR6604路由器，配置一块FIP-110线卡，SR6604路由器下联核心交换机，SR6604路由器同时与下联分支网点建立IPSEC链接，SR6604因为作为出口路由器，所以也配置了NAT服务。

二、   问题描述：

随着客户业务的大量增长，IPSEC隧道数大量增加，某天客户发现从核心交换机上ping SR6604路由器直连接口地址有大量丢包现象，在两端设备上配置流统，然后ping 20个包，统计如下：

三、   过程分析：

问题发生后收集了诊断信息，查看接口统计信息，接口下有大量的因为队列溢出导致的丢包统计：

GigabitEthernet2/0/1 current state: UP

Line protocol current state: UP

Description: TO-(MNJT-F1-C-SecBlande-FW-01)

The Maximum Transmit Unit is 1500

Internet Address is 10.10.3.2/30 Primary

IP Packet Frame Type: PKTFMT_ETHNT_2,  Hardware Address: c4ca-d932-7676

IPv6 Packet Frame Type: PKTFMT_ETHNT_2,  Hardware Address: c4ca-d932-7676

Media type is optical fiber, loopback not set, promiscuous mode not set

1000Mb/s, Full-duplex, link type is autonegotiation

Output flow-control is disabled, input flow-control is disabled

SFP Transceiver Info:

Vendor name:WTD    Port hardware type:1000_BASE_LX_AN_SFP.

link length: single mode 9/125μm 10000 Meters.

Output queue : (Urgent queuing : Size/Length/Discards)  0/100/0

Output queue : (Protocol queuing : Size/Length/Discards)  0/500/0

Output queue : (FIFO queuing : Size/Length/Discards)  0/75/22888

Last clearing of counters: Never

    Last 300 seconds input rate 7692305.00 bytes/sec, 61538440 bits/sec, 15404.27 packets/sec

    Last 300 seconds output rate 7434690.50 bytes/sec, 59477520 bits/sec, 12146.25 packets/sec

    Input: 89003762 packets, 41529584008 bytes, 5022158 no buffers

           0 broadcasts, 1144 multicasts, 0 pauses

           0 errors, 0 runts, 0 giants

           0 crc, 0 align errors, 0 overruns

           0 dribbles, 0 drops

    Output:71996075 packets, 47690256378 bytes

           3 broadcasts, 1547 multicasts, 0 pauses

           0 errors, 0 underruns, 0 collisions

           0 deferred, 0 lost carriers

查看统计信息VCPU4/5/6/7（FIP-110线卡为双核4线程处理器）均存在大量的no buffer统计：

Distribute Flow VCPU information:

PORT  0: Total Tokens = 512      Used Tokens =  0

PORT  1: Total Tokens = 512      Used Tokens = 513

Forwarding VCPU information:

VCPU  4: TotalPkts = 85       Overflows(no buffers) = 1996279

VCPU  5: TotalPkts = 18       Overflows(no buffers) = 236790

VCPU  6: TotalPkts = 4        Overflows(no buffers) = 381987

VCPU  7: TotalPkts = 406      Overflows(no buffers) = 2400392

在隐含模式下查看VCPU利用率，发现4/5/6/7四个VCPU利用率偏高：

[MNJT-R1-C-SR6604-01-hidecmd]tsh main sl 2 vcpu all cpu-usage                  

  VCPU usage information of slot 2:                                            

  VCPU Last Summary                                                            

   1    0%   0%                                                                

   2    0%   0%                                                                

   3    0%   0%                                                                

   4   59%  69%                                                                

   5   57%  62%                                                                

   6   50%  62%                                                                

   7   64%  72

综合SR6604上建立了大量的IPSEC隧道以及NAT会话，所以由此推断为因为FIP-110线卡性能不足导致。

四、   解决方法：

客户更换高性能线卡FIP-210后恢复正常。