S8500产品接口板CPU利用率过高问题处理方法

S8500产品接口板CPU利用率过高问题处理方法

一、故障处理流程

                 

                                                                                                                                图1  接口板CPU占用率过高故障处理流程

二、故障处理步骤

 

1）步骤1

使用命令display cpu slot连续查询接口板CPU的占用率，确定接口板的CPU占用率是否异常。

如果接口板CPU占用率持续在50％以上，说明CPU有异常任务在处理，需要确认CPU高的具体原因，请转步骤2。

注意：在通过TELNET方式登录使用display diagnostic-information收集诊断信息时，查询结果会显示CPU占用率较高，此时需要在信息收集完毕后，等待一段时间再查询相关槽位的CPU占用率，确定是由于收集诊断信息导致的CPU占用率升高还是任务异常导致的CPU占用率升高。

在通过TELNET方式登录查询CPU占用率时，两次查询的时间间隔要稍长一些（10秒），避免TELNET交互报文较多导致CPU占用率高。

 

2）步骤2

使用命令_system-monitor监控并查询主控板CPU各个任务运行情况。下面以S8512为例查询4号槽位任务运行情况：

打开调试开关，进入隐含模式

<S8500>terminal debugging

Current terminal debugging is on

<S8500>terminal monitor

Current terminal monitor is on

<S8500>sys

[S8500]_

[S8500-hidecmd]

打开监控开关，监控5秒后关闭监控开关，并查询指定槽位单板的任务信息

[S8500-hidecmd]_system-monitor start

[S8500-hidecmd]_system-monitor stop

[S8500-hidecmd]_system-monitor print 4

[S8500-hidecmd]

*0.8361548 Quidway DRVL2/8/DBG_DBG:Slot=4;

Total Monitoring time 10718 ms

Task Name        Max Run(ms)     Max Pend(ms)    Running(ms)     Percent(%)

---------------------------------------------------

RPCQ             0               0               0               0.00

WEIL             10              98              4991            46.56

SYST             1               87              103             0.96

IPCQ             1               118             22              0.20

RPCQ             1               88              85              0.79

MDCT             1               98              39              0.36

MDCR             0               0               0               0.00

VP               0               0               0               0.00

STND             0               0               0               0.00

INFO             0               5087            0               0.00

SOCK             1               188             18              0.16

VMON             0               0               0               0.00

MPM              0               0               0               0.00

*0.8362153 Quidway DRVL2/8/DBG_DBG:Slot=4;

RPR              1               98              44              0.41

DEV              1               173             31              0.28

IFCB             0               0               0               0.00

HAND             1               1061            2               0.01

PRX0             87              13              4522            42.19

PRX1             0               0               0               0.00

BOTT             1               2101            2               0.01

zhTx             4               189             394             3.67

ISRL             1               98              89              0.83

L2ST             1               98              94              0.87

L2Ma             1               4990            33              0.30

L2Ch             0               0               0               0.00

L2PS             4               1058            45              0.41

DL3              0               0               0               0.00

dgsd             0               0               0               0.00

DIAG             1               1061            2               0.01

QACL             1               98              95              0.88

L3MC             1               98              98              0.91

 

3）步骤3

根据步骤2的查询结果，进行故障分析。

接口板任务信息查看和主控板相同，需要分析哪些任务导致接口板CPU异常。

下面介绍一些常见导致主控板CPU高的任务及处理方法：

PRX0，接口板收包任务。该任务占用率高，说明极有可能受到ARP、IP等报文攻击，需要对上CPU的报文进行分析。在1235或者1270之后的版本，可以通过cpu-mirror命令把CPU收到的报文镜像到物理端口，然后再通过PC抓包分析，确定攻击源，并且采用ACL过滤等方式切断攻击源。

[S8500]cpu-mirror slot 2 mirrored-to e2/1/1

注意：该命令在1226-1233版本和1265-1269版本存在致命缺陷，请不要使用；在其他版本中使用该命令抓包完成后请立即通过undo命令关闭该功能。

如果无法使用该命令，建议收集CPU异常的接口板收到协议报文的情况，主要是两个命令_txerr display slot <slot_num> clear和_rxpkt slot <slot_num> length <len_num> numbers <num>，特别是_rxpkt slot <slot_num>，建议收集上送CPU的报文数目不低于1000个。

在收集如上信息前，需要记住保存收集的信息。建议收集信息时按照如下步骤收集，且_txerr display slot <slot_num> clear至少执行5次：

<Quidway>sys                                                                   

<Quidway>system-view                                                            

System View: return to User View with Ctrl+Z.                                  

[Quidway]en_diag                                                               

WARNING: Commands under this mode,especially for flash and nvram, may be destruc

tive,so you must save your configuration first and use them with  guidance of pr

ofessional men.                                                                 

[Quidway-testdiag]_txerr display slot 3 clear                                  

Tx queue status:                                                                

  TxResPkt=0           TxResBf=0           TxResFailPkt=0                      

                                                                               

Tx queue status:                                                                

  TxQueLen=0             TxQuePtr=1642                                         

  TxTaskPtr=1642         TxInQueueErr=0                                        

                                                                                

Rec int statistics:                                                            

 intGenScan =612387                                                            

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                      

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

                                                                                

Rec control statistics:                                                        

 RecInt =489898                                                                

Rec Max speed statistics:                                                       

 1              0              0              0                                

 0              0              6              6                                

 12                                                                             

Rec AU statistics:                                                             

 0                                                                             

Rec discard statistics:                                                         

 0              0              0              0                                

 0              0              0              0                                

Rec AU discard statistics:                                                      

 0                                                                             

[Quidway-testdiag]_txerr display slot 3 clear                                  

Tx queue status:                                                                

  TxResPkt=0           TxResBf=0           TxResFailPkt=0                      

                                                                               

Tx queue status:                                                               

  TxQueLen=0             TxQuePtr=1644                                         

  TxTaskPtr=1644         TxInQueueErr=0                                        

                                                                               

Rec int statistics:                                                             

 intGenScan =612397                                                            

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                      

 616   624   632   0     584   616   624   632   0     584                     

 616   624   632   0     584   616   624   632   0     584                     

                                                                                

Rec control statistics:                                                        

 RecInt =8                                                                     

Rec Max speed statistics:                                                      

 0              0              0              0                                

 0              0              2              2                                

 4                                                                             

Rec AU statistics:                                                              

 0                                                                             

Rec discard statistics:                                                        

 0              0              0              0                                

 0              0              0              0                                

Rec AU discard statistics:                                                     

 0                                                                              

[Quidway-testdiag]_rxpkt slot 3 length 100 numbers 1000         

zhTx，接口板发包任务。该任务繁忙时通常PRX0任务也比较繁忙，可以和PRX0任务结合在一起分析。该任务占用率高，说明极有可能受到IP报文攻击，CPU一直在忙于发送应答报文。

L2PS，接口板端口状态轮循任务。该任务占用率高，表明当前单板UP的端口较多，或者读取端口状态时出错，如果一直比较高需要联系800进一步处理。

 

4）步骤4

如果无法确定任务占用率高属于正常现象还是存在异常，请联系800进一步处理。