某局点终结者本体注册不上

无

终结者本体注册不上，现场使用的自动注册，查看之前有同型号的本体已经上线，配置正常

WT-3F                          200   R/M   WT1024-X-Q      219801A1LQP1920000JZ

WT-4F                          77    R/M   WT1024-X-Q      219801A1LQP19200007Y

WT-5F                          9     R/M   WT1024-X-Q      219801A1LQP1920000MW

WT-6F                          50    R/M   WT1024-X-Q      219801A1LQP1920000D9

WT-7F                          78    R/M   WT1024-X-Q      219801A1LQP1920000GM

WT-8F                          107   R/M   WT1024-X-Q      219801A1LQP1920000TN

WT-9F                       4984  I     WT1024-X-Q      219801A1LQP1920000CZ

AP ip地址：172.25.200.49

本体版本：WT1024Q-CMW710-R2423P01和能上线的本体的版本一致的

查看上次掉线原因是超时

Latest IP address             : 172.25.200.49

Current AC IP                 : N/A

Tunnel down reason            : Wait request timer expired

查看AP debug 跟AC能正常交互

*Aug 27 16:19:23:965 2020 WT-9F CWC/7/TMR: Discovery Interval timer expired.uiAPID = 1.

*Aug 27 16:19:23:966 2020 WT-9F CWC/7/EVENT: Discovered AC by method of IPv4 broadcast successfully.

*Aug 27 16:19:23:966 2020 WT-9F CWC/7/EVENT: AP selected AC IP 172.25.200.254 Priority 4 successfully.

*Aug 27 16:19:23:968 2020 WT-9F CWC/7/FSM: Enter Join state.

*Aug 27 16:19:23:968 2020 WT-9F CWC/7/EVENT: Fill join req.wtu=1.model=WT1024-X-Q.wt drv info:[].subslot=0.licensetype=3.ret=0x40010001.

*Aug 27 16:19:23:968 2020 WT-9F CWC/7/EVENT: Fill join req.ap=1.model=WT1024-X-Q.licensetype=3.carry wt info is unnecessary.

*Aug 27 16:19:23:970 2020 WT-9F CWC/7/SND_PKT: Sent join request with SeqNum 1 to AC 172.25.200.254:5246.

*Aug 27 16:19:23:970 2020 WT-9F CWC/7/SND_PKT: Verbose info for join request sent to AC 172.25.200.254:5246, length=199.

*Aug 27 16:19:23:990 2020 WT-9F CWC/7/RCV_PKT: Received a fragment from AC at 172.25.200.254:5246.

*Aug 27 16:19:23:990 2020 WT-9F CWC/7/RCV_PKT: Verbose info for a fragment from AC 172.25.200.254:5246, length=1448.

00 10 02 80 00 00 0B 40 00 00 00 00 25 00 11 00

00 63 A2 00 5C 00 09 00 00 13 78 7B 00 00 00 00

00 25 00 11 00 00 63 A2 00 5C 00 09 00 00 13 78

7C 00 00 00 00 00 25 00 11 00 00 63 A2 00 5C 00

09 00 00 13 78 7D 00 00 00 00 00 25 00 11 00 00

63 A2 00 5C 00 09 00 00 13 78 7E 00 00 00 00 00

25 00 11 00 00 63 A2 00 5C 00 09 00 00 13 78 7F

00 00 00 00 00 25 00 11 00 00 63 A2 00 5C 00 09

00 00 13 78 80 00 00 00 00 00 25 00 11 00 00 63

A2 00 5C 00 09 00 00 13 78 81 00 00 00 00 00 25

00 11 00 00 63 A2 00 5C 00 09 00 00 13 78 82 00

00 00 00 00 25 00 11 00 00 63 A2 00 5C 00 09 00

00 13 78 83 00 00 00 00 00 25 00 11 00 00 63 A2

00 5C 00 09 00 00 13 78 84 00 00 00 00 00 25 00

11 00 00 63 A2 00 5C 00 09 00 00 13 78 85 00 00

00 00 00 25 00 11 00 00 63 A2 00 5C 00 09 00 00

...Omitted

.

*Aug 27 16:19:23:992 2020 WT-9F CWC/7/RCV_PKT: Received a fragment from AC at 172.25.200.254:5246.

*Aug 27 16:19:23:993 2020 WT-9F CWC/7/RCV_PKT: Verbose info for a fragment from AC 172.25.200.254:5246, length=361.

*Aug 27 16:20:53:587 2020 WT-9F CWC/7/FSM: Enter Data Check state.//进入了数据检查阶段

*Aug 27 16:20:53:589 2020 WT-9F CWC/7/SND_PKT: Sent change state event request with SeqNum 3 to AC 172.25.200.254:5246.//向AC发送了SeqNum3

*Aug 27 16:20:53:590 2020 WT-9F CWC/7/TMR: [Port 3] MaxDiscovery Interval timer expired.//后续超时

*Aug 27 16:20:53:591 2020 WT-9F CWC/7/TMR: Retransmission Timer of TunnelId:1 expired. uiSeqNum = 0, uiIPCMsgType = 0x0, enPattern = 0.

*Aug 27 16:20:53:591 2020 WT-9F CWC/7/SND_PKT: Sent change state event request with SeqNum 3 to AC 172.25.200.254:5246.

*Aug 27 16:20:53:592 2020 WT-9F CWC/7/SND_PKT: Verbose info for change state event request sent to AC 172.25.200.254:5246, length=24.

00 10 02 00 00 00 00 00 00 00 00 0B 03 00 0B 00

00 21 00 04 00 00 00 00

查看AC debug

*Aug 27 16:19:24:545 2020 WX3508H CWS/7/SND_PKT: Sent all fragments of configuration response with SeqNum 2 to AP at 172.25.200.49:32079.

*Aug 27 16:19:24:545 2020 WX3508H CWS/7/FSM: Enter Config state.

*Aug 27 16:20:54:167 2020 WX3508H CWS/7/ERROR: Failed to match change state event request with SeqNum 3 from IP address 172.25.200.49:32079 //如下报错

于是在ac和ap上同时debug有如下信息：

ac上debug：

*Aug 27 16:19:18:307 2020 WX3508H CWS/7/ERROR: Failed to match change state event request with SeqNum 3 from IP address 172.25.200.49:32077

*Aug 27 16:19:18:309 2020 WX3508H CWS/7/ERROR: Failed to match change state event request with SeqNum 3 from IP address 172.25.200.49:32077

ap上debug

*Aug 27 16:17:51:969 2020 WT-9F CWC/7/RCV_PKT: Assembled configuration response with SeqNum 2 from AC at 172.25.200.254:5246.

*Aug 27 16:19:17:742 2020 WT-9F CWC/7/FSM: Enter Data Check state.

*Aug 27 16:19:17:744 2020 WT-9F CWC/7/SND_PKT: Sent change state event request with SeqNum 3 to AC 172.25.200.254:5246.

*Aug 27 16:19:17:745 2020 WT-9F CWC/7/SND_PKT: Verbose info for change state event request sent to AC 172.25.200.254:5246, length=24. 00 10 02 00 00 00 00 00 00 00 00 0B 03 00 0B 00 00 21 00 04 00 00 00 00 .

*Aug 27 16:19:17:745 2020 WT-9F CWC/7/TMR: [Port 5] MaxDiscovery Interval timer expired.

*Aug 27 16:19:17:746 2020 WT-9F CWC/7/TMR: [Port 17] MaxDiscovery Interval timer expired.

*Aug 27 16:19:17:746 2020 WT-9F CWC/7/TMR: [Port 23] MaxDiscovery Interval timer expired.

ac上debug信息显示，在和ap交互change（SeqNum 3）报文时出了问题，再结合ap的debug发现当ap和ac交互完config（SeqNum 2）报文后，到发送change请求用了1分26秒，判断是本体上处理慢。

在远程过程中还发现当再本体上看display wlan ap 的命令时会卡住，判断是apmgr进程异常，99%都是因为被内核挂住，内核有问题，查看secondary_log_buf

可以看到 secondary log buffer 里的内容如下，有Poe硬件故障。

<0>[   13.159000] 0:please shutdown wtu port

<0>[   14.377000] 0:please shutdown wtu port

<0>[   15.592000] 0:please shutdown wtu port

<0>[   15.592000] 0:ERROR poe <{FILE}> 2382 DRV_POE_KeepLive vcpu(0): Failed to send POE command!

<0>[   21.807000] 0:ERROR poe <{FILE}> 846 DRV_POE_UART_Parse vcpu(0): Time out or error happened when getting semaphore RtnS!

<0>[   23.021000] 0:please shutdown wtu port

所以最终定位问题原因为：Poe硬件故障导致内核工作不正常，进一步导致apmgr工作不正常，所以apmgr报文间隔特别大，ac上的信息表示报文超时了。

设备硬件故障，走备件流程