某局点S6800异常重启问题

/

/

设备：S6800 R2609  H35

问题：两台堆叠，其中slot1发生了  AutoUpdateReboot重启，需要分析原因。

Slot 1:

Uptime is 0 weeks,0 days,3 hours,30 minutes

S6800-54QF with 2 Processor

BOARD TYPE:         S6800-54QF

DRAM:               2048M bytes

FLASH:              512M bytes

PCB 1 Version:      VER.A

FPGA Version:       NONE

Bootrom Version:    157

CPLD 1 Version:     002

CPLD 2 Version:     001

Release Version:    H3C S6800-54QF-2609

Patch Version:      Release 2609H35

Reboot Cause:       AutoUpdateReboot

[SubSlot 0] 48SFP Plus+6QSFP Plus

*Mar 25 04:41:50:348 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 SCMD/3/JOBINFO: The service pkg_upgrade is running...

*Mar 25 04:41:50:348 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 LAGG/6/LAGG_TRACE: Notify LAG leave: port 0x3b group 0x654 ret 0

*Mar 25 04:41:50:351 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 LAGGK/6/LAGG_TRACE: Notify port 0x3b leaved lag group 0x654 ret 0x0.

*Mar 25 04:41:50:363 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 PKG/4/PKG: [ISSU]: Sync soft reboot list to all MDC , MDC 1, return 0, MDC return 0

1．  设备重启原因为AutoUpdateReboot，查看重启记录，梳理设备重启过程如下

Slot 1 因为异常先异常重启了

%Mar 25 04:41:44:587 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 HA/5/HA_STANDBY_TO_MASTER: Standby board in slot 2 changed to master.

%Mar 25 04:41:45:029 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 DEV/3/BOARD_REMOVED: Board was removed from slot 1, type is S6800-54QF.

重启加入堆叠的过程，因为之前安装R2609H35补丁的时候，应该是没有执行install commit，重启的时候是没有带着补丁

这点从如下信息可以确认，没重启的slot2的下发启动包没有补丁，说明没有执行commit。

===============boot-loader infomation=============== 

Software images on slot 1:

Current software images:

  flash:/s6800-cmw710-boot-r2609.bin

  flash:/s6800-cmw710-system-r2609.bin

  flash:/S6800-CMW710-SYSTEM-R2609H35.bin

Main startup software images:

  flash:/s6800-cmw710-boot-r2609.bin

  flash:/s6800-cmw710-system-r2609.bin

  flash:/S6800-CMW710-SYSTEM-R2609H35.bin

Backup startup software images:

  None

Software images on slot 2:

Current software images:

  flash:/s6800-cmw710-boot-r2609.bin

  flash:/s6800-cmw710-system-r2609.bin

  flash:/S6800-CMW710-SYSTEM-R2609H35.bin

Main startup software images:

  flash:/s6800-cmw710-boot-r2609.bin

  flash:/s6800-cmw710-system-r2609.bin

Backup startup software images:

重启加入堆叠的时候。需要同步补丁，slot 1 再次重启，所以重启原因为AutoUpdateReboot。

%@466%Mar 25 04:46:49:268 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 STM/5/STM_AUTO_UPDATING: Don't reboot the slot 1. It is loading files.

%@467%Mar 25 04:46:50:206 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 DEV/2/BOARD_STATE_FAULT: Board state changed to Fault on slot 1, type is unknown.

%@468%Mar 25 04:47:09:386 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 DEV/4/BOARD_LOADING: Board in slot 1 is loading software images.

%@469%Mar 25 04:47:19:870 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 DEV/5/LOAD_FINISHED: Board in slot 1 has finished loading software images.

%@470%Mar 25 04:47:58:776 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 STM/5/STM_AUTO_UPDATE_FINISHED: File loading finished on slot 1.

%@483%Mar 25 04:54:16:606 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 DEV/2/BOARD_STATE_FAULT: Board state changed to Fault on slot 1, type is unknown.

%@484%Mar 25 04:54:24:250 2023 NJ-GL4F-D7&8-ASW-2.RP.S6800 DEV/5/BOARD_STATE_NORMAL: Board state changed to Normal on slot 1, type is S6800-54QF.

2．  Slot 1 第一异常重启从目前记录看，没有记录到重启原因，kernel exception以及reboot都没有记录

==display kernel deadloop 20 verbose slot 1 =============== 

No information to display.

=================================================================

  ===============display kernel exception 10 verbose slot 1 =============== 

No information to display.

=================================================================

  ===============display kernel reboot 20 verbose slot 1 =============== 

No information to display.

=================================================================

Secondary log buf 也没有记录到重启异常栈

<4>[   85.514549] 7:DEV_KDBG: Md:EDEV Lv:EMERG Ln: 425 Error: Input invalid IfIndex=-1

<4>---------- secondary log buffer [2] ----------

S6800，对重启类故障有多个方面的监控，对于异常信息栈的记录也比较完备。出现异常重启栈信息未记录的原因，目前从代码上分析

可能是发生Cache error引起的情况 ,  目前推测产品异常重启并无记录的过程：设备先出现cache error异常, 在处理cache error重启的流程中，get_module的时候触发异常嵌套，导致最后多核ipi发不出去，导致死循环，但是在异常嵌套情况下，死循环监控也无法记录下信息，直到长时间不喂狗，触发狗叫重启，但是重启时的信息也记录不下。

Pcie cache error 目前在R2609H18补丁已有优化。

•       问题现象：极低概率设备运行过程中异常重启。

•       问题产生条件：pcie链路抖动或信号不稳定。

现场打了补丁没有install commit，所有在slot1重启加入堆叠的时候，需要同步补丁，所以重启原因为AutoUpdateReboot。