CAS平台多台CVK主机异常fence案例

产品型号：

CAS 云计算管理平台

故障类型：

软件漏洞类

涉及软件版本：
    V5.0 (E0526H03)及以下版本

故障描述：

CAS平台部分CVK主机在前台管理界面出现状态异常及心跳中断的告警。同时，告警的CVk主机无法ssh连接，且无法ping通。登录到ILO界面显示花屏。

通过日志分析，发现CAS前台主机状态出现异常的原因为：主机管理网心跳超时90s 触发主机CVK fence。了解到出现告警时, 深信服公司安全设备正在对CAS平台管理的主机进行定时安全漏洞扫描任务，且漏扫时输入了系统root口令。结合现场服务器型号，提供补丁包对主机内核进行优化，然后重新进行配置了root口令漏扫测试，CVK主机crash生成了dump日志。通过dump日志，定位为：在漏洞过程中，短时间内出现大量ssh并发连接，触发CAS底层操作系统（Ubantu）的漏洞，导致服务器卡死，网络异常。

如主机故障前，同一时间15:11:01出现并发连接/断开操作

Sep 24 15:10:34 HBZWY-INTCVK12 sshd[28330]: pam_unix(sshd:session): session closed for user root

Sep 24 15:10:43 HBZWY-INTCVK12 sshd[28226]: pam_unix(sshd:session): session closed for user root

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28535]: pam_unix(cron:session): session opened for user root by (uid=0)

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28537]: pam_unix(cron:session): session opened for user root by (uid=0)

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28536]: pam_unix(cron:session): session opened for user root by (uid=0)

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28538]: pam_unix(cron:session): session opened for user root by (uid=0)

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28539]: pam_unix(cron:session): session opened for user root by (uid=0)

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28539]: pam_unix(cron:session): session closed for user root

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28537]: pam_unix(cron:session): session closed for user root

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28536]: pam_unix(cron:session): session closed for user root

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28535]: pam_unix(cron:session): session closed for user root

Sep 24 15:11:01 HBZWY-INTCVK12 CRON[28538]: pam_unix(cron:session): session closed for user root

                          Sep 24 15:12:33 HBZWY-INTCVK12 sshd[29991]: pam_unix(sshd:session): session opened for user root by (uid=0)

10.130.161.31为深信服漏扫设备的管理地址，在15:13主机crash前出现bad  protocol的异常，15：15主机已经重启恢复。

Sep 24 15:12:34 HBZWY-INTCVK12 sshd[29991]: Received disconnect from 10.130.161.21: 11: disconnected by user

Sep 24 15:12:34 HBZWY-INTCVK12 sshd[29991]: pam_unix(sshd:session): session closed for user root

Sep 24 15:12:43 HBZWY-INTCVK12 sshd[29886]: pam_unix(sshd:session): session closed for user root

Sep 24 15:12:45 HBZWY-INTCVK12 sshd[30081]: Bad protocol version identification '\026\003\001' from 10.130.161.31

Sep 24 15:12:45 HBZWY-INTCVK12 sshd[30083]: Bad protocol version identification 'GET / HTTP/1.0' from 10.130.161.31

Sep 24 15:15:46 HBZWY-INTCVK12 sshd[2322]: Server listening on 0.0.0.0 port 22.

Sep 24 15:15:46 HBZWY-INTCVK12 sshd[2322]: Server listening on :: port 22.

Sep 24 15:15:54 HBZWY-INTCVK12 su[2551]: Successful su for dsva by root

Sep 24 15:15:54 HBZWY-INTCVK12 su[2551]: + ??? root:dsva

Sep 24 15:15:54 HBZWY-INTCVK12 su[2551]: pam_unix(su:session): session opened for user dsva by (uid=0)

Sep 24 15:15:55 HBZWY-INTCVK12 su[2815]: Successful su for mysql by root

Sep 24 15:15:55 HBZWY-INTCVK12 su[2815]: + /dev/console root:mysql

Sep 24 15:15:55 HBZWY-INTCVK12 su[2815]: pam_unix(su:session): session opened for user mysql by (uid=0)

Sep 24 15:15:55 HBZWY-INTCVK12 su[2815]: pam_unix(su:session): session closed for user mysql

Sep 24 15:15:58 HBZWY-INTCVK12 sshd[2322]: Received signal 15; terminating.

Sep 24 15:15:58 HBZWY-INTCVK12 sshd[3729]: Server listening on 0.0.0.0 port 22.

Sep 24 15:15:58 HBZWY-INTCVK12 sshd[3729]: Server listening on :: port 22.

Sep 24 15:15:58 HBZWY-INTCVK12 sshd[3729]: Received signal 15; terminating.

Sep 24 15:15:58 HBZWY-INTCVK12 sshd[3817]: Server listening on 0.0.0.0 port 22.

Sep 24 15:15:58 HBZWY-INTCVK12 sshd[3817]: Server listening on :: port 22.

Sep 24 15:16:01 HBZWY-INTCVK12 sshd[4257]: Accepted password for root from 10.130.161.36 port 63784 ssh2

Sep 24 15:16:01 HBZWY-INTCVK12 sshd[4257]: pam_unix(sshd:session): session opened for user root by (uid=0)

Sep 24 15:16:01 HBZWY-INTCVK12 CRON[4347]: pam_unix(cron:session): session opened for user root by (uid=0)

如CVKcrash重启后（漏扫未执行完继续进行），15:17，日志里短时间内有大量的SSH访问：

Sep 24 15:17:08 HBZWY-INTCVK12 sshd[8970]: Accepted password for root from 10.130.161.31 port 41393 ssh2

Sep 24 15:17:08 HBZWY-INTCVK12 sshd[8985]: Accepted password for root from 10.130.161.31 port 41394 ssh2

Sep 24 15:17:08 HBZWY-INTCVK12 sshd[8987]: Accepted password for root from 10.130.161.31 port 41395 ssh2

Sep 24 15:17:08 HBZWY-INTCVK12 sshd[9011]: Accepted password for root from 10.130.161.31 port 41396 ssh2

Sep 24 15:17:08 HBZWY-INTCVK12 sshd[9011]: Accepted password for root from 10.130.161.31 port 41396 ssh2

Sep 24 15:17:09 HBZWY-INTCVK12 sshd[9025]: Accepted password for root from 10.130.161.31 port 41397 ssh2

Sep 24 15:17:09 HBZWY-INTCVK12 sshd[9041]: Accepted password for root from 10.130.161.31 port 41398 ssh2

Sep 24 15:17:09 HBZWY-INTCVK12 sshd[9043]: Accepted password for root from 10.130.161.31 port 41399 ssh2

Sep 24 15:17:09 HBZWY-INTCVK12 sshd[9063]: Accepted password for root from 10.130.161.31 port 41400 ssh2

Sep 24 15:17:10 HBZWY-INTCVK12 sshd[9085]: Accepted password for root from 10.130.161.31 port 41402 ssh2

Sep 24 15:17:10 HBZWY-INTCVK12 sshd[9102]: Accepted password for root from 10.130.161.31 port 41403 ssh2

Sep 24 15:17:10 HBZWY-INTCVK12 sshd[9106]: Accepted password for root from 10.130.161.31 port 41404 ssh2

主机crash时内核调用栈抛出异常如下，此内核抛出异常与linux 已知漏洞： 

***.***/patchwork/patch/1033267/       [v4] tty: fix race between flush_to_ldisc and tty_open描述一致。

1、临时规避方案：通过了解其他局点、其他厂家产品，一般情况下进行等保测评时，除了基线扫描，其他扫描项都采取外部扫描的方式，即不需要提供root口令；因此，建议现场进行安全测评时，除了基线扫描外，其他如漏洞扫描，不提供root口令。

2、此bug将在CASE0535新版本进行修复，新版本预计19年10月下旬发布。届时可通过升级彻底解决。