DataEngine集群异常处理方法

节点node1不再发送心跳，集群异常。

node1.hde.h3c.com is not sending heartbeats.

[root@node1 ~]# jps

Error occurred during initialization of VM

java.lang.OutOfMemoryError: unable to create new native thread

[root@node1 ~]# su - hdfs

[hdfs@node1 ~]$ jps

Error occurred during initialization of VM

java.lang.OutOfMemoryError: unable to create new native thread

[hdfs@node1 ~]$ exit

[root@node1 ~]# jps

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Cannot create GC thread. Out of system resources.

# An error report file with more information is saved as:

# /root/hs_err_pid34510.log

[root@node1 ~]# jps

Error occurred during initialization of VM

java.lang.OutOfMemoryError: unable to create new native thread

         at java.lang.Thread.start0(Native Method)

         at java.lang.Thread.start(Thread.java:714)

         at java.lang.ref.Finalizer.<clinit>(Finalizer.java:226)

[root@node1 ~]# jps

24930 pentaho-application-launcher-6.0.0.0-353.jar

6589 -- process information unavailable

4132 -- process information unavailable

6515 -- process information unavailable

5598 -- process information unavailable

7506 -- process information unavailable

8968 -- process information unavailable

5292 -- process information unavailable

1996 Jps

[root@node1 ~]# su - hdfs

[hdfs@node1 ~]$ jps

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Cannot create GC thread. Out of system resources.

# An error report file with more information is saved as:

# /home/hdfs/hs_err_pid45812.log

初步排查，是当前进程数超过最大进程数。

[root@node1 home]# ps -eLf | wc -l

-bash: fork: Cannot allocate memory

[root@node1 home]# sysctl kernel.pid_max

-bash: fork: Cannot allocate memory

查看node2上相关值做参考

[root@node2 ~]# ps -eLf | wc -l

61614

[root@node2 ~]# sysctl kernel.pid_max

kernel.pid_max = 65536

可以看出node2上当前进程数据也接近最大进程数了。

增大最大进程数

分别在node1和node2上修改最大进程数

echo 1000000 > /proc/sys/kernel/pid_max

echo "kernel.pid_max=1000000 " >> /etc/sysctl.conf

sysctl –p

重启ambari服务后，恢复正常。

集群异常，可先查看进程是否运行正常，然后再逐步排查。