节点node1不再发送心跳,集群异常。
node1.hde.h3c.com is not sending heartbeats.
[root@node1 ~]# jps
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
[root@node1 ~]# su - hdfs
[hdfs@node1 ~]$ jps
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
[hdfs@node1 ~]$ exit
[root@node1 ~]# jps
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /root/hs_err_pid34510.log
[root@node1 ~]# jps
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.lang.ref.Finalizer.<clinit>(Finalizer.java:226)
[root@node1 ~]# jps
24930 pentaho-application-launcher-6.0.0.0-353.jar
6589 -- process information unavailable
4132 -- process information unavailable
6515 -- process information unavailable
5598 -- process information unavailable
7506 -- process information unavailable
8968 -- process information unavailable
5292 -- process information unavailable
1996 Jps
[root@node1 ~]# su - hdfs
[hdfs@node1 ~]$ jps
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# /home/hdfs/hs_err_pid45812.log
初步排查,是当前进程数超过最大进程数。
[root@node1 home]# ps -eLf | wc -l
-bash: fork: Cannot allocate memory
[root@node1 home]# sysctl kernel.pid_max
-bash: fork: Cannot allocate memory
查看node2上相关值做参考
[root@node2 ~]# ps -eLf | wc -l
61614
[root@node2 ~]# sysctl kernel.pid_max
kernel.pid_max = 65536
可以看出node2上当前进程数据也接近最大进程数了。
增大最大进程数
分别在node1和node2上修改最大进程数
echo 1000000 > /proc/sys/kernel/pid_max
echo "kernel.pid_max=1000000 " >> /etc/sysctl.conf
sysctl –p
重启ambari服务后,恢复正常。
集群异常,可先查看进程是否运行正常,然后再逐步排查。
该案例暂时没有网友评论
✖
案例意见反馈
亲~登录后才可以操作哦!
确定你的邮箱还未认证,请认证邮箱或绑定手机后进行当前操作