My elastic search clusters keep crashing for no reason. All I can see in the logs is :
INFO | jvm 4 | 2015/04/17 19:10:09 | *** glibc detected *** /usr/java/jre1.7.0_04/bin/java: malloc(): memory corruption: 0x00007f6a2c4d8560 ***
I have a cluster with one master and two slaves. I do have an old version ("0.20.2") but it has never been a problem so far... I can not update easily and will only consider it as a last resort.
I don't really know where to start; So far, I have changed the following setting:
wrapper.java.additional.3=-Xss256ktowrapper.java.additional.3=-Xss512k- and have deleted all the indices (this needed some cleaning)
Unfortunately, the problem is still happening even without any indices..
java is java version "1.7.0_04" Java(TM) SE Runtime Environment (build 1.7.0_04-b20) Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
Before the wrapper setting change I was also getting
INFO | jvm 1 | 2015/04/17 16:03:05 | 7fffe41fe000-7fffe4200000 r-xp 00000000 00:00 0 [vdso]
INFO | jvm 1 | 2015/04/17 16:03:05 | ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
STATUS | wrapper | 2015/04/17 16:03:05 | JVM process is gone.
ERROR | wrapper | 2015/04/17 16:03:05 | JVM exited unexpectedly.
STATUS | wrapper | 2015/04/17 16:03:10 | Launching a JVM...
INFO | jvm 2 | 2015/04/17 16:03:10 | WrapperManager: Initializing...
STATUS | wrapper | 2015/04/17 16:03:39 | JVM received a signal UNKNOWN (6).
STATUS | wrapper | 2015/04/17 16:03:39 | JVM process is gone.
service.log
Happy to provide whatever else information is needed to help troubleshoot this.
