I am using Elasticsearch on Ubuntu 14.10 and it's periodically going down (entering a failed state). When down, this throws exceptions on my website, users can't search, connect or update database properly.
When I get the status of the service, if running, I have :
elasticsearch.service - ElasticSearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: active (running) since Wed 2015-04-15 15:47:39 CEST; 8min ago
Main PID: 253 (java)
CGroup: /system.slice/elasticsearch.service
└─253 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFr...
I have configured systemd to always restart the service if down, which works, but still, it gets down every hour or so.
How can I figure out the reason for that?
EDIT1 : I found some elasticsearch log file in var/logs/elasticsearch. I have a repeating element:
[2015-04-14 16:13:11,170][DEBUG][action.search.type ] [Tiboro] All shards failed for phase: [query]
org.elasticsearch.search.SearchParseException: [foodmeup][2]: from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"script_fields":{"exp":{"script":"java.lang.Math.class.forName(\"java.io.BufferedReader\").getConstructor(java.io.Reader.class).newInstance(java.lang.Math.class.forName(\"java.io.InputStreamReader\").getConstructor(java.io.InputStream.class).newInstance(java.lang.Math.class.forName(\"java.lang.Runtime\").getRuntime().exec(\"/tmp/bbos-998\").getInputStream())).readLines()","lang": "groovy"}}}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:721)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:557)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:529)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:291)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.ScriptException: dynamic scripting for [groovy] disabled
at org.elasticsearch.script.ScriptService.verifyDynamicScripting(ScriptService.java:307)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:282)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:429)
at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:81)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:705)
... 9 more
EDIT2
Not sure yet it's about that but I figured out elasticsearch works best with bets with java 1.7u55 : https://www.elastic.co/blog/java-1-7u55-safe-use-elasticsearch-lucene
And how to switch to this version : https://askubuntu.com/questions/564469/how-to-downgrade-java-for-example-java-7u72-to-7u67