3

I am using Elasticsearch on Ubuntu 14.10 and it's periodically going down (entering a failed state). When down, this throws exceptions on my website, users can't search, connect or update database properly.

When I get the status of the service, if running, I have :

elasticsearch.service - ElasticSearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: active (running) since Wed 2015-04-15 15:47:39 CEST; 8min ago
 Main PID: 253 (java)
   CGroup: /system.slice/elasticsearch.service
           └─253 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFr...

I have configured systemd to always restart the service if down, which works, but still, it gets down every hour or so.

How can I figure out the reason for that?

EDIT1 : I found some elasticsearch log file in var/logs/elasticsearch. I have a repeating element: 

[2015-04-14 16:13:11,170][DEBUG][action.search.type       ] [Tiboro] All shards failed for phase: [query]
org.elasticsearch.search.SearchParseException: [foodmeup][2]: from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"script_fields":{"exp":{"script":"java.lang.Math.class.forName(\"java.io.BufferedReader\").getConstructor(java.io.Reader.class).newInstance(java.lang.Math.class.forName(\"java.io.InputStreamReader\").getConstructor(java.io.InputStream.class).newInstance(java.lang.Math.class.forName(\"java.lang.Runtime\").getRuntime().exec(\"/tmp/bbos-998\").getInputStream())).readLines()","lang": "groovy"}}}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:721)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:557)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:529)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:291)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.script.ScriptException: dynamic scripting for [groovy] disabled
    at org.elasticsearch.script.ScriptService.verifyDynamicScripting(ScriptService.java:307)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:282)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:429)
    at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:81)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:705)
    ... 9 more

EDIT2

Not sure yet it's about that but I figured out elasticsearch works best with bets with java 1.7u55 : https://www.elastic.co/blog/java-1-7u55-safe-use-elasticsearch-lucene

And how to switch to this version : https://askubuntu.com/questions/564469/how-to-downgrade-java-for-example-java-7u72-to-7u67

Community
  • 1
Sébastien
  • 233
  • 3
  • 8
  • Indeed, I found some logs. Does that speak to you ? Why is elasticsearch calling a script on its server ? – Sébastien Apr 15 '15 at 18:19
  • as per the systemctl status, would that be a failure of java ? (the main PID is java) – Sébastien Apr 15 '15 at 18:21
  • I don't know the answer. My concern (as a moderator) was to help you improve the question, so that someone who does know the answer will be able to do so. – Michael Hampton Apr 15 '15 at 18:23
  • it seems what i explain in my last edit made the trick, more than 4h of runtime, it wouldn't run that long before – Sébastien Apr 15 '15 at 22:41

0 Answers0