We're running Elasticsearch 1.7.2 on ubuntu 14.04, the system has enough memory and other services on the server work just fine. But sometimes elastic shutsdown without any thing logged to it's log, or anything logged to syslog. How can I investigate this issue? This is part of the log that shows when it was down:
[2015-11-24 02:10:54,121][DEBUG][index.shard ] [Keen Marlow] [haystack_backup][4] updating index_buffer_size from [64mb] to
[4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard ] [Keen Marlow] [product_index_backup][0] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard ] [Keen Marlow] [product_index_backup][1] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard ] [Keen Marlow] [product_index_backup][2] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard ] [Keen Marlow] [product_index_backup][3] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard ] [Keen Marlow] [product_index_backup][4] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:11:24,015][INFO ][com.asquera.elasticsearch.plugins.http.HttpBasicServer] [Keen Marlow] Authorization:xx, Host:xx.xx.xx.xx:yy, Path:/, :null, Request-IP:xx.xx.xx.xx, Client-IP:null, X-Client-IPnull
[2015-11-24 02:12:15,416][INFO ][com.asquera.elasticsearch.plugins.http.HttpBasicServer] [Keen Marlow] Authorization:Basic xxxx, Host:xx.xx.xx.xx:yy, Path:/product_index/product/webservice.ProductPage.282933/, :null, Request-IP:xx.xx.xx.xx
, Client-IP:null, X-Client-IPnull
[2015-11-24 02:25:18,402][INFO ][node ] [Stakar] version[1.7.2], pid[16597], build[e43676b/2015-09-14T09:49:53Z]
[2015-11-24 02:25:18,402][INFO ][node ] [Stakar] initializing ...
[2015-11-24 02:25:18,403][DEBUG][node ] [Stakar] using home [/usr/share/elasticsearch], config [/etc/elasticsearch],
data [[/var/lib/elasticsearch]], logs [/var/log/elasticsearch], work [/tmp/elasticsearch], plugins [/usr/share/elasticsearch/plugins]
[2015-11-24 02:25:18,550][DEBUG][plugins ] [Stakar] lucene property is not set in plugin es-plugin.properties file. Ski
pping test.
[2015-11-24 02:25:18,551][DEBUG][plugins ] [Stakar] [/usr/share/elasticsearch/plugins/http-basic-server-plugin/_site] d
irectory does not exist.
[2015-11-24 02:25:18,552][DEBUG][plugins ] [Stakar] [/usr/share/elasticsearch/plugins/http-basic/_site] directory does
not exist.
[2015-11-24 02:25:18,553][INFO ][plugins ] [Stakar] loaded [http-basic-server-plugin], sites []
[2015-11-24 02:25:18,581][DEBUG][common.compress.lzf ] using encoder [VanillaChunkDecoder] and decoder[{}]
[2015-11-24 02:25:18,605][DEBUG][env ] [Stakar] using node location [[NodePath{path=/var/lib/elasticsearch/torob/no
des/0}]], local_node_id [0]
Notice the 13 minute gap between 2:12 & 2:25, when I started elastic again. There is nothing relevant in syslog or auth.log. What should I do to investigate it further?
UPDATE As I said, there is nothing related to this in syslog. This is part of it that covers that part of time:
Nov 24 01:19:01 hostname CRON[13408]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Nov 24 02:19:01 hostname CRON[16223]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Nov 24 03:19:01 hostname CRON[22518]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
