I don't remember changing anything at all on my logstash server. As of just yesterday it was working fine! And I used it to run some queries.
However.. today when I went to my logstash page, I see only thing message come up:
Connection Failed
Possibility #1: Your elasticsearch server is down or unreachable
This can be caused by a network outage, or a failure of the Elasticsearch process. If you have recently run a query that required a terms facet to be executed it is possible the process has run out of memory and stopped. Be sure to check your Elasticsearch logs for any sign of memory pressure.
Possibility #2: You are running Elasticsearch 1.4 or higher
Elasticsearch 1.4 ships with a security setting that prevents Kibana from connecting. You will need to set http.cors.allow-origin in your elasticsearch.yml to the correct protocol, hostname, and port (if not 80) that your access Kibana from. Note that if you are running Kibana in a sub-url, you should exclude the sub-url path and only include the protocol, hostname and port. For example, http://mycompany.com:8080, not http://mycompany.com:8080/kibana.
Click back, or the home button, when you have resolved the connection issue
And I can easily tell that elasticsearch 1.1.1 is running on that host:
[root@logs:~] #ps -ef | grep elasticsearch | grep -i -v -e grep -e screen
root 16666 9640 6 09:00 pts/1 00:05:49 /etc/alternatives/javahome/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC - XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/local/elasticsearch-1.1.1 -cp :/usr/local/elasticsearch-1.1.1/lib/elasticsearch-1.1.1.jar:/usr/local/elasticsearch- 1.1.1/lib/*:/usr/local/elasticsearch-1.1.1/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
And I can see, in fact that elastic search is listening on the right ports:
[root@logs:~] #netstat -tulpn | grep -i listen | grep java
tcp 0 0 :::2541 :::* LISTEN 16722/java
tcp 0 0 :::9200 :::* LISTEN 16666/java
tcp 0 0 :::9300 :::* LISTEN 16666/java
tcp 0 0 :::9301 :::* LISTEN 16722/java
All of the above ports are for elasticsearch. I use port 2541 for logstash. That's fine too.
And logstash is doing it's thing as well. I usually leave logstash running in verbose tucked away in a screen session. And when I go to that screen session I can see all the log information flying by.
I've tried stopping and starting both elasticsearch as well as logstash. But I still see the same message.
Can someone please let me know what's going on? How do I correct this?
Thanks
