Monit cannot connect to Solr/Jetty instance

Question

I have my Solr/Jetty up and running well on Ubuntu 12.04. I am trying to hookup my monit conf to monitor it, but it is having trouble connecting to the service.

I have Solr running on port 8983 and can connect via http://localhost:8983/solr/, but cannot connect via localhost, 127.0.0.1, or 0.0.0.0 in my monit conf file.

Monit conf file:

check process solr with pidfile "/var/run/jetty.pid"
start program = "/usr/sbin/service jetty start"
stop program = "/usr/sbin/service jetty stop"
    if failed host 127.0.0.1 port 8983 protocol http then restart
    if totalmem is greater than 7268 MB for 10 cycles then restart
    if 5 restarts within 5 cycles then timeout

Netstat:

root@ip-10-110-37-29:~# netstat -lnp | grep 8983
tcp        0      0 0.0.0.0:8983            0.0.0.0:*               LISTEN      16033/java

I have tried different permutations of if failed lines, but always get the following monit errors in my logs:

'solr' failed, cannot open a connection to INET[localhost:8983] via TCP

What am I doing wrong here?

[Here](http://pastebin.com/nd4eYfHU) is the jetty config, and [here](http://dev.eclipse.org/svnroot/rt/org.eclipse.jetty/jetty/trunk/jetty-distribution/src/main/resources/bin/jetty.sh) is the jetty init.d script — coneybeare, Aug 17 '14 at 03:28

score 1 · Accepted Answer · edited May 23 '17 at 19:01

I believe the problem laid in the fact that the new(er) Solr web-interface uses a ton of Javascript and other redirects, something maybe the basic monit http checker can’t handle. Turns out there is a special page for this at either http://localhost:8983/admin/ping or, in case of my multicore setup, http://localhost:8983/solr/<MY_COLLECTION_NAME>/admin/ping

There was also an issue with the start delay in jetty. I used monit to start jetty, which was checking and restarting right away, before jetty was up and able to respond. Because my cycle time was so low (10), the jetty stop program command was run before jetty was even started!

So, moving forward, I solved it by starting the jetty server on my own before I install the monit script, then tweaking the monit script to restart less frequently upon failed http connection:

check process solr with pidfile "/var/run/jetty.pid"
start program = "/usr/sbin/service jetty start"
stop program = "/usr/sbin/service jetty stop"
    if failed host localhost port 8983 protocol http and request "/solr/<MY_COLLECTION_NAME>/admin/ping" for 3 cycles then restart
    if totalmem is greater than 1024 MB for 15 cycles then restart
    if 5 restarts within 15 cycles then timeout

Monit cannot connect to Solr/Jetty instance

1 Answers1