1

I get the error "monit failed protocol test [HTTP] at [localhost]:8080" every so often. The application, tomcat, has not stopped and is still running just fine. I know port 8080 is open because I can get to it in the browser on my personal PC ( so NOT localhost ). What else can I check to see what is happening?

Some logs in response to questions from someone else:

It happened again this morning at 3:26 or 3:27 am this morning, so here are, first of all, the monit log for the time in question:

[EDT May 19 03:26:01] info     : Reinitializing monit daemon
[EDT May 19 03:26:01] info     : 'newapp.turnsmith.com' Monit reloaded
[EDT May 19 03:26:01] error    : 'tomcat' failed protocol test [HTTP] at [localhost]:8080 [TCP/IP] -- HTTP error: Server returned status 404
[EDT May 19 03:26:01] info     : 'tomcat' start: '/etc/init.d/tomcat start'
[EDT May 19 03:26:01] error    : 'tomcat' failed protocol test [HTTP] at [localhost]:8080 [TCP/IP] -- HTTP error: Server returned status 404
[EDT May 19 03:27:01] error    : 'tomcat' failed protocol test [HTTP] at [localhost]:8080 [TCP/IP] -- HTTP error: Server returned status 404
[EDT May 19 03:27:01] info     : 'tomcat' start: '/etc/init.d/tomcat start'

And here is the log from http, error_log:

[Sun May 19 03:26:01.573204 2019] [auth_digest:notice] [pid 3625] AH01757: generating secret for digest authentication ...
[Sun May 19 03:26:01.573817 2019] [lbmethod_heartbeat:notice] [pid 3625] AH02282: No slotmem from mod_heartmonitor
[Sun May 19 03:26:01.573856 2019] [mpm_prefork:notice] [pid 3625] AH00163: Apache/2.4.6 (CentOS) configured -- resuming normal operations
[Sun May 19 03:26:01.573859 2019] [core:notice] [pid 3625] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

The error actually happens more, but we only get the "alert" email that one time.

Also, I don't know if this will help, but here is monit config for tomcat ( /etc/monit.d/tomcat ):

check host tomcat with address localhost
 stop program = "/etc/init.d/tomcat stop"
 start program = "/etc/init.d/tomcat start"
 if failed port 8080 and protocol http then alert
 if failed port 8080 and protocol http then start

Update 5/20/2019: Based on @asktyagi advice, I changed my config to the below, seeing if it works:

check host tomcat with address localhost
 stop program = "/etc/init.d/tomcat stop"
 start program = "/etc/init.d/tomcat start"
 if failed port 8080 and protocol http retry 5 then alert
 if failed port 8080 and protocol http retry 5 then start
Tony B
  • 254
  • 2
  • 12

2 Answers2

1

Can you check monit and http logs at the time of alert? some possibilities are: http port exhausted or http under pressure/busy. If so try to use RETRY. Edit: Also you can try to remove protocol and use like below if failed port {port number} then restart

asktyagi
  • 2,401
  • 1
  • 5
  • 19
  • Added more information to main post, but just to clarify: What do you mean by "RETRY"? I don't see port exhausting or pressure or anything, but maybe I am looking in the wrong logs? – Tony B May 19 '19 at 19:37
  • Some time first attempt may be unsuccessful due to unforeseen issues, so just try to increase the number of monit retry. – asktyagi May 20 '19 at 01:56
  • OK, found it now, I think, and will give it a try. I guess I should have said I was not that experienced with monit. Anyway, it is weird, but it consistently happens in the morning when monit does some reloading of some sort. This does not happen on my other machines. But there are differences between OS and tomcat version used, so I guess that could be it. – Tony B May 20 '19 at 16:07
  • Still same problem, and got an alert at 3:25 am when monit did it's reload. This error happens a lot throughout the day, but only get the alert once a day. I tried 20 retries, and it does not seem to go away. Now, with 20 retries, tomcat is not always "restarted", so it is am improvement. Just something seems weird. – Tony B May 21 '19 at 15:04
  • can you try to use only port bases monit monitoring and check it works, sometime protocol behave differently. – asktyagi May 21 '19 at 15:11
  • What do you mean? I am not the most experienced person with monit, so a little clarification would help. I don't see anything referring to "port bases" in monit documentation. Searching for "port bases" related to monit does not really reveal anything in Google, so I might be using the wrong terminology. – Tony B May 21 '19 at 15:19
  • 1
    for example use like this **if failed port {port number} then restart** – asktyagi May 21 '19 at 15:22
  • if you want credit, just update your answer. As per my answer, the key was getting rid of the http check. – Tony B May 22 '19 at 21:13
  • Thanks edited my answer. – asktyagi May 23 '19 at 03:01
0

Based on advice from @asktyagi, I got it working much better with the below config:

check host tomcat with address localhost
 stop program = "/etc/init.d/tomcat stop"
 start program = "/etc/init.d/tomcat start"
 ## if failed port 8080 and protocol http for 20 cycles then alert
 if failed port 8080 then alert
 ##if failed port 8080 and protocol http for 40 cycles then start
 if failed port 8080 then start

In my case, "Retry" did not help although it did decrease the occurrences. Also, based on help from here, I tried the "for x cycles" idea and that actually decreased it from multiple times per minute to only once per minute. Obviously, I am sticking with the idea from @asktyagi as it, so far, eliminates the problem, but did want to tell people what I figured out.

Tony B
  • 254
  • 2
  • 12