tomcat6 on CentOS 6: cannot stop/restart service

Question

I have this problem with stopping and starting tomcat6 (package from the repos). I've seen with several CentOS 6 and RHEL 6 boxes.

The symptoms are that when I want to restart or stop tomcat6 it just fails. This seems to be only happening after it has been running for a while. I have a fresh CentOS 6 install and was able to restart it, but not anymore.

This is what i see:

 # service tomcat6 restart
 Stopping tomcat6:                                          [FAILED]
 Starting tomcat6:                                          [FAILED]

When I try through /usr/sbin/tomcat6:

# /usr/sbin/tomcat6 stop
/usr/sbin/tomcat6: line 60: /logs/catalina.out: No such file or directory

And output from /var/log/tomcat6/catalina.out:

Oct 22, 2012 4:53:31 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
    at java.net.Socket.connect(Socket.java:546)
    at java.net.Socket.connect(Socket.java:495)
    at java.net.Socket.<init>(Socket.java:392)
    at java.net.Socket.<init>(Socket.java:206)
    at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:424)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

I've searched the net and I can see that I am not alone, but haven't found a proper solution, hence my question.

BTW: I'm not too familiar with tomcat. Oh and: first post! So be nice ;)

Answer 1

Problem

The problem with the standard Tomcat shutdown script is that it is simply not tough enough. When you are using your distro's service scripts to stop Tomcat you'll eventually simply be calling Tomcat's own shutdown script. CentOS is no different in this respect. For this reason you need to be familiar with what Tomcat's shutdown script can do for you and most importantly what it doesn't do: It doesn't actually guarantee that Tomcat dies. Far from it.

The thing is that it is quite easy for Tomcat to get itself into some kind of situation where it cannot be killed via its management port .. or even via a TERM signal.

Let's review how Tomcat can be killed in order of escalation:

1. Killing Tomcat via its management port. This is the default and this is what Tomcat's own shutdown script attempts first. It won't work if the Java process is hung for one reason or another. All of the reports you see on the Internet where people complain that Tomcat won't stop is more or less always in this alley. This isn't really Tomcat's fault. Its quite easy to create a web-application that will make the whole container hang or impossible to stop via this method.

2. Killing Tomcat by sending the Java process a TERM signal. This is a stronger way to kill Tomcat than in (1) and Tomcat's own shutdown script will attempt this but its only activated if you've set the CATALINA_PID variable in your Tomcat setenv.sh. (which is highly recommendable in any case). Killing a Unix/Linux process by sending it a TERM signal is the default for the kill command from the OS. It is the polite way to tell a Unix/Linux process to die. Unfortunately even this will on rare occasions not kill the Tomcat process.

3. Killing Tomcat by sending the Java process a KILL signal. (from OS command line this would be kill -9 <pid>). This will always kill the process and should be the last resort. The problem here is that the standard Tomcat shutdown script will never attempt this even if method (1) and (2) fails. So if you really want to make sure that Tomcat has been killed then you have no choice but to implement your own wrapper script around Tomcat's own shutdown script.

If running Tomcat in a production environment you really need to consider if you can live with a situation where Tomcat doesn't die (or isn't restarted) when you run e.g. service tomcat restart. You may be doing this via cron and it that case you'll for sure expect a deterministic outcome, right?

Recommendations

• Always use a Tomcat setenv.sh file where you define CATALINA_PID. This will at least give you method #2 above. Tomcat's setenv.sh file doesn't exist by default so you must create it yourself.

• Create a wrapper script around Tomcat's own scripts that makes sure that Tomcat really dies.

At the place where I work we've implemented this on all hosts that run Tomcat as a service. Its the same problem/solution for any variant of Unix/Linux.

Answer 2

I ran into this today. In my case the cause was a stale pid file.

Tomcat started fine after doing a rm /var/run/tomcat6.pid.

Answer 3

I've faced the same problem. For me, It happened due to file permission/owner problem. When the init.d or /usr/sbin/tomcat6 script is unable to read the /etc/tomcat6/tomcat6.conf file, then value of ${CATALINA_BASE} becomes empty. So ${CATALINA_BASE}/logs/catalina.out in line 60 becomes /logs/catalina.out.

Answer 4

I am running Tomcat7 on CentOS; previously I was running Tomcat6.

In order to restart Tomcat, I always just go into the bin directory of my Tomcat installation and run shutdown.sh, then startup.sh You should be able to do it this way as a fallback method