Tomcat Cluster Farm Deployer issue

Question

I have setup apache-tomcat cluster with mod_jk on ubuntu server. I have some issues with FarmDeployer which deploys war files to other web servers in the cluster and I got some hope that it works as I posted and got FarmDeployer working here.

The war deployer works perfectly with the below configuration from the post but not working trustfully as it sometimes gives 404 error when accessed the page:

 <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

    <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
        <Channel className="org.apache.catalina.tribes.group.GroupChannel">
            <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                  address="192.168.1.4"
                  port="4000"
                  selectorTimeout="100"
                  maxThreads="6"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
                <Member className="org.apache.catalina.tribes.membership.StaticMember"
                      port="4000"
                      securePort="-1"
                      host="192.168.1.2"
                      domain="staging-cluster"
                      uniqueId="{0,1,2,3,4,5,6,7,8,9}"/>
            </Interceptor>
        </Channel>
        <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
            tempDir="/usr/share/tomcat/temp/"
            deployDir="/usr/share/tomcat/webapps/"
            watchDir="/usr/share/tomcat/watch/"
            watchEnabled="true"/>
    </Cluster>
  </Host>

---

The following is the configuration(from tomcat cluster docs) which works perfectly for HA and LB clustering. But it fails in Farm Deployer and there is no trace in logs that why the farm deployer is not working.

<Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">

  <Manager className="org.apache.catalina.ha.session.DeltaManager" 

          expireSessionsOnShutdown="false"

          notifyListenersOnReplication="true"/>



  <Channel className="org.apache.catalina.tribes.group.GroupChannel">

    <Membership className="org.apache.catalina.tribes.membership.McastService" 

          address="228.0.0.4" 

          port="45564" frequency="500" 

          dropTime="3000"/>

    <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

     <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>

    </Sender>

    <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" 

           address="auto" port="4000" autoBind="100"

           selectorTimeout="5000" maxThreads="6"/>

    <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>

    <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>

  </Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>

<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>

<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>

<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>

</Cluster>


<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
            tempDir="/usr/share/tomcat/temp/"
            deployDir="/usr/share/tomcat/webapps/"
            watchDir="/usr/share/tomcat/watch/"
            watchEnabled="true"/>


</Host>

Anybody has any idea or any working configuration on how to make farm deployer working perfectly with the above HA & LB configuration?.

Thank you!

---

Update:1

I configured with 3 nodes as per your configuration and I could not access the site with the domain name(point to LB) and it's giving tomcat ROOT page but with the ip/myapp it's rendering.

The war file myapp.war is deployed to all the servers but it's not extracting on other 2 nodes other than manager node(deployer web server).

logs on other 2 webservers:

dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 56}:4000,{-64, -88, 2, 56},4000, alive=2908632,id={64 21 -13 -74 -38 -98 70 117 -117 77 40 -79 124 -110 56 59 }, payload={}, command={}, domain={}, ]]
    18 Aug, 2011 3:17:33 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
    INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 56}:4000,{-64, -88, 2, 56},4000, alive=1020,id={-65 -93 19 -47 -112 -115 78 -60 -118 99 120 107 -51 115 -88 115 }, payload={}, command={}, domain={}, ]
    18 Aug, 2011 3:19:55 AM org.apache.catalina.core.StandardContext stop
    INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/myapp] has not been started
    18 Aug, 2011 3:19:55 AM org.apache.catalina.startup.HostConfig checkResources
    INFO: Undeploying context [/myapp]
    18 Aug, 2011 3:19:59 AM org.apache.catalina.startup.HostConfig checkResources
    INFO: Undeploying context [/myapp]
    18 Aug, 2011 3:20:01 AM org.apache.catalina.startup.HostConfig deployWAR
    INFO: Deploying web application archive myapp.war
    18 Aug, 2011 3:20:01 AM org.apache.catalina.startup.ContextConfig init
    SEVERE: Exception fixing docBase: {0} 
    java.util.zip.ZipException: invalid CEN header (bad signature)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:131)
        at java.util.jar.JarFile.<init>(JarFile.java:150)
        at java.util.jar.JarFile.<init>(JarFile.java:87)
        at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:90)
        at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:66)
        at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:86)
        at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
        at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
        at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:141)
        at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:883)
        at org.apache.catalina.startup.ContextConfig.init(ContextConfig.java:1012)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:279)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
        at org.apache.catalina.core.StandardContext.init(StandardContext.java:5338)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4086)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:515)
        at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1231)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
        at org.apache.catalina.ha.deploy.FarmWarDeployer.check(FarmWarDeployer.java:623)
        at org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived(FarmWarDeployer.java:246)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:899)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:880)
        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:269)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
    18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext resourcesStart
    SEVERE: Error starting static Resources
    java.lang.IllegalArgumentException: Invalid or unreadable WAR file : invalid CEN header (bad signature)
        at org.apache.naming.resources.WARDirContext.setDocBase(WARDirContext.java:135)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3957)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4126)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:515)
        at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1231)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:297)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
        at org.apache.catalina.ha.deploy.FarmWarDeployer.check(FarmWarDeployer.java:623)
        at org.apache.catalina.ha.deploy.FarmWarDeployer.messageReceived(FarmWarDeployer.java:246)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:899)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:880)
        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:269)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:110)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:79)
        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:241)
        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:225)
        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:188)
        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:91)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
    18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext start
    SEVERE: Error in resourceStart()
    18 Aug, 2011 3:20:01 AM org.apache.catalina.ha.deploy.FileMessageFactory writeMessage
    WARNING: Receive Message again -- Sender ActTimeout to short [ path: /myapp war: /usr/share/tomcat/webapps/myapp.war data: [B@107108e data length: 6535 ]
    18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext start
    SEVERE: Error getConfigured
    18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext start
    SEVERE: Context [/myapp] startup failed due to previous errors
    18 Aug, 2011 3:20:01 AM org.apache.catalina.core.StandardContext stop
    INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/myapp] has not been started
    18 Aug, 2011 3:20:09 AM org.apache.catalina.startup.HostConfig deployWAR
    INFO: Deploying web application archive myapp.war
    18 Aug, 2011 3:20:09 AM org.apache.catalina.startup.ContextConfig init
    SEVERE: Exception fixing docBase: {0} 
    java.util.zip.ZipException: invalid CEN header (bad signature)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:131)
        at java.util.jar.JarFile.<init>(JarFile.java:150)
        at java.util.jar.JarFile.<init>(JarFile.java:87)
        at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:90)
        at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:66)
        at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:86)
        at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
        at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89)
        at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:141)
        at org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:883)
        at org.apache.catalina.startup.ContextConfig.init(ContextConfig.java:1012)
        at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:279)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
        at org.apache.catalina.core.StandardContext.init(StandardContext.java:5338)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4086)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:719)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490)
        at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1217)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
        at java.lang.Thread.run(Thread.java:636)
    18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext resourcesStart
    SEVERE: Error starting static Resources
    java.lang.IllegalArgumentException: Invalid or unreadable WAR file : invalid CEN header (bad signature)
        at org.apache.naming.resources.WARDirContext.setDocBase(WARDirContext.java:135)
        at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3957)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4126)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:830)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:719)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490)
        at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1217)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:293)
        at 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
        at java.lang.Thread.run(Thread.java:636)
    18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext start
    SEVERE: Error in resourceStart()
    18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext start
    SEVERE: Error getConfigured
    18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext start
    SEVERE: Context [/myapp] startup failed due to previous errors
    18 Aug, 2011 3:20:09 AM org.apache.catalina.core.StandardContext stop
    INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[www.mydomain.com].[/myapp] has not been started
    18 Aug, 2011 3:28:09 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
    WARNING: Context manager doesn't exist:/myapp
    18 Aug, 2011 3:28:09 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
    WARNING: Context manager doesn't exist:/myapp

---

logs on deployer web server:

18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.18
18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base /usr/share/tomcat/webapps/myapp does not exist or is not a readable directory
    at org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:141)
    at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:3957)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4126)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:719)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
    at org.apache.catalina.core.StandardService.start(StandardService.java:516)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:578)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext start
SEVERE: Error in resourceStart()
18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext start
SEVERE: Error getConfigured
18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [] startup failed due to previous errors
18 Aug, 2011 4:50:57 PM org.apache.catalina.core.StandardContext stop
INFO: Container org.apache.catalina.core.ContainerBase.[Catalina].[www.mydomain.com].[/] 
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.deploy.FarmWarDeployer start
INFO: Cluster deployment is watching /usr/share/tomcat/watch/ for changes.
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.deploy.FarmWarDeployer start
INFO: Cluster FarmWarDeployer started.
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.JvmRouteBinderValve start
INFO: JvmRouteBinderValve started
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /manager to cluster element Host with name localhost
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /manager
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [/manager]: skipping state transfer. No members active in cluster group.
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /docs to cluster element Host with name localhost

INFO: Starting clustering manager at 
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager []: skipping state transfer. No members active in cluster group.
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /host-manager to cluster element Host with name localhost
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /host-manager
18 Aug, 2011 4:50:59 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [/host-manager]: skipping state transfer. No members active in cluster group.
18 Aug, 2011 4:51:00 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /examples to cluster element Host with name localhost
18 Aug, 2011 4:51:00 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /examples
18 Aug, 2011 4:51:00 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [/examples]: skipping state transfer. No members active in cluster group.
18 Aug, 2011 4:51:00 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-9080
18 Aug, 2011 4:51:00 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
18 Aug, 2011 4:51:00 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/57  config=null
18 Aug, 2011 4:51:00 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3206 ms
18 Aug, 2011 4:51:01 PM org.apache.catalina.tribes.io.BufferPool getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.101:4002,192.168.1.101,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ]
18 Aug, 2011 4:51:04 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.101:4002,192.168.1.101,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ]
18 Aug, 2011 4:51:06 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 53}:4001,{-64, -88, 2, 53},4001, alive=1007,id={80 88 -34 -11 98 72 71 -70 -75 -107 -48 -91 -90 50 -126 116 }, payload={}, command={}, domain={}, ]
18 Aug, 2011 4:51:09 PM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck
WARNING: Member added, even though we werent notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.102:4001,192.168..102,4001, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ]
18 Aug, 2011 4:51:09 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.1.102:4001,192.168.1.102,4001, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ]
18 Aug, 2011 4:51:20 PM org.apache.catalina.ha.deploy.WarWatcher check
INFO: check cluster wars at /usr/share/tomcat/watch
18 Aug, 2011 4:51:20 PM org.apache.catalina.ha.deploy.FarmWarDeployer fileModified
INFO: Installing webapp[/myapp] from /usr/share/tomcat/webapps/myapp.war
18 Aug, 2011 4:51:20 PM org.apache.catalina.ha.deploy.FarmWarDeployer remove
INFO: Cluster wide remove of web app /myapp
18 Aug, 2011 4:51:26 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive myapp.war
18 Aug, 2011 4:51:27 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /myapp to cluster element Host with name localhost
18 Aug, 2011 4:51:27 PM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /myapp
18 Aug, 2011 4:51:27 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [/myapp], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 2, 101}:4002,{-64, -88, 2, 101},4002, alive=26275,id={-120 -66 103 32 -56 -122 77 51 -95 20 25 88 83 -69 -59 -115 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds.
18 Aug, 2011 4:52:27 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
SEVERE: Manager [/myapp]: No session state send at 18/8/11 4:51 PM received, timing out after 60,073 ms.
18 Aug, 2011 4:52:44 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive myapp.war
18 Aug, 2011 4:52:45 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
18 Aug, 2011 4:52:48 PM org.apache.catalina.core.ApplicationContext log
INFO: org.tuckey.web.filters.urlrewrite.utils.Log ERROR: logLevelConf: null
18 Aug, 2011 4:52:48 PM org.apache.catalina.core.ApplicationContext log
INFO: org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: loaded (conf ok)
18 Aug, 2011 4:52:48 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'SpringAction'
18 Aug, 2011 4:53:00 PM org.apache.catalina.ha.deploy.WarWatcher check
INFO: check cluster wars at /usr/share/tomcat/watch
18 Aug, 2011 4:53:20 PM org.apache.catalina.ha.deploy.WarWatcher check

---

Update:2 for EDIT

ifconfig listed the below entry for eth0 on all web servers

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

Executed the below command on all the web servers

route add -host 228.0.0.4 dev eth0

ping to 228.0.0.4 didn't work on any of the system

lsof -i :4000 listed similar to what is listed to you(could not paste here as characters exceeded for this post) on deployer web server(also on other servers):

IMPROVEMENT: I happily Used the Multicast configuration changing auto to ip in Receiver and I saw myapp.war on another web server when I placed the war file in watch directory on deployer web server. The Problem is, war is only extracting on the deployer web server(creating a direcotry) but not on other servers.

Answer 1

How do I update for 3 web servers?

Do the same as when doing with 2 nodes.

On the deployment node, add one more member in StaticMembershipInterceptor:

<Member className="org.apache.catalina.tribes.membership.StaticMember"
                          port="4002"
                          securePort="-1"
                          host="192.168.5.208"
                          domain="staging-cluster"
                          uniqueId="{0,1,2,3,4,5,6,7,8,9}"/>

On the node 3, create a cluster like belows:

        <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster">
            <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="192.168.5.208"
                      port="4002"
                      selectorTimeout="100"
                      maxThreads="6"/>
            </Channel>
            <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
                tempDir="/opt/tomcat6/tempdir/"
                deployDir="/opt/tomcat6/webapps/"
                watchDir="/opt/tomcat6/watchdir/"
                watchEnabled="false"/>
        </Cluster>

And copy a *.war file to watchdir directory on the deployment node, the below lines will appears in catalina.out:

INFO: check cluster wars at /usr/share/tomcat6/watchdir
Aug 17, 2011 2:07:53 PM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck
WARNING: Member added, even though we werent notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.5.208:40
02,192.168.5.208,4002, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 
103 45 99 ...(15)}, ]
Aug 17, 2011 2:07:53 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://192.168.5.208:4002,192.168.5.208,4002, 
alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={115 116 97 103 105 110 103 45 99 ...(15)}, ]
Aug 17, 2011 2:08:12 PM org.apache.catalina.ha.deploy.WarWatcher check
INFO: check cluster wars at /usr/share/tomcat6/watchdir
Aug 17, 2011 2:08:12 PM org.apache.catalina.ha.deploy.FarmWarDeployer fileModified
INFO: Installing webapp[/cas] from /usr/share/tomcat6/webapps/cas.war
Aug 17, 2011 2:08:12 PM org.apache.catalina.ha.deploy.FarmWarDeployer remove
INFO: Cluster wide remove of web app /cas
Aug 17, 2011 2:08:13 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive cas.war

Check the webapps folder on the node 3:

drwxr-xr-x  8 root    root        4096 Aug 17 14:08 cas
-rw-r--r--  1 root    root    19962865 Aug 17 14:08 cas.war
...

and the catalina.out:

Aug 17, 2011 2:08:13 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive cas.war

---

EDIT

Invalid or unreadable WAR file : invalid CEN header (bad signature)

It looks like the WAR file is corrupted when deploying to 2 other nodes. How often does it happen?

Isn't it possible making FarmDeployer work with multicast from tomcat cluster docs which works perfectly for me apart from FarmDeployer?

I figured it out! The reason is you haven't added a multicast route for your network interface, so the cluster message cannot be send/receive between Tomcat nodes:

lsof -i :4000
COMMAND  PID   USER   FD   TYPE    DEVICE SIZE NODE NAME
java    9968 tomcat   37u  IPv6 121232775       TCP localhost:terabase (LISTEN)

Do it by following steps:

• Make sure that your network interface is enabled for multicast:

  UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

• Add a multicast route:

  route add -host 228.0.0.4 dev bond1

(you should change bond1 to the network interface that receives the cluster message)

Testing with ping -c 4 228.0.0.4 didn't work for me. So, I downloaded multicast test tool, run it and I see the belows response:

./bin/multicast-test.sh -local 192.168.5.149 -group 228.0.0.4:45564
PATH=/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/ossec/bin
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)

Oracle Coherence Version 3.3.1/389
 Grid Edition: Development mode
Copyright (c) 2000-2007 Oracle. All rights reserved.

Starting test on ip=/192.168.5.149, group=/228.0.0.4:45564, ttl=4
Configuring multicast socket...
Starting listener...
Fri Aug 19 10:19:14 ICT 2011: Sent packet 1.
Fri Aug 19 10:19:14 ICT 2011: Received test packet 1 from self (sent 9ms ago).
Fri Aug 19 10:19:14 ICT 2011: Received 69 bytes from an unknown multicast application at 192.168.5.199: ???
Fri Aug 19 10:19:14 ICT 2011: Received 69 bytes from an unknown multicast application at 192.168.5.149: ???

Or you can check with lsof, netstat, ...

lsof -i :4000
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE NODE NAME
java    11084 tomcat   37u  IPv6 121236631       TCP localhost:terabase (LISTEN)
java    11084 tomcat   59u  IPv6 121236830       TCP localhost:terabase->192.168.5.199:43519 (ESTABLISHED)
java    11084 tomcat   66u  IPv6 121236835       TCP localhost:55538->192.168.5.199:terabase (ESTABLISHED)
java    11084 tomcat   67u  IPv6 121236836       TCP localhost:55539->192.168.5.199:terabase (ESTABLISHED)

You are ready for configure Tomcat clustering with multicast.

Enable clustering configuration same as mentioned in the doc, but change <Receiver address from "auto" to specified interface:

<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                      address="192.168.5.149"
                      port="4000"
                      autoBind="100"
                      selectorTimeout="5000"
                      maxThreads="6"/>

Do the same for all the nodes in the cluster to see it works.

---

EDIT 2

IMPROVEMENT: I happily Used the Multicast configuration changing auto to ip in Receiver and I saw myapp.war on another web server when I placed the war file in watch directory on deployer web server. The Problem is, war is only extracting on the deployer web server(creating a direcotry) but not on other servers.

It means that the connection between deployment node and the other nodes is established. Did you still get the same error "Invalid or unreadable WAR file : invalid CEN header (bad signature)"? Everytime?

---

java.util.zip.ZipException: invalid END header (bad central directory offset)

According to the bug 45851, you should upgrade your Tomcat version (at least 6.0.19) and try again. I'm using version 6.0.29 and as you can see, it works fine.

---

It's needing tomcat restart on other web servers then it copies and extracts and the site is available with other-web-severs-ip/myapp but not with domain name(404 error). Where do you point the site domain name, I pointed the site domain name in the DNS to LB and created a virtual host in all the tomcat web servers with same domain name

No. It seems that you haven't cleared thoroughly. No need to create virtual host in all Tomcat nodes.

Assuming that you have Apache HTTP server and Tomcat on node 1, with IP 192.168.1.101. Two other Tomcat nodes at 1.102, 1.103. jvmRoute corresponding are: tomcat1, tomcat2, tomcat3.

The mod_jk.conf on Apache HTTP server:

LoadModule          jk_module modules/mod_jk.so

JkWorkersFile       /etc/httpd/conf/workers.properties
JkShmFile           /var/log/httpd/jk.shm

JkLogFile           /var/log/httpd/mod_jk.log
JkLogLevel          info
JkLogStampFormat    "[%a %b %d %H:%M:%S %Y] "

JkMount /myapp/*    cluster

The workers.properties is something like this:

worker.list = cluster

worker.tomcat1.port=8009        
worker.tomcat1.host=localhost
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1

worker.tomcat2.port=8009        
worker.tomcat2.host=192.168.1.102
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1

worker.tomcat3.port=8009        
worker.tomcat3.host=192.168.1.103
worker.tomcat3.type=ajp13
worker.tomcat3.lbfactor=1

worker.cluster.type=lb
worker.cluster.balance_workers=tomcat1,tomcat2,tomcat3
worker.cluster.sticky_session=1

Try again.