Static Tomcat cluster, Apache load balancer and replicating sessions

The normal way to set up a Tomcat 6 cluster is to use multicast messages to manage cluster membership, with Tribes replication of session data (or deltas) to all known members. When multicasting is not supported in your network environment, static membership is an alternative you can try. It’s not as flexible, in that you cannot dynamically size a running cluster, but it has all the essential features such as session synchronization, balancing, stickiness and failover.

Assuming you are reasonably familiar with Apache httpd and Apache Tomcat, here’s the OS-neutral recipe.

You will need:

• Apache httpd 2.2
• Apache Tomcat 6.0.xx (and we will assume it’s on the latest Java)
• Connectors 1.2.xx (latest version of mod_jk)
• A distributable Web application.

For this little tutorial, let’s assume that the Apache load balancer will be on 10.0.0.123 (port 80), and the two Tomcats will be on 10.0.0.111:80 and 10.0.0.222:80 respectively. Both Tomcats will also be using AJP on port 8009. Finally, we assume your DNS (or hosts file) will have cluster.example.com mapped to the load balancer on 10.0.0.123. If you want SSL, you’ll need a secure port (e.g. 8433) assigned to another connector, though you’ll need to terminate the SSL at the load balancer, then set secure=”true” and SSLEnabled=”false” in the corresponding <Connector> element. We are also reserving port 4444 for replication traffic. Obviously you can change all these values to suit your own requirements, to fit your network, avoid IP/port clashes etc.

Step 1 : set up Apache 2.2 as a load balancer

Download the latest version of mod_jk.so and place it into the modules directory in Apache. Edit the conf/httpd.conf file so that the following lines are placed immediately after the existing list of LoadModule entries:

LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"
JkMount / balancer
JkMount /* balancer

Make sure you are not using virtual hosts by commenting out the Include that loads the vhosts file:

#Include conf/extra/httpd-vhosts.conf

Edit the “Listen” entry so that it matches the loadbalancer address:

Listen: 10.0.0.123:80

Create a file called conf/workers.properties with the following:

worker.list=balancer
worker.node1.port=8009
worker.node1.host=10.0.0.111
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.node2.port=8009
worker.node2.host=10.0.0.222
worker.node2.type=ajp13
worker.node2.lbfactor=1
worker.balancer.type=lb
worker.balancer.balance_workers=node1,node2
worker.balancer.method=B

Step 2 : set up the first Tomcat node

Most of the magic happens in server.xml, which should look something like this:

<?xml version='1.0' encoding='utf-8'?>
<Server port="8006" shutdown="s3cretzhutdown">
  . . .
  <Service name="Catalina">
    . . .
    <Connector address="10.0.0.111" port="80" protocol="HTTP/1.1" redirectPort="8433"
               connectionTimeout="10000" enableLookups="false" maxPostSize="20000" executor="tcThreadPool" />
    <Connector address="10.0.0.111" port="8009" protocol="AJP/1.3" redirectPort="8433"/>
    <Engine name="Catalina" defaultHost="cluster" jvmRoute="node1">
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
               channelSendOptions="6" channelStartOptions="3">
        <Manager className="org.apache.catalina.ha.session.DeltaManager"
                 expireSessionsOnShutdown="false" notifyListenersOnReplication="true" />
        <Channel className="org.apache.catalina.tribes.group.GroupChannel">
          <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                    autoBind="0" selectorTimeout="5000" maxThreads="6"
                    address="10.0.0.111" port="4444"
          />
          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
          </Sender>
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor" />
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor" />
          <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">
            <Member className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1"
                    host="10.0.0.222" port="4444"
            />
            . . .
          </Interceptor>
        </Channel>
        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*.gif;.*.jpg;.*.png;.*.css" />
        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve" />
        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener" />
        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />
      </Cluster>
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase" />
      <Host name="cluster" appBase="webapps" unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
        <Alias>cluster.example.com</Alias>
        . . .
      </Host>
    </Engine>
  </Service>
</Server>

The channelStartOptions value of 3 will disable multicasting. We also don’t use the <Membership> element in this configuration, which is only for multicast environments. Instead, we explicitly list the other members of the cluster as <Member> elements within the StaticMembershipInteceptor. In this tutorial, from the perspective of the .111 node, there is only one other member (.222), so we have only one <Member> element. Add more if you have a bigger cluster. Don’t add the current node, since it makes no sense for a node to be sending replication data to itself.

The <Member> elements do not have a uniqueId attribute set. In the current version of Tomcat, attempting to set a value will cause a warning. The value seems to default to 16 zeros. This does not appear to cause any problem, because presumably the session manager is determining uniqueness based on socket identity. However, a Tomcat node that has just rebooted will be indistinguishable from it’s pre-reboot incarnation, despite the fact that its memory has been flushed, so relying only on the socket to recognize cluster members might not be 100% reliable. Nevertheless, in the absence of any clear documentation of this feature, we’re just going to assume this is normal.

Repeat this formula for the other Tomcat nodes in your cluster.

Step 3 : Configure a distributed Web application with session replication

In the WEB-INF/web.xml file, make sure the application is marked as distributable:

<destributable/>

Also make sure you have a META-INF/context.xml configured to tell Tomcat to replicate sessions:

<Context distributable="true"></Context>

All data placed into a session must be serializable. Keep in mind that Tomcat only knows about session data being changed when you use the session.setAttribute() method, so any indirect manipulation of complex session objects may go unnoticed unless you force the “dirty” side-effect of a redundant operation such as session.setAttribute(“x”,session.getAttribute(“x”)).

Step 4 : Test

Run Apache first. Then run the two Tomcats. You should see something like the following in the Tomcat logs:

**-***-**** **:**:** org.apache.catalina.ha.tcp.SimpleTcpCluster start
INFO: Cluster is about to start
**-***-**** **:**:** org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.0.0.111:4444
**-***-**** **:**:** org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl\
  [tcp://10.0.0.222:4444,10.0.0.222,4444, alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 },\
  payload={}, command={}, domain={}, ]
**-***-**** **:**:** org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor session.xml
**-***-**** **:**:** org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /session to cluster element Engine with name Catalina
**-***-**** **:**:** org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /myapp
**-***-**** **:**:** org.apache.catalina.tribes.io.BufferPool getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl
**-***-**** **:**:** org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/myapp], requesting session state from\
  org.apache.catalina.tribes.membership.MemberImpl[tcp://10.0.0.222:4444,10.0.0.222,4444, \
  alive=0,id={0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 }, payload={}, command={}, domain={}, ]. \
  This operation will timeout if no session state has been received within 60 seconds.
**-***-**** **:**:** org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions

If you have cluster.example.com in your DNS/hosts, you can test the cluster by browsing to cluster.example.com:80/myapp and interacting with your distributable application. If you kill one of the Tomcats, your session will not be affected. If you examine the cookies during such tests you will see the “.node1” or “.node2” at the end of the JSESSIONID cookies, which gives you a clue as to which of the Tomcats is handling your requests.

Tagged with: [Apache | clustering | Tomcat]

Categorised as: Coding

Posted by Rotan on January 6, 2012 at 10:46 pm