Setting up Clustering on Apache/Jboss/Tomcat using Jakarta mod_jk

Clustering Jboss for two reasons  load balancing and failover.

Fail-over is probably the most important issue for web applications. If the front end load balancer detects that one of the nodes has gone down it will redirect all the traffic to the second instance and your clients, apart from any on the failed node, won’t even notice. Well that’s the theory – if the site is under high load you may find that the surviving Tomcat cannot cope with the increased load and fails in short order. You need to size and correctly configure your hardware and software.

You can also manually fail-over a system to upgrade the hardware, JVM or other application without end users being aware. If your management doesn’t tolerate any downtime this is an extremely important feature. Your JVM’s can be installed on separate or the same physical host. The latter leaves you exposed to hardware problems.

There are two easy ways to increase application performance without tuning the application software itself. Buy faster hardware or add more hardware. If you add a second server running Jboss you can use load balancing to distribute the work between the two servers. Requests come in to the Apache front end, static content is served directly by Apache and any dynamic requests forwarded to the Tomcats based on some algorithm. The Apache Jakarta mod_jk connector is the most popular way of doing this.

In this exercise we will configure two physical computers each running one Apache web server and two tomcats for both failover and load balancing. The example ignores any back-end data storage considerations but will discuss load balancing between the two computers.

The simplest technique for load balancing between the two Apache front end servers is the DNS Round Robin strategy. This performs load balancing by returning a different server IP address each time a (DNS) name server is queried by a client. It is usually found as front-end service on large clusters of servers to provide a server address that is geographically close to the client. Normally an intelligent name server such as lbnamed is used. This will poll the web servers in the list to check if they are online and not overloaded.

DNS Load balancing has some significant drawbacks. Addresses may be cached by clients and local DNS resolvers and it can be hard to accurately manage load. The system simply shuffles the order of the address records each time the name server is queried. No account is taken of real server load or network congestion.

The advantage of using a load balancer compared to using round robin DNS is that it takes care of the load on the web server nodes and will direct requests to the node with the least load. It can also remove failed nodes from the list. Some round robin DNS software can do this, the problem is they have little control about how client’s cache IP addresses.

Many web applications (e.g. forum software, shopping carts, etc.) make use of sessions. If you are in a session on Apache node 1, you would lose that session if suddenly node 2 served your requests. This assumes that the web servers don’t share their session data, say via a database.

For DNS load balancing this is not a problem, the IP address will resolve to a single server. However a front end load balancer must be aware of sessions and always redirect requests from the same user to the same backend server.

There are a number of packages to support front end load balancing. For example RedHat Piranha or Ultramonkey. There are also hardware/firmware solutions from companies such as Cisco.

Apache httpd

MaxClients

MaxClients ? (RAM - size_all_other_processes)/(size_apache_process(es))

Apache.MaxClients = 0.8 * (Tomcat-1.maxThreads + … Tomcat-n.maxThreads)

400 = 0.8 * (250 + 250)

MaxRequestsPerChild

Redirection requests to the mod_jk load balancer

We will need to load the mod_jk module (a shared library) into Apache when it starts and tell the module where to find the workers.properties file and where to write logs (useful for debugging/crashes).
/etc/httpd/conf/httpd.conf

1. httpd.conf:
———————————–
# Include mod_jk’s specific configuration file
Include conf/mod-jk.conf

2. mod-jk.conf:
———————————
# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so
# Where to find workers.properties
JkWorkersFile conf/workers.properties
# Where to put jk logs
JkLogFile logs/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel info
# Select the log format
JkLogStampFormat “[%a %b %d %H:%M:%S %Y]”
# JkOptions indicates to send SSK KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
# JkRequestLogFormat
JkRequestLogFormat “%w %V %T”
# Mount your applications
JkMount /myapp/* loadbalancer
JkMount /jspellhtml2k4/* loadbalancer
JkMount /jspelliframe2k4/* loadbalancer
JkMount /jspellhtml2k4 loadbalancer
JkMount /jspelliframe2k4 loadbalancer

# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# /examples/*=loadbalancer
JkMountFile conf/uriworkermap.properties

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
# Note: Replaced JkShmFile logs/jk.shm due to SELinux issues. Refer to
# https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=225452
JkShmFile run/jk.shm

# Add jkstatus for managing runtime data
<Location /jkstatus>
JkMount status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>

2. uriworkermap.properties
————————————————-
# Simple worker configuration file
# Mount the Servlet context to the ajp13 worker
/jmx-console=loadbalancer
/jmx-console/*=loadbalancer
/web-console=loadbalancer
/web-console/*=loadbalancer
/myapp/*=loadbalancer
/myapp=loadbalancer
/jspellhtml2k4/*=loadbalancer
/jspellhtml2k4=loadbalancer
/jspelliframe2k4/*=loadbalancer
/jspelliframe2k4=loadbalancer

The following is a suggested configuration for two physical servers with a pair Tomcat instances running on each. The rational is that the front end load balancer will distribute load between the two physical hosts. The f/e load balancer is itself configured for fail-over using a virtual IP address and a heartbeat type operation to determine if the balancer is still operational.

Each physical server has an Apache web server and two tomcat servlet engines. Apache will serve static content either from an NFS mounted disk or from a replicated file system using something like Web. Apache will load balance between the two local  jboss but if these become overloaded or crash it can forward requests on to the other physical server.

The Apache mod_jk load-balancer can distribute work using one of three methods:

• B (busyness): choose the worker with the lowest number of requests currently in progress
• R (requests): choose the worker that has processed the lowest number of requests overall
• T (traffic): choose the worker that transmitted the lowest number of bytes

The sticky_session is set to true to maintain a session between the client and Tomcat.

worker.list=loadbalancer

worker.loadbalancer.type=lb
worker.loadbalancer.method=B
worker.loadbalancer.balance_workers=worker1,worker2,worker3,worker4
worker.loadbalancer.sticky_session=true

This is common configuration for all workers. Note if Prefork MPM is used you will want to set the cachesize property. This should reflect the estimated average concurrent number of users for the Tomcat instance. Cache_timeout should be set to close cached sockets. This reduces the number of idle threads on the Tomcat server. Note that connection_pool_size and connection_pool_timeout should be used with mod_jk 1.2.16 and later.

If there is a firewall between Apache and Jboss t the socket_keepalive property should be used. Socket_timeout will tell Apache to close an ajp13 connection after some inactivity time. This also reduces idle threads on Tomcat.

# common config
worker.basic.type=ajp13
worker.basic.socket_timeout=300
worker.basic.lbfactor=5

3. workers.properties
————————————————-
# The configuration directives are valid
# for the mod_jk version 1.2.18 and later
#
worker.list=loadbalancer,status

# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=myapp-myapp.myapp.com
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.node1.connection_pool_size=10
worker.node1.cachesize=10

# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8009
worker.node2.host=myapp-myapp1.myapp.com
worker.node2.type=ajp13
worker.node2.lbfactor=1
worker.node2.connection_pool_size=10
worker.node2.cachesize=10

# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=1

# Status worker for managing load balancer
worker.status.type=status

2.    Make Sure all below given settings are available in JBoss configurations in all nodes of  Jboss cluster.

JBoss Server Side Configuration Changes:
———————————————–

1. Remove/comment out below two ports from ja.properties file.
———————————————————————-
vi /opt/jboss/jboss-5.0/jboss-as/server/template_cluster/conf/ja.properties file.
Search for below two lines
OAPort=3528
OASSLPort=3529
Commentout both lines like below and save the file.
#OAPort=3528
#OASSLPort=3529

2. Add jvmRoute
——————
vi /opt/jboss/jboss-5.06/jboss-as/server/template_cluster/deploy/jboss-web.deployer/server.xml

Search For Below Text:
<Engine name=”jboss.web” defaultHost=”localhost”>
Change it to:
<Engine name=”jboss.web” defaultHost=”localhost” jvmRoute=”node1″>

Note: This configuration needs to be done in all the nodes and all the configuration files need to be changed accordingly.
Check the workers.properties file to know which host is node1 or nodex
Example: For node2 the jvmRoute will be node2(HostName can be found out from workers.properties file)
<Engine name=”jboss.web” defaultHost=”localhost” jvmRoute=”node2″>

3. Change UseJK to true
————————-
vi /opt/jboss/jboss-5.0/jboss-as/server/template_cluster/deploy/jboss-web.deployer/META-INF/jboss-service.xml

Search For Below Text:
<attribute name=”UseJK”>false</attribute>
Change it to:
<attribute name=”UseJK”>true</attribute>

Note: This configuration needs to be done in all the nodes and all the configuration files need to be changed accordingly.

4. Make Sure Below properties are present in jboss-service.xml config file.
——————————————————————————
vi /opt/jboss/jboss-5.0/jboss-as/server/template_cluster/deploy/jboss-web-cluster.sar/META-INF/jboss-service.xml

<mbean code=”org.jboss.cache.aop.TreeCacheAop” name=”jboss.cache:service=TomcatClusteringCache”>
<depends>jboss:service=Naming</depends>
<depends>jboss:service=TransactionManager</depends>
<depends>jboss.aop:service=AspectDeployer</depends>
<attribute name=”TransactionManagerLookupClass”>org.jboss.cache.BatchModeTransactionManagerLookup</attribute>
<attribute name=”IsolationLevel”>REPEATABLE_READ</attribute>
<attribute name=”CacheMode”>REPL_ASYNC</attribute>
<attribute name=”ClusterName”>Tomcat-${jboss.partition.name:Cluster}</attribute>
<attribute name=”UseMarshalling”>false</attribute>
<attribute name=”InactiveOnStartup”>false</attribute>
<attribute name=”ClusterConfig”>
… …
</attribute>
<attribute name=”LockAcquisitionTimeout”>15000</attribute>
</mbean>

The sticky_session is set to true to maintain a session between the client and Tomcat.

worker.list=loadbalancer

worker.loadbalancer.type=lb
worker.loadbalancer.method=B
worker.loadbalancer.balance_workers=worker1,worker2,worker3,worker4
worker.loadbalancer.sticky_session=true

This is common configuration for all workers. Note if Prefork MPM is used you will want to set the cachesize property. This should reflect the estimated average concurrent number of users for the Tomcat instance. Cache_timeout should be set to close cached sockets. This reduces the number of idle threads on the Tomcat server. Note that connection_pool_size and connection_pool_timeout should be used with mod_jk 1.2.16 and later.

If there is a firewall between Apache and Tomcat the socket_keepalive property should be used. Socket_timeout will tell Apache to close an ajp13 connection after some inactivity time. This also reduces idle threads on Tomcat.

# common config
worker.basic.type=ajp13
worker.basic.socket_timeout=300
worker.basic.lbfactor=5

These are the two workers. Notice the use of the local_worker property. The load-balancer will always favour local_workers with a value set to 1.

worker.worker1.host=server1
worker.worker1.port=8009
worker.worker1.local_worker=1
worker.worker1.reference=worker.basic

worker.worker2.host=server1
worker.worker2.port=8010
worker.worker2.local_worker=1
worker.worker2.reference=worker.basic

worker.worker3.host=server2
worker.worker3.port=8009
worker.worker3.local_worker=0
worker.worker3.reference=worker.basic

worker.worker4.host=server2
worker.worker4.port=8010
worker.worker4.local_worker=0
worker.worker4.reference=worker.basic

Tomcat Configuration

<Connector port="8899"  minSpareThreads="5" maxThreads="10" enableLookups="false" redirectPort="8443" acceptCount="10" debug="0" connectionTimeout="60000"/>

<Connector port="8009" enableLookups="false" redirectPort="8443" 
protocol="AJP/1.3" maxThreads="400" minSpareThreads="50"  maxSpareThreads="300" />

<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">