16.5. HTTP Services
HTTP session replication is used to replicate the state associated with your web clients on other nodes of a cluster. Thus, in the event one of your node crashes, another node in the cluster will be able to recover. Two distinct functions must be performed:
Session state replication
Load-balance of incoming invocations
State replication is directly handled by JBoss. When you run JBoss in the all configuration, session state replication is enabled by default. Just deploy your web application and its session state is already replicated across all JBoss instances in the cluster.
However, Load-balancing is a different story, it is not handled by JBoss itself and requires additional software. As a very common scenario, we will demonstrate how to setup Apache and mod_jk. This activity could be either performed by specialized hardware switches or routers (Cisco LoadDirector for example) or any other dedicated software though.
Note
A load-balancer tracks the HTTP requests and, depending on the session to which is linked the request, it dispatches the request to the appropriate node. This is called a load-balancer with sticky-sessions: once a session is created on a node, every future request will also be processed by the same node. Using a load-balancer that supports sticky-sessions without replicating the sessions allows you to scale very well without the cost of session state replication: each query will always be handled by the same node. But in the case a node dies, the state of all client sessions hosted by this node are lost (the shopping carts, for example) and the clients will most probably need to login on another node and restart with a new session. In many situations, it is acceptable not to replicate HTTP sessions because all critical state is stored in the database. In other situations, loosing a client session is not acceptable and, in this case, session state replication is the price one has to pay.
Apache is a well-known web server which can be extended by plugging modules. One of these modules, mod_jk (and the newest mod_jk2) has been specifically designed to allow forward requests from Apache to a Servlet container. Furthermore, it is also able to load-balance HTTP calls to a set of Servlet containers while maintaining sticky sessions, and this is what is actually interesting for us.
16.5.1. Download the software
First of all, make sure that you have Apache installed. You can download Apache directly from Apache web site at http://httpd.apache.org/. Its installation is pretty straightforward and requires no specific configuration. As several versions of Apache exist, we advise you to use version 2.0.x. We will consider, for the next sections, that you have installed Apache in the APACHE_HOME directory.
Next, download mod_jk binaries. Several versions of mod_jk exist as well. We strongly advise you to use mod_jk 1.2.x, as both mod_jk and mod_jk2 are deprecated, unsupported and no further developments are going on in the community. The mod_jk 1.2.x binary can be downloaded from http://www.apache.org/dist/jakarta/tomcat-connectors/jk/binaries/. Rename the downloaded file to mod_jk.so and copy it under APACHE_HOME/modules/.
16.5.2. Configure Apache to load mod_jk
Modify APACHE_HOME/conf/httpd.conf and add a single line at the end of the file:
# Include mod_jk’s specific configuration file
Include conf/mod-jk.conf
Next, create a new file named APACHE_HOME/conf/mod-jk.conf:
# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so
# Where to find workers.properties
JkWorkersFile conf/workers.properties
# Where to put jk logs
JkLogFile logs/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel info
# Select the log format
JkLogStampFormat “[%a %b %d %H:%M:%S %Y]”
# JkOptions indicates to send SSK KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
# JkRequestLogFormat
JkRequestLogFormat “%w %V %T”
# Mount your applications
JkMount /application/* loadbalancer
# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# /examples/*=loadbalancer
JkMountFile conf/uriworkermap.properties
# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm
# Add jkstatus for managing runtime data
<Location /jkstatus/>
JkMount status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>
Please note that two settings are very important:
The LoadModule directive must reference the mod_jk library you have downloaded in the previous section. You must indicate the exact same name with the “modules” file path prefix.
The JkMount directive tells Apache which URLs it should forward to the mod_jk module (and, in turn, to the Servlet containers). In the above file, all requests with URL path /application/* are sent to the mod_jk load-balancer. This way, you can configure Apache to server static contents (or PHP contents) directly and only use the loadbalancer for Java applications. If you only use mod_jk as a loadbalancer, you can also forward all URLs (i.e., /*) to mod_jk.
In addition to the JkMount directive, you can also use the JkMountFile directive to specify a mount points configuration file, which contains multiple Tomcat forwarding URL mappings. You just need to create a uriworkermap.properties file in the APACHE_HOME/conf directory. The format of the file is /url=worker_name. To get things started, paste the following example into the file you created:
# Simple worker configuration file
# Mount the Servlet context to the ajp13 worker
/jmx-console=loadbalancer
/jmx-console/*=loadbalancer
/web-console=loadbalancer
/web-console/*=loadbalancer
This will configure mod_jk to forward requests to /jmx-console and /web-console to Tomcat.
You will most probably not change the other settings in mod_jk.conf. They are used to tell mod_jk where to put its logging file, which logging level to use and so on.
16.5.3. Configure worker nodes in mod_jk
Next, you need to configure mod_jk workers file conf/workers.properties. This file specify where are located the different Servlet containers and how calls should be load-balanced across them. The configuration file contains one section for each target servlet container and one global section. For a two nodes setup, the file could look like this:
# Define list of workers that will be used
# for mapping requests
worker.list=loadbalancer,status
# Define Node1
# modify the host as your host IP or DNS name.
worker.node1.port=8009
worker.node1.host=node1.mydomain.com
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.node1.cachesize=10
# Define Node2
# modify the host as your host IP or DNS name.
worker.node2.port=8009
worker.node2.host= node2.mydomain.com
worker.node2.type=ajp13
worker.node2.lbfactor=1
worker.node2.cachesize=10
# Load-balancing behaviour
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1,node2
worker.loadbalancer.sticky_session=1
#worker.list=loadbalancer
# Status worker for managing load balancer
worker.status.type=status
Basically, the above file configures mod_jk to perform weighted round-robin load balancing with sticky sessions between two servlet containers (JBoss Tomcat) node1 and node2 listening on port 8009.
In the works.properties file, each node is defined using the worker.XXX naming convention where XXX represents an arbitrary name you choose for one of the target Servlet container. For each worker, you must give the host name (or IP address) and port number of the AJP13 connector running in the Servlet container.
The lbfactor attribute is the load-balancing factor for this specific worker. It is used to define the priority (or weight) a node should have over other nodes. The higher this number is, the more HTTP requests it will receive. This setting can be used to differentiate servers with different processing power.
The cachesize attribute defines the size of the thread pools associated to the Servlet container (i.e. the number of concurrent requests it will forward to the Servlet container). Make sure this number does not outnumber the number of threads configured on the AJP13 connector of the Servlet container. Please review http://jakarta.apache.org/tomcat/connectors-doc/config/workers.html for comments on cachesize for Apache 1.3.x.
The last part of the conf/workers.properties file defines the loadbalancer worker. The only thing you must change is the worker.loadbalancer.balanced_workers line: it must list all workers previously defined in the same file: load-balancing will happen over these workers.
The sticky_session property specifies the cluster behavior for HTTP sessions. If you specify worker.loadbalancer.sticky_session=0, each request will be load balanced between node1 and node2. But when a user opens a session on one server, it is a good idea to always forward this user’s requests to the same server. This is called a “sticky session”, as the client is always using the same server he reached on his first request. Otherwise the user’s session data would need to be synchronized between both servers (session replication, see Section 16.5.5, “Configure HTTP session state replication”). To enable session stickiness, you need to set worker.loadbalancer.sticky_session to 1.
Note
A non-loadbalanced setup with a single node required the worker.list=node1 entry before mod_jk would function correctly.
16.5.4. Configure JBoss
Finally, we must configure the JBoss Tomcat instances on all clustered nodes so that they can expect requests forwarded from the mod_jk loadbalancer.
On each clustered JBoss node, we have to name the node according to the name specified in workers.properties. For instance, on JBoss instance node1, edit the JBOSS_HOME/server/all/deploy/jbossweb-tomcat50.sar/server.xml file (replace /all with your own server name if necessary). Locate the <Engine> element and add an attribute jvmRoute:
<Engine defaultHost=”localhost” jvmRoute=”node1″>
… …
</Engine>
Then, for each JBoss Tomcat instance in the cluster, we need to tell it to add the jvmRoute value to its session cookies so that mod_jk can route incoming requests. Edit the JBOSS_HOME/server/all/deploy/jbossweb-tomcat50.sar/META-INF/jboss-service.xml file (replace /all with your own server name). Locate the <attribute> element with a name of UseJK, and set its value to true:
<attribute >true</attribute>
At this point, you have a fully working Apache+mod_jk load-balancer setup that will balance call to the Servlet containers of your cluster while taking care of session stickiness (clients will always use the same Servlet container).
Note
For more updated information on using mod_jk 1.2 with JBoss Tomcat, please refer to the JBoss wiki page at http://wiki.jboss.org/wiki/Wiki.jsp?page=UsingMod_jk1.2WithJBoss.
16.5.5. Configure HTTP session state replication
In Section 16.5.3, “Configure worker nodes in mod_jk”, we covered how to use sticky sessions to make sure that a client in a session always hits the same server node in order to maintain the session state. However, that is not an ideal solution. The load might be unevenly distributed over the nodes over time and if a node goes down, all its session data is lost. A better and more reliable solution is to replicate session data across all nodes in the cluster. This way, the client can hit any server node and obtain the same session states.
The jboss.cache:service=TomcatClusteringCache MBean makes use of JBoss Cache to provide HTTP session replication service to the HTTP load balancer in a JBoss Tomcat cluster. This MBean is defined in the deploy/tc5-cluster-service.xml file.
Note
If you are not running from the all configuration, session replication can be enabled by copying deploy/tc5-cluster-service.xml, lib/jgroups.jar and lib/jboss-cache.jar from server/all into your server configuration directory.
Below is a typical tc5-cluster-service.xml file. The configuration attributes in the TomcatClusteringCache MBean is very similar to those in Section 17.2, “JBossCache Configuration”.
<mbean code=”org.jboss.cache.TreeCache”
>
<depends>jboss:service=Naming</depends>
<depends>jboss:service=TransactionManager</depends>
<attribute >
org.jboss.cache.JBossTransactionManagerLookup
</attribute>
<attribute >REPEATABLE_READ</attribute>
<attribute >REPL_ASYNC</attribute>
<attribute >Tomcat-Cluster</attribute>
<attribute >
… …
</attribute>
<attribute >15000</attribute>
</mbean>
The detailed configuration for the TreeCache MBean is covered in Section 17.2, “JBossCache Configuration”. Below, we will just example several attributes that are most relevant to the HTTP cluster session replication.
TransactionManagerLookupClass sets the transaction manager factory. The default value is org.jboss.cache.JBossTransactionManagerLookup. It expects to find the transaction manager under java:/TransactionManager.
IsolationLevel sets the isolation level for updates to the transactional distributed cache. The valid values are SERIALIZABLE, REPEATABLE_READ, READ_COMMITTED, READ_UNCOMMITTED, and NONE. These isolation levels mean the same thing as isolation levels on the database. The default isolation of REPEATABLE_READ makes sense for most web applications.
CacheMode controls how the cache is replicated. The valid values are REPL_SYNC and REPL_ASYNC, which determine whether changes are made synchronously or asynchronously. Using synchronous replication makes sure changes propagated to the cluster before the web request completes. However, synchronous replication is much slower. For asyncrhonous access, you will want to enable and tune the replication queue.
ClusterName specifies the name of the cluster that the cache works within. The default cluster name is Tomcat-Cluster. All the nodes should use the same cluster name. Although session replication can share the same channel (multicast address and port) with other clustered services in JBoss, replication should have it’s own cluster name.
ClusterConfig configures the underlying JGroups stack. The most import configuration elements are the muliticast adress and port, mcast_addr and mcast_port respectively, to use for clustered communication. These values should make sense for your network. Please refer to Section 17.1, “JGroups Configuration” for more information.
LockAcquisitionTimeout sets the maximum number of milliseconds to wait for a lock acquisition. The default value is 15000.
UseReplQueue determines whether to enable the replication queue when using asynchronous replication. This allows multiple cache updates to be bundled together to improve performance. The replication queue properties are controlled by the ReplQueueInterval and ReplQueueMaxElements properties.
ReplQueueInterval specifies the time in milliseconds JBoss Cache will wait before sending items in the replication queue.
ReplQueueMaxElements: specifies the maximum number of elements allowed in the replication queue before JBoss Cache will send an update.
16.5.6. Enabling session replication in your application
To enable clustering of your web application you must it as distributable in the web.xml descriptor. Here’s an example:
<?xml version=”1.0″?>
<web-app xmlns=”http://java.sun.com/xml/ns/j2ee“
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xsi:schemaLocation=”http://java.sun.com/xml/ns/j2ee
http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd”
version=”2.4″>
<distributable/>
<!– … –>
</web-app>
You can futher configure session replication using the replication-config element in the jboss-web.xml file. Here is an example:
<jboss-web>
<replication-config>
<replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger>
<replication-granularity>SESSION</replication-granularity>
</replication-config>
</jboss-web>
The replication-trigger element determines what triggers a session replication (or when is a session is considered dirty). It has 4 options:
SET: With this policy, the session is considered dirty only when an attribute is set in the session. If your application always writes changed value back into the session, this option will be most optimized in term of performance. If an object is retrieved from the session and modified without being written back into the session, the change to that object will not be replicated.
SET_AND_GET: With this policy, any attribute that is get or set will be marked as dirty. If an object is retrieved from the session and modified without being written back into the session, the change to that object will be replicated. This option can have significant performance implications.
SET_AND_NON_PRIMITIVE_GET: This policy is similar to the SET_AND_GET policy except that only non-primitive get operations are considered dirty. For example, the http session request may retrieve a non-primitive object instance from the attribute and then modify the instance. If we don’t specify that non-primitive get is considered dirty, then the modification will not be replication properly. This is the default value.
ACCESS: This option causes the session to be marked as dirty whenever it is accessed. Since a the session is accessed during each HTTP request, it will be replicated with each request. The access time stamp in the session instance will be updated as well. Since the time stamp may not be updated in other clustering nodes because of no replication, the session in other nodes may expire before the active node if the HTTP request does not retrieve or modify any session attributes. When this option is set, the session timestamps will be synchronized throughout the cluster nodes. Note that use of this option can have a significant performance impact, so use it with caution.
The replication-granularity element controls the size of the replication units. The supported values are:
session: Replication is per session instance. As long as it is considered modified when the snapshot manager is called, the whole session object will be serialized.
attribute: Replication is only for the dirty attributes plus some session data, like, lastAccessTime. For session that carries large amount of data, this option can increase replication performance.
If your sessions are generally small, session is the better policy. If your session is larger and some partsare infrequently accessed, attribute replication will be more effective.
16.5.7. Monitoring session replication
If you have deployed and accessed your application, go to the jboss.cache:service=TomcatClusteringCache MBean and invoke the printDetails operation. You should see output resembling the following.
/JSESSION
/quote
/FB04767C454BAB3B2E462A27CB571330
VERSION: 6
FB04767C454BAB3B2E462A27CB571330: org.jboss.invocation.MarshalledValue@1f13a81c
/AxCI8Ovt5VQTfNyYy9Bomw**
VERSION: 4
AxCI8Ovt5VQTfNyYy9Bomw**: org.jboss.invocation.MarshalledValue@e076e4c8
This output shows two separate web sessions, in one application named quote, that are being shared via JBossCache. This example uses a replication-granularity of session. Had attribute level replication been used, there would be additional entries showing each replicated session attribute. In either case, the replicated values are stored in an opaque MarshelledValue container. There aren’t currently any tools that allow you to inspect the contents of the replicated session values. If you don’t see any output, either the application was not correctly marked as distributable or you haven’t accessed a part of application that places values in the HTTP session. The org.jboss.cache and org.jboss.web logging categories provide additional insight into session replication useful for debugging purposes.
16.5.8. Using Single Sign On
JBoss supports clustered single sign-on, allowing a user to authenticate to one application on a JBoss server and to be recognized on all applications, on that same machine or on another node in the cluster, that are deployed on the same virtual host. Authentication replication is handled by the HTTP session replication service. Although session replication does not need to be explicitly enabled for the applications in question, the tc5-cluster-service.xml file does need to be deployed.
To enable single sign-on, you must add the ClusteredSingleSignOn valve to the appropriate Host elements of the tomcat server.xml file. The valve configuration is shown here:
<Valve className=”org.jboss.web.tomcat.tc5.sso.ClusteredSingleSignOn” />
Recent Comments