Multi-Site Dynamic Apache Configurations

In a continuing effort to make my hosting cloud more dynamic, I have had to become relatively creative with my Apache configurations so that my farm can remain relatively flexible with little or no reconfiguration on newly introduced servers. This presents a big problem when working with a multi-site Apache configuration, and an even bigger problem when SSL certificates are involved. As many of you probably already know, you need to have one network interface (virtual or physical) dedicated to a particular domain SSL certificate. It’s easy enough to grab a wildcard SSL certificate from GoDaddy and bind a single network interface to all subdomains, but when you have a farm configuration where you’re hosting multiple domains from the same web servers, then you’ll need to have a dedicated NIC for each wildcard domain SSL certificate. That’s ok, we can handle this by creating virtual NICs (eth0:1) and giving them their own IP address, and Apache is none the wiser. But, this is where our Apache configuration starts to become a problem. With Apache, we know that we can listen by IP address, but not by interface, so in order for us to keep a common configuration between all of the servers in our farm, we’ll need to figure out a way for it to be able to dynamically handle binding when the IP address is not statically defined.

If you’re not fully understanding the issue right now, that’s ok. Let me outline the environment consideration a little bit better with a pseudo-diagram:

                Load Balancer -
                VIP1 = 10.1.1.2 (www.example1.com)
                                              - www01.example1.com
                                              - www02.example1.com
                                              - www03.example1.com
                VIP2 = 10.1.1.3 (www.example2.com)
                                              - www01.example2.com
                                              - www02.example2.com
                                              - www03.example2.com

                SERVER1:
                www01.example1.com:                      www01.example2.com:
                \                                                   /
                eth0 = 10.1.1.10                             eth0:1 = 10.1.1.11

                SERVER2:
                www02.example1.com:                      www02.example2.com:
                \                                                   /
                eth0 = 10.1.1.20                              eth0:1 = 10.1.1.21

                SERVER3:
                www03.example1.com:                      www03.example2.com:
                \                                                   /
                eth0 = 10.1.1.30                              eth0:1 = 10.1.1.31

This is a relatively common and efficient setup, and in fact may be extrapolated by many more sites and domains hosted from the same server farms. Our Apache configuration is relatively simple to setup by telling the VirtualHost sections what IP to listen on, as such (www01 example):

Listen 10.1.1.10:80
<VirtualHost 10.1.1.10:80>
        DocumentRoot /var/www/example1_root
        CustomLog logs/www.example1.com-access_log combined
        ErrorLog logs/www.example1.com-error_log
</VirtualHost>

Listen 10.1.1.11:80
<VirtualHost 10.1.1.11:80>
        DocumentRoot /var/www/example2_root
        CustomLog logs/www.example2.com-access_log combined
        ErrorLog logs/www.example2.com-error_log
</VirtualHost>

The problem with this setup is that as we deploy a new box into the farm, we’ll need to reconfigure Apache to listen on the new IP addresses that it gets for it’s eth0 and eth0:1 … That’s ok, and really can be done with relative ease, but what if we want to have it so that we can deploy a new box (say, from VMware template?) with a preloaded configuration, and have it just work (mmm, Linked Clones anyone? )? For this case, we’ll need to completely redesign the way that our Apache configuration is structured so that we don’t need to do any configuration after a box is deployed, and so that we can have a common configuration between www01,www02,www03, etc… This is nice for management too because we’ll always know that each farm server has the same configuration, and we’ll never have to worry about typos or anything like that. Sounds nice, so I’ll stop the suspense.

Now, Apache gives us the nice and native ability to import environment variables into our configuration, which means that we can figure out everything that we need for our configuration prior to it being loaded, and pass those values on. This means, through our advanced (eh?) bash scripting skills, that we can figure out what the IP addresses of our interfaces are, populate an environment variable, and pass that knowledge on to Apache. We’ll start with /etc/sysconfig/httpd. From the httpd startup script, /etc/sysconfig/httpd gets inherited prior to loading the Apache configuration, so we can declare in here any variables that we will need and they will be retained within Apache’s execution scope.

The bare metal…

/etc/sysconfig/httpd:

# Configuration file for the httpd service.

#
# The default processing model (MPM) is the process-based
# 'prefork' model.  A thread-based model, 'worker', is also
# available, but does not work with some modules (such as PHP).
# The service must be stopped before changing this variable.
#
#HTTPD=/usr/sbin/httpd.worker

#
# To pass additional options (for instance, -D definitions) to the
# httpd binary at startup, set OPTIONS here.
#
#OPTIONS=

#
# By default, the httpd process is started in the C locale; to
# change the locale in which the server runs, the HTTPD_LANG
# variable can be set.
#
#HTTPD_LANG=C

IPADDR_ETH0=$(ifconfig eth0 | grep inet | grep -v inet6 | awk '{print $2}' | sed -e s/addr://g)
IPADDR_ETH01=$(ifconfig eth0:1 | grep inet | grep -v inet6 | awk '{print $2}' | sed -e s/addr://g)

export IPADDR_ETH0 IPADDR_ETH01

/etc/httpd/conf.d/vhosts.conf:

 # Pass in our environment variables where we've defined the IP addresses
 # for each of our NICs
 PassEnv IPADDR_ETH0
 PassEnv IPADDR_ETH01

 # Listen on 80 and 443 on our www0x.example1.com NIC
 NameVirtualHost ${IPADDR_ETH0}:80
 NameVirtualHost ${IPADDR_ETH0}:443

 # Listen on 80 and 443 on our www0x.example2.com NIC
 NameVirtualHost ${IPADDR_ETH01}:80
 NameVirtualHost ${IPADDR_ETH01}:443

 # Redirect all of our hosts to HTTPs -- Regular HTTP is soooo lame 

 <VirtualHost ${IPADDR_ETH0}:80>
     RewriteEngine on
     RewriteRule ^/(.*)$ https://%{HTTP_HOST}/$1 [L,R]
 </VirtualHost>

 <VirtualHost ${IPADDR_ETH01}:80>
     RewriteEngine on
     RewriteRule ^/(.*)$ https://%{HTTP_HOST}/$1 [L,R]
 </VirtualHost>

 # www.example1.com SSL example
 <VirtualHost ${IPADDR_ETH0}:443>
     DocumentRoot /var/www/example1
     CustomLog logs/${HTTP_HOST}-access_log combined
     ErrorLog logs/${HTTP_HOST}-error_log
     TransferLog logs/${HTTP_HOST}-access_log
     SSLEngine on
     SSLCertificateFile /etc/httpd/conf.d/certs/_.example1.com.crt
     SSLCertificateKeyFile /etc/httpd/conf.d/certs/example1.com.key
     SSLCertificateChainFile /etc/httpd/conf.d/certs/gd_bundle.crt
 </VirtualHost>

 # www.example2.com SSL example
 <VirtualHost ${IPADDR_ETH01}:443>
     DocumentRoot /var/www/example2
     CustomLog logs/${HTTP_HOST}-access_log combined
     ErrorLog logs/${HTTP_HOST}-error_log
     TransferLog logs/${HTTP_HOST}-access_log
     SSLEngine on
     SSLCertificateFile /etc/httpd/conf.d/certs/_.example2.com.crt
     SSLCertificateKeyFile /etc/httpd/conf.d/certs/example2.com.key
     SSLCertificateChainFile /etc/httpd/conf.d/certs/gd_bundle.crt
 </VirtualHost>

And in doing it this way, we can dynamically create Apache configurations for each new box that we put into the farm. Using this method, we’ll be able to expand or reduce the size of the farm with little impact or effort.

As always, feel free to contact me with any questions, comments, or corrections