load balanced and redundant solution for a squid proxy server. My primary goals were simplicity and redundancy. I started with a stand-alone squid proxy server (CentOS 6, squid, NTLM authentication). Alone, this works great for an AD environment; single sign-on authentication so no password prompt, integrates with AD groups for various access levels, auto-proxy config using DHCP and DNS, etc. (anyway, I should post this config later).
First step was building an identical squid server (I just cloned the vm, changed the name and IP). I then set up a unison cron job to sync the squid configs, so any change on one would propogate to the other:
* * * * * /usr/bin/unison /etc/squid ssh://proxy2//etc/squid -batch >> /dev/null 2>&1
Once I had two functional proxy servers, my first attempt to add redundancy was relying on most browsers ability to switch to a second proxy server if the first goes down. I used an auto-configuration script to push out these settings. Also note I incorporated IP-based load balancing as well. In the proxy.pac:
function FindProxyForURL(url, host) {
var proxy1="proxy1:8080"
var proxy2="proxy2:8080"
var myip=myIpAddress()
var ipbits=myip.split(".")
var myseg=parseInt(ipbits[3])
if(myseg==Math.floor(myseg/2)*2) {
var proxone=proxy1
var proxtwo=proxy2
}
else {
var proxone=proxy2
var proxtwo=proxy1
}
return "PROXY "+proxone+"; PROXY "+proxtwo+";";
}
This works, but the problem I discovered is some browsers can take a long time to determine if a proxy is down – about 15 seconds. And some browsers check for every host. Most website these days have links, images, ads to several different hosts, so just loading a singe home page can literally take minutes if one of the proxy servers is down. And, if you close your browser and re-open it, it has to check all over again.
Then I thought I would try a real load-balancing solution. I found Piranha which is Red-Hat’s load-balancing solution with optional redundancy. It’s very similiar to Microsft’s NLB for those familiar with it, but my biggest complaint about it is it typically is designed to run on separate servers than your proxy servers (or whatever IP service you are load balancing). That means adding two more servers to the mix. I wanted simple, and turning two servers into four isn’t my kind of simple. So, why not try to get Piranha running on the proxy servers themselves? Here’s my layout:
proxy1: 10.120.100.60 proxy2: 10.120.100.61 Squid port: 8080 Available IP address to be used as the virtual IP: 10.120.100.62
I installed Piranha on each server:
# yum install piranha
and set the Piranha password:
# piranha-passwd
I had to enable iptables; I had turned it off initially. I turned it on and wiped the config – of course, don’t wipe yours if you are using it for other stuff! (more on why we need iptables later):
# chkconfig iptables on # service iptables start # iptables -F # iptables -t nat -F # iptables -t mangle -F # iptables -X # service iptables save
Enable IP forwarding (not sure this is required for our all-in-one design, but it’s standard for Piranha configs anyway).
# sysctl -w net.ipv4.ip_forward=1
To make this peristent across reboots, edit that line in /etc/sysctl.conf
Turned on the web-based Piranha config on the first server:
# service piranha-gui start
The Piranha GUI runs a web server on port 3636. Browse to it and login. Username is “piranha” and the password that you set earlier:
On the GLOBAL SETTINGS page, set the Primary server public IP to first server’s IP address. Network type should be Direct Routing. Private IP should be blank.
On the REDUNDANCY page, plug in the IP address of your second server
Add a virtual server. Use an available unused IP address on the same subnet as your Piranha servers.
Add two Real Servers, which in this case are the same as your Piranha servers. Use their IP addresses, and you can leave the port blank, since that was defined on the virtual server.
In my case, using squid, the default Monitoring Scripts work since a HTTP GET provides a response.
Now that the GUI config is done, copy the config from server 1 to server 2:
# scp /etc/sysconfig/ha/lvs.cf proxy2:/etc/sysconfig/ha/lvs.cf
The config need to be identical on both servers. For reference, here’s what mine looks like:
serial_no = 26
primary = 10.120.100.60
service = lvs
backup_active = 1
backup = 10.120.100.61
heartbeat = 1
heartbeat_port = 539
keepalive = 3
deadtime = 6
network = direct
debug_level = NONE
monitor_links = 1
syncdaemon = 0
virtual proxy {
active = 1
address = 10.120.100.62 eth0:1
vip_nmask = 255.255.255.0
port = 8080
persistent = 60
send = "GET / HTTP/1.0\r\n\r\n"
expect = "HTTP"
use_regex = 0
load_monitor = none
scheduler = wlc
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 0
server proxy1 {
address = 10.120.100.60
active = 1
weight = 1
}
server proxy2 {
address = 10.120.100.61
active = 1
weight = 1
}
}
Now your ready to start. Lets start up the primary server first. One server 1:
# chkconfig pulse on # service pulse start
Watch /var/log/messages for the following:
Feb 3 10:33:10 proxy1 pulse[2850]: STARTING PULSE AS MASTER Feb 3 10:33:13 proxy1 pulse[2850]: partner dead: activating lvs Feb 3 10:33:13 proxy1 lvs[2853]: starting virtual service Proxy active: 8080 Feb 3 10:33:13 proxy1 lvs[2853]: create_monitor for Proxy/proxy1 running as pid 2861 Feb 3 10:33:13 proxy1 lvs[2853]: create_monitor for Proxy/proxy2 running as pid 2862 Feb 3 10:33:13 proxy1 nanny[2862]: starting LVS client monitor for 10.120.100.62:8080 -> 10.120.100.61:8080 Feb 3 10:33:13 proxy1 nanny[2861]: starting LVS client monitor for 10.120.100.62:8080 -> 10.120.100.60:8080 Feb 3 10:33:14 proxy1 nanny[2861]: [ active ] making 10.120.100.60:8080 available Feb 3 10:33:14 proxy1 nanny[2862]: [ active ] making 10.120.100.61:8080 available Feb 3 10:33:14 proxy1 ntpd[1528]: Listening on interface #7 eth0:1, 10.120.100.62#123 Enabled Feb 3 10:33:18 proxy1 pulse[2855]: gratuitous lvs arps finished
ifconfig should show the virtual IP address bound to eth0:1
eth0:1 Link encap:Ethernet HWaddr 00:0C:29:C3:50:80
inet addr:10.120.100.62 Bcast:10.120.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Start pulse on the backup server:
# chkconfig pulse on # service pulse start
all you should see in /var/log/messages is:
Feb 3 10:35:45 proxy2 pulse[2820]: STARTING PULSE AS BACKUP
If you tried to use your new Virtual IP address at this point, it won’t work. We need to do some iptables work. On each server:
# iptables -t nat -A PREROUTING -p tcp -d 10.120.100.62 --dport 8080 -j REDIRECT # service iptables save
That’s it. Now test everything, make sure your clients still work while you stop/start pulse, stop/start squid, reboot, etc. Worst case, clients may be unable to use the service for a few seconds. If you ever make changes, be sure and copy the lvs.cf to the other server and then reload pulse. Unfortunately, the GUI does not provide a method to copy the configs and reload the services.
Notes:
I was using VMware 4.1 as the host servers. Originally, I could not get the Redundancy heartbeat monitor to work – both servers could not see each other, so they both became active, assigning themselves the same IP address. Turns out, I had to do two things to the VM guests:
- Use the E1000 Network card instead of the VMXNET.
- add “pci=nomsi” to the kernel line in /boot/grub/grub.conf (maybe – not sure if this did anything)
You may notice that even after you stop pulse on both servers, your clients can still connect? Why? iptables is still listening to that virtual IP address, even if it’s not bound to an adapter. And as long as your clients and/or switch has the mac-address cached, traffic will still be sent to the last known port with that IP.
Recent Comments