Cluster load balancing runs on software from the Linux Virtual Server (LVS) project. The master server has an extra IP address referred to as the Virtual IP; this address is where all incoming requests for cluster services should be sent, and is the address which can fail over to a Backup Master in high availability setups. LVS is used to distribute raw TCP port connections like http or smtp requests across an entire cluster and can be customized to distribute connections on any port.

While every node still has its own IP address and is reachable via SSH, the Virtual IP exists only on the master. When a request is received, the master will check its load balancing tables to see which nodes are currently registered to handle service requests. Based on a certain scheduling algorithm, the master will decide which node it will forward the current request to. The load balancing configuration can be viewed from the web interface or the command line.

1. View configuration on the web interface

Here is a screenshot showing the LVS Load Balancing page in the web interface:

All basic information can easily be seen in one place. The Virtual IP, scheduling algorithm, which ports are registered and which nodes are assigned to them, etc. No changes can be made here, but the entire configuration can be reviewed at a glance.

Load balancing can only be reconfigured at the command line. To view the current status, type the following at a bash prompt on the master server:

ipvsadm -Ln 

This will show registered ports, active nodes, the Virtual IP address, and the scheduling algorithm. To make changes, there are several important steps to consider.

To illustrate the steps involved, we'll walk through adding a sample application to the cluster and setting it up for load balancing across many nodes. Note that the cluster comes pre-configured to load balance web applications. Anything that can be deployed as a .WAR file to the integrated Terracotta / Tomcat installation will be load balanced and distributed automatically to all nodes listed in /cluster/http_nodes.

First, create a file that hold the names of the nodes that should run this application when they come online. For our example, let's create ”/cluster/myapp_nodes”.

vi /cluster/myapp_nodes

Type in the name of a node on the first line, and the name of another node on the second line. If you only have one node, one node name is fine. Just remember to always put one node name per line. Now save and close the file.

Next we'll modify /cluster/bin/lvs-init. This file tells LVS which ports it should listen on. If you have an application that takes requests on port 5000 and you want to load balance this application across the cluster, you will first need to add a line to lvs-init, like this:

 /sbin/ipvsadm -A -t $VIP:80 -p 300 -s $SCHED

In the line above, port 80 is registered with LVS. Substitute the desired port for “80” when adding the line. The ”-p 300” is optional, and tells LVS to use persistent connection handling for this service with a timeout of 300 seconds. Persistence means that repeated connections from a particular client will be sent to the same back-end server, and that after 300 seconds with no connections, the connection will be forgotten.

Next you should make a copy of the file /cluster/mon/mon.d/cluster_http.monitor. The current contents of this file are shown below:

#
##################################################
# Copyright (c) 2008, RapidScale Clusters LLC    #
#                                                #
##################################################
# cluster_http.monitor
#
# Monitor online nodes for functional
# http services
# Exit 1 if any online node fails

 . /cluster/mon/mon_common

if [ ! -d /tmp/http_failed ]
  then
        mkdir -p /tmp/http_failed
fi

rm -rf /tmp/http_failed/*

for i in `/bin/cat /cluster/http_nodes`
  do
        ## If node is online, test it. Throw error if any failures.
        if test "`cat $BASEDIR/$i`" = "1"
          then
            $MONITORS/http.monitor -t 3 $i && continue || touch tmp/http_failed/$i && exit 1

        ## If any node is online but in a failed state, throw error so http_down.alert can re-test it and put


        ## it back in load balancing if it has recovered or mark it offline if it has become un-pingable
        elif test "`cat $BASEDIR/$i`" = "2"
          then
            exit 1
        fi
  done
exit 0

This is a very simple script that lists the nodes in /cluster/http_nodes, checks to see if they are online (do they have a “1” in /cluster/online/nodename file?) and tries to open a connection to port 80 if they are. If the connection succeeds, nothing happens and the script checks the next node in the list. If the connection does not return an “OK” response, then the script exits with a “1” return code.

To add your application to load balancing and monitoring, you'll have to modify a copy of this file to fit your application. It should check port 5000 (to continue our example), and should list the nodes in the file that you created before ( i.e., change /cluster/http_nodes to /cluster/myapp_nodes ).

Note that the http script calls another script named “http.monitor” from the default monitors directory. In this directory are all the scripts included with the Mon monitoring system, and there may already be one for your application. Check in /cluster/mon/mon.d for an appropriate script. If you can't find one, you may have to write it. All it needs to do is take a node name or IP as an argument and attempt a connection on a certain port. Based on the response, it should either exit with a “1” or a “0”. For more examples, check here.

Test your file by running your new script against a server running your app:

./myapp.monitor my_running_server

This should have returned “0” as the exit code. You check with:

echo $?

Next, modify the file /cluster/mon/mon_common. Insert a line like this:

MYAPP_NODES=`cat /cluster/myapp_nodes`

This will include your application nodes in the rest of the monitoring system, as soon as we put an entry in /cluster/mon/mon.cf. This is the master configuration file for Mon. With the other ”“hostgroup” entries near the top of the file, add a line like this:

hostgroup myapp_nodes $MYAPP_NODES

And with the other service groups, add one for your application. Simply copy the service definition for http (see below):

watch http_nodes
    service http_failures
        description Check http status for online http nodes
        interval 10s
        randskew 3s
        monitor cluster_http.monitor
        period hr {12am-11pm}
        alert http_down.alert
        numalerts 1
        alert cluster_mail.alert 
  This e-mail address is being protected from spambots. You need JavaScript enabled to view it
 
        numalerts 1

and modify it the refer to your application.

Now that monitoring is configured, there are two remaining steps. First, what should happen when a node comes online? This is controlled by the file /cluster/mon/alert.d/node_up.alert.

Mon knows which nodes are online by checking the status the nodes file in /cluster/online/“nodename”. If it can ping a node which was previously marked offline, this triggers node_up.alert. It puts a “1” in /cluster/online/“nodename” and then check the node for any services that it should participate in. To add your application to the cluster, you will need to include a service check in node_up.alert.

As before, simply copy the section for http, paste it above or below the http check, and make the relevant changes. There is a clearly marked section of node_up.alert for this purpose, shown here:

## Service checks that should happen on boot up go here
##

## HTTP Service Check

        if test "`grep -c $i $HTTP_NODES`" != "0" -a "$NODE" == "1"
          then
            initlog -n nodeup.alert -s "$i found in /cluster/http_nodes. Checking for http services:"
            $MONITORS/http.monitor -t 3 $i && /sbin/ipvsadm -a -t $VIP:80 -r $i -g && initlog -n nodeup.alert -s "$i http service check passed. Node added to LVS" || initlog -n nodeup.alert -s "$i http service check failed! Node not added to load balancing." && echo 2 > $BASEDIR/$i
          fi

After changing the “http” wording to “myapp” and changing the messages that will be logged to syslog when this script is triggered, you just need to save and exit the file.

Last, we need to define what should happen when a node running our application fails a service check. The http service handles this with the /cluster/mon/alert.d/http_down.alert script, which we'll use as our template one more time. It is somewhat longer than the other scripts, but is easy to modify. The entire script is here:

#
##################################################
# Copyright (c) 2008, RapidScale Clusters LLC    #
#                                                #
##################################################
# http_down.alert
#
# Actions to take when any node has failed an http check:
# 1. Identify the node
# 2. echo 2 > /cluster/online/node_name (2 means error)
# 3. Remove it from LVS tables
# 4. Reboot it
#

# Time in seconds to wait for response before declaring node failed
RESPONSE_TIME=3

 . /cluster/mon/mon_common

# Test for failed http services

for i in `cat /cluster/http_nodes`
  do
         if test "`cat $BASEDIR/$i`" == "1"
           then
             $MONITORS/http.monitor -t $RESPONSE_TIME $i
             if [ $? != 0 ]
               then
                 initlog -n http_down.alert -s "$i has failed an http check"
                 /sbin/ipvsadm -d -t $VIP:80 -r $i && initlog -n http_down.alert -s "$i has been removed from the http load balancing pool"
                 ping -c 1 $i
                 if [ $? = 0 ]
                   then
                     rm -f $BASEDIR/$i
                     echo 2 > $BASEDIR/$i && initlog -n http_down.alert -s "$i responds to ping, but http services have failed"
                   else
                     rm -f $BASEDIR/$i
                     echo 0 > $BASEDIR/$i && initlog -n http_down.alert -s "$i does not respond to ping. Setting state to offline"
                   fi
               fi

             /etc/init.d/mon reload
 
elif test "`cat $BASEDIR/$i`" == "2"
           then
             rm -f $BASEDIR/$i
             echo 1 > $BASEDIR/$i
             initlog -n http_down.alert -s "$i was alive but in a failed state for http. Re-checking services..."

             ## Make sure the node is not still in LVS!
             /sbin/ipvsadm -d -t $VIP:80 -r $i

             ## Still online?
             ping -c 1 $i
             if [ $? != 0 ]
               then
                rm -f $BASEDIR/$i
                echo 0 > $BASEDIR/$i && initlog -n http_down.alert -s "$i does not respond to ping. State changed from failed to offline"
                /etc/init.d/mon reload
                /etc/init.d/mon reload
                break
               fi

             ## Service restored?
             $MONITORS/http.monitor -t $RESPONSE_TIME $i
             if [ $? = 0 ]
               then
                 $SSH $i "ifconfig lo:1 $VIP netmask 255.255.255.255 up"
                 /sbin/ipvsadm -a -t $VIP:80 -r $i -g
                 initlog -n http_down.alert -s "$i http service check passed. Node added to LVS"
               else
                 initlog -n http_down.alert -s "$i http service check failed. Node will be re-checked at next interval."
                 echo 2 > $BASEDIR/$i
               fi

             /etc/init.d/mon reload
         fi
  done

exit 0

Replacing the “http” wording with the appropriate “myapp” filenames will adapt this file for your application. Save this new script as /cluster/mon/alert.d/my_app_down.alert, and make sure the master configuration file for Mon (/cluster/mon/mon.cf) references this new file in the Alert section of the watch group you created earlier.