Please note that this article is about practical side of security audit.
You can easily find many teoretical materials on this subject (for example at wikipedia http://en.wikipedia.org/wiki/Information_technology_security_audit).

One of our customers asked us to make a security audit of his server. He had suspicions that his system has been compromised (this indirectly has been confirmed by the increased load, which was not so high before). He had a CentOS 5.4 server running Apache, MySQL, vsftpd server and so on. In a nutshell, it was a typical LAMP configuration.

[Start:: definining security perimeter]

Before proceeding to the audit, we have discussed with customer detailed list of components over which we have to focus. This stage helped us to determine security perimeter. Final list consisted of the following items:

* System kernel and modules
* System services
* Logging
* Backups
* Firewall rules
* General security policy

[Analysing system:: detailed picture of the perimeter]

Given the high probability of breaking the system, we immediately started to collect information and security checks. It is nearly impossible to create strict scenario of such work. It always include some kind of improvisation. However we came to few steps that help us. These steps convenient to use in the form of questions.

— Who is working on a server right now?

To answer this question, we have used system utilities such as ‘w’ and ‘last’.

• w - Displays list of logged users and shows what commands they run at the moment.”

• last -i -  Will show who, when and from which ip addresses logined to server.

Logs were analyzed as well. We have checked /var/log/audit/audit.log and /var/log/secure. After that we checked for suspicious entries /etc/passwd, /etc/shadow, /etc/group, /etc/sudoers

— What processes are running?

— Which ones should run on this host?

Right, these are two questions. And this step is most important. It needs the maximum attention and highest  concentration. You should rembember that there is possibility of changing the system binaries. So we used few ways of checking the running processes.

• ps auxf

• pstree

• cat /proc/*/stat | awk ‘{print $1,$2}’

There is common trick used by hackers together with binaries replacement. They put to their applications immutable flag, which is not seen by usual ls. But there is an effective way of finding such binaries:

• lsattr `echo $PATH | tr ‘:’ ‘ ‘` | grep i–

In most cases the process list shows us that there was intrusion and this case was the one of them. We have found two crond processess running: /usr/sbin/cron and /usr/sbin/crond. Path to default cron daemon binary on rh-based systems is /usr/sbin/crond. Second one was listening on port 2283 and telnet to this port opened a root shell on the server.
To find out what is doing a process on server usually are used strace and lsof utilities. If you didn’t get enough information from these utilities – then can be checked /proc/PID filesystem:

• lsof -p PID - Displays files, sockets and ports opened by a process.

• strace -p PID -F – Can help to trace what a process doing (call and signals) and its forks.

• mc -sd – /proc/PID Browsins proc using a file manager.

Here you can find a little description of procfs pseudo filesystem.

/proc/PID/cmdline -  Command which started the process
/proc/PID/cwd - Symlink to working directory.
/proc/PID/environ - Enviroment variables which affect the process.
/proc/PID/exe - Symlink to executable file
/proc/PID/fd - Directory with symlinks to opened file descriptors
/proc/PID/root - Symlink to filesystem root of this process
/proc/PID/status - Contains basuc information on process status
/proc/PID/task – Directory with hardlinks to child processes

— What processes opened ports of connections?

Besides all of above we are checking the network activity for all processes. This can be easily done using netstat, lsof and nmap utilities:

• netstat -tapn – Displays network activity on server

• lsof -i -n – Will display same result as netstat

• nmap -sT -sU -p 1-65535 localhost – Scanning for open ports

— What automated checks say?

At the last we check the system using some automated utilities: rkhunter, chkrootkit and clamdscan. They can be installed using default package manager (yum install application-name)

• rkhunter -с – Checks the system using series of automated checks. Before start is better to start rkhunder –update.

• chkrootkit – Checks system in the same way as rkhunder does, but has more false-positive alerts. Analyzing chkrootkit -x | less is more easy.

• clamscan - Checks file for injections, trojans, shell-scripts. Time of checking depends on harddrive size.

Check for / may take long time, so it is better to check separate folders / mount points (/home, /tmp, etc)

In our case rkhunder output was not so usefull, but clamscan found few interesting files”

/home/vhosts/domain.tpl/httpdocs/tmp/cc.php: PHP.Hide FOUND
/home/vhosts/domain.tpl/httpdocs/tmp/mass.pl: Perl.Defacer FOUND

Files were created few hour ago, that increased the possibility to find intruision signs in logs. After checking the apache logs for domain we could find entries about accessing these scripts from web. These scripts were accessed from the same ip, that simplified for us next steps of analyzing the atatckers actiosn. First attacker requests were looking like this:

xxx.xxx.xxx.xxx [DATE:11:19:24 /index.php?option=com_ckforms&controller=ck
data&view=ckformsdata&layout=detail&task=detail&fid=-2+union+select+1,2,3,
concat(0x3a,username,0x3a,email,0x3a,activation),5,6,7,8,9,10,11,12,13,14,
15,16,17,18,19+from+jos_users--

In this request we noticed two known things. First of all, option=com_ckforms, gives us idea that site has Joomla installed on it. Second, the last part of request is a SQL-injection. As we found later, attacke used ckforms-index-sql-injection (56988) vulnerability to gain administrator rights in cms.

FTP logs contained same ip. Attacker used ftp to upload few perl scripts and a kernel exploit. Firstg things that we did - we checked the config file for Joomla to check if mysql and ftp password for this domain are the same. We found that passwords were the same, so most probably attackers used same technology to gain the passwords.

Thus we could understand how attackers tactic and causes of successful attack:

• Old Joomla version

• Same password for ftp and mysql

• Weak firewall policy, that allowed to open ports for incoming connections

• Old version of kernel prone to attacks (CVE-2009-3547)

Elimination of foresaid causes is only a part of work we are doing during audit. This case is not so tipic as during analyzis we found that system is already compromised. In most cases we have to find the ways of break-ins. There is some irony in the fact that attacker did a part of work for us.

[Risk analysis]

Another part of the work that we perform in such cases is
risk analysis. The process of analysis is divided into three phases:
1) Analysis of processes related to perimeter of security
2) Analysis in a perimeter
3) Analysis of the human factor (random error, the dismissed employee)

In this article we will not go into details of the analysis, we describe only
basic concepts. The concept is to find opportunities to compromise the perimeter. Each of such possibilities, depending on the situation, is assigned a level. In our practice, we use three levels “High”, “medium” and “low”. Each level includes certain points. Correspondingly more points perimeter gains – worse it is protected.

[Finding the appropriate solution]

We must understand that absolute security is not a subject to discuss in real life. Security is a process that assumes a cyclical and even better permanent search for solutions and methods of their use. At the same time we should not forget about comfort and usability. In most cases enhancing security makes us to decrease comfort or usability. For anyone is not so pleasant to take off shoes at the airport, but the possibility to be in the same plane with a terrorrist is more unpleasant. It is very important to search suitable compromises. For example using ftp is not secure, because data and moreover passwords are transmitted in an unencrypted form. A good solution might be to use ssh authentication with rsa keys (+ passphrase). However, the probability of successful man-in-the-middle attack to ftp is not so high. It is worth to carefully consider the risks before changing the working processes that are already used. Ftp problem can be solved in several ways. Though its replacement by sftp is reliable but not always is acceptable. In this case we are allowed to access ftp from certain ips only, which solves the problem and thus reduces the discomfort to the minimum.

[Implementation and Monitoring]

Of course setting up the firewall with the default policy to DENY is only a part of measures aimed at improve the security. We preffer to use nagios and nrpe. We even built a distributed cluster based on it, to reduce to minimum false-positive alerts. Distributed monitoring will be discussed here but later, in other article. Of course we performed update for all software on the server and set up a common security policy (using sudo, configured passwords lifetime, etc.). We have developed procedures for updating and checking the server, which help client to be informed of how things are going on on his server. Customer continues to use our services and we are pleased in our partnership. We hope this article helped you to discover and learn something new from the world of security.
Thank you for your attention.

http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

Assumptions:

1. Server1 IP: 10.1.1.1
2. Server2 IP: 10.1.1.2
3. We will completely replicate all databases
4. Servers are not in production yet. If they are – be careful when rsyncing data

Notes:

• This setup can be used in HA environments, provided that only one of the servers is used for writes (INSERTs, UPDATEs, DELETEs) at a given moment
• In case one of the servers crashes, and data corruption occurs (by corrupting binlogs or innodb data files), there is the possibility that replication will fail and data will have to be re-synchronized from the known good server to the other one

Run (on 1st server, replace “password” with actual password):
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'10.1.1.%' IDENTIFIED BY 'password';

Stop mysql on both servers, rsync the mysql datadir from 1st to 2nd server, delete any existing binlogs or relay logs.

The global my.cnf should be quite similar on both servers (including any parameters used for tuning, like max_heap_table_size, etc), apart from the replication-related settings (very important: sync_binlog, and auto_increment_*, which ensure fail-safe replication):
Server1:
server-id = 1
master-host = 10.1.1.2
master-user = repl
master-password = password
sync_binlog=1
auto_increment_increment=2
auto_increment_offset=1
#adjusted due to mysql 5.x placing relay logs in wrong directory
relay-log=/var/lib/mysql/slave-relay.log
relay-log-index=/var/lib/mysql/slave-relay-log.index
relay-log-info-file=/var/lib/mysql/slave-relay.info

Server2:
server-id = 2
master-host = 10.1.1.1
master-user = repl
master-password = password
sync_binlog=1
auto_increment_increment=2
auto_increment_offset=2
#adjusted due to mysql 5.x placing relay logs in wrong directory
relay-log=/var/lib/mysql/slave-relay.log
relay-log-index=/var/lib/mysql/slave-relay-log.index
relay-log-info-file=/var/lib/mysql/slave-relay.info

Start mysql on both servers, run:
SHOW MASTER STATUS;
SHOW SLAVE STATUS \G
on both servers to check the status.

In order to achieve a HA solution, we can setup heartbeat on both servers, similar to the setup in http://www.rslab.net/2010/03/service-high-availability-using-open-source-tools/

We can use mon or monit to detect mysql failures.

In case we are using monit, following configuration can be used:

check process mysqld with pidfile /var/run/mysql/mysql.pid
if failed host 127.0.0.1 port 3306 protocol mysql then exec "/etc/monit-scripts/stop-heartbeat.sh"

/etc/monit-scripts/stop-heartbeat.sh itself should contain:
#!/bin/bash
/etc/init.d/mysql stop
/usr/lib64/heartbeat/hb_standby local

sync_binlog=1 indicated above will add a performance loss, since every query / transaction has to be logged immediately to the binlog. It can be safely disabled, but in case of hardware failures on one of the MySQL servers, there’s no guarantee that after it will come back up, replication will be intact.

The most simple example (not directly related to the Internet or IT, but illustrating the main idea):

Imagine the modern telephone system. Each country has it’s own area code, each state/district/region has it’s own area code too, inside the country. Each smaller region has an area code too.

For example, you want to call someone in Vienna, Virginia, with the local number 123 4567.

You will have to lookup the U.S international code first (+1 in this case). After, you will need to find the Virginia’s area code. You will use something like a phone-book for this. In the phone book – we can notice that Virginia, U.S has multiple area codes (276, 434, 540, 571, 703, 757, 804 in this particular example). In the same phone book, we find that Vienna’s area code is 703, so, we will have to dial: +1 703 123 4567 (where 123 4567 is the local phone number).

So, to find the intermediary codes you have to dial in order to be able to reach the local phone number, you have consulted a phone book, or the equivalent of a “Directory Service” (The information in the phone book being the directory itself).

A similar logic is hidden, for example, behind the DNS (Domain Name System) hierarchical naming system. We won’t go to deep into DNS structure, but in order to find valid information about a specific domain name, you will have to take similar steps as with the phone system.

Let’s say we want to find the IP address of the 4th level domain headquarters.servercompany.co.uk. We firstly need to know who controls the .uk TLD. After that – we will need to find who controls the .co.uk 2nd level domain, and so on until we obtain the needed IP address.

In theory, a “directory” refers to a database of information that is highly optimized for read operations, which should provide the ability of browse, lookup or search information using some specific criteria.

On the internet, we deal with hierarchical data structures too.
Most notable are: DNS and IP networks.
We gave a DNS-related example earlier. The situation with IPv4 IP adresses is similar, but instead of dividing domain names into TLD, 2nd level, 3rd level domains, etc, IP networks can be divided by applying shorter bitmask to the base 2 notation of an initial ip address.

Thus, 74.125.39.99 is part of the 74.125.39.0/24 network, which in turn is part of 74.125.36.0/22, which is part of 74.125.0.0/18, which is also a subnet of the 74.112.0.0/12 network, and so on.

By definition, RWhois or Referral Whois is an application protocol for querying of distributed databases of Internet domain names, address allocations, and other directory information.
It is in some way an extension of the Whois protocol, which was meant to be used earlier for the same purpose, but which was appropriate only during the beginning of the Internet, more exactly on the ARPANET.
As the internet evolved – a single server for keeping all this data is not enough, nor is it scalable.

Rwhois enhances Whois, and adds hierarchy and scalability to it.
Rwhois is documented in detail in RFC 2167, which supersedes a few older standards.
The most used rwhois server daemon is rwhoisd (http://projects.arin.net/rwhois/) developed by ARIN.
We will proceed with the standard install and basic configuration of the rwhoisd server.
The install process will be OS-independent, so we will install it from the source tarball provided by ARIN, using the last (1.5.9.5) version.
==========================================================
[root@server src]# wget http://projects.arin.net/rwhois/ftp/rwhoisd-1.5.9.5.tar.gz
[root@server src]# tar zxf rwhoisd-1.5.9.5.tar.gz
[root@server src]# cd rwhoisd-1.5.9.5
If we are installing this on Linux, we will have to adjust a small detail, because of different syntax of the `sort` parameters on FreeBSD and Linux:
[root@server rwhoisd-1.5.9.5]# nano mkdb/index.c
Before the next lines:
#ifdef NEW_STYLE_BIN_SORT
#define SORT_COMMAND "sort -o %s -k 5,5 -k 4,4n -t : %s"
#else
#define SORT_COMMAND "sort -o %s +4 +3 -t : %s "
#endif

add:
#define NEW_STYLE_BIN_SORT
Compiling and installing:
[root@server rwhoisd-1.5.9.5]# ./configure --prefix=/usr/local/rwhois
[root@server rwhoisd-1.5.9.5]# make
[root@server rwhoisd-1.5.9.5]# make install
Creating an init script for the rwhoisd daemon:
[root@server rwhoisd-1.5.9.5]# nano /etc/init.d/rwhoisd
#!/bin/bash
#
# /etc/rc.d/init.d/rwhoisd
#
# Source function library.
. /etc/init.d/functions
test -x /usr/local/rwhois/sbin/rwhoisd || exit 0
RETVAL=0
#
# See how we were called.
#
prog="rwhoisd"
start() {
# Check if atd is already running
if [ ! -f /var/lock/subsys/rwhoisd ]; then
echo -n $"Starting $prog: "
daemon /usr/local/rwhois/sbin/rwhoisd -d -c /usr/local/rwhois/etc/demo/rwhoisd.conf
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/rwhoisd
echo
fi
return $RETVAL
}
stop() {
echo -n $"Stopping $prog: "
killproc /usr/local/rwhois/sbin/rwhoisd
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/rwhoisd
echo
return $RETVAL
}
restart() {
stop
start
}
reload() {
restart
}
case "$1" in
start)
start
;;
stop)
stop
;;
reload|restart)
restart
;;
condrestart)
if [ -f /var/lock/subsys/rwoisd ]; then
restart
fi
;;
*)
echo $"Usage: $0 {start|stop|restart|condrestart}"
exit 1
esac
exit $?
exit $RETVAL

[root@server rwhoisd-1.5.9.5]# chmod +x /etc/init.d/rwhoisd
[root@server rwhoisd-1.5.9.5]# chkconfig --add rwhoisd
[root@server rwhoisd-1.5.9.5]# chkconfig rwhoisd on
[root@server rwhoisd-1.5.9.5]# mkdir /usr/local/rwhois/etc/demo
[root@server rwhoisd-1.5.9.5]# rsync -a /usr/local/rwhois/etc/rwhoisd/samples/ /usr/local/rwhois/etc/demo/
[root@server rwhoisd-1.5.9.5]# cd /usr/local/rwhois/etc/demo/
[root@server demo]# nano rwhoisd.conf
Adjust the following parameters:
—————–
root-dir: /usr/local/rwhois/etc/demo
local-host: rwhois.rservers.com
local-port: 4321
userid: rwhois
max-children: 30
—————–
[root@server demo]# chown -R rwhois:rwhois /usr/local/rwhois/etc/demo/net-*
[root@server demo]# touch /usr/local/rwhois/etc/demo/rwhoisd.log
[root@server demo]# chown -R rwhois:rwhois /usr/local/rwhois/etc/demo/rwhoisd.log
[root@server demo]# /usr/local/rwhois/bin/rwhois_indexer -i -c /usr/local/rwhois/etc/demo/rwhoisd.conf -v -C network -s txt
[root@server demo]# /etc/init.d/rwhoisd restart

To add new networks to this rwhois system – you will need to copy an existing subnet’s directory from /usr/local/rwhois/etc/demo/ to a new one, and adjust rwhoisd.auth_area to include the new network.

Also, these procedures can be automated, and integrated with a web interface which would greatly simplify management.

First of all, we need fast storage servers.

One would think that a few bulky servers with lots of HDD space, fast RAID10 arrays and Gig uplinks would be enough. In a real-world situation – this is not too correct. You will soon notice that you can’t get even 2-300Mbit traffic from these servers, since the bottleneck in this situation would be the hard drives, which will have to do random multiple reads and writes concurrently.

A better option would be to use a few raid1 arrays. or even better, a few separate drives, since MTBF is pretty high for the current HDD models on the market.

This way, for the storage servers, you could use Dual or Quad CPU machines, with 10 – 20 x 1TB hard drives (how many drives – depends on the SCSI/SAS/SATA controller model), also the system partitions should be placed better on a separate RAID1 array. Usually – 2×36 or 2×72 GB SAS/SCSI drives would be a better setup.
This will ensure necessary performance at a reasonable cost.

With high performance storage that provides high disk I/O, comes the need in faster network links. Putting these servers on 100Mbps links would mean to underutilise the disk resources. 1Gbps links are more than welcome, but, be aware that the maximum throughput of a gig link in practice is somewhere around 700-800Mbps, and if your hosting provider is billing you for the allocated bandwidth – it is more convenient to get 800Mbps allotment from them, than get 1Gig and pay the extra 200-300Mbps while not using them.

Apart from high-performance hardware, we must use appropriate software for this job.
The most popular and most used open-source web server – Apache – won’t give us the appropriate performance since it is not best-suited for serving (lots of) static content.
Instead, we consider nginx (http://sysoev.ru/nginx/ ) a much better alternative.
Nginx – a very fast and small footprint http server (it also has IMAP/POP3/SMTP proxy functions, but that is not related to the current subject), makes use of kqueue (FreeBSD), epoll (Linux), and sendfile (both OSs).

Now, the following highly depends on your application’s architecture, and also on the possible amount of visitors that you are going to have, but there are some common ways to achieve the final design of all the infrastructure. Also, if your budget permits it, it is way better to have more resources from the start, and to expand later, than to find very quick that your implementation is not scalable, and be forced to re-design parts of the system and add new resources in a hurry.

Ask yourself:
- how will the users upload files, is this FTP? HTTP?
- how will they download the files? FTP/HTTP?
- do the end-users authenticate when downloading?
- will you separate users with paid accounts, and the others with free access?

If FTP is involved – you will need to use a ftp server that supports virtual user accounts, virtual quotas, and can be integrated with MySQL (or any other DBMS that you are going to use)
Candidates:
- pure-ftpd
- proftpd
- vsftpd

For the web part, this meaning the site, the eventual payment system, it is highly recommended to start with a few servers, not a single one, though this is the part that can be adjusted and scaled easier (than the DB backend for example).

To reduce the load on the web servers, don’t use only Apache. Separate the static content (images, js, css files) from the scripts:
1. separate the static content to a different subdomain (this needs to be done within the application), and serve it through nginx
2. at the webserver level, setting up nginx in front of apache, and serve all content through nginx. requests to the dynamic content – should be proxied to apache.
3. in case you are using php-coded scripts, same as (2), but get rid of apache and proxy the requests to the .php files to a FastCGI backend (like PHP with the Php-fpm patch)

Once you run out of resources, you can easily add a new web server, mount the web content from a common NFS share, and:
1. Round-robin dns load-balancing
2. Add a software load balancer in front of the web servers, either nginx (in this case you can have multiple apache or fastcgi backends and serve static content dirrectly from the balancer), or Haproxy (which can only proxy the requests, but has advanced health checks, ACLs and other interesting features)
3. Add a hardware load balancer (Like alteon / radware / big-ip)

The most sensitive part: DB servers.
We will give examples assuming that you are going to use MySQL.

Implementing a successful topology from the start greatly depends on your application structure.
It is great if the application is able to separate database reads and writes, this gives you greater flexibility. It is also great if the application can use multiple DB servers.

When comes to hardware:
Use fast 15k rpm SCSI/SAS drives, RAID1 or RAID10. Get lots of RAM, DB servers love huge amounts of RAM.
Try to avoid RAID5 if possible, since it’s write performance is slower than RAID1/RAID10, and also a RAID5 is slower if one of the drives is failed.

Use a scalable architecture.
As stated above with regards to the application structure, it is great if we can separate reads and writes to different servers. In this case we can use Master-Slave replication, and even better, Master-Multiple Slave replication. DB Inserts / Updates / Deletes should be performed on the master, all Selects can be performed on the slaves.

On the master: Store Databases and binary logs on different RAID1 arrays. Different physical devices – faster random reads/writes.

The most sensitive server is the Master. You can double it with an intermediary slave, which can became a Master in case the current Master fails (think about some automatization here. Tip: heartbeat / wackamole)

A sample diagram is attached, which includes the last few discussed details:

References:

http://dev.mysql.com/doc/refman/5.0/en/replication.html

http://en.wikipedia.org/wiki/RAID

http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5_performance

http://sysoev.ru/nginx/

http://haproxy.1wt.eu/

http://www.pureftpd.org/project/pure-ftpd

http://www.proftpd.org/

http://vsftpd.beasts.org/

• power failures
• network device failures
• sysadmin’s mistakes

Most of these factors can be excluded, or their impact can be reduced to a minimum by a careful initial design and planning, and eliminating any SPOF (*single point of failure), whenever possible of course.

We also have to define the notion of downtime: it is the amount of time when the service is unavailable, or the system fails to provide the services it should be providing.

Downtime can be: Planned or Unplanned
In a HA (*high-availability) environment, we need to exclude unplanned downtimes.
Planned downtime – is the result of maintenance procedures executed on the system. This may include:

• hardware components replacement
• applying security patches or OS updates, that require a system reboot
• performing hardware upgrades
• system redesign

In this article we are designing a high-availability solution for a web service, using open-source software.



There are multiple ways of achieving a HA web service. Most common include:
1. DNS load-balancing – this implies using two or more A records in the dns zone for a web site, with a low TTL, thus the requests will be round-robin distributed to all the servers (or “nodes”) that the A dns records point to.
Example: google.com (though they aren’t using DNS load balancing only. They also seem to use geodns, anycast and other related technologies)

;; QUESTION SECTION:
;google.com.            IN    A

;; ANSWER SECTION:
google.com.        300    IN    A    209.85.135.99
google.com.        300    IN    A    209.85.135.103
google.com.        300    IN    A    209.85.135.104
google.com.        300    IN    A    209.85.135.105
google.com.        300    IN    A    209.85.135.106
google.com.        300    IN    A    209.85.135.147


This method has some drawbacks in a simple 2-3 node cluster.
If one node goes down, manual (or automatic) modification of the DNS records for this domain is required, and at least a few minutes (interval equal to the DNS TTL), a part of the requests will still go to the affected node.

We will adopt the following schema (that can be extended of course, but we are using a minimum of resources):

• two servers, that will have both the role of web servers, file servers and load balancers. Servers are running latest CentOS 5.4
• two networks, one public, that links to the ISP/Datacenter network, and an OOB network (Local to these servers)

Following open-source software will be used:

• unison (http://www.cis.upenn.edu/~bcpierce/unison/) – open source file synchronization tool
• nginx (http://sysoev.ru/nginx/) – small-footprint and very fast HTTP server, especially for static content serving.
• heartbeat (http://linux-ha.org/) – “a daemon that provides cluster infrastructure (communication and membership) services to its clients”
• apache (http://httpd.apache.org/) – most known and popular open-source HTTP server
• mon (http://mon.wiki.kernel.org/index.php/Main_Page) – “tool for monitoring the availability of services, and sending alerts on prescribed events”
• mod_rpaf for apache (http://stderr.net/apache/rpaf/) – changes the remote address of the client visible to other Apache modules.

All software is installed from standard repositories, except nginx and mon, which will be installed from the EPEL repositories (http://fedoraproject.org/wiki/EPEL)

Both servers should have similar configuration

Typical network wiring:


We will use the following IP adresses.
1. Public network:
2.2.2.1 – gateway
2.2.2.2 – srv1
2.2.2.3 – srv2
2.2.2.4, 2.2.2.5 – are used for the sites, and will be migrated from one server to another automatically by heartbeat.

2. Private network:
10.0.0.2 – srv1
10.0.0.3 – srv2
10.0.0.4, 10.0.0.5 – used for the OOB VIPs.

First of all, we will add the EPEL repository:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm


Install all required software:
yum install nginx heartbeat mon httpd php php-mysql unison httpd-devel


mod_rpaf will be installed manually:
wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
tar zxf mod_rpaf-0.6.tar.gz
cd mod_rpaf-0.6
apxs -cia mod_rpaf-2.0.c


Adjust the apache config by adding:
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.5
RPAFheader X-Forwarded-For

next to the
LoadModule rpaf_module libexec/apache2/mod_rpaf-2.0.so directive.

Also make sure to install any other php modules needed.

Heartbeat configuration (config files are similar on both servers):
Please refer to the documetation in case some parameter is not completely clear.
File /etc/ha.d/ha.cf:
—— cut ——-
node srv1.example.com
node srv2.example.com
bcast eth0
bcast eth1
udpport 694
ucast eth0 2.2.2.2
ucast eth0 2.2.2.3
ucast eth1 10.0.0.2
ucast eth1 10.0.0.3
ping 2.2.2.1
baud 19200
crm off
use_logd on
keepalive 1
deadtime 10
initdead 30
—— cut ——-

File /etc/ha.d/haresources:
—— cut ——-
srv1.example.com 2.2.2.4/24 10.0.0.4/24
srv2.example.com 2.2.2.5/24 10.0.0.5/24
—— cut ——-

In case you are using some iptables-based firewall, make sure to allow all traffic between the two servers, on both public and private networks.

Start the heartbeat service on both servers:
service heartbeat start

You should notice that 2.2.2.4, 10.0.0.4 IPs will be brought up automatically on srv1, and 2.2.2.5, 10.0.0.5 on srv2.

Next, to the file synchronization setup.
Assuming that all virtual hosts are stored in /home/websites, we will setup the following script on srv1 and add it to the root user’s crontab, to run once per minute (or more, if the content isn’t changing frequently):
/root/bin/usync.sh:

#!/usr/local/bin/bash
LOCK="/root/.unison";
if [ -f $LOCK ];
then
echo "unison already running";
exit 1;
fi
touch $LOCK
/usr/bin/unison -batch
rm -f $LOCK

Also we must create the unison config in /root/.unison/default.prf

# Unison preferences file
root = /home/websites/
root = ssh://10.0.0.3//home/websites/
ignore = Path {logs/*}
log = true

Please note above that you can exclude some specific paths that don’t need to be synchronized, like logs or temporary files.

Configure a sample vhost in apache (we will setup apache to listen on 8000 port only, adjust the Listen directive if needed):

NameVirtualHost *:8000
<VirtualHost *:8000>
DocumentRoot "/home/websites/example.com"
ServerName example.com
ServerAlias www.example.com
<Directory "/home/websites/example.com">
Allow from all
Options -Indexes FollowSymLinks
</Directory>
ErrorLog /home/websites/logs/example.com-error.log
CustomLog /home/websites/logs/example.com-access.log combined
ErrorDocument 404 /error/index.php
</VirtualHost>


Configure the vhost in nginx:
upstream backend {
# add the internal VIPs here, the requests will be forwarded to all ip:port combinations listed here.
# in case hardware configuration is different on these servers - you can adjust the 'weight' parameter
server 10.0.0.4:8000 weight=50;
server 10.0.0.5:8000 weight=50;
}
server
{
listen  80;
server_name example.com www.example.com;
location /
{
proxy_pass     http://backends;
proxy_redirect  off;
log_not_found   off;

proxy_buffer_size          4k;
proxy_buffers              4 32k;
proxy_busy_buffers_size    64k;
proxy_temp_file_write_size 64k;
}
location ~* ^.+\.(jpg|jpeg|gif|png|css|js|ico|zip|rar|swf)$ {
root  /home/websites/example.com;
access_log /home/logs/example.com-access.log main;
error_page 404 = @fallback;
}
location @fallback {
proxy_pass http://backends;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
}

Ok, start apache, nginx.

You should use the public VIPs (2.2.2.4, 2.2.2.5) to access the web sites.
In case one of the servers will fail, heartbeat will ensure the ips are brought up automatically on the 2nd server.
Now we have to configure automatic failover in case the servers do not fail, but the web service is inaccessible on one of them. We are going to use ‘mon’ for this.

File: /etc/mon/mon.cf
### global options
cfbasedir   = /etc/mon
pidfile     = /var/run/mon.pid
statedir    = /var/lib/mon/state.d
logdir      = /var/lib/mon/log.d
dtlogfile   = /var/lib/mon/log.d/downtime.log
alertdir    = /usr/lib64/mon/alert.d
mondir      = /usr/lib64/mon/mon.d
maxprocs    = 20
histlength  = 100
randstart   = 60s
authtype    = pam
userfile    = /etc/mon/userfile

service nginx
interval 30s
monitor http.monitor -p 80 -u /nginx-status 192.168.1.232
period wd {Mon-Sun}
alert stop-heartbeat.alert
alert mail.alert -S "Nginx on Node 1 Down" monitoring@example2.com
upalert start-heartbeat.alert
alertevery 1h


We will need to make two simple scripts that will stop/start heartbeat on demand:
/usr/lib64/mon/alert.d/stop-heartbeat.alert:
#!/bin/bash
/etc/init.d/heartbeat stop


and /usr/lib64/mon/alert.d/start-heartbeat.alert:
#!/bin/bash
/etc/init.d/heartbeat start


As you may have noticed, we did not mention MySQL in our setup,
This is left as an exercise to the reader, as well as the initial server and network setup, and eventual ftp/sftp, dns or firewall configuration.
A HA MySQL cluster will be discussed in a later article.

http://en.wikipedia.org/wiki/Downtime
http://en.wikipedia.org/wiki/High_availability
http://linux-ha.org/
http://www.cis.upenn.edu/~bcpierce/unison/
http://sysoev.ru/nginx/
http://httpd.apache.org/
http://mon.wiki.kernel.org/index.php/Main_Page
http://stderr.net/apache/rpaf/

Alteon 180/184 series load balancers is a cheap solution to handle at least 1 Gigabit of load balanced web traffic, when using DSR.

As per Wikipedia, load balancing is a technique to spread work between two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, and minimize response time. Using multiple components with load balancing, instead of a single component, may increase reliability through redundancy.




DSR is a way for outbound traffic to bypass the load balancer, sending traffic directly to the default router of that network.
DSR uses the loopback interface on a server to spoof the address of the VIP (virtual ip address) on the load balancer when sending traffic out, making it look as the load balancer sent the packet instead of the server, thus eliminating the need for the load balancer to process that traffic. The loopback interface is a special kind of network interface inside the machine. Usually, it is used only by the operating system for internal network communications, but it can be used for other purposes, such as DSR.

DEVICE=lo:0
IPADDR=10.0.0.1
NETMASK=255.255.255.255
ONBOOT=yes




On Linux running kernel 2.6  there is one more thing to adjust,  because the Linux boxes will respond to the ARP requests for that VIP, when they are not supposed to. This can be prevented by specific kernel arp settings, added to the configuration file /etc/sysctl.conf  (2.6 kernel only) and rebooting the server:


net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.eth0.arp_ignore=1
net.ipv4.conf.eth1.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.eth0.arp_announce=2
net.ipv4.conf.eth1.arp_announce=2




The following commands may be used to change the settings interactively during runtime:


echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/eth1/arp_announce




Unfortunately there seems to be no general and simple solution for for kernel 2.4. For additional information about the nature of the problem (and other solutions) check http://linux-ip.net/html/ether-arp.html#ether-arp-flux .



2. Configure the web server to bind to both the real IP address (so the load balancer can still perform health checks) and the new loopback IP address.



A config snippet example for Apache  :


<VirtualHost 10.0.0.1 10.0.0.100>
ServerAdmin support@example.com
DocumentRoot /var/www/html
ServerName www1.example.com
ErrorLog logs/error_log
CustomLog logs/access_log common
</VirtualHost>


3. Point the default route on the web servers directly towards the router (rather than through the load balancer).



In this example, run on the Linux server :



route add default gw 10.0.0.254



and save this configuration in the init scripts (/etc/sysconfig/network OR /etc/sysconfig/network-scripts/ifcfg-ethX) .

4. Configure the load balancer to enable DSR.

In this test scenario I`m using the following ip addresses :


10.0.0.1 and 10.0.0.2 as web servers RIPs
10.0.0.100 the Alteon VIP
10.0.0.254 the router, acting as gateway for servers as well as for the Alteon.

A sample config of the alteon load balancer with two real servers behind a single VIP address.




The most important advantage of using DSR is performance, because the load balancer handles about one packet in for every eight packets out, depending on the traffic profile, the load balancer does substantially less work.
The disadvantages of using DSR is a most complex setup and because this method may perform only layer 4 load balancing, as layer 7 (URL parsing and cookie persistence)  requires the ability to completely proxy a connection.



References:

1. http://en.wikipedia.org/wiki/Load_balancer
2. http://lbwiki.com/index.php/DSR
3. http://www.inlab.de/balanceng/faq.html
4. Tony Bourke: Server Load Balancing, O’Reilly, ISBN 0-596-00050-2

The goal was to collect cpu usage by

1. apache and mod_php
2. mysql
3. cgi/daemons
4. mail

The tool will get actual resource usage every 5 minutes and save the data in a dbm database.
Actually we started with apache resources usage.
The utility is collecting cpu usage per user , vhost and even requested url.
The first try could be downloaded here, available for Cpanel servers only.
I do not recommend at all to run it on non-Cpanel servers.