Creating a HA cluster using two Centos 6.x servers

Important info in regards to CentOS 6.5

Due to consequence of multilib support and PATH handling heartbeat’s init scripts wont work on Centos 6.5. The easiest way to fix it is to install heartbeat 3.0.4-2 that is not currently in the epel repo, but can be downloaded from here:

Note: you will need both heartbeat-3.0.4-2.el6.x86_64.rpm and heartbeat-libs-3.0.4-2.el6.x86_64.rpm packages. In regards to further development of this but you can check this bugzilla bug.


  • Two ‘base’ CentOS servers (architecture is irrelevant)
  • A  lot of time

Notes about this guide

  • I will presume that both CentOS servers were installed following this guide, with a partitioning scheme similar to this.
  • Unless otherwise specified, with red for primary and blue for secondary, run each command on both servers.

Before we begin make sure that both servers are up to date:

yum -y update

Installing packages

We need to add a repo first, import the public key:

rpm --import

and than add the repo:

rpm -Uvh

now we install drbd, it’s kernel module and heartbeat, by running:

yum -y install drbd84-utils kmod-drbd84 heartbeat

Note: As of writing this document the latest drbd version is 8.4, your milage might differ, change the above command accordingly.

Configuring DRBD

DRBD refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.

Presuming, that you followed the custom partitioning scheme how-to you should have some free space in your LVM that you can allocate to be used by drbd. Let us create a logical volume, named lv_drbd by running:

lvcreate -L 35GB --name lv_drbd vg_linuxserver

where 35GB is the size of the block device, and vg_linuxserver is your volume group name; usually it’s the same as your host name preceded by a vg_ prefix. You can check your host name by typing:

uname -n

or just check the partitioning table by typing:

fdisk -l

We need to create a config file, by running:

nano /etc/drbd.conf

delete everything inside it (ctrl+k deletes entire lines) and paste this:

global { usage-count no; }
resource repdata {
 protocol C;
 startup { wfc-timeout 0; degr-wfc-timeout 120; }
 disk { on-io-error detach; } # or panic, ...
 net { cram-hmac-alg "sha1"; shared-secret "somepassword"; } # don't forget to choose a secret for auth !
 syncer { rate 10M; }
 on linuxserver1 {
 device /dev/drbd0;
 disk /dev/vg_linuxserver1/lv_drbd;
 meta-disk internal;
 on linuxserver2 {
 device /dev/drbd0;
 disk /dev/vg_linuxserver2/lv_drbd;
 meta-disk internal;

This will create a repdata resource shared between linuxserver1 and linuxserver2. The servers IP addresses are and respectively. Adjust those settings to match your setup. Don’t forget to change somepassword to something more secure.

Initialize the meta-data area by running:

drbdadm create-md repdata

and start drbd:

service drbd start

check it’s status by running:

cat /proc/drbd


service drbd status

As you can see , both nodes are secondary, which is normal. we need to decide which node will act as a primary now, that will initiate the first ‘full sync’ between the two nodes. Run this ONLY on the primary node:

drbdadm -- --overwrite-data-of-peer primary repdata

You can monitor the progress (on either node) of the sync by running:

watch -n 1 cat /proc/drbd

When all is done format and mount it. Run this ONLY on the primary node:

mkfs.ext4 /dev/drbd0 ; mkdir /repdata ; mount /dev/drbd0 /repdata

but don’t forget to create the mount point on the secondary node as well. Run this ONLY on the secondary node:

mkdir /repdata

Configuring Heartbeat

Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them.

Let’s create the authkeys. Run:

nano /etc/ha.d/authkeys

and paste:

auth 2
2 sha1 somepassword

change somepassword to a stronger one.

Change the access rights, else heartbeat won’t start:

chmod 600 /etc/ha.d/authkeys

Create the file, the main config file for heartbeat by running:

nano /etc/ha.d/

and paste:

udpport 695
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
bcast eth0
auto_failback on
node linuxserver1
node linuxserver2

Pay close attention the the node lines, they should be the hostnames of the two linux servers. If you want to run more than one HA cluster inside the same network change the udpport to a different, unused one.

Now to the haresources file. Run:

nano /etc/ha.d/haresources

and paste:

linuxserver1 IPaddr:: drbddisk::repdata Filesystem::/dev/drbd0::/repdata::ext4 httpd

This basically means that linuxserver1 is the primary node, the node ip address will be on a subnet. The filesystem is a drbddisk, namely /dev/drdb0, that should be mounted on the /repdata mountpoint as ext4. The service that should be started is httpd (apache). Adjust it to fir your needs.

And finally start the service:

service heartbeat start


  • make sure that the ports listed above are open on both servers
  • make sure that drbd and heartbeat start at boot time
  • make sure to NOT start the service at boot time, since it is managed by heartbeat
  • make sure that the service in question actually uses the /repdata mount point.

That’s it!

Leave a Reply

Your email address will not be published. Required fields are marked *