Getting started with HashiCorp Serf

In my post about Apache Mesos I briefly mentioned Serf.

Serf (from Hashicorp, who also make Vagrant and Packer) is a decentralised service discovery tool with support for custom events.

By installing a Serf agent on each node in a network, and (maybe) bootstrapping each agent with the IP address of another agent, you are quickly provided with a scalable membership system with the ability to propagate events across the network.

Once it’s installed and agents are started, running serf members from any node will produce output similar to this:

vagrant@master10:~$ serf members
master10     192.168.100.131:7946    alive    role=master
zk10     192.168.100.130:7946    alive    role=zookeeper
slave10     192.168.100.135:7946    alive    role=slave
mongodb10     192.168.100.136:7946    alive    role=mongodb

Which is when you realise you’ve still got a Mesos cluster running that you’d forgotten about…

The output from Serf shows the hostname, IP address, status and any tags the Serf agent is configured with. In this case, I’ve set a role tag which lets us quickly find a particular instance type on the network:

vagrant@master10:~$ serf members | grep mongodb
mongodb10     192.168.100.136:7946    alive    role=mongodb

This, with its event system, makes Serf ideal for efficiently maintaining cluster node state, and reducing or eliminating application configuration.

In a Mesos cluster, it lets us make core parts of the infrastructure (like ZooKeepers and Mesos masters) simple to scale with no manual configuration.

Serf has lots of other potential uses too, some of them documented on the Serf website.

Getting started with Serf

Installation

Installing Serf couldn’t be easier.

• Download Serf
• Extract the downloaded archive
• Move the ‘serf’ binary into your PATH (full instructions from Serf)

You should be able to run serf from the command line and see a list of available commands.

Trying it out

To try Serf, you need two console windows open and you’re ready!

• Run serf agent from one console
• Run serf members from another

You should see something like this:

vagrant@example:~$ serf members
example     127.0.0.1:7946    alive

That output shows a cluster containing your local machine (with a hostname of ‘example’), available at 127.0.0.1, and that the node is alive.

It’s that simple!

Starting Serf automatically

This isn’t much use on its own - we need Serf to start every time a node boots up. As soon as a new node comes online, the cluster finds out about it immediately, and the new node can configure itself.

Serf provides us with example scripts for upstart and systemd.

For Ubuntu, copy upstart.conf to /etc/init/serf-agent.conf then run start serf-agent (you might need to modify the upstart script if Serf isn’t installed to /usr/local/bin/serf).

Configuration

Now we’ve got our Serf agent running, we need to configure it so it knows what to do.

You can configure Serf using either command line options (useful if you’re talking to a remote Serf agent or using non-standard ports), or you can provide configuration files (which are JSON files, loaded from the configuration directory in alphabetical order).

If you’ve used the Ubuntu upstart script, creating config.json in /etc/serf will work.

All of the configuration options are documented on the Serf website.

The examples below are in JSON, but they can all be provided as command line arguments instead.

IP addresses

This caught me out a few times - Serf, by default, will advertise the bind address (usually the IP address of your first network interface, e.g. eth0).

In a Vagrant environment, you will always have a NAT connection as your first interface (the one Vagrant uses to communicate with the VM). This was causing my agents to advertise an IP which other nodes couldn’t connect to.

To fix this, Serf lets us override the IP address it advertises to the cluster:

{
    "advertise": "192.168.100.200"
}

Setting tags

Serf used to provide a ‘role’ command line option (it still does, but its deprecated). In its place, we have tags, which are far more flexible.

Tags are key-value pairs which provide metadata about the agent. In the example above, I’ve created a tag named role which describes the purpose of the node.

{
    "tags": {
        "role": "mongodb"
    }
}

You can set multiple tags, but there is a limit - the Serf documentation doesn’t specify the limit, except to say

There is a byte size limit for the maximum number of tags, but in practice dozens of tags may be used.

You can also replace tags while the Serf agent is still running using the serf tags command, though changes aren’t persisted to configuration files.

Protocol

You shouldn’t need to set the protocol - it should default to the latest version (currently 3).

It does, when started from the command line. But it didn’t seem to when started using upstart and a configuration directory. Easy to fix though:

{
    "protocol": 3
}

This might not be a bad practice anyway, you can update Serf on all nodes without worrying about protocol compatibility.

Forming a cluster

I’ll cover this in more detail later, but you can set either start_join or discover to join your agent to a cluster.

Scripting it

Since Serf is ideal for a cloud environment, its useful to script its installation and configuration.

Here’s an example using bash similar to the one in my vagrant-mongodb example. It installs Serf, configures upstart, and writes an example JSON configuration file.

Because its from a Vagrant build, it uses a workaround to find the correct IP.

wget https://dl.bintray.com/mitchellh/serf/0.4.5_linux_amd64.zip
unzip 0.4.5_linux_amd64.zip
mv serf /usr/local/bin
rm 0.4.5_linux_amd64.zip
wget https://raw.github.com/hashicorp/serf/c15b5f15dec3d735dae1a6d1f3455d4d7b5685e7/ops-misc/upstart.conf
mv upstart.conf /etc/init/serf-agent.conf
mkdir /etc/serf
ip=`ip addr list eth1 | grep "inet " | cut -d ' ' -f6 | cut -d/ -f1`
echo { \"start_join\": [\"$1\"], \"protocol\": 3, \"tags\": { \"role\": \"mongodb\" }, \"advertise\": \"$ip\" } | tee /etc/serf/config.json
exec start serf-agent

Forming a cluster

So far we have just a single Serf agent. The next step is to setup another Serf agent, and join them together, forming a (small) cluster of Serf agents.

Using multicast DNS (mDNS)

Serf supports multicast DNS, so in a contained environment with multicast support we don’t need to provide it with a neighbour.

Using the discover configuration option, we provide Serf with a cluster name which it will use to automatically discover Serf peers.

{
    "discover": "mycluster"
}

In a cloud environment this removes the need to bootstrap Serf, making it truly autonomous.

Providing a neighbour

If we can’t use multicast DNS, we can provide a neighbour and Serf will discover the rest of the cluster from there.

This could be problematic, but if we’re in a Mesos cluster it becomes easy. We know at least one Zookeeper must always be available, so we can give Serf the hostnames of our known Zookeeper instances:

{
    "start_join": [ "zk1", "zk2", "zk3" ]
}

Or, if we get the Zookeepers to update a load balancer (using Serf!) when they join or leave the cluster, we can make our configuration even easier:

{
    "start_join": [ "zk" ]
}

We can also use the same technique to configure Serf on the Zookeeper nodes.

How clusters are formed

A cluster is formed as soon as one agent discovers another (whether this is through multicast DNS or using a known neighbour).

As soon as a cluster is formed, agents will share information between them using the Gossip Protocol.

If agents from two existing clusters discover each other, the two clusters will become a single cluster. Full membership information is propagated to every node.

Once two clusters have merged, it would be difficult to split them without restarting all agents in the cluster or forcing agents to leave the cluster using the force-leave command (and preventing them from discovering each other again!).

Nodes leaving

If a node chooses to leave the cluster (e.g. scaling down or restarting nodes), other nodes in the cluster will be informed with a leave event.

It’s membership information will be updated to show a ‘left’ state:

example     192.168.100.200:7946    left    role=mongodb

A node leaving the cluster is treated differently to a failure.

This is determined by the signal sent to the Serf agent to terminate the process. An interrupt signal (Ctrl+C or kill -2) will tell the node to leave the cluster, while a kill signal (kill -9) will be treated as a node failure.

Node failures

When a node fails, other nodes are informed with a failed event.

It’s membership information will be updated to show a ‘failed’ state:

example     192.168.100.200:7946    failed    role=mongodb

Knowledge of the failed node is kept by other Serf agents in the cluster. They will periodically attempt to reconnect to the node, and eventually remove the node if further attempts are unsuccessful.

Events

Serf uses events to propagate membership information across the cluster, either member-join, member-leave, member-failed or member-update.

You can also send custom events (which use the user event type), and provide a custom event name and data payload to send with it:

serf event dosomething "{ \"foo\": \"bar\" }"

Events with the same name within a short time frame are coalesced into one event, although this can be disabled using the -coalesce=false command line argument.

This makes Serf useful as an automation tool - for example, to install applications on cluster nodes or configure ZooKeeper or Mesos instances.

Event handlers

Event handlers are scripts which are executed as a shell command in response to the events.

Shell environment

Within the shell created by Serf, we have the following environment variables available:

• SERF_EVENT - the event type
• SERF_SELF_NAME - the current node name
• SERF_SELF_ROLE - the role of the node, but presumably deprecated
• SERF_TAG_${TAG} - one for each tag set (uppercased)
• SERF_USER_EVENT - the user event type, if SERF_EVENT is ‘user’
• SERF_USER_LTIME - the Lamport timestamp of the event, if SERF_EVENT is ‘user’

Any data payload given by the event is piped to STDIN.

Creating event handlers

Serf’s event handler syntax is quite flexible, and lets you listen to all events or filter based on event type.

• The most basic option is to invoke a script for every event:

{
    "event_handlers": [
        "dosomething.sh"
    ]
}

• You can listen for a specific event type:

{
    "event_handlers": [
        "member-join=dosomething.sh"
    ]
}

• You can specify multiple event types:

{
    "event_handlers": [
        "member-join,member-leave=dosomething.sh"
    ]
}

• You can listen to just user events:

{
    "event_handlers": [
        "user=dosomething.sh"
    ]
}

• You can listen for specific user event types:

{
    "event_handlers": [
        "user:dosomething=dosomething.sh"
    ]
}

Multiple event handlers can be specified, and all event handlers which match for an event will be invoked.

Reloading configuration

Serf can reload its configuration without restarting the agent.

To do this, send a SIGHUP signal to the Serf process, for example using killall serf -HUP or kill -1 PID.

You could even use custom user events to rewrite Serf configuration files and reload them across the entire cluster.