MongoDB Replica Sets with Ubuntu Server and Juju

** This is an updated post reflecting the project formerly known as Ensemble, Juju **

I have always liked MongoDB and, recently Juju so, it was a matter of time until I came up with a MongoDB charm for Juju.

Here are some of the goals I set out to accomplish when I started working on this charm:

• stand alone deployment. 
• replica sets.  More information about replica sets can be found here.
• master and server relationships
• Don't try to solve all deployment scenarios just concentrate on the above ones for now.

Let's start with the stand-alone deployment first and, we'll add the other functionality a bit later.

Before we go into creating the directories and files, I should probably mention Charm Tools.  Charm Tools is ( as the name implies ) a set of tools that facilitates the creation of charms for juju.

You can get charm-tools on most supported release of Ubuntu in the Juju ppa:

sudo add-apt-repository ppa:juju/pkgs
sudo apt-get update
sudo apt-get install charm-tools

After installing charm-tools, go to the directory where you will be creating your charms and type the following to get started:

• charm create mongodb

The above commands will look in your cache for a package called mongodb and create a "skeleton" structure with the metadata.yaml, hooks and descriptions already done for you into a directory called ( you guessed it ), mongodb.

The structure should look something like this:

mongodb:
mongodb/metadata.yaml
mongodb/hooks
mongodb/hooks/install
mongodb/hooks/start
mongodb/hooks/stop
mongodb/hooks/relation-name-relation-joined
mongodb/hooks/relation-name-relation-departed
mongodb/hooks/relation-name-relation-changed
mongodb/hooks/relation-name-relation-broken

metadata.yaml
At this point in the development, the metadata.yaml file should look very similar to this:

name: mongodb
revision: 1
summary: An object/document-oriented database (metapackage)
description: |
  MongoDB is a high-performance, open source, schema-free document-
  oriented  data store that's easy to deploy, manage and use. It's
  network accessible, written in C++ and offers the following features :
  * Collection oriented storage - easy storage of object-     style data
  * Full index support, including on inner objects   * Query profiling
  * Replication and fail-over support   * Efficient storage of binary
  data including large     objects (e.g. videos)   * Auto-sharding for
  cloud-level scalability (Q209) High performance, scalability, and
  reasonable depth of functionality are the goals for the project.  This
  is a metapackage that depends on all the mongodb parts.
provides:
  relation-name:
    interface: interface-name
requires:
  relation-name:
    interface: interface-name
peers:
  relation-name:
    interface: interface-name

For our purposes, let's change the emphasized lines to the following:

provides:

  database:

    interface: mongodb

peers:

  replica-set:

    interface: mongodb-replica-set

The peers section will be used when we start working with replica sets so, let's just ignore that one for now.

provides: is the way we "announce" what our particular charm ...well ... provides.  In this case we provide a database interface by the name of mongodb.


Not much else to do with metadata.yaml file as charm create did the brunt of work here for us.

hooks/install
charm create also took care of providing us with a basic install script based on the mongodb package already available in Ubuntu.  It should look very similar to this:



#!/bin/bash
# Here do anything needed to install the service
# i.e. apt-get install -y foo  or  bzr branch http://myserver/mycode /srv/webroot


apt-get install -y mongodb

After some trial and error and some debugging, here is what I came up with:

#!/bin/bash
# Here do anything needed to install the service
# i.e. apt-get install -y foo  or  bzr branch http://myserver/mycode /srv/webroot


set -ux


#################################################################################
# Install some utility packages needed for installation
#################################################################################
rm -f /etc/apt/sources.list.d/facter-plugins-ppa-oneiric.list
echo deb http://ppa.launchpad.net/facter-plugins/ppa/ubuntu oneiric main  >> /etc/apt/sources.list.d/facter-plugins-ppa-oneiric.list
echo deb-src http://ppa.launchpad.net/facter-plugins/ppa/ubuntu oneiric main  >> /etc/apt/sources.list.d/facter-plugins-ppa-oneiric.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B696B50DD8914A9290A4923D6383E098F7D4BE4B


#apt-add-repository ppa:facter-plugins/ppa
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y install facter facter-customfacts-plugin


##################################################################################
# Set some variables that we'll need for later
##################################################################################


DEFAULT_REPLSET_NAME="myset"
HOSTNAME=`hostname -f`
EPOCH=`date +%s`
fact-add replset-name ${DEFAULT_REPLSET_NAME}
fact-add install-time ${EPOCH}




##################################################################################
# Install mongodb
##################################################################################


DEBIAN_FRONTEND=noninteractive apt-get install -y mongodb




##################################################################################
# Change the default mongodb configuration to bind to relfect that we are a master
##################################################################################


sed -e "s/#master = true/master = true/" -e "s/bind_ip/#bind_ip/" -i /etc/mongodb.conf




##################################################################################
# Reconfigure the upstart script to include the replica-set option.
# We'll need this so, when we add nodes, they can all talk to each other.
# Replica sets can only talk to each other if they all belong to the same
# set.  In our case, we have defaulted to "myset".
##################################################################################
sed -i -e "s/ -- / -- --replSet ${DEFAULT_REPLSET_NAME} /" /etc/init/mongodb.conf




##################################################################################
# stop then start ( *** not restart **** )  mongodb so we can finish the configuration
##################################################################################
service mongodb stop
# There is a bug in the upstart script that leaves a lock file orphaned.... Let's wipe that file out
rm -f /var/lib/mongodb/mongod.lock
service mongodb start




##################################################################################
# Register the port
##################################################################################
[ -x /usr/bin/open-port ] && open-port 27017/TCP

I have tried to comment the install script so you have an idea of what's going on ...


hooks/start
This is the script that Juju will call to start mongodb.  Here is what mine looks like:



#!/bin/bash
# Here put anything that is needed to start the service.
# Note that currently this is run directly after install
# i.e. 'service apache2 start'


service mongodb status && service mongodb restart || service mongodb start


It's simple enough.

hooks/stop

#!/bin/bash
# This will be run when the service is being torn down, allowing you to disable
# it in various ways..
# For example, if your web app uses a text file to signal to the load balancer
# that it is live... you could remove it and sleep for a bit to allow the load
# balancer to stop sending traffic.
# rm /srv/webroot/server-live.txt && sleep 30


service mongodb stop
rm -f /var/lib/mongodb/mongod.lock

This is the script that Juju calls when it needs to stop a service.

hooks/relation-name-relation-[joined|changed|broken|departed]

These files are templates for the relationships ( provides, requires, peers, etc. ) declared in the metadata.yaml file.  Here is a look at the ones that I have for mongodb:

• Per the metadata.yaml, we need to define the following relationships:
• database
• replica-set

Based on that information, here are the files that I created for this charm:

database-relation-joined

#!/bin/bash

# This must be renamed to the name of the relation. The goal here is to

# affect any change needed by relationships being formed

# This script should be idempotent.

set -ux

relation-set hostname=`hostname -f` replset=`facter replset-name`

echo $JUJU_REMOTE_UNIT joined

replica-set-relation-joined

#!/bin/bash

# This must be renamed to the name of the relation. The goal here is to

# affect any change needed by relationships being formed

# This script should be idempotent.

set -ux

relation-set hostname=`hostname -f` replset=`facter replset-name` install-time=`facter install-time`

echo $JUJU_REMOTE_UNIT joined

replica-set-relation-changed

#!/bin/bash

# This must be renamed to the name of the relation. The goal here is to

# affect any change needed by relationships being formed, modified, or broken

# This script should be idempotent.

##################################################################################

# Set debugging information

##################################################################################

set -ux

##################################################################################

# Set some global variables

##################################################################################

MY_HOSTNAME=`hostname -f`

MY_REPLSET=`facter replset-name`

MY_INSTALL_TIME=`facter install-time`

MASTER_HOSTNAME=${MY_HOSTNAME}

MASTER_REPLSET=${MY_REPLSET}

MASTER_INSTALL_TIME=${MY_INSTALL_TIME}

echo "My hosntmae: ${MY_HOSTNAME}"

echo "My ReplSet: ${MY_REPLSET}"

echo "My install time: ${MY_INSTALL_TIME}"

##################################################################################

# Here we need to find out which is the first node ( we record the install time ).

# The one with the lowest install time is the master.

# Initialize the master node.

# Add the other nodes to the master's replica set.

##################################################################################

# Find the master ( lowest install time )

for MEMBER in `relation-list`

do

   HOSTNAME=`relation-get hostname ${MEMBER}`

   REPLSET=`relation-get replset ${MEMBER}`

   INSTALL_TIME=`relation-get install-time ${MEMBER}`

   [ ${INSTALL_TIME} -lt ${MASTER_INSTALL_TIME} ] && MASTER_INSTALL_TIME=${INSTALL_TIME}

done

echo "Master install-time: ${MASTER_INSTALL_TIME}"

# We should now have the lowest member of this relationship.  Let's get all of the information about it.

for MEMBER in `relation-list`

do

   HOSTNAME=`relation-get hostname ${MEMBER}`

   REPLSET=`relation-get replset ${MEMBER}`

   INSTALL_TIME=`relation-get install-time ${MEMBER}`

   if [ ${INSTALL_TIME} -eq ${MASTER_INSTALL_TIME} ]; then

      MASTER_HOSTNAME=${HOSTNAME}

      MASTER_REPLSET=${REPLSET}

   fi

done

echo "Master Hostname: ${MASTER_HOSTNAME}"

echo "Master ReplSet: ${MASTER_REPLSET}"

echo "Master install time: ${MASTER_INSTALL_TIME}"

# We should now have all the information about the master node.

# If the node has already been initialized, it will just inform you

# about it with no other consequence.

if [ ${MASTER_INSTALL_TIME} -eq ${MY_INSTALL_TIME} ]; then

   mongo --eval "rs.initiate()"

else

   mongo --host ${MASTER_HOSTNAME} --eval "rs.initiate()"

fi

# Now we need to add the rest of nodes to the replica set

for MEMBER in `relation-list`

do

   HOSTNAME=`relation-get hostname ${MEMBER}`

   REPLSET=`relation-get replset ${MEMBER}`

   INSTALL_TIME=`relation-get install-time ${MEMBER}`

   if [ ${MASTER_INSTALL_TIME} -ne ${INSTALL_TIME} ]; then

      if [ ${INSTALL_TIME} -eq ${MY_INSTALL_TIME} ]; then

         mongo --eval "rs.add(\""${HOSTNAME}"\")"

      else

         mongo --host ${MASTER_HOSTNAME} --eval "rs.add(\""${HOSTNAME}"\")"

      fi

   fi

done

echo $JUJU_REMOTE_UNIT modified its settings

echo Relation settings:

relation-get

echo Relation members:

relation-list

You can delete the relation-name* files now that you have created the real ones needed for this charm.

In case typing all of this is not to your liking, the charm can be found here.

You can now deploy this charm as follows:

• juju bootstrap # bootstraps the system
• juju deploy --repository . mongodb # deploys mongodb
• juju status # to see what's going on 

The above commands satisfy one of our design goals, standalone deployment.  Let's check out the replica sets.  Type this:

• juju add-unit mongodb

And that's all that is needed to add a new mongodb node that will automatically create a replica set with the existing node.  You can continue to "add-unit" to add more nodes to the replica set.  Notice that all of the configuration, is taken care of with the replica-set-relation-joined and replica-set-relation-changed hook scripts that we wrote above.

The beauty of this charm is that the user doesn't really have to know exactly what is needed to get a replica set cluster up and running.  Juju charms are self-contained and idempotent.  This means portability.