Building Riak on OmniOS

When we first set out about using Riak, we chose to develop against the Yokozuna release – an alpha-stage integration of Solr in to the core of Riak (but not “riak-core” as it were), replacing the never-quite-amazing Riak Search. Although still in alpha, we didn’t have time to re-architect our server infrastructure and API once against Riak Search, and then again against Solr once Riak 2.0 is released.

We followed the basic constructs of getting it installed on Ubuntu Linux 12.10 or 13.04 (both of which we run in production for our API servers), and it worked fine.

But the more we researched about the best way to back up Riak, the more we figured out that taking a node down, copying its data (or, in our case, we could snapshot an EBS volume in AWS), spinning the node back up, and then waiting for it to catch up again (and then doing that in a rolling fashion for N nodes) was just silly.

So after chatting again with the awesome team at Wildbit about some of their solutions, we decided to give OmniOS a try. It’s an operating system built by OmniTI on top of illumos, a fork of the relatively-defunct Open Solaris project, by many of the people that worked on Open Solaris in its day.

Needless to say, UNIX is the mainframe of our day. It’s incredibly rock-solid, and every release is basically a Long Term Support release (LTS) because, short of serious security updates, an installation will run solidly – well – forever.

But we didn’t just hop from Linux to UNIX because I’m nostalgic for the Sun Sparcstations and Xterms in the basement of the CoRE building at Rutgers University.

Well, maybe a little bit.

But mostly, it was for ZFS.

ZFS allows us to take live-snapshots of any file system (including intelligent diffs instead of full snapshots) and ship them off to a backup server. It means zero downtime for a node with full backups. Wildbit has some of its servers snapshot once per second – that’s how granular you can get with your backups. Restoration is incredibly easy, and did I mention there is zero node downtime?

I’ll write more about our final ZFS snapshot setup when we’ve finalized the architecture. But first, we had to get Riak building on OmniOS – both for development and production.

Before I just spit code on to the page, you should know the biggest issue we had to tackle was that everything was being built 32 bit. OmniOS runs with dual instruction set support (x86 and x86-64), and in many scenarios, defaults to 32 bit. Once we figured out that’s why we were being denied disk access (we were getting “Too many files open” errors even with our open file limit set to 65536) and running in to other issues, getting a final build script became much easier.

Let’s get started

NOTE: Many, many thanks go to to the people on the #riak and #omnios IRC channels on Freenode, as well as the Riak Users and OmniOS Discuss mailing lists for all of their help.

We’ll start by installing the following packages:

NOTE: Thanks to Eric Sproul from OmniTI for pointing out that they maintain root certificates in a package, I’ve moved the CA certificate install in to this list.

# Install compilers, gmake, etc.
pkg install pkg:/developer/versioning/git
pkg install pkg:/developer/build/gnu-make
pkg install pkg:/developer/gcc46
pkg install pkg:/system/header
pkg install pkg:/system/library/math/header-math
pkg install pkg:/developer/object-file
pkg install pkg:/network/netcat
pkg install pkg:/web/ca-bundle

These packages replace things like build-essentials, et. al. in Linux. We’re using GCC 4.6 instead of 4.7 (4.7.2 on OmniOS currently) because of a compilation bug that is fixed in 4.7.3 and 4.8 (but there’s no package for either of those versions yet).

Next, download and install Java JDK from Oracle. Make sure to use Oracle’s JDK, not OpenJDK, and you must, must, must download the 64 bit versions. We downloaded the JDK i586 and x64 .gz files and put them in a private S3 bucket for ease of access.

# Download files from S3 in to folder such as /opt/<YOUR NAME>/java
gzip -dc jdk-7u25-solaris-i586.gz | tar xf -
gzip -dc jdk-7u25-solaris-x64.gz | tar xf -

We have all these fun tools, so let’s set up some PATH’ing:

NEWPATH=/opt/gcc-4.6.3/bin/:/usr/gnu/bin:/opt/flyclops/java/jdk1.7.0_25/bin/amd64:/opt/flyclops/java/jdk1.7.0_25/bin:/opt/omni/bin:$PATH 
export PATH=$NEWPATH
echo "export PATH=$NEWPATH" >> /root/.profile

Now we’ll create an unprivileged user to run Riak under, and the directory for holding Riak. We create a directory at /riak, which I’m sure we’ll get yelled at for in the comments:

# Make riak user
useradd -m -d /export/home/riak riak
usermod -s /bin/bash riak
echo "export PATH=$NEWPATH" >> /export/home/riak/.profile

# Create the riak dir
mkdir /riak
chown riak:other /riak

Our final build tool is kerl – we use it for easily building and installing Erlang.

# Install kerl
curl -O --silent https://raw.github.com/spawngrid/kerl/master/kerl; chmod a+x kerl
mv kerl /usr/bin

Last but not least, we want to make sure our open file limit is nice and high. There are two ways to do this in OmniOS, this shows both. The first is user-specific and more correct, the other is system-wide:

# Add project settings for riak
sudo projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak

# Add limits to /etc/system
bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"
bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"

# Set this session's limit high
ulimit -n 65536

Building Erlang and Riak as the `riak` user

We want to run the rest of the commands as the unprivileged riak user. If you’re going to do this by hand, you can simply call sudo su - riak. If you’re doing this via a shell script, create a second script for building Erlang and Riak, and launch it via your first being run as root. Example:

# Run stage 2 builds as riak user
sudo -u riak sh /path/to/riak_build_stage2.sh

Now that we’re riak we can build Erlang. As of this writing, R15B01 is required for Riak 1.4.1, and R15B02 is required for the latest Yokozuna build. Choose appropriately. Please note the --enable-m64-build flag – this is very important. It (along with other commands further down) will ensure a 64 bit build, not a 32 bit build.

# Build Erlang
# MAKE SURE WE USE --enable-m64-build
echo 'KERL_CONFIGURE_OPTIONS="--disable-hipe --enable-smp-support --enable-threads --enable-kernel-poll --enable-m64-build"' > ~/.kerlrc
kerl build R15B02 r15b02

# Install and activate Erlang
mkdir -p /riak/erlang/r15b02
kerl install r15b02 /riak/erlang/r15b02
. /riak/erlang/r15b02/activate

Finally, download (however you choose) a Riak source package and build it. Here we’ll download the Yokozuna 0.8.0 release and build it. Again, note that we’re setting CFLAGS to use -m64 to ensure a 64 bit build.

# Get riak
cd /riak
wget -nv http://data.riakcs.net:8080/yokozuna/riak-yokozuna-0.8.0-src.tar.gz
tar zxvf riak-yokozuna-0.8.0-src.tar.gz
mv riak-yokozuna-0.8.0-src riak

# Build riak
cd /riak/riak
CC=gcc CXX=g++ CFLAGS="-m64" make

# "Install" riak, using "make rel" or "make stagedevrel"
CC=gcc CXX=g++ CFLAGS="-m64" make rel

There you have it. Riak (and if you like, Yokozuna) on OmniOS (currently, R151006p). We’ve done all the build experimenting, so hopefully this will save you some time!

  • Theo Schlossnagle

    Out of curiosity, why do you disable hipe in your erlang build?

  • Eric Sproul

    OmniOS ships CA certs in pkg:/web/ca-bundle, including both the individual cert files (/etc/ssl/certs) as well as the single PEM (/etc/ssl/cacert.pem). We extract them from the same version of Mozilla NSS that we ship so that regardless of whether you use the files directly or use the NSS libs, you see the same set.

    • http://davemartorana.com Dave Martorana

      Thanks @ericsproul:disqus! I updated the post to install that package instead. Cheers!

  • Hari John Kuriakose

    Did anyone face any serious performance issues? For me, Basho Bench is giving me much lower iops for OmniOS when compared to Ubuntu. My Riak cluster comprises of 5 Amazon EC2 large instances with 8GB ram each. I did all the AWS tuning proposed by Basho, and I am using the riak-js client library.

    • Guest

      Turns out the lower iops was mainly due to different configuration of the Amazon EBS volumes. Had a nice go with OmniOS and ZFS, but ultimately settled with Ubuntu itself. Wanted to clear the suspicion regarding the performance, in case someone got any wrong impression.