posted in DevOps on 2013-08-19 00:00:00 UTC by
When we first set out about using Riak, we chose to develop against the Yokozuna release - an alpha-stage integration of Solr in to the core of Riak (but not “riak-core” as it were), replacing the never-quite-amazing Riak Search. Although still in alpha, we didn’t have time to re-architect our server infrastructure and API once against Riak Search, and then again against Solr once Riak 2.0 is released.
We followed the basic constructs of getting it installed on Ubuntu Linux 12.10 or 13.04 (both of which we run in production for our API servers), and it worked fine.
But the more we researched about the best way to back up Riak, the more we figured out that taking a node down, copying its data (or, in our case, we could snapshot an EBS volume in AWS), spinning the node back up, and then waiting for it to catch up again (and then doing that in a rolling fashion for N nodes) was just silly.
So after chatting again with the awesome team at Wildbit about some of their solutions, we decided to give OmniOS a try. It’s an operating system built by OmniTI on top of illumos, a fork of the relatively-defunct Open Solaris project, by many of the people that worked on Open Solaris in its day.
Needless to say, UNIX is the mainframe of our day. It’s incredibly rock-solid, and every release is basically a Long Term Support release (LTS) because, short of serious security updates, an installation will run solidly - well - forever.
But we didn’t just hop from Linux to UNIX because I’m nostalgic for the Sun Sparcstations and Xterms in the basement of the CoRE building at Rutgers University.
Image: Lovingly stolen from http://eceweb1.rutgers.edu/~bushnell/
Well, maybe a little bit.
But mostly, it was for ZFS.
ZFS allows us to take live-snapshots of any file system (including intelligent diffs instead of full snapshots) and ship them off to a backup server. It means zero downtime for a node with full backups. Wildbit has some of its servers snapshot once per second - that’s how granular you can get with your backups. Restoration is incredibly easy, and did I mention there is zero node downtime?
I’ll write more about our final ZFS snapshot setup when we’ve finalized the architecture. But first, we had to get Riak building on OmniOS - both for development and production.
Before I just spit code on to the page, you should know the biggest issue we had to tackle was that everything was being built 32 bit. OmniOS runs with dual instruction set support (x86 and x86-64), and in many scenarios, defaults to 32 bit. Once we figured out that’s why we were being denied disk access (we were getting “Too many files open” errors even with our open file limit set to 65536) and running in to other issues, getting a final build script became much easier.
Let’s get started
We’ll start by installing the following packages:
NOTE: Thanks to Eric Sproul from OmniTI for pointing out that they maintain root certificates in a package, I’ve moved the CA certificate install in to this list.
1 2 3 4 5 6 7 8 9
# Install compilers, gmake, etc. pkg install pkg:/developer/versioning/git pkg install pkg:/developer/build/gnu-make pkg install pkg:/developer/gcc46 pkg install pkg:/system/header pkg install pkg:/system/library/math/header-math pkg install pkg:/developer/object-file pkg install pkg:/network/netcat pkg install pkg:/web/ca-bundle
These packages replace things like build-essentials, et. al. in Linux. We’re using GCC 4.6 instead of 4.7 (4.7.2 on OmniOS currently) because of a compilation bug that is fixed in 4.7.3 and 4.8 (but there’s no package for either of those versions yet).
Next, download and install Java JDK from Oracle. Make sure to use Oracle’s JDK, not OpenJDK, and you must, must, must download the 64 bit versions. We downloaded the JDK i586 and x64 .gz files and put them in a private S3 bucket for ease of access.
1 2 3
# Download files from S3 in to folder such as /opt/<YOUR NAME>/java gzip -dc jdk-7u25-solaris-i586.gz | tar xf - gzip -dc jdk-7u25-solaris-x64.gz | tar xf -
We have all these fun tools, so let’s set up some PATH’ing:
1 2 3 4
NEWPATH=/opt/gcc-4.6.3/bin/:/usr/gnu/bin:/opt/flyclops/java/jdk1.7.0_25/bin/amd64:/opt/flyclops/java/jdk1.7.0_25/bin:/opt/omni/bin:$PATH export PATH=$NEWPATH echo "export PATH=$NEWPATH" >> /root/.profile
Now we’ll create an unprivileged user to run Riak under, and the directory for holding Riak. We create a directory at /riak, which I’m sure we’ll get yelled at for in the comments:
1 2 3 4 5 6 7 8
# Make riak user useradd -m -d /export/home/riak riak usermod -s /bin/bash riak echo "export PATH=$NEWPATH" >> /export/home/riak/.profile # Create the riak dir mkdir /riak chown riak:other /riak
Our final build tool is kerl - we use it for easily building and installing Erlang.
1 2 3
# Install kerl curl -O -silent https://raw.github.com/spawngrid/kerl/master/kerl; chmod a+x kerl mv kerl /usr/bin
Last but not least, we want to make sure our open file limit is nice and high. There are two ways to do this in OmniOS, this shows both. The first is user-specific and more correct, the other is system-wide:
1 2 3 4 5 6 7 8 9
# Add project settings for riak sudo projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak # Add limits to /etc/system bash -c "echo ‘set rlim_fd_max=65536' >> /etc/system" bash -c "echo ‘set rlim_fd_cur=65536' >> /etc/system" # Set this session's limit high ulimit -n 65536
Building Erlang and Riak as the `riak` user
We want to run the rest of the commands as the unprivileged
riak user. If you’re going to do this by hand, you can simply call
sudo su - riak. If you’re doing this via a shell script, create a second script for building Erlang and Riak, and launch it via your first being run as root. Example:
# Run stage 2 builds as riak user sudo -u riak sh /path/to/riak_build_stage2.sh
Now that we’re
riak we can build Erlang. As of this writing, R15B01 is required for Riak 1.4.1, and R15B02 is required for the latest Yokozuna build. Choose appropriately. Please note the
--enable-m64-build flag - this is very important. It (along with other commands further down) will ensure a 64 bit build, not a 32 bit build.
1 2 3 4 5 6 7 8 9 10
# Build Erlang # MAKE SURE WE USE -enable-m64-build echo 'KERL_CONFIGURE_OPTIONS="-disable-hipe -enable-smp-support -enable-threads -enable-kernel-poll -enable-m64-build"' > ~/.kerlrc kerl build R15B02 r15b02 # Install and activate Erlang mkdir -p /riak/erlang/r15b02 kerl install r15b02 /riak/erlang/r15b02 . /riak/erlang/r15b02/activate
Finally, download (however you choose) a Riak source package and build it. Here we’ll download the Yokozuna 0.8.0 release and build it. Again, note that we’re setting
CFLAGS to use
-m64 to ensure a 64 bit build.
1 2 3 4 5 6 7 8 9 10 11 12
# Get riak cd /riak wget -nv http://data.riakcs.net:8080/yokozuna/riak-yokozuna-0.8.0-src.tar.gz tar zxvf riak-yokozuna-0.8.0-src.tar.gz mv riak-yokozuna-0.8.0-src riak # Build riak cd /riak/riak CC=gcc CXX=g++ CFLAGS="-m64" make # "Install" riak, using "make rel" or "make stagedevrel" CC=gcc CXX=g++ CFLAGS="-m64" make rel
There you have it. Riak (and if you like, Yokozuna) on OmniOS (currently, R151006p). We’ve done all the build experimenting, so hopefully this will save you some time!