posted in DevOps on 2013-06-21 00:00:00 UTC by Dave Martorana
Flyclops’s entire stack runs on AWS, and uses CloudFormation and AutoScale Groups to launch servers on the fly, with no intervention on our part. We don’t use custom AMIs, but vanilla Ubuntu server images (12.10 at the time of writing this) and Ubuntu’s very cool CloudInit functionality to bootstrap building servers with shell scripts. I want to use the same identical method to bootstrap my Vagrant boxes for local development. Here’s how to do it.
(Note: This is a cross-post from here.)
I’ll go further into how we bootstrap - templated shell files that compile in to multipart mime messages, gzipped and put on S3, and a single bootstrap-to-the-bootstrap script being passed in to the server as user data - in another post. But I wanted my local servers to run almost identical boot scripts via Vagrant so my dev environment was almost identical to my production environment.
Ubuntu provides cloud images that are used in EC2, as well as OpenStack, Rackspace, etc., and even have versions for Vagrant. The Vagrant versions have their own cloud-init boot scripts built in to set up networking and whatnot… but I want them to then run my scripts. Obviously.
Vagrant has this really nice thing where you can supply a shell script to run the first time your boxes start. Simply put something similar to this to have it run your file:
# Run our shell script on provisioning config.vm.provision :shell, :path => "vagrant_build.sh"
Next we want to provide a very small shell script that clears out cloud-init run-info and provides us access to have it run again. Here’s the script, we’ll step through it below.
PLEASE NOTE: This method of working with cloud-init does not work on Ubuntu 12.04 or earlier - cloud-init was significantly updated between 12.04 and 12.10.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
# Check to see if we have done this already if [ -f /.vagrant_build_done ]; then echo "Found, not running." exit fi # Make the box think it hasn't init-ed yet rm -rf /var/lib/cloud/instance/* rm -rf /var/lib/cloud/seed/nocloud-net/user-data # Seed our own init scripts cat << 'END_OF_FILE_CONTENTS' > /var/lib/cloud/seed/nocloud-net/user-data Content-Type: multipart/mixed; boundary="===============apiserversStackMultipartMessage==" MIME-Version: 1.0 -===============apiserversStackMultipartMessage== #include https://someS3bucket.s3.amazonaws.com/somefolder/vagrant/someDateStampedFile.gz -===============apiserversStackMultipartMessage==- END_OF_FILE_CONTENTS # Re-run cloud-init cloud-init init cloud-init modules -mode init cloud-init modules -mode config cloud-init modules -mode final # Do not let this run again touch /.vagrant_build_done
First things first - we don’t want this to optionally repeat rerunning cloud-init, so we check for a record that this has run before:
1 2 3 4 5
# Check to see if we have done this already if [ -f /.vagrant_build_done ]; then echo "Found, not running." exit fi
Assuming we don’t find anything, we’re off to the races. Lines 8-9 clean up the existing cloud-init trail, almost re-virgining the machine. That said, we’re going to re-use the meta-data file from the vanilla image’s nocloud-net seed, and simply supply our own user-data. We also clear out the “instance” directory so our new scripts can be written there by cloud-init.
1 2 3
# Make the box think it hasn't init-ed yet rm -rf /var/lib/cloud/instance/* rm -rf /var/lib/cloud/seed/nocloud-net/user-data
Lines 12-23 do the heavy-lifting of moving a script in to place for us. Again, we’re putting this in
and using cat to move the contents of the file for us.
Note: This method of using bash scripts to write files is ugly, but works reliably. I use this method a lot - but I have a custom Jinja2 filter that writes out the ugly for me, keeping my template files nice and clean.
1 2 3 4 5 6 7 8 9 10 11 12
# Seed our own init script cat << 'END_OF_FILE_CONTENTS' > /var/lib/cloud/seed/nocloud-net/user-data Content-Type: multipart/mixed; boundary="===============apiserversStackMultipartMessage==" MIME-Version: 1.0 -===============apiserversStackMultipartMessage== #include https://someS3bucket.s3.amazonaws.com/somefolder/vagrant/someDateStampedFile.gz -===============apiserversStackMultipartMessage==- END_OF_FILE_CONTENTS
Notice on line 18, we’re actually writing out
which tells cloud-init to “include” scripts at that url. According to the docs,
> the content read from the URL can be gzipped, mime-multi-part, or plain text
so I store a gzipped multipart mime file. Works like a charm, and for EC2, gets us around the user-data size limit for supplying scripts to cloud-init.
Now comes the fun - with cloud-init cleaned out and our new user-data in, we simply force-run cloud-init again.
1 2 3 4 5
# Re-run cloud-init cloud-init init cloud-init modules -mode init cloud-init modules -mode config cloud-init modules -mode final
First we initialize it, then run “modules” through initialization, configuration, and finalization. The line
cloud-init modules -mode final
is the most important for us. Through initialization and configuration, cloud-init has downloaded our gzip file from S3, has unzipped and separated out all of the parts of our multipart-mime message in to component shell scripts, but has not yet run those shell scripts. Line 29 asks cloud-init to now run those scripts.
Line 32 is just so that if Vagrant re-executes this build script at any point in time, we don’t run cloud-init through the initialization process again.
So that’s it. Despite the length of this post, it’s a very short script. I should note that you can also add in any other initialization needed specifically for Vagrant boxes after this - like sym-linking in /vagrant (the shared directory) in to a particular location, etc., to finalize the box for local development.