Technology at Flyclops

GoCM: Super-simple asynchronous GCM listener in Go

posted in Code on 2014-03-18 19:44:19 UTC by Dave Martorana

Build Status

We have been using pyapns, a very light-weight, fast, Python/Twisted based asynchronous push notification server for Apple Push Notification Services. It sent our very first push notification ever, and has sent well over 1.5 billion more since. I originally stumbled upon it while reading the Instagram Engineering Blog, and this post in particular.

With pyapns, we toss off an alert asynchronously and never block the API from returning as quickly as possible. It’s rock-solid.

GCM, or Google Cloud Messaging, is the technology that is used to send notifications to Android devices. Unlike APNS, GCM has an HTTP REST interface to GCM servers that is used to send “downstream messages” (server-to-device).

The problem is that sending a message to GCM over HTTP is a synchronous call. Our average API response time is ~70ms, but sending a message synchronously added between 125-250ms on top of that.

So we needed to make GCM just as “fast” as APNS.

It’s likely not obvious yet, but we’re slowly beginning to move pieces of our monolithic Python-based API server application to a multi-service architecture built primarily on Go. At the speed that we’re answering API requests, we’ve hit the need a massively concurrent language that’s also capable of taking advantage of multi-core, multi-CPU systems. But more on this in later post.

No matter what language our API is built on, now or in the future, pyapns will likely continue to run our APNS push notifications. So, we needed something similar for GCM.

Enter GoCM.

GoCM follows some of the basic conventions of pyapns, but almost none of the architecture. It accepts a push request and returns immediately, handing the work off to a series of asynchronous goroutines. It can handle thousands of messages at a time, and doesn’t block the API.

GoCM listens via the HTTP protocol on port 5601 (or one of your choosing) and can accept messages to send from any application written in any language - from a BASH script to Python to C++ and everything inbetween.

It also has a (currently minor) reporting interface, and gracefully handles GCM canonical responses, allowing for them to be handled asynchronously as well.

Open Source

While we rely on a lot of open source software, and contribute code to many OSS projects, GoCM is the first piece of OSS originally authored at Flyclops that is being released for public use. If you’d like to give GoCM a try, head over to our GitHub repository and check it out.

GoCM is enabling us to grow with our Android traffic while still keeping those nice, fast response times. Maybe you can get some use out of it too.

GCM is also beginning to push it’s long-open-connection solution for messaging that uses CSS/XMPP. However, there are several hurdles - first, it requires special device permissions, and with hundreds of thousands of copies of our games in the wild, it would take months to get them all updated to use the newer technology - and until then, we’d have to continue sending messages over the HTTP REST interface. Secondly, we have no need currently for GCM’s XMPP bidirectional protocol.

Amazing, big, portable (and cheap!) whiteboards

posted in Office on 2014-02-18 20:34:05 UTC by Dave Martorana, Parker Whitney

Every business needs whiteboards - large whiteboards. Some go for fancy frosted-glass whiteboards that are absolutely amazing, but require complicated mounting, very careful moving, and a bunch of money ($500-$700 and up).

Even large-sized wood whiteboards are incredibly expensive. We wanted whiteboard space at least 3′ x 5′, and we wanted several of them. The prices were in the $250-700 range, and were flimsy whiteboards requiring wall mounting to have any real sturdiness or stability.

So we got creative, and have some sweet whiteboards to show for it.

We decided we wanted them to be portable (we work at Philly Game Forge and our desk situation is constantly evolving), large, and most importantly, resiliant and self-supporting. I knew you could buy whiteboard paint, and started there. Friends of ours painted whole walls in IdeaPaint, awesome stuff that can cost north of $150 per GALLON.

Then we found this great company - Magic Murals - that sells custom-sized white-board roll-on material that is thick and beautiful. The price was right, so all we needed was something to make in to a whiteboard.

We headed to Home Depot and bought ourselves a couple of doors. 32″ x 80″ was a nice size, and two of them side-by-side would be brilliant. These doors do not come with pre-drilled door knob holes, so they’re perfect. Light (16 lbs), strong and tall, they can simply be leaned against a wall - no mounting needed.

Next we special-ordered the white-board material from Magic Murals. We bought it oversized, of course - 7′ x 3′ (or 84″ x 36″) so when we applied the material we would have overhang we could trim later. Each cut cost us $85. Each door cost $26. So far we’re in for $111.

A quick sanding of the door surface (and some wiping clean with a damp cloth), and the door was ready for application. We laid out the material, and peeled off the back, using eraser-blocks to smooth the material on to the doors.


 

After it was all applied, we let the glue cure to the door, and then using a utility knife, made quick work of trimming the material to the door size.

  

That was it! Now we have two large (7.5′ x 2.5′) white boards that cost $111 each, that we can use in a variety of ways. Want a huge surface? Put them next to each other. Multiple meetings? Split them up and move them to different locations. You can flip the boards to make use of the whole vertical space. You could even mount a rail on the wall, and lift the whiteboards in to place in a conference room. It’s the flexibility of use that makes them amazing.


Most importantly, if as we put these through their paces, we really really love them, we’ll order more material and apply it to the back side of the doors, thus making the whiteboards dual-sided as well.

We couldn’t be happier. We’ll post an update in a few weeks to let you know how they’re standing up.

Using Packer for faster scaling

posted in DevOps on 2014-02-06 00:00:00 UTC by Dave Martorana

We have been playing with Packer for a little while, thanks to the behest of a friend of Flyclops. While I haven’t completely explained how we manage our server stack(s), I’ve hinted before at the following configuration for any of our ephemeral servers that automatically scale up and down with load:

  • AWS CloudFormation templates
  • Vanilla Ubuntu servers
  • Just-in-time provisioning of specific servers at scale/deploy-time using CloudInit

This worked well for a good amount of time. Along with the use of Fabric, a single command-line compiles all of our CloudFormation templates, provisioning scripts, variables for scale alarms and triggers, update our stack parameters and notifications, and push new code to running servers. All of our scripts, config files for servers, etc., are Jinja2 templates that get compiled before being uploaded to Amazon.

The biggest bottle-neck was our scale-up. The process worked like this:

  1. Boot a new EC2 instance at desired size, using a vanilla Ubuntu AMI
  2. Attach server to the load balancer with a long initial wait period
  3. Run a series of in-order shell scripts (via CloudInit) to build the server in to whatever it needs to be
  4. Load balancer detects a fully-functional server
  5. Traffic begins to be routed to the new server

The beauty of our system was also its biggest fault. Our servers never got stale, and as they scaled up and down, benefitted from bug fixes to the core OS. Every server that started handling work was freshly built, and any error in the build process would kill the instance and boot a new server - so the fault-tolerance was fantastic, too.

But the downsides were many as well. The provisioning process was highly optimized and still took over 6 minutes to get a server from boot to answering traffic. Provisioning on the fly required access to several external services (APT repositories, PyPi, Beanstalk/Github, and so-on. Any service disruption to any of these would cause a failed build (and we were unable to scale until the external service issue had been resolved).

Caching as a bad first attempt

We went through several rounds of attempting to remove external dependencies to scaling - from committing code to the pip2pi repository to include the on-the-fly creation of an S3 bucket to serve as a PyPi mirror to bundling git snapshots, etc.

Eventually, the staleness we were trying to avoid was now possible in many other services we were attempting to force caches of on to the AWS network, and we were maintaining a lot more code.

Enter Packer

Packer brought us almost back to our roots. By ripping out massive amounts of support code and adding only a bit to include Packer, we were able to recreate on-the-fly builds of local VMs that were almost identical to those running in our stack. From then, it was a very easy process to pack the VM in to an AMI, and use that AMI, not a vanilla Ubuntu AMI, in scaling. Here’s the entirety of the pack command in our code (note, this is a Fabric command, and uses Fabric function calls):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#
#
@task
@runs_once
def pack(builder=None):
    '''
    Pack for the current
    environment and name
    '''
    if not builder:
        abort('Please provide a builder name for packing')

    _confirm_stack()
    _get_stack_context()

    env.template_context['builder'] = builder

    # This function compiles all of our provisioning
    # shell scripts
    _compile_userdata()

    # This compiles our packer config file
    packer_dir = _get_packer_folder()
    packer_file = join(packer_dir, 'packer.json')
    compiled_packer_file = join(packer_dir, 'packer_compiled.json')
    print 'Using packer file: %s' % packer_file

    # Get the compiled packer file
    packer_compiled_contents = _get_jina2_compiled_string(packer_file)
    with open(compiled_packer_file, 'w') as f:
        f.write(packer_compiled_contents)

    # Move in to the directory and pack it
    local("cd %s && packer build -only=%s packer_compiled.json && cd .." % (
        packer_dir,
        builder
    ))

    # Clean up
    local("rm %s/*" % join(packer_dir, 'scripts'))
    local("rm %s" % compiled_packer_file)
    local("rm -rf %s" % join(packer_dir, 'packer_cache'))

Of course, we need to grab the AMI that packer just created. You can use tagging and all that fun, but here’s a quick way to do it with boto:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
#
#
def _get_ami_for_stack():
    ec2_conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    images = ec2_conn.get_all_images(
        owners='self',
        filters={
            'name': '*%s*' % env.environment # This might be 'production' or 'staging'
        }
    )

    images.sort(key=lambda i: i.name)
    images.reverse()

    ami_image = images[0]
    return ami_image.id

Now we place the Packer-created AMI instance ID in to our CloudFormation template (in the AutoScaleGroup section) and we’re off to the races.

The benefits are plentiful:

  • AMI is built at pack time, not scale time
  • Can still use CloudInit to build, just happens at pack-time, not scale time
  • No reliance on any external services to bring another server online
  • Updated AMI is a command-line statement away
  • Can pack local VMs using identical code
  • Removal of all caching code

But most importantly…

Because of the fact that no code has to run for our services to scale up, we’ve gone from 6 minute scale-up times to approximately the time it takes for the EC2 instance to boot and the ELB to hit it once. That is currently < 60 seconds.

Packer has been a fantastic win for speeding up and simplifying deployments for us. I recommend you look in to it for your own stack.

Shell scripting: logging fractional elapsed time

posted in DevOps on 2013-10-21 18:21:09 UTC by Dave Martorana

I wanted a more robust way to track the amount of time building certain parts of our servers takes. As I describe in another post (and want to dramatically expand on in a future post), we use a series of logically separated shell scripts to build our servers from vanilla instances. So here’s a bit of a tip about logging elapsed time.

Here’s a basic shell script:

1
2
3
4
5
6
7
#!/bin/sh
START=`date +%s`
 
# Do stuff here
 
END=`date +%s`
ELAPSED=$(( $END - $START ))

Seems smart enough - and then I can take the ELAPSED variable and log it or whatever. But this method only works in whole seconds, and that’s silly. So this is a bit more elaborate, but much more exact:

1
2
3
4
5
6
7
#!/bin/sh
START=`date +%s%N`
 
# Do stuff here
 
END=`date +%s%N`
ELAPSED=`echo "scale=8; ($END - $START) / 1000000000" | bc`

This is the difference between “2 seconds” and “2.255322 seconds”, which is important to us, since some shell scripts may only take a fraction of a second.

So what’s going on?

First, we change the date format to

1
date +%s%N

Adding the %N gets us nanoseconds instead of seconds - as whole numbers, that is. (Note: %N doesn’t work on BSD variants.) So we divide by 1 billion, and pipe the calculation string through bc, a unix calculator, and it will return our seconds difference to 8 decimal places.

This changes my output from:

Backup complete in 2 seconds

to:

Backup complete in 2.83732828 seconds

Much better :)

Tagged:
#bash  #bc  #linux  #logging  #sh  #shell  #time  #unix

Replacing "pip bundle"

posted in Code on 2013-09-20 04:31:38 UTC by Dave Martorana

2014-02-06: This solution was a band-aid house of cards, and we eventually ended up moving to Packer, which removes the need to bundle Python dependencies at all.

Our API application is written in Python, using a lightweight framework called Flask, sitting behind nginx proxying to uWSGI. As part of our code-commit process, we bundled all of our Python library dependencies and committed them to our code repo - this removes the dependency on PyPi being up and available (which it often isn’t).

But bundle is going away.

Pip’s bundle command was not very popular, and that (and a lack of ongoing development) has led to the pip maintainers deciding to deprecate the code. When I last bundled our dependencies, I received the following message in the terminal:

1
2
3
4
5
6
7
8
###############################################
##                                           ##
##  Due to lack of interest and maintenance, ##
##  'pip bundle' and support for installing  ##
##  from *.pybundle files is now deprecated, ##
##  and will be removed in pip v1.5.         ##
##                                           ##
###############################################

Well that sucks.

So, not really knowing the alternatives and not finding much with LMGTFY, I turned to Stack Overflow with this question: Is there a decent alternative to “pip bundle”?

TL;DR - check out pip wheel.

Basically, wheel creates (or installs) binary packages of dependencies. What we want to do is create a cache of our dependencies and store them in our source repo.

NOTE: Because the packages are pre-compiled for those that require compiling (think MySql-python, etc.) wheel will create platform-specific builds. If you are developing on OS X and using x86_64 Linux in production, you’ll have to cache your production binaries from Linux, not OS X.

So… Here are the steps.

Continuing to rely on requirements.txt

We need to modify our file just a little bit. If you have a pure dependency list, you’re good to go. If you have any pointers to code repositories (think git) you need a minor change. Here’s the before and after:

1
2
3
4
5
6
riak==2.0.0
riak-pb==1.4.1.1
pytest==2.3.5

-e git+git@your.gitrepo.com:/repo0.git#egg=repo0
-e git+git@your.gitrepo.com:/repo1.git#egg=repo1

Wheel doesn’t respect the -e flag, and has trouble with SSH based git links, so go ahead and put in the https equivalents. There is also no need to name the “egg” as wheel is basically a replacement for eggs.

1
2
3
4
5
6
7
  
riak==2.0.0  
riak-pb==1.4.1.1
pytest==2.3.5

git+https://your.gitrepo.com/repo0.git
git+https://your.gitrepo.com/repo1.git

Cache the dependencies as wheel packages

Now you can call pip to download and cache all of your dependencies. I like to put them in local/wheel (they are .whl, or “wheel” bundles, basically glorified zip files). You will require the “wheel” package for this part, but not for installing the already bundled packages.

Due to the pre-compiled nature of wheel, package names are ended in whichever platform they were compiled. For pure-python packages, which can be installed anywhere, packages end in -none-any.whl. For instance, the boto package for Amazon AWS:

boto-2.3.0-py27-none-any.whl

However, MySql-python and the like, that require binary compilation will result in file names that are platform specific. Note the difference for OS X and Linux (in our case, Ubuntu 13.04):

To cache the wheel packages, run the following line:

1
$ pip install wheel && pip wheel -wheel-dir=local/wheel -r requirements.txt

This isn’t nearly as convenient as say, using pip bundle to create a single requirements.pybundle file, but it works just fine.

Add to git, or whatever you use

Commit the local/wheel directory to your repo, so the bundles are available for you to install at production-time.

Installing on production servers

This is where we ran in to problems. Despite being cached, any git-based packages still go out to git when you run the following command on the production server:

1
$ pip install -use-wheel -no-index -find-links=local/wheels -r requirements.txt

This breaks our desire to not have to rely on any external service for installing requirements. What’s worse is that the package in question is in fact in the ./local/wheel directory. So a little bit of command-line magic, and installing the packages by name works just as well:

1
$ ls local/wheel/*.whl $1 | while read x; do pip install -use-wheel -no-index -find-links=local/wheels $x; done

This basically lists the local/wheel directory, and passes the results in to pip install --use-wheel which also has the --find-links argument that tells pip to look for any dependencies in the local/wheel folder as well. --no-index keeps pip from looking at PyPi.

NOTE: If you have multiple binary packages for different platforms, you’ll have to modify the command above to ignore binary packages that are not built for the specific platform you’re installing to.

Final word

Those are the basics. This can be automated in all sorts of ways - even zipping up all the wheel files in to a single file to get you pretty close to a .pybundle file. It’s up to you - but hopefully this will help as you are torn away from the arms of pip bundle.

2013-09-23: Edited to better represent the binary nature of pre-compiled packages and their platform-specificness.

Tagged:
#bundle  #pip  #python  #servers