One of the most baffling parts of Amazon Web Services’ service is that, while Route 53 supports AAAA records, almost none of AWS supports IPv6 addresses. That’s slooooowly changing, (CloudFront, for instance, AWS’s CDN, just received support), but which services do and don’t support it are very unclear at best, and disastrous at worst.

DISASTROUS

We run DNS through Route53 (R53), and build our server stacks on top of CloudFormation (CF), AWS’s stack-description templating service. Our CF template describes the R53 DNS records and how they should point to Elastic Load Balancers (ELB) that sit in front of our public-facing servers. Our ELBs - our entire stack - sits inside of a Virtual Private Cloud (VPC), which is not only the best way to run AWS resources, but, it turns out, the only way AWS will let new accounts run servers at all.

For those of us that have been using AWS for years, there is AWS “Classic” which runs EC2 servers and ELBs as stand-alone servers. Bizarrely, ELBs in “Classic” mode will answer requests over an IPv6 IP address, while newer ELBs inside a VPC won’t. However, even more bizarrely, CloudFormation and Route53 will allow you to set up AAAA records pointing to ELBs inside of a VPC.

See where this breaks? AWS’s own services will gladly set up an IPv6 DNS entry, point it to an ELB, and then… drop all connections across that IP address.

WHAT??

Yeah. So, just about every mobile operating system looks up and uses IPv4 addresses. Many resources now have both IPv4 and IPv6 addresses, but those IPv6 addresses, while being the future of the internet, are relatively ignored.

This fall, Apple made a change to iOS in their iOS 10 release, making networking in iOS 10 prefer IPv6 addresses. The DNS servers on AT&T’s mobile internet connections (unlike Verizon, Sprint, T-Mobile, etc) have been upgraded to start replying with IPv6 addresses where available.

Because CloudFormation assigned IPv6 addresses in DNS to our ELBs, DNS responded to iOS 10 devices on AT&T’s mobile network with those IPv6 addresses. However, when our software then tried to issue a call to our backend API, AWS unceremoniously dropped the connections outright.

So for at least a couple weeks, as people quietly upgraded their phones to iOS 10, we had tens of thousands of players that could play our multi-player games on their home WIFI, but not while using mobile data.

It took us some time to track down the issue - only on iOS 10, only on AT&T, only on mobile data. While I don’t have exact figures, there was a reduction in play, and most certainly a direct impact on revenue.

What should happen?

Route53 should deny AAAA records using AWS’s proprietary pointer-type record to ELB endpoints in a VPC - or any endpoint, for that matter, in the array of services they provide and support that don’t yet play well with IPv6.

Takeaway

BE VERY CAREFUL rolling out support for IPv6 if running your server stack on AWS - inside or outside a VPC. Support is spotty at best, and Amazon’s own lack of understanding of the severity of issues caused by their lackadaisical application of IPv6 addresses to services that have little-to-no support for those addresses can cause huge headaches.

Update 2016-11-30: uploaded a new screenshot of the file watcher screen in 2016.3

IntelliJ IDEA is arguably the best Go IDE on the planet. The go-lang-idea-plugin turns IDEA into a full-featured IDE, and does so incredibly well. In fact, since moving to it, I’m not sure how I used anything else.

In any case, the one thing really missing from the IDE is auto-formatting with gofmt or formatting+imports with goimports - but it’s pretty easy to get working.

Setup

First, install IDEA and the go-lang-idea-plugin… plugin. Next, install the “File Watchers” plugin from Intellij:

Automatically calling goimports

Unfortunately, you’ll need to set up a file watcher for each project, but it’s not too tedious. With your project open, go to the Preferences screen, and open the “Tools” tree. Select “File Watchers” and click the + button to create a new file watcher.

Fill in the information similarly to below (make sure “Program” points to your goimports binary):

A few tips:

• Putting -w in the “Arguments” field will tell goimports to overwrite the file, not dump its output to stdout.
• Make sure to un-check “Immediate file synchronization” or the IDE will spaz out on you
• I like to check “Trigger watcher regardless of syntax errors,” but YMMV

That’s it. Click save on any document in your project, and (assuming there are no errors that break goimports) your document will be formatted and imports will be automatically filled in.

This last little step is all that Sublime Text or Atom (or Vim-Go) had on IntelliJ IDEA. After this, in my humble opinion, IntelliJ IDEA stands alone as the best Go IDE out there.

Note: To use gofmt instead of goimports, simply replace the “Program” field with the path to gofmt instead of goimports and you’ll get the results you’re looking for.

Attempting to put my money where my mouth is, I just finished moving all three of our websites to a new, static-only, HTTPS-secured infrastructure.

Most recent setup

I moved the flyclops.com site to a statically-rendered site some time ago, serving it over Amazon S3. That went fairly well, but it’s a very minimal website, and doesn’t require anything dynamic.

The tech blog, which you’re reading now, was hosted on Wordpress, as are most blogs in this world. But over time, “features” became unnecessary (or even fairly useless) - think comments. Other features, like caching plug-ins, were nothing more than a means to an end - near-statically loading sites. But it was kludgy.

Our press kit site was based on presskit() 2.0, a PHP-based press-kit solution that is fantastic.

Enter FREE SSL

LetsEncrypt was released out of beta. This was a turning point. I spend a lot of time making noise about three-letter agencies and snooping and the necessity of encryption across as much of life as possible. Considering Flyclops is a profitable enterprise, there wasn’t much excuse for not having HTTPS across all our sites to begin with. With LetsEncrypt being production-ready, there was no more excuse at all.

Shortly thereafter, Amazon AWS released AWS Certificate Manager. AWS Certificate Manager offers free, auto-renewing SSL certificates for anything hosted on AWS services. That was now two free SSL services to choose from. Being hosted almost entirely on AWS, ACM was a natural choice - it’s also easier to manage from a renewal standpoint, in that there is no renewal overhead as with LetsEncrypt. There are plenty of tools that make LetsEncrypt renewing almost entirely hands-off, but that would still be something to manage.

What to do with PHP?

Now we had to choose - spin up servers to run PHP-based websites in AWS? That sounded plenty unnecessary. Instead, I took a closer look at Hugo, the Go-based static site generator I had glanced at when I moved the main website to a static site. I quickly ported the static site to Hugo, and was very happy with the process. (It helps that we recently finished moving our entire backend infrastructure to Go-written services - Hugo felt like a natural fit after that.)

I became determined to get our blog there as well, and installed the Wordpress to Hugo Exporter plug-in (thank you all you wonderful open-source devs out there). I exported my content, and spent about a day or so setting up templates, partials, taxonomies, and archetypes in Hugo. The process wasn’t entirely painless, but “minimally” painful. All-in-all, easier than I expected.

All that was left was the press kit. Lo-and-behold, I was not the first person to want to push the wonderful presskit() to a static environment, and I happened upon dopresskit-static on Github. A fork and a few tweaks later, and I had all our websites generating static output.

Hosting

Static sites need static hosting, and S3 was a natural fit… except it really isn’t a totally natural environment. To boot, you can’t use AWS Certificate Manager-based SSL certificates with S3. Besides, Amazon has this sweet CDN service called Cloudfront that would deliver static content so much faster than S3.

The setup

First, I generated the following SSL certificates in AWS Certificate Manager:

• One for flyclops.com and www.flyclops.com
• One wildcard certificate for *.flyclops.com

I set up three S3 buckets (one per site) to act as origins for content. I then set up three Cloudfront distributions, one for each bucket, with the appropriate CNAMEs. Cloudfront distributions let you specify:

• CDN origin - an S3 bucket, or BYO
• A secure certificate from AWS Certificate Manager
• CNAMEs for the distribution
• A default file to serve (index.html)
• What to do/serve for different HTTP error codes (404 page). Hint: Depending on how you set up S3 origins, they will return 403 errors for missing documents, not 404.
• Whether to serve both HTTP/HTTPS, HTTPS only, or redirect HTTP to HTTPS (we use redirect)

Distributions take a few minutes to set up, and once they were published, I went into DNS (conveniently, Route53 on AWS) and updated the flyclops.com, www.flyclops.com, presskit.flyclops.com, and tech.flyclops.com to point to the appropriate CDN distributions.

Publishing

All that was left to do was to set up publishing. At first, I just assumed I’d push public content to S3 manually as I updated it. But then I took a look at the “Deployments” capabilities of our git service, Beanstalk by Wildbit. It turns out, Beanstalk’s deployments are incredibly powerful and feature-rich. Publishing, selectively, even, a git branch to an S3 bucket, with pattern-matched inclusions and exclusions, automatically or manually, was extremely easy to set up.

(Don’t use Beanstalk for git hosting? Check out DeployBot!!)

And bonus - it will automatically invalidate any updated files on Cloudfront! That is spectacular!

Click the image above for a look at all the features

Now I write content, use Hugo to generate the static version of the site, commit the changes to git, push them to my Beanstalk remotes, and a few seconds later, they’re published!

So now, at least, my money is now firmly where my mouth is - all of our sites are HTTPS only. To boot, they’re completely static, and CDN-served from an edge node close to you.

Now we can concentrate on more important things, and not worry about constant security updates to Wordpress and PHP, hosting issues, SSL costs or renewals, site or page speeds, ease of crawl-ability for web crawlers, and a host of other things.

Not too shabby for a couple days work.

This is the easiest way to install NSQ on FreeBSD. There are other, probably more proper ways of doing so, but it’s as close as you can get to the Linux install instructions.

First, you’ll need bash if you don’t already have it:

1
2
3
4
5

# Install BASH for installation
pkg install -y bash

# Install Go
pkg install -y go

Make sure you set up your Go paths:

1
2
3

# Set up Go paths
export GOPATH="<gopath>"
export PATH=$PATH:$GOPATH/bin

Next, install GPM, the package manager Bitly chose to use with NSQ. In order to install GPM and use it, you’ll need to temporarily mount fdescfs, the file-descriptor file system for FreeBSD. You don’t need this as a permanent mount, so I use it in the install script only:

1
2
3
4
5
6

# Install GPM
git clone https://github.com/pote/gpm.git gpm
cd gpm
git checkout -b v1.3.2
./configure
make install

Now that we have GPM set up, we can move to NSQ:

1
2
3
4
5
6

# Install
go get -d github.com/bitly/nsq || true
cd "$GOPATH/src/github.com/bitly/nsq"
git checkout -b v0.3.5
gpm install
go get ./...

That’s it. nsqd, nsqlookup, etc., will be in your $GOPATH/bin directory.

We use Sublime Text (²⁄₃) pretty heavily at Flyclops.

We (and especially I) deal with a lot of JSON files. In my case, the majority of our backend is defined by AWS CloudFormation template files - and man, can they be long. So I wanted a way to browse them quickly.

So, I wrote this wonky, almost-working plugin for Sublime Text 3 (which is a “beta” but substantially better than 2) called JsonTree. It works by hitting a key combination (cmd-ctrl-j in OS X, or ctrl-alt-j in Windows/Linux) and brings up a tree-view of your JSON document that is browsable and searchable. Screenshot:

JsonTree has been accepted into the Sublime Package Repository. Grab the Package Control plugin and install JsonTree from there.

This is the second open-source code bit from Flyclops.