Golang in Production
For the last month we have been running a couple services written in Google’s Golang programming language.
We are also currently in the process of creating another new product with a significant portion written in Go.
Most of the Sparkart and story teller products are currently written in Ruby. We have spent years honing the skill set of the engineering and product development team using Ruby as our weapon of choice. We had a new challenge when we decided to launch americas cup on a totally new platform. One of the first things we noticed, even when using fibers and ruby 1.9.3 was that the performance of the system, particularly the CPU and Memory utilization left a lot to be desired. With some tuning we were able to scale the system to the throughput requirements of the site.
Later a feature was pitched to allow the storyteller platform to cache arbitrary URL’s dynamically rewriting the URL so an asset could be served from our CDN. This simple service seemed like a great use case for go.
Moxy is the caching proxy that we developed internally for storyteller, it’s the technology behind the cache transform in our template language Prose. The basic idea is that it takes any URL, signs it with a HMAC and generates a Moxy URL. Creating the service took about an hour. Then we compiled it and deployed it to our test environment. We were blown away by the performance. During testing the application was using kilobytes of memory and very, very little CPU. We were pretty excited.
To date the service has been running for a month continuously, and processed millions of URL’s all while using minimal resources.
A late night emergency and Go to the rescue
The weekend before a major event, we found a severe issue in a clients Wordpress install. The plugin they were using was returning errors with a HTTP status code of 200. Our internal caching system was categorizing these requests as valid, and then destroying the previously cached entry which was causing all kinds of problems on the site. We had just gotten our caching system for storyteller stable and were a bit hesitant to go in and make major changes for one Wordpress site. However since we already had Moxy running, it seemed like a reasonable workaround to update it to handle this one Wordpress site until we could analyze the plugin and make an update.
Changing Moxy to handle proxying and analyzing the JSON response took about two hours, including testing and deploying the service on a weekend. Thankfully we had come across a post on the Bitly blog regarding go-simplejson a library they created in Go for dealing with arbitrarily complex JSON objects.
All in all we have been really happy with our experiments in go. We are currently working on updating the goamz package to include dynamoDB support.