Two years ago, 99designs had localized sites for a handful of English speaking countries, and our dev team had little experience in multilingual web development. But we felt that translating our site was an important step, removing yet another barrier for designers and customers all over the world to work together. Today we serve localized content to customers in 18 countries, across six languages. Here’s how we got there, and some of the road blocks we ran into.

Starting local

The most difficult aspect to internationalizing is language, so we started with localization: everything but language. In particular, this means region-appropriate content and currency. A six-month development effort saw us refactor our core PHP codebase to support local domains for a large number of countries (e.g. 99designs.de), where customers could see local content and users could pay and receive payments in local currencies. At the end of this process, each time we launched a regional domain we began redirecting users to that domain from our Varnish layer, based on GeoIP lookups. The process has changed little since then, and continued to serve us well in our recent launch in Singapore.

Languages and translation

With localization working, it was time to make hard decisions about how we would go about removing the language barrier for non-English speakers (i.e. the majority of the world).There were a lot of questions for us to answer.

• What languages will we offer users in a given region?
• How will users choose their language?
• How will we present translated strings to users?
• How will strings be queued for translation?
• Who will do the translation?

What languages to offer?

Rather than making region, language and currency all user selectable, we chose to restrict language and currency availability to a user’s region. This was a trade-off which made working with local content easier: if our German region doesn’t support Spanish, we avoid having to write Spanish marketing copy for it. Our one caveat was for all regions to support English as a valid language. As an international language of trade, this lessens any negative impact of region pinning.

Translating strings

There were two main approaches we considered for translation: use a traditional GNU gettext approach and begin escaping strings, or else try a translation proxy such as Smartling. gettext had several advantages: it has a long history, and is well supported by web frameworks; it’s easily embedded; and translations just become additional artifacts which can be easily version controlled. However, it would require a decent refactoring of our existing PHP codebase, and left open issues of how to source translations.

In Smartling’s approach, a user’s request is proxied through Smartling’s servers, in turn requesting the English version of our site and applying translations to the response before the user receives it. When a translation is missing, the English version is served and the string is added to a queue to be translated. Pulling this off would mean reducing substantially the amount of code to be changed, a great win. However, it risked us relying on a third-party for our uptime and performance.

In the end, we went with Smartling for several reasons. They provided a source of translators, and expertise in internationalization which we were lacking. Uptime and performance risks were mitigated somewhat by two factors. Firstly, Smartling’s proxy would be served out of the US-East AWS region, the same region our entire stack is served from, increasing the likelihood that their stack and ours would sink or swim together. Secondly, since our English language domains would continue to be served normally, the bulk of our traffic would still bypass the proxy and be under our direct control.

Preparing our site

We set our course and got to work. There was substantially more to do than we first realized, mostly spread over three areas.

Escaping user-generated content

Strings on our site which contained user content quickly filled our translation queue (think “Logo design for Greg” vs “Logo design for Sarah”). Contest titles, descriptions, usernames, comments, you name it, anything sourced from a user had to be found and wrapped in a <span class="sl_notranslate"> tag. This amounted to a significant ongoing audit of the pages on our site, fixing them as we went.

Preparing Javascript for translation

Our Javascript similarly needed to be prepared for translation, with rich client-side pages the worst hit. All strings needed to be hoisted to a part of the JS file which could be marked up for translation. String concatenation was no longer ok, since it made flawed assumptions about the grammar of other languages. Strings served through a JSON API were likewise hidden from translation, meaning we had to find other ways to serve the same data.

Making our design more flexible

In our design and layout, we could no longer be pixel-perfect, since translated strings for common navigation elements were often much longer in the target language. Instead, it forced us to develop a more robust design which could accommodate the variation in string width. We stopped using CSS transforms to vary the case of text stylistically, since other languages are more sensitive to case changes than English.

The wins snowball

After 9 months of hard work, we were proud to launch a German language version of our site, a huge milestone for us. With the hardest work now done, the following 9 months saw us launch French, Italian, Spanish and Dutch-language sites. Over time, the amount of new engineering work reduced with each launch, so that the non-technical aspects of marketing to, supporting and translating a new region now dominate the time to launch a new language.

The challenges

We also encountered several unexpected challenges.

Client-side templating

We mentioned earlier that the richer the client-side JS, the more work required to ensure smooth translation. The biggest barrier for us was our use of Mustache templates, which were initially untranslatable on the fly. To their credit, Smartling vastly improved their support for Mustache during our development, allowing us to clear this hurdle.

Translating non-web artifacts

It should be no surprise: translation by proxy is a strategy for web pages, but not a strong one for other non-web artifacts. In particular, for a long time translating emails was a pain, and in the worst case consisted of engineers and country managers basically emailing templates for translation back and forward. After some time, we worked around this issue by using Smartling’s API in combination with gettext for email translation.

Exponential growth of translation strings

Over time, we repeatedly found our translation queue clogged with huge numbers of strings awaiting translation. Many of these cases were bugs where we hadn’t appropriately marked up user-generated content, but the most stubborn were due to our long-tail marketing efforts. Having a page for each combination of industry, product category and city led to an explosion of strings to translate. Tackling these properly would require a natural language generation engine with some understanding of each language’s grammar. For now we’ve simply excluded these pages from our translation efforts.

The future

This has been an overview of the engineering work involved in localizing and translating a site like ours to other languages. Ultimately, we feel that the translation proxy approach we took cut down our time to market significantly; we’d recommend it to other companies who are similarly expanding. Now that several sites are up and running, we’ll continue to use a mix of the proxy and gettext approaches, where each is most appropriate.

We’re proud to be able to ship our site in multiple languages, and keen to keep breaking down barriers between businesses and designers wherever they may be, enabling them to work together in the languages in which they’re most comfortable.

Discuss on Hacker News or Reddit.

I recently found myself wanting the features of the rails asset pipeline in my golang project at work. Since there isn’t much in the way of asset pipelining for golang yet, I built it. Turns out, sprockets is really easy to integrate. Here’s how you can go about setting it up for your project.

Assets in development

First things first - lets get it to the ‘it works on my machine’ stage. I’ve put together a sample repo using the asset pipeline, which you can use as a guide.

The setup for your app will be similar:

• The assets folder contains your stylesheets, javascript, etc (this directory name is set in sprockets/environment.rb).
• You’ll need a similar Rakefile to build assets (and maybe launch the server)
• You might store the sprockets directory somewhere else - update the Rakefile to match.
• Use a Gemfile and the bundler rubygem to manage dependencies.
• Edit the rakefile to change the port the asset server runs on.

When your app starts up (in development), it should make a request to http://localhost:11111/assets/manifest.json, which provides a JSON hash linking asset names (e.g. “application.css”) to the relative URLs the compiled assets can be fetched from. To generate a link to an asset in your app, use the JSON hash you fetched to lookup the URL. For example, the URL for “application.css” this might look like http://localhost:11111/application-8e5bf6909b33895a72899ee43f5a9d53.css.

That should be all you need for development - you should be able to see SASS/Coffeescript assets compiled and loading normally. Hooray!

Assets in production

For production we want to pre-compile assets rather than regenerating them each time they change.

rake assets will create a ‘public’ folder containing ‘manifest.json’ (same format as before). Get this directory onto your production servers. git add -Af public/ will add it to source control if you deploy via git.

When generating a link to an asset, simply look up manifest.json from the filesystem rather than from HTTP.

Fin

If you’ve followed these steps, you’ll have a fullly functioning asset pipeline for your golang project. The whole thing, including deployment, took me well under a day to add to our app. The resulting assets are minified, concatenated, and gzipped (for size). They are also fingerprinted, so you can serve them with an unlimited cache lifetime and reap the benefits.

Although I set this up for golang, there’s nothing go-specific about it. The same technique works just as well for any language or framework without a mature asset pipeline. If you find yourself in need, just use this pattern and you can be up and running in no time.

At 99designs, we try to make sure we’re always fixing bugs as well as writing code. It can be easy to neglect bugs when you’re busy churning out new features.

We use GitHub issues to track bugs in our various applications. GitHub issues integrate well with our codebase, commits and pull requests, but the reporting facilities are a bit limited.

As our team grows, it’s become increasingly important for us to be able to answer key questions about bugs, including:

• How many bugs are currently open?
• Have we each remembered to spend time working on bug fixes this sprint?
• Are we closing more bugs than we’re opening?

To help answer these questions, a few of our team spent a number of hack days implementing a bug dashboard named GitHub Survivor.

Unlike the similarly-named reality TV show, GitHub Survivor doesn’t feature eliminations, gruelling physical challenges, or Jeff Probst. However, it does pit developers against one another — in a light-hearted way.

We display GitHub Survivor on a big screen in the office, where all the team can see it. We’ve found it helps keep our minds on bugs — it reminds us to make a small effort every sprint, gradually bringing the bug count closer to zero.

A bug leaderboard occupies the bulk of the screen. It shows who’s closed the most bugs this sprint (may they be laden with Praise and Whisky!) and who’s forgotten to spend some time fixing bugs (may they toil in the maintenance of a thousand Malbolge programs!).

There are charts showing the number of bugs opened and closed in recent sprints, the open bug count over time, and a big indicator showing the current open bug count.

The source is available for you to inspect and adapt to your needs. Please try it out, make improvements and contribute them back! We hope you find it useful.

We’re passionate about building high-quality software at 99designs, and this is just one way we measure whether we’re doing a good job of that. If you’re similarly interested in building cool things in an awesome environment, check out our open positions!

Join the discussion at Hacker News and Reddit.

What if it were possible to call methods with spaces in their name directly from Ruby?

If you’ve seen Gary Bernhardt’s awesome talk where he digs into some of the quirks of Ruby, you’ll know that it’s pretty trivial to get bare words in Ruby:

1
2
3
4
5
6

1.9.3-p286 :001 > def method_missing(*args)
1.9.3-p286 :002?>   args.join(" ")
1.9.3-p286 :003?>   end
 => nil
1.9.3-p286 :004 > ruby has bare words
SystemStackError: stack level too deep

Wait.. What?

Disappointing to say the least. Obviously something is amiss. It turns out this is just a quirk in irb; if you try instead

1
2
3
4
5
6
7

1.9.3-p286 :001 > def self.method_missing(*args)
1.9.3-p286 :002?>   args.join(" ")
1.9.3-p286 :003?>   end
=> nil
1.9.3-p286 :004 > ruby has bare words
=> "ruby has bare words"
1.9.3-p286 :005 >

Cool, so what else can we do with this? It’s trivial to define a method with a space in its name, and calling it isn’t terribly difficult:

1
2
3
4
5
6
7
8

1.9.3-p286 :005 > self.class.send(:define_method, :"i have a space") do
1.9.3-p286 :006 >     puts "I has a space"
1.9.3-p286 :007?>   end
=> #<Proc:0x007ff89c1e0b58@(irb):5 (lambda)>
1.9.3-p286 :008 > send(:"i have a space")
I has a space
=> nil
1.9.3-p286 :009 >

But having created such a monstrosity, how do you call it from the repl? Or for that matter, from an actual Ruby program? This is obviously something you should be doing in production…

1
2
3
4
5
6
7
8
9
10
11
12
13

self.instance_exec do
def method_missing(sym, *args)
  # Splat args if passed in from a parent call
  if args.length == 1 && args[0].is_a?(Array) && args[0][0].class == NameError
    args = args[0]
  end

  method_names, arguments = args.partition { |a| a.class == NameError }
  method([sym.to_s, *method_names.map(&:name)].join(" ")).call(*arguments)
rescue NameError => e
  return [e, *arguments]
end
end

Bam. You may be looking at this baffled (or if you’re reasonably tight with metaprogramming in Ruby, sharpening/setting fire to something with a view to causing me significant bodily harm).

Walking through this, we first of all act on whatever self is; in most cases this will be the local scope. If we didn’t do this, we’d be defining the method on Object, which can cause all kinds of headaches when you’re trying to debug.

Immediately after this, we unpack arguments if they look like they were created by an earlier instance of this method. This is unwieldy, but unfortunately Ruby’s single return values and the recursion we’re employing here make it necessary. We could definitely define a subclass of Array to make the test cleaner and the implementation more robust, but I preferred to keep this as short as possible and use the bare minimum number of Ruby primitives.

Once we’ve unpacked our arguments, we do the real magic. First off, we split our arguments into NameErrors, the container we’re using for our missing method names, and everything else (the legitimate arguments we were called with).

We try to find a method with the current name (as we’ll be building our method name right to left with recursive calls to method_missing), and failing that we pack up our current attempt with our arguments, and return it for the next pass.

There are enough issues with this (if you defined the methods foo bar baz and bar baz, a call to foo bar baz would call foo with bar bazs return) to make it unwieldy. On the other hand; if those bugs are the only thing stopping you from putting this into production, you’ve probably got larger issues.

If this large scale abuse of the language excites you, you might be interested to know that we’re hiring.

At this point you’re probably eager to know.. does it work?

1
2
3
4
5
6
7
8
9

1.9.3-p286 :001 > load "bare_words.rb"
1.9.3-p286 :002 > self.class.send(:define_method, :"i has a space") do |name, greeting|
1.9.3-p286 :003 >     puts "#{greeting}, #{name}!"
1.9.3-p286 :004?>   end
=> #<Proc:0x007fc6b41872c0@(irb):2 (lambda)>
1.9.3-p286 :005 > i has a space "richo", "Hello"
Hello, richo!
=> nil
1.9.3-p286 :006 >

Join the discussion at Hacker News and Reddit.

If you had happened to have been wandering around the 99designs office today you would have heard hysterical laughter and cries of “yarrrr”.

Something we try to do is foster a good “DevOps” working culture. One of the critical components of DevOps is a collaborative way of working and close working relationship between those working in development and operations. And other parts of the business for that matter. Company culture is something that is tricky to improve if it’s not working, but immediately obvious when it is working.

A key component of good company culture is good communication. We make extensive use of IRC as a communication medium. Naturally we use it for technical discussions like “How does this component of code work”, but just like companies like GitHub and Etsy we use IRC within the company as a very effective method of documenting what both dev and ops are doing day to day. We also have a bot that lives in IRC and does useful things for us. Our bot is called agent99. She makes our life easier and has the ability to do such things as deploy new versions of code to our production website.

She also has a bit of character. She can find memes to punctuate the moment, or fetch funny pictures of cats when asked.

We’ve actually open-sourced agent99 as well as many other pieces of code. We like to contribute code that we think might be useful to others back to the community.

So what was the source of hysterical laughter? Well today is Talk Like a Pirate Day. So one of our staff hacked together a plugin to make agent99 send a “yarr” over the office speakers. Hilarity ensued. It was quicky repatched, so that “yarr” only works one golden day a year.

This is a little example of where all are empowered to use technology to make it a more enjoyable workplace. And a work environment that is both fun and technically challenging is a competitive advantage in that helps attract and retain motivated and happy employees!

Here’s a sneak peek of some experimental stuff we’ve been working on at 99designs.
99designs color explorer is a tool for browsing designs by color.

Nerdy technical details

For those interested in the technical details, color explorer uses colorific to extract colors from logo designs (We’ve written more about colorific previously). It’s also using colordb, a side project of mine designed to efficiently index and search by color using some fancy perceptual nearest-neighbour algorithms.

Colordb turned out to be super easy to implement thanks to a few excellent Python libraries: Rtree, Flask and colormath.

Rome wasn’t built in a day—but the 99designs color explorer definitely was built in a day thanks to these great libraries.

R-trees

R-trees are a useful data structure for indexing spatial data such as the location of planets in the solar system or coffee shops in your local area. In this case, colordb uses an R-tree to index colors since colors can also be kind of spatial and three dimensional (colordb uses the Lab color space that has L, a, and b dimensions instead of the typical x, y and z). Normally you’d use an R-tree to help find nearby coffee shops but I’ve used it instead to find nearby colors.

The best part about R-trees is that they let you do nearest-neighbour searches efficiently. This means that you can search for a color and it will return results with colors that are similar—it means the search doesn’t have to be exact.

By indexing colors in the Lab color space, colordb is also able to give search results that are perceptually similar. Color differences in the Lab color space are perceptually uniform. This means that comparing colors in the Lab color space matches how human eye perceives color.

Flask

Flask is a micro-framework for making little web apps in Python. Flask made it dead-easy to add a simple HTTP interface to colordb in order to integrate colordb with the rest of the 99designs codebase.

Putting it all together

So, there’s a couple of things that make color explorer work:

We used colorific to extract the top two colors in a design and insert them into colordb. At the time of writing, there are ~120,000 designs indexed in colordb.

The color explorer backend code connects to colordb over HTTP to search for a color. Colordb returns designs that contain a color similar to the one searched for. Color explorer is then able to serve up the HTML markup with the matching designs into what you see on the page.

That’s about all there is really. It’s quite simple.

Why?

So, what’s the point of all this? Well, mostly because I thought it’d be fun to make. :-)

But there’s another reason. I’m a big fan of using clever computer science to make software simpler. I love the idea of making user’s lives easier by using sophisticated algorithms to reduce complexity.

Color explorer is an experiment in taking an open ended problem such as “search designs by color” and using clever computer science to create a solution that is simple, powerful and easy to use.

Some other great examples of this principle in practice are Google’s search and Apple’s Siri.

I think it’s pretty amazing that Google is able to hide an incredibly sophisticated search engine behind a single text input. Likewise, Apple’s Siri is able to combine sophisticated machine learning, voice recognition and language analysis to enable people to perform tasks simply by speaking into their phone.

Isn’t that remarkable?

Acknowledgements

A big thanks to Josh Amos for his help on the UI and visual style.

This experiment was built as part of a 99designs R&D day. Every second Friday we have a day to spend on an interesting work related side-project. So, a big thank you to 99designs for letting our dev and design team loose to make cool and crazy stuff.

By the way, if this sounds like your cup of tea—did I mention that 99designs is hiring? :-)

Discuss

Join the discussion at Hacker News and Reddit.

At 99designs we love great design, and a big part of good design is use of color. We were interested to see how designers make use of color in their designs, so we built an automatic color extractor to enable us to analyse color usage at a massive scale.

Introduction

Think of a design you love. Part of the story it tells is in the colors it uses, the contrast of light and shade, and the subtle emotions those colors convey.

It’s pretty easy for a human to tell what colors are important in a design. Take this logo below for example. Most people identifying the important colors would come up with something like this:

Images are made up of pixels, and if you just count the colors of every pixel, you don’t get anything like the list of colors above. This post is about our journey working out how to automatically work out an image’s color palette that is close to what a real person would pick.

Problem 1: Detecting background color

Quick quiz: What’s the primary color in this image?

Answer: White!

The problem here is that simply counting the number of pixels with a given color the background color nearly always dominates. We need to work out the background color so we can exclude it.

We found a simple approach that works well in most cases. We found that if the pixels in the corners of the image are the same color, that color was the background.

Let’s look again. In this case, the corner pixels are: #ffffff, #ffffff, #ffffff, #ffffff – all white. We can thus safely exclude white as a background color, leaving red as the most frequent color. Nice!

Problem 2: Too many colors

Quiz time again, how many colors are in this image?

4 colors right? Not quite. Look closer at the G for example:

There’s actually 255 different colors (an occult number for computer scientists). If we take the top four, we get:

Aww, not that great. It has two very similar blues, and misses the yellow entirely.

The problem is that colors that are not exactly the same are treated differently to a computer. All the different shades and variants of colors mess up the counts. Humans easily group sets of colors together though, and ideally our program would do the same.

So how can we judge if two colors look the same to the human eye or not? Fortunately, a bit of color theory comes in handy here.

Aside: Color theory

On a computer, colors are usually represented in the RGB color space. This means that a color is made up of three components: Red, Green and Blue. To work out the distance between two colors you can use the Euclidian distance of the components.

Comparing colors in the RGB color space works ok, but it’s not perfect. Differences in RGB don’t accurately match how the human eye perceives color. For example, yellow often appears brighter to humans than a blue of the same brightness. Also, humans can perceive smaller differences in green hues than in pink.

A better way to compare colors is to convert to the Lab color space. Lab is designed to allow comparison in a way that matches how the human eye perceives color. A color in the Lab color space has three components: “L” represents lightness, the “a” component ranges from green to magenta, the “b” component ranges from blue to yellow. The distance between colors in the Lab color space is often called the delta-E. A delta-E of less than 1.0 means that the human eye cannot tell the difference between two colors.

We can take advantage of this to group similar colors together in a way that matches how the eye would do it naturally.

Merging colors together

By combining visually similar colors, we’re able to address most of the earlier problems. The color theory above tells us which colors to merge: pairs with a low delta-E are visually similar, and can be safely combined. On our noisy image, this leads to a clearer list of colors:

Much better!

As it turns out, there’s a whole lot of situations in which extra colors get added: antialiasing on edges of shapes, image compression artifacts, textures and gradients all add to the number of different colors that occur, even if they don’t change the overall palette. This technique helps to deal with these issues. It can leave behind very low counts of some noisy colors – an additional threshold filter helps to clean up colors that don’t occur very often.

Problem 3: “interesting” colors

Now we can work out what colors are used in a design, but it turns out that a lot of the colors we find aren’t that visually interesting.

Which colors are the most interesting here?

When we ask people, they usually pick the brighter and more distinctive colors. But the palette this image uses is much more like this:

In fact, it seems like grays and subdued shades are often used used as fillers to give highlights more impact. How could we isolate these distinctive colors?

Excellent question! Lab coats on!

After flicking through designs until our retinas got tired, we turned to color theory again. It turns out color theory has a name for the concept of color interestingness: “saturation”.

Cutting out colors with a low saturation gives you much more interesting results.

We can now automatically work out the palette for an image that closely matches what a human would pick.

Introducing Colorific

At 99designs we love open source – so we’re releasing Colorific, our automatic color palette detector. Check it out at github.

Python users can install Colorific via the pip packaging system:

pip install colorific

Colorific has been tested on Python 2.7, but if you have any problems please submit an issue on github.

Usage

Colorific is designed to run in a streaming manner. You feed in image filenames as input and colorific spits out the filename and the color palette as output.

1
2

$ echo myimage.png | colorific
myimage.png #3e453f,#2ea3b7,#bee6ea,#51544c,#373d38 #ffffff

What’s next?

Tune in next time and we’ll tell you all about how we apply Colorific at a massive scale to analyse the huge number of incoming designs at 99designs.

The 99designs Research and Development programme enables members of our development team to spend a full day every two weeks (what we Australians call a “fortnight,” much to the amusement of our American colleagues) to work on projects we think are important. Read on to find out why 99designs does this, how it works in practical terms, and the successes and challenges we’ve encountered.

Why we do it

Everyone has a proverbial “itch to scratch” and our R&D programme allows us to, well, scratch it. Companies allocating time for their best and brightest to innovate is not a new school of thought. Many, technology based and otherwise, have realised that the return on investment of allowing employees to work autonomously on projects of their own choosing is substantial.

99designs’ programme serves as an incubator for ideas, and it follows that one tangible result is fantastic new products. Also, we learn new skills by dabbling with brand new technologies and then applying the lessons learned in our day-to-day tasks. And then, of course, there is the more difficult to measure result: greater staff satisfaction.

We’re not alone

A good example of a company that has been doing this for a very long time is the one that brought us the little yellow notes you’ll likely see if you glance around you right now, scribbled with clever quips or very insecure password reminders and stuck to computer monitors. Bill Coyne, a former senior vice president of R&D at 3M, put forward one of the best arguments I’ve encountered about why it’s key for the company’s engineers to spend 15 percent of their time pursuing their own ideas:

Most of the inventions that 3M depends upon today came out of that kind of individual initiative… You don’t make a difference by just following orders.

The 15 percent rule was instituted after an employee at the then-struggling company went a bit rogue at work and invented the world’s first masking tape back in 1925. Post-It notes hit the market in 1980. The company now employs 7,350 researchers and has sales of $27 billion a year.

In the technology sector, Atlassian and Google are two well-known evangelists of R&D time. Google’s “20 percent time” paved the way for key products such as Gmail, Chrome and Google News. Atlassian credits its own 20-percent programme with helping to keep its developers so happy that it doesn’t need to pour a lot of money into recruiting new talent. Prospects simply hear about its innovative culture and climb over one another to join them.

How we do it

99designs’ R&D days are scheduled at the end of our fortnightly development sprints, so that we have one every second Friday. The day gives us a nice buffer in which to get creative after our projects and other business tasks are wrapped up Thursday and before a new sprint starts Monday.

We like to hold an informal meeting the day before our R&D day to share ideas about what we’d like to tackle. Not everyone has a killer idea every time, and it’s not uncommon for a colleague to describe a project that sounds a lot cooler then what you had in mind. The get-together allows teams to self organise and game plans to develop. Another benefit of holding a prep meeting is that it gets us all amped up to hack on whatever we want all day the following day.

The Monday after an R&D day, each team or individual gives a five-minute presentation on what they achieved, what they learned and whether it’s something they want to pursue further.

Successes so far

The 99designs R&D program was implemented six months ago and in those 10 or so days we’ve managed to come up with some useful tools and fun apps.

One of our very first projects to be adopted as a mainstream product feature was an admin toolbar. This project came from a few developers who were frustrated with how difficult it was to search for and manage contests, so during our very first R&D day we decided to form a team and build a solution. To the delight of our staff, the toolbar is now part of the application and used on a daily basis.

Another frustration among developers was the lack of visibility and subsequent difficulty managing errors within our application. Yes, we have error logs. But who in their right mind likes trawling through them to get to the bottom of an issue? “Triage” is the aptly named error management system we developed over eight R&D days. This open source, ongoing project will soon be rolled out into our application and will make our lives a lot easier.

On the fun side, one of our developers created a nice little app called Mapviz, which provides a real-time visualisation of all customer log-ins, contest launches and design entries on a world map. Our marketing team was blown away by it.

Challenges

One challenge we face is coming up with a way to involve all of our teams, including support and marketing, in the R&D programme. Involving non-technical team members from time to time could allow the development team to get inspiration and ideas from a different angle. Then there’s the challenge that comes with being a global company – the Pacific Ocean creates quite a barrier between our Melbourne and San Francisco offices, making it tough for us to consistently communicate ideas efficiently and effectively.

Secondly, is an R&D day twice a month the best approach for fostering innovation? Our friends at Flippa have taken on what they like to call “Triple Time,” in which the development team dedicates three consecutive days every month to R&D. This approach might work well for us, however, giving it a go will be a challenge when we are so used to our current programme.

Another interesting approach is the “FedEx day” championed by Atlassian, during which a team is given one day to build and demonstrate a finished or near-complete product. Could introducing a deliver-in-a-day element hinder or supplement our current programme?

Looking ahead

The evidence is clear that we’re off to a good start – we’re delivering some great products, helping to foster an innovative company culture and learning new technical skills. However, like everything at 99designs, the programme is continually evolving and improving.

I’m curious to learn more about what other companies are doing to foster innovation. Does your company have an R&D program? How is it structured – as a percentage of time, designated days, another format? How else are employees encouraged to innovate?

This post describes feature flipping, an approach to development that helps solve some of the issues associated with risk management and quality assurance when a fast moving development team expands.

Continuous deployment in large teams

Developers at 99designs use agile and lean startup methodologies, but as our development team gets bigger, deployments happen more frequently. This volume of change brings increased risk for the stability of our site. It can also be quite a challenge to measure the success of a single new feature on a rapidly changing website with multiple new features operating at any given time. In our earlier days we’d demo new features on a staging server, but as 99designs has grown, using several staging servers to demo all our new features at once has become clunky. So how do we solve these issues? Feature flipping.

What is feature flipping?

Feature flipping enables the ability to turn site features on or off on a per-user basis. You’ve probably encountered it before with companies like Google or Facebook when they’re rolling out major changes. A few examples include the recent UI changes to Google Docs and Google Mail, and Facebook’s new Timeline. Our approach was inspired by our friends at Learnable, and slots in well with the Lean Startup methodology of releasing minimum viable products, measuring, and adapting through fast feedback.

Rolling out features incrementally gives companies the ability to ensure the appropriateness and stability of a feature. Some companies make this visible to users via an opt in/out approach, but in our case we use this internally as an improved form of A/B testing. Traditionally we would only A/B test on landing pages and static content, but feature flipping also allows us to experiment with pervasive site functionality.

How it works at 99

Before running through our approach, let’s first examine it from a developers perspective and from a site admin’s perspective.

Developers

As a developer, you create a new feature simply by registering it in a specific module in our codebase.

1

features.add(name='raptors', description='Unleashes raptors over the site')

Now that we’ve defined our feature, it’s available for use in a conditional statement, which you use to guard feature-specific code.

1
2

if feature('raptors'):
    unleash_raptors()

Boom! Our raptors feature can now be toggled on or off for any user who interacts with our site.

To kick off an incremental roll out, a commit is required with a defined set of eligibility criteria a user must meet before the feature is enabled for them. For example, the user must be a designer and needs to have submitted a minimum of 5 entries.

1

feature('raptors').rollout(role='designer', min_entries=5)

Of course, we make sure to test the new behaviour in addition to the old.

Admin

99designs staff get a nice interface where they can turn on and off features for themselves or a specific user. Here they can also view the progress of any experiments hanging off this feature. Many of our stakeholders are located remotely; by giving them the power to enable features themselves, they can try these out in production and provide feedback. This removes the need for a dedicated staging environment for demoing.

Under the hood

When we first considered how best to implement features, we recognized that every feature we introduced would require us to perform an additional user-feature check. This would clearly result in rapid growth of queries as we began checking features for every user on every page of our site.

For logged out users, we decide to to go with a cookie based solution. However, for logged-in users we wanted our user features to be persistent. We thought about using MySQL, but based on Redis benchmarks and its inbuilt sets data structure, going this route seemed an ideal solution. We decided to go with a combined cookie and Redis approach to minimise the number of database lookups without impacting the performance of our site.

These two methods work well together. When a user logs into 99designs, cookie based features are synched up with Redis by taking a simple union of features enabled and persisting the resulting set. This allows us to determine exactly who has what features as well as when a user obtained those features.

Challenges

Developing with a multi-version mindset is probably one of the most challenging parts of feature flipping. What was once a simple deployment requiring a migration now entails more thought to allow the old and new versions to function independently of one other.

Deprecated code can start to accumulate when you start leaving code wrapped in feature checks. We try to curtail this as much as possible by coming back to clean up unused code paths once a feature has been fully rolled out. Also, we generally have a grace period even for features we’re decided on, so that we can roll back if the need arises.

Unit and integration tests become more difficult when you’re working with an exponential number of feature combinations at any given time. This can lead to an outsized volume of test cases, a problem we’re still solving.

What now?

Feature flipping is our approach to solving several problems we face as 99designs expands. Managing features gives us fine-grained control over exactly what our users see, and is paving the way for us to experiment and adapt to our users much faster than was previously possible.

Big companies often execute this approach to development seamlessly, while there’s more of a learning curve for small to mid-sized companies. In our case, we’re still learning but we’re far enough along to be reaping the benefits.

Where to go from here? We suggest you explore a few open source solutions that deal with features:

• flip - Declare and manage features.
• rollout - Conditionally roll out features with redis.
• degrade - Keeps track of features, degrades functionality if errors are too high.

Although our user facing site is the most visible part of what we do, it’s only half the story. The other half takes part in asychronous queues, which work overtime behind the scenes, and process hundreds of millions of tasks each year. In this post I’ll explain a bit about why we use queueing at 99designs, and how it all works.

A Bit of Background

If you’ve never heard of asynchronous task queues before, the idea behind them is pretty simple. Say you had a task you needed to do such as buying some milk but you didn’t have the time to take care of it yourself, so instead you leave a note for a friend/spouse/roommate asking them to do it for you when they have a chance. Congratulations, you’ve just implemented an asynchronous task queue.

Why We Use Queues

Now obviously our web apps aren’t busy ordering milk, more common uses for a queue are things like talking to third party API’s, sending emails, or performing computationally expensive tasks like image resizing. But why do we need a queue at all? Wouldn’t it be easier to just do the work a user requires immediately? Well, there’s a few reasons:

The first reason is speed: When we’re talking to a third party API we have to face reality; unless that third party is physically located next to our infrastructure, there’s going to be latency involved. All it would take is the addition of a few API calls and we could easily end up doubling or tripling our response time, leading to a sluggish site and unhappy users. However if we push these API calls into our queue instead, we can return a response to our users immediately while our queues take as long as they like to talk to the API.

The second reason is reliability: We don’t live in a world of 100% uptime, services do go down, and when they do it’s important that our users aren’t the ones that suffer. If we were to make our API calls directly in the users requests we wouldn’t have any good options in the event of a failure. We could retry the call right away in the hope that it was just a momentary glitch, but more than likely we’ll either have to show the user an error, or silently discard whatever we were trying to do. Queues neatly get around this problem since they can happily continue retrying over and over in the background, and all the while our users never need to know anything is wrong.

The final reason to use a queue is for scalability. If we had a surge in requests that involved something CPU intensive like resizing images, we might have a problem if all of our apps were responsible for this. Not only would the increased CPU load slow down other image resize requests, it could very well slow down requests across the entire site. What we need to do is isolate this workload from the user’s experience, so that it doesn’t matter if it happens quickly or slowly. This is where queues shine. Even if our queues become overloaded, the rest of the site will remain responsive.

How We Implement It At 99designs

Understanding why we use queues is one thing, actually implementing it is quite another. In the case of 99designs we chose beanstalk as our queue, it’s key features being performance, reliability and simplicity. Rather than having just one centralized queue which our servers push tasks to, we instead have a separate queue on every app server. Queuing a task locally is always going to be very quick, and having multiple queues gives us a healthy dose of redundancy in the event of a beanstalk failure — something I’ve yet to see.

Of course on its own a queue doesn’t do anything; simply pushing tasks onto it doesn’t magically make them happen. For that you need a worker. Since a lot of what 99designs does is PHP-based we had to develop our own custom worker daemon, as there weren’t any open source solutions for beanstalk. Each worker node maintains a pool of worker processes to connect to each queue and listen for new tasks. In the event that our queues start to back up, we can easily launch a new worker instance thanks to AWS and have it processing tasks within minutes.

The final part that ties all of it together is how we deal with failures. In the event that a task fails it doesn’t simply disappear into the ether. Instead we’ll release the task with a delay, essentially putting it back onto the queue after a certain amount of time has passed. If a task continues to fail we use an exponential backoff strategy to prevent failing tasks from clogging up our queues. If after multiple releases the task still has not finished successfully then we “bury” it. Buried tasks are then manually inspected at a later date, and a decision is made to either fix the problem (if possible), or delete the task if it is deemed unrecoverable.

Work in progress

There’s still more that we’d like to do with our queueing systems. If too many tasks fail, and are buried, we get alerts which wake us in the night. Sometimes, this happens because an external service we use goes down, something we can’t do much about anyway. Unfortunately, our tasks don’t yet coordinate their back off in any way; each assumes it’s the only one failing. Ideally, we’d track failures against external APIs, and simply delay groups of tasks for longer, giving third party services time to recover.

In closing

Asynchronous tasks are a crucial, and often overlooked, part of how any large site operates. We hope this post gives some insight into why queues are important, and how they can improve your site’s performance and reliability. Whilst there are many different queuing architectures possible, we also hope our recipe serves as a useful pattern for other sites.

99designs hosts design contests, and has been growing rapidly year on year. This post gives an overview of the infrastructure which powers our site, how we make sure it continues to hum smoothly, and the challenges we face as we scale.

A little context

Since its humble origins in ad-hoc contests within Sitepoint’s forums, 99designs has turned out to be one of the success stories of the Melbourne start-up scene. Although we now have offices in both Melbourne and San Francisco, the development team is in the Melbourne office, close to where the action all began. Our team here has 8 devs, 2 dev ops, 2 ux/designers, and is expanding. Half the people here arrived within the past year, myself included, amounting to a lot of growth and change over a limited amount of time.

Our site sees hundreds of thousands of unique visitors a month, generating pageviews in the tens of millions. Since we deal with graphic design, many of our pages are asset heavy — these pageviews fan out to some 40 times as many requests. Whilst there are many larger sites on the net, we thought this was enough to warrant sharing the way we do things.

Requests in layers

The easiest way to describe how we serve requests is to talk about it in layers, each of which solves a set of problems we face. We’ll cover six pieces of this puzzle: load balancing, acceleration, application, asynchronous tasks, storage and transient data.

Load balancing

Let’s start at the beginning of a request. When a user visits 99designs.com, the request firstly hits our Elastic Load Balancer (ELB). The load balancer is a highly reliable service which ensures that requests are spread evenly between the Varnish servers beneath it. It also performs active health checks, so that requests only hit healthy servers, and SSL unwrapping, allowing us to work with an unencrypted stack from there on down. On the SSL front, using a separate ELB for each domain turns out to be a convenient way of running multiple secure domains.

Acceleration

Our acceleration layer consists of several Varnish servers, which allow us to serve a large amount of media with only a limited app stack beneath them. We have a long-tail of static media, so we run Varnish with a file-based rather than in-memory storage backend. Varnish is fast, and incredibly configurable through its inbuilt DSL. Furthermore, its command-line tools for inspecting live traffic are second to none, and are incredibly useful in tracking down odd site behaviour.

Application

Dynamic or otherwise uncached requests are served from our PHP application layer, using Apache/mod_php. Our polyglot team takes inspiration from some of the best frameworks in the Ruby, Python and Javascript worlds, and we’re not above porting over what we need. We also open source what we can, for example our lightweight web application framework Ergo. Designs which users submit aren’t stored on every app server, but are instead stored in S3. Since end-user latency is so poor for S3, we serve designs through our app layer, and cache them locally after each request.

Asynchronous tasks

We strive to have a responsive site where users aren’t kept waiting, but often requests might need to do some extended work, or access an external API. Integrated with our application layer is an asyncronous layer which tackles this problem. We queue up tasks using Beanstalk in-memory task queues on each of our app servers, using the Pheanstalk bindings. Beanstalk is known to be lightweight and performant, with the trade-off that we lose some visibility into the immediate contents of our queues. A pool of PHP workers listens to these queues, and takes care of anything lengthy or requiring access to an external API. Tasks which need to run at a particular time are stored instead in the database, and added to the queues by cron when they fall due.

Storage

Our storage layer features Amazon’s managed MySQL service (RDS) as the primary, authoritative and persistent store for our crucial data. An RDS instance configured to use multiple availability zones provides master-master replication, providing crucial redundancy for our DB layer. This feature has already saved our bacon multiple times: the fail over has been smooth enough that by the time we realised anything was wrong, another master was correctly serving requests. Its rolling backups provide a means of disaster recovery. We load-balance reads across multiple slaves as a means of maintaining performance as the load on our database increases. For media files and data blobs, we use S3 for redundant and highly-available storage, with periodic backups to Rackspace Cloudfiles for disaster recovery.

Transient data

Aside from our database proper, there are three services which we use primarily for transient data: Memcached, MongoDB and Redis. Memcached runs locally on every application server, with a peering arragement between servers, and helps us reduce our database queries dramatically. We log errors and statistics to capped collections in MongoDB, providing us with more insight into our system’s performance. Redis captures per-user information about which features are enabled at any given time; it supports our development stragegy around dark launches, soft launches and incremental feature rollouts.

Software as infrastructure

99designs strongly follows the “software as infrastructure” mantra. Like many companies now, we don’t own any hardware ourselves, preferring to remain flexible, and relying heavily on Amazon’s cloud offering. Growing as we have has meant a lot of change in a limited period of time, and has built into our culture a distrust for documentation and the dual-maintenance problem it creates. Instead, we focus on automation of as much as possible.

We currently use Rightscale to manage our server configurations, which basically amounts to using a managed form of Chef for provisioning new servers. We make sure each server type has a recipe which allows us to spin up replacements at a moment’s notice. This means we can treat servers as disposable, and mean it.

The layers and services which make up our infrastructure amount to a fair number of moving parts, so monitoring and keeping track of the distributed application state is important. We do that through a number of services, incuding a large number custom monitoring pages, NewRelic, CloudWatch, Statsd and others. Two large monitoring screens feature prominently in our office, making sure we’re aware of changes to the site. Despite all this information, we’re continuously working to get a better understanding around site behaviour and performance.

Challenges

Whilst the team here has some pride in our accomplishments, there’s a lot we still have to work on. Here’s some of our biggest challenges:

1. Scaling back infrastructure, rather than just scaling out. As our site changes and our customer base grows, the load we place on our backend systems can vary dramatically. One way to deal with this is to over-provision, so as to meet such spikes without issue. A challenge for us is to automate and stress-test even more of our infrastructure, so that we can bring up new servers even faster and more reliably. This would allow us to confidently reduce capacity when we have excess, rather than simply expanding.

2. Providing a strong experience for international customers. We have a diverse and international customer-base, yet all the action is currently served out of Amazon’s US-East data center. This leads to quite a disparity in customer experience. We’re currently trialling CDNs in order to get static media to our international customers faster, and likewise looking at other ways we can improve performance.

3. Balancing feature growth with stability. Being responsive to our customers means being able to push out new features quickly. In some companies, this causes a tension between developers who need to get code out, and ops who are woken in the night by the consequences of a hasty change. We’re attacking this problem from multiple angles: stronger acceptance testing should give us better sanity checks on new code that goes out; feature flipping allows us to incrementally role out new features to only a subset of users; and finally, we’re working on further automation in order to allow our developers to be more active in our infrastructure, meaning they can really own a change from the moment it’s coded to the moment users see it in production.

Watch this space

This post has given an overview of our current stack, and some of the broad challenges we face, but we’ve got a lot more to say about our development style, and the things which make our culture. We’ve benefited greatly from the open source community and the expertise of those who share their experiences. Now that we’ve grown, we’re keen to give back a little too, in the hope that others can benefit from what we’ve learned.