by Lars Yencken

Two years ago, 99designs had localized sites for a handful of English speaking countries, and our dev team had little experience in multilingual web development. But we felt that translating our site was an important step, removing yet another barrier for designers and customers all over the world to work together. Today we serve localized content to customers in 18 countries, across six languages. Here’s how we got there, and some of the road blocks we ran into.

Starting local

The most difficult aspect to internationalizing is language, so we started with localization: everything but language. In particular, this means region-appropriate content and currency. A six-month development effort saw us refactor our core PHP codebase to support local domains for a large number of countries (e.g. 99designs.de), where customers could see local content and users could pay and receive payments in local currencies. At the end of this process, each time we launched a regional domain we began redirecting users to that domain from our Varnish layer, based on GeoIP lookups. The process has changed little since then, and continued to serve us well in our recent launch in Singapore.

Languages and translation

With localization working, it was time to make hard decisions about how we would go about removing the language barrier for non-English speakers (i.e. the majority of the world).There were a lot of questions for us to answer.

• What languages will we offer users in a given region?
• How will users choose their language?
• How will we present translated strings to users?
• How will strings be queued for translation?
• Who will do the translation?

What languages to offer?

Rather than making region, language and currency all user selectable, we chose to restrict language and currency availability to a user’s region. This was a trade-off which made working with local content easier: if our German region doesn’t support Spanish, we avoid having to write Spanish marketing copy for it. Our one caveat was for all regions to support English as a valid language. As an international language of trade, this lessens any negative impact of region pinning.

Translating strings

There were two main approaches we considered for translation: use a traditional GNU gettext approach and begin escaping strings, or else try a translation proxy such as Smartling. gettext had several advantages: it has a long history, and is well supported by web frameworks; it’s easily embedded; and translations just become additional artifacts which can be easily version controlled. However, it would require a decent refactoring of our existing PHP codebase, and left open issues of how to source translations.

In Smartling’s approach, a user’s request is proxied through Smartling’s servers, in turn requesting the English version of our site and applying translations to the response before the user receives it. When a translation is missing, the English version is served and the string is added to a queue to be translated. Pulling this off would mean reducing substantially the amount of code to be changed, a great win. However, it risked us relying on a third-party for our uptime and performance.

In the end, we went with Smartling for several reasons. They provided a source of translators, and expertise in internationalization which we were lacking. Uptime and performance risks were mitigated somewhat by two factors. Firstly, Smartling’s proxy would be served out of the US-East AWS region, the same region our entire stack is served from, increasing the likelihood that their stack and ours would sink or swim together. Secondly, since our English language domains would continue to be served normally, the bulk of our traffic would still bypass the proxy and be under our direct control.

Preparing our site

We set our course and got to work. There was substantially more to do than we first realized, mostly spread over three areas.

Escaping user-generated content

Strings on our site which contained user content quickly filled our translation queue (think “Logo design for Greg” vs “Logo design for Sarah”). Contest titles, descriptions, usernames, comments, you name it, anything sourced from a user had to be found and wrapped in a <span class="sl_notranslate"> tag. This amounted to a significant ongoing audit of the pages on our site, fixing them as we went.

Preparing Javascript for translation

Our Javascript similarly needed to be prepared for translation, with rich client-side pages the worst hit. All strings needed to be hoisted to a part of the JS file which could be marked up for translation. String concatenation was no longer ok, since it made flawed assumptions about the grammar of other languages. Strings served through a JSON API were likewise hidden from translation, meaning we had to find other ways to serve the same data.

Making our design more flexible

In our design and layout, we could no longer be pixel-perfect, since translated strings for common navigation elements were often much longer in the target language. Instead, it forced us to develop a more robust design which could accommodate the variation in string width. We stopped using CSS transforms to vary the case of text stylistically, since other languages are more sensitive to case changes than English.

The wins snowball

After 9 months of hard work, we were proud to launch a German language version of our site, a huge milestone for us. With the hardest work now done, the following 9 months saw us launch French, Italian, Spanish and Dutch-language sites. Over time, the amount of new engineering work reduced with each launch, so that the non-technical aspects of marketing to, supporting and translating a new region now dominate the time to launch a new language.

The challenges

We also encountered several unexpected challenges.

Client-side templating

We mentioned earlier that the richer the client-side JS, the more work required to ensure smooth translation. The biggest barrier for us was our use of Mustache templates, which were initially untranslatable on the fly. To their credit, Smartling vastly improved their support for Mustache during our development, allowing us to clear this hurdle.

Translating non-web artifacts

It should be no surprise: translation by proxy is a strategy for web pages, but not a strong one for other non-web artifacts. In particular, for a long time translating emails was a pain, and in the worst case consisted of engineers and country managers basically emailing templates for translation back and forward. After some time, we worked around this issue by using Smartling’s API in combination with gettext for email translation.

Exponential growth of translation strings

Over time, we repeatedly found our translation queue clogged with huge numbers of strings awaiting translation. Many of these cases were bugs where we hadn’t appropriately marked up user-generated content, but the most stubborn were due to our long-tail marketing efforts. Having a page for each combination of industry, product category and city led to an explosion of strings to translate. Tackling these properly would require a natural language generation engine with some understanding of each language’s grammar. For now we’ve simply excluded these pages from our translation efforts.

The future

This has been an overview of the engineering work involved in localizing and translating a site like ours to other languages. Ultimately, we feel that the translation proxy approach we took cut down our time to market significantly; we’d recommend it to other companies who are similarly expanding. Now that several sites are up and running, we’ll continue to use a mix of the proxy and gettext approaches, where each is most appropriate.

We’re proud to be able to ship our site in multiple languages, and keen to keep breaking down barriers between businesses and designers wherever they may be, enabling them to work together in the languages in which they’re most comfortable.

Discuss on Hacker News or Reddit.

by Daniel Heath

I recently found myself wanting the features of the rails asset pipeline in my golang project at work. Since there isn’t much in the way of asset pipelining for golang yet, I built it. Turns out, sprockets is really easy to integrate. Here’s how you can go about setting it up for your project.

Assets in development

First things first - lets get it to the ‘it works on my machine’ stage. I’ve put together a sample repo using the asset pipeline, which you can use as a guide.

The setup for your app will be similar:

• The assets folder contains your stylesheets, javascript, etc (this directory name is set in sprockets/environment.rb).
• You’ll need a similar Rakefile to build assets (and maybe launch the server)
• You might store the sprockets directory somewhere else - update the Rakefile to match.
• Use a Gemfile and the bundler rubygem to manage dependencies.
• Edit the rakefile to change the port the asset server runs on.

When your app starts up (in development), it should make a request to http://localhost:11111/assets/manifest.json, which provides a JSON hash linking asset names (e.g. “application.css”) to the relative URLs the compiled assets can be fetched from. To generate a link to an asset in your app, use the JSON hash you fetched to lookup the URL. For example, the URL for “application.css” this might look like http://localhost:11111/application-8e5bf6909b33895a72899ee43f5a9d53.css.

That should be all you need for development - you should be able to see SASS/Coffeescript assets compiled and loading normally. Hooray!

Assets in production

For production we want to pre-compile assets rather than regenerating them each time they change.

rake assets will create a ‘public’ folder containing ‘manifest.json’ (same format as before). Get this directory onto your production servers. git add -Af public/ will add it to source control if you deploy via git.

When generating a link to an asset, simply look up manifest.json from the filesystem rather than from HTTP.

Fin

If you’ve followed these steps, you’ll have a fullly functioning asset pipeline for your golang project. The whole thing, including deployment, took me well under a day to add to our app. The resulting assets are minified, concatenated, and gzipped (for size). They are also fingerprinted, so you can serve them with an unlimited cache lifetime and reap the benefits.

Although I set this up for golang, there’s nothing go-specific about it. The same technique works just as well for any language or framework without a mature asset pipeline. If you find yourself in need, just use this pattern and you can be up and running in no time.

by Stuart Campbell

At 99designs, we try to make sure we’re always fixing bugs as well as writing code. It can be easy to neglect bugs when you’re busy churning out new features.

We use GitHub issues to track bugs in our various applications. GitHub issues integrate well with our codebase, commits and pull requests, but the reporting facilities are a bit limited.

As our team grows, it’s become increasingly important for us to be able to answer key questions about bugs, including:

• How many bugs are currently open?
• Have we each remembered to spend time working on bug fixes this sprint?
• Are we closing more bugs than we’re opening?

To help answer these questions, a few of our team spent a number of hack days implementing a bug dashboard named GitHub Survivor.

Unlike the similarly-named reality TV show, GitHub Survivor doesn’t feature eliminations, gruelling physical challenges, or Jeff Probst. However, it does pit developers against one another — in a light-hearted way.

We display GitHub Survivor on a big screen in the office, where all the team can see it. We’ve found it helps keep our minds on bugs — it reminds us to make a small effort every sprint, gradually bringing the bug count closer to zero.

A bug leaderboard occupies the bulk of the screen. It shows who’s closed the most bugs this sprint (may they be laden with Praise and Whisky!) and who’s forgotten to spend some time fixing bugs (may they toil in the maintenance of a thousand Malbolge programs!).

There are charts showing the number of bugs opened and closed in recent sprints, the open bug count over time, and a big indicator showing the current open bug count.

The source is available for you to inspect and adapt to your needs. Please try it out, make improvements and contribute them back! We hope you find it useful.

We’re passionate about building high-quality software at 99designs, and this is just one way we measure whether we’re doing a good job of that. If you’re similarly interested in building cool things in an awesome environment, check out our open positions!

Join the discussion at Hacker News and Reddit.

by Richo Healey

What if it were possible to call methods with spaces in their name directly from Ruby?

If you’ve seen Gary Bernhardt’s awesome talk where he digs into some of the quirks of Ruby, you’ll know that it’s pretty trivial to get bare words in Ruby:

1
2
3
4
5
6

1.9.3-p286 :001 > def method_missing(*args)
1.9.3-p286 :002?>   args.join(" ")
1.9.3-p286 :003?>   end
 => nil
1.9.3-p286 :004 > ruby has bare words
SystemStackError: stack level too deep

Wait.. What?

Disappointing to say the least. Obviously something is amiss. It turns out this is just a quirk in irb; if you try instead

1
2
3
4
5
6
7

1.9.3-p286 :001 > def self.method_missing(*args)
1.9.3-p286 :002?>   args.join(" ")
1.9.3-p286 :003?>   end
=> nil
1.9.3-p286 :004 > ruby has bare words
=> "ruby has bare words"
1.9.3-p286 :005 >

Cool, so what else can we do with this? It’s trivial to define a method with a space in its name, and calling it isn’t terribly difficult:

1
2
3
4
5
6
7
8

1.9.3-p286 :005 > self.class.send(:define_method, :"i have a space") do
1.9.3-p286 :006 >     puts "I has a space"
1.9.3-p286 :007?>   end
=> #<Proc:0x007ff89c1e0b58@(irb):5 (lambda)>
1.9.3-p286 :008 > send(:"i have a space")
I has a space
=> nil
1.9.3-p286 :009 >

But having created such a monstrosity, how do you call it from the repl? Or for that matter, from an actual Ruby program? This is obviously something you should be doing in production…

1
2
3
4
5
6
7
8
9
10
11
12
13

self.instance_exec do
def method_missing(sym, *args)
  # Splat args if passed in from a parent call
  if args.length == 1 && args[0].is_a?(Array) && args[0][0].class == NameError
    args = args[0]
  end

  method_names, arguments = args.partition { |a| a.class == NameError }
  method([sym.to_s, *method_names.map(&:name)].join(" ")).call(*arguments)
rescue NameError => e
  return [e, *arguments]
end
end

Bam. You may be looking at this baffled (or if you’re reasonably tight with metaprogramming in Ruby, sharpening/setting fire to something with a view to causing me significant bodily harm).

Walking through this, we first of all act on whatever self is; in most cases this will be the local scope. If we didn’t do this, we’d be defining the method on Object, which can cause all kinds of headaches when you’re trying to debug.

Immediately after this, we unpack arguments if they look like they were created by an earlier instance of this method. This is unwieldy, but unfortunately Ruby’s single return values and the recursion we’re employing here make it necessary. We could definitely define a subclass of Array to make the test cleaner and the implementation more robust, but I preferred to keep this as short as possible and use the bare minimum number of Ruby primitives.

Once we’ve unpacked our arguments, we do the real magic. First off, we split our arguments into NameErrors, the container we’re using for our missing method names, and everything else (the legitimate arguments we were called with).

We try to find a method with the current name (as we’ll be building our method name right to left with recursive calls to method_missing), and failing that we pack up our current attempt with our arguments, and return it for the next pass.

There are enough issues with this (if you defined the methods foo bar baz and bar baz, a call to foo bar baz would call foo with bar bazs return) to make it unwieldy. On the other hand; if those bugs are the only thing stopping you from putting this into production, you’ve probably got larger issues.

If this large scale abuse of the language excites you, you might be interested to know that we’re hiring.

At this point you’re probably eager to know.. does it work?

1
2
3
4
5
6
7
8
9

1.9.3-p286 :001 > load "bare_words.rb"
1.9.3-p286 :002 > self.class.send(:define_method, :"i has a space") do |name, greeting|
1.9.3-p286 :003 >     puts "#{greeting}, #{name}!"
1.9.3-p286 :004?>   end
=> #<Proc:0x007fc6b41872c0@(irb):2 (lambda)>
1.9.3-p286 :005 > i has a space "richo", "Hello"
Hello, richo!
=> nil
1.9.3-p286 :006 >

Join the discussion at Hacker News and Reddit.

by David Lutz

If you had happened to have been wandering around the 99designs office today you would have heard hysterical laughter and cries of “yarrrr”.

Something we try to do is foster a good “DevOps” working culture. One of the critical components of DevOps is a collaborative way of working and close working relationship between those working in development and operations. And other parts of the business for that matter. Company culture is something that is tricky to improve if it’s not working, but immediately obvious when it is working.

A key component of good company culture is good communication. We make extensive use of IRC as a communication medium. Naturally we use it for technical discussions like “How does this component of code work”, but just like companies like GitHub and Etsy we use IRC within the company as a very effective method of documenting what both dev and ops are doing day to day. We also have a bot that lives in IRC and does useful things for us. Our bot is called agent99. She makes our life easier and has the ability to do such things as deploy new versions of code to our production website.

She also has a bit of character. She can find memes to punctuate the moment, or fetch funny pictures of cats when asked.

We’ve actually open-sourced agent99 as well as many other pieces of code. We like to contribute code that we think might be useful to others back to the community.

So what was the source of hysterical laughter? Well today is Talk Like a Pirate Day. So one of our staff hacked together a plugin to make agent99 send a “yarr” over the office speakers. Hilarity ensued. It was quicky repatched, so that “yarr” only works one golden day a year.

This is a little example of where all are empowered to use technology to make it a more enjoyable workplace. And a work environment that is both fun and technically challenging is a competitive advantage in that helps attract and retain motivated and happy employees!

by Dennis Hotson

Here’s a sneak peek of some experimental stuff we’ve been working on at 99designs.
99designs color explorer is a tool for browsing designs by color.

Nerdy technical details

For those interested in the technical details, color explorer uses colorific to extract colors from logo designs (We’ve written more about colorific previously). It’s also using colordb, a side project of mine designed to efficiently index and search by color using some fancy perceptual nearest-neighbour algorithms.

Colordb turned out to be super easy to implement thanks to a few excellent Python libraries: Rtree, Flask and colormath.

Rome wasn’t built in a day—but the 99designs color explorer definitely was built in a day thanks to these great libraries.

R-trees

R-trees are a useful data structure for indexing spatial data such as the location of planets in the solar system or coffee shops in your local area. In this case, colordb uses an R-tree to index colors since colors can also be kind of spatial and three dimensional (colordb uses the Lab color space that has L, a, and b dimensions instead of the typical x, y and z). Normally you’d use an R-tree to help find nearby coffee shops but I’ve used it instead to find nearby colors.

The best part about R-trees is that they let you do nearest-neighbour searches efficiently. This means that you can search for a color and it will return results with colors that are similar—it means the search doesn’t have to be exact.

By indexing colors in the Lab color space, colordb is also able to give search results that are perceptually similar. Color differences in the Lab color space are perceptually uniform. This means that comparing colors in the Lab color space matches how human eye perceives color.

Flask

Flask is a micro-framework for making little web apps in Python. Flask made it dead-easy to add a simple HTTP interface to colordb in order to integrate colordb with the rest of the 99designs codebase.

Putting it all together

So, there’s a couple of things that make color explorer work:

We used colorific to extract the top two colors in a design and insert them into colordb. At the time of writing, there are ~120,000 designs indexed in colordb.

The color explorer backend code connects to colordb over HTTP to search for a color. Colordb returns designs that contain a color similar to the one searched for. Color explorer is then able to serve up the HTML markup with the matching designs into what you see on the page.

That’s about all there is really. It’s quite simple.

Why?

So, what’s the point of all this? Well, mostly because I thought it’d be fun to make. :-)

But there’s another reason. I’m a big fan of using clever computer science to make software simpler. I love the idea of making user’s lives easier by using sophisticated algorithms to reduce complexity.

Color explorer is an experiment in taking an open ended problem such as “search designs by color” and using clever computer science to create a solution that is simple, powerful and easy to use.

Some other great examples of this principle in practice are Google’s search and Apple’s Siri.

I think it’s pretty amazing that Google is able to hide an incredibly sophisticated search engine behind a single text input. Likewise, Apple’s Siri is able to combine sophisticated machine learning, voice recognition and language analysis to enable people to perform tasks simply by speaking into their phone.

Isn’t that remarkable?

Acknowledgements

A big thanks to Josh Amos for his help on the UI and visual style.

This experiment was built as part of a 99designs R&D day. Every second Friday we have a day to spend on an interesting work related side-project. So, a big thank you to 99designs for letting our dev and design team loose to make cool and crazy stuff.

By the way, if this sounds like your cup of tea—did I mention that 99designs is hiring? :-)

Discuss

Join the discussion at Hacker News and Reddit.

by Dennis Hotson and Lars Yencken

At 99designs we love great design, and a big part of good design is use of color. We were interested to see how designers make use of color in their designs, so we built an automatic color extractor to enable us to analyse color usage at a massive scale.

Introduction

Think of a design you love. Part of the story it tells is in the colors it uses, the contrast of light and shade, and the subtle emotions those colors convey.

It’s pretty easy for a human to tell what colors are important in a design. Take this logo below for example. Most people identifying the important colors would come up with something like this:

Images are made up of pixels, and if you just count the colors of every pixel, you don’t get anything like the list of colors above. This post is about our journey working out how to automatically work out an image’s color palette that is close to what a real person would pick.

Problem 1: Detecting background color

Quick quiz: What’s the primary color in this image?

Answer: White!

The problem here is that simply counting the number of pixels with a given color the background color nearly always dominates. We need to work out the background color so we can exclude it.

We found a simple approach that works well in most cases. We found that if the pixels in the corners of the image are the same color, that color was the background.

Let’s look again. In this case, the corner pixels are: #ffffff, #ffffff, #ffffff, #ffffff – all white. We can thus safely exclude white as a background color, leaving red as the most frequent color. Nice!

Problem 2: Too many colors

Quiz time again, how many colors are in this image?

4 colors right? Not quite. Look closer at the G for example:

There’s actually 255 different colors (an occult number for computer scientists). If we take the top four, we get:

Aww, not that great. It has two very similar blues, and misses the yellow entirely.

The problem is that colors that are not exactly the same are treated differently to a computer. All the different shades and variants of colors mess up the counts. Humans easily group sets of colors together though, and ideally our program would do the same.

So how can we judge if two colors look the same to the human eye or not? Fortunately, a bit of color theory comes in handy here.

Aside: Color theory

On a computer, colors are usually represented in the RGB color space. This means that a color is made up of three components: Red, Green and Blue. To work out the distance between two colors you can use the Euclidian distance of the components.

Comparing colors in the RGB color space works ok, but it’s not perfect. Differences in RGB don’t accurately match how the human eye perceives color. For example, yellow often appears brighter to humans than a blue of the same brightness. Also, humans can perceive smaller differences in green hues than in pink.

A better way to compare colors is to convert to the Lab color space. Lab is designed to allow comparison in a way that matches how the human eye perceives color. A color in the Lab color space has three components: “L” represents lightness, the “a” component ranges from green to magenta, the “b” component ranges from blue to yellow. The distance between colors in the Lab color space is often called the delta-E. A delta-E of less than 1.0 means that the human eye cannot tell the difference between two colors.

We can take advantage of this to group similar colors together in a way that matches how the eye would do it naturally.

Merging colors together

By combining visually similar colors, we’re able to address most of the earlier problems. The color theory above tells us which colors to merge: pairs with a low delta-E are visually similar, and can be safely combined. On our noisy image, this leads to a clearer list of colors:

Much better!

As it turns out, there’s a whole lot of situations in which extra colors get added: antialiasing on edges of shapes, image compression artifacts, textures and gradients all add to the number of different colors that occur, even if they don’t change the overall palette. This technique helps to deal with these issues. It can leave behind very low counts of some noisy colors – an additional threshold filter helps to clean up colors that don’t occur very often.

Problem 3: “interesting” colors

Now we can work out what colors are used in a design, but it turns out that a lot of the colors we find aren’t that visually interesting.

Which colors are the most interesting here?

When we ask people, they usually pick the brighter and more distinctive colors. But the palette this image uses is much more like this:

In fact, it seems like grays and subdued shades are often used used as fillers to give highlights more impact. How could we isolate these distinctive colors?

Excellent question! Lab coats on!

After flicking through designs until our retinas got tired, we turned to color theory again. It turns out color theory has a name for the concept of color interestingness: “saturation”.

Cutting out colors with a low saturation gives you much more interesting results.

We can now automatically work out the palette for an image that closely matches what a human would pick.

Introducing Colorific

At 99designs we love open source – so we’re releasing Colorific, our automatic color palette detector. Check it out at github.

Python users can install Colorific via the pip packaging system:

pip install colorific

Colorific has been tested on Python 2.7, but if you have any problems please submit an issue on github.

Usage

Colorific is designed to run in a streaming manner. You feed in image filenames as input and colorific spits out the filename and the color palette as output.

1
2

$ echo myimage.png | colorific
myimage.png #3e453f,#2ea3b7,#bee6ea,#51544c,#373d38 #ffffff

What’s next?

Tune in next time and we’ll tell you all about how we apply Colorific at a massive scale to analyse the huge number of incoming designs at 99designs.

by Michael De Wildt

The 99designs Research and Development programme enables members of our development team to spend a full day every two weeks (what we Australians call a “fortnight,” much to the amusement of our American colleagues) to work on projects we think are important. Read on to find out why 99designs does this, how it works in practical terms, and the successes and challenges we’ve encountered.

Why we do it

Everyone has a proverbial “itch to scratch” and our R&D programme allows us to, well, scratch it. Companies allocating time for their best and brightest to innovate is not a new school of thought. Many, technology based and otherwise, have realised that the return on investment of allowing employees to work autonomously on projects of their own choosing is substantial.

99designs’ programme serves as an incubator for ideas, and it follows that one tangible result is fantastic new products. Also, we learn new skills by dabbling with brand new technologies and then applying the lessons learned in our day-to-day tasks. And then, of course, there is the more difficult to measure result: greater staff satisfaction.

We’re not alone

A good example of a company that has been doing this for a very long time is the one that brought us the little yellow notes you’ll likely see if you glance around you right now, scribbled with clever quips or very insecure password reminders and stuck to computer monitors. Bill Coyne, a former senior vice president of R&D at 3M, put forward one of the best arguments I’ve encountered about why it’s key for the company’s engineers to spend 15 percent of their time pursuing their own ideas:

Most of the inventions that 3M depends upon today came out of that kind of individual initiative… You don’t make a difference by just following orders.

The 15 percent rule was instituted after an employee at the then-struggling company went a bit rogue at work and invented the world’s first masking tape back in 1925. Post-It notes hit the market in 1980. The company now employs 7,350 researchers and has sales of $27 billion a year.

In the technology sector, Atlassian and Google are two well-known evangelists of R&D time. Google’s “20 percent time” paved the way for key products such as Gmail, Chrome and Google News. Atlassian credits its own 20-percent programme with helping to keep its developers so happy that it doesn’t need to pour a lot of money into recruiting new talent. Prospects simply hear about its innovative culture and climb over one another to join them.

How we do it

99designs’ R&D days are scheduled at the end of our fortnightly development sprints, so that we have one every second Friday. The day gives us a nice buffer in which to get creative after our projects and other business tasks are wrapped up Thursday and before a new sprint starts Monday.

We like to hold an informal meeting the day before our R&D day to share ideas about what we’d like to tackle. Not everyone has a killer idea every time, and it’s not uncommon for a colleague to describe a project that sounds a lot cooler then what you had in mind. The get-together allows teams to self organise and game plans to develop. Another benefit of holding a prep meeting is that it gets us all amped up to hack on whatever we want all day the following day.

The Monday after an R&D day, each team or individual gives a five-minute presentation on what they achieved, what they learned and whether it’s something they want to pursue further.

Successes so far

The 99designs R&D program was implemented six months ago and in those 10 or so days we’ve managed to come up with some useful tools and fun apps.

One of our very first projects to be adopted as a mainstream product feature was an admin toolbar. This project came from a few developers who were frustrated with how difficult it was to search for and manage contests, so during our very first R&D day we decided to form a team and build a solution. To the delight of our staff, the toolbar is now part of the application and used on a daily basis.

Another frustration among developers was the lack of visibility and subsequent difficulty managing errors within our application. Yes, we have error logs. But who in their right mind likes trawling through them to get to the bottom of an issue? “Triage” is the aptly named error management system we developed over eight R&D days. This open source, ongoing project will soon be rolled out into our application and will make our lives a lot easier.

On the fun side, one of our developers created a nice little app called Mapviz, which provides a real-time visualisation of all customer log-ins, contest launches and design entries on a world map. Our marketing team was blown away by it.

Challenges

One challenge we face is coming up with a way to involve all of our teams, including support and marketing, in the R&D programme. Involving non-technical team members from time to time could allow the development team to get inspiration and ideas from a different angle. Then there’s the challenge that comes with being a global company – the Pacific Ocean creates quite a barrier between our Melbourne and San Francisco offices, making it tough for us to consistently communicate ideas efficiently and effectively.

Secondly, is an R&D day twice a month the best approach for fostering innovation? Our friends at Flippa have taken on what they like to call “Triple Time,” in which the development team dedicates three consecutive days every month to R&D. This approach might work well for us, however, giving it a go will be a challenge when we are so used to our current programme.

Another interesting approach is the “FedEx day” championed by Atlassian, during which a team is given one day to build and demonstrate a finished or near-complete product. Could introducing a deliver-in-a-day element hinder or supplement our current programme?

Looking ahead

The evidence is clear that we’re off to a good start – we’re delivering some great products, helping to foster an innovative company culture and learning new technical skills. However, like everything at 99designs, the programme is continually evolving and improving.

I’m curious to learn more about what other companies are doing to foster innovation. Does your company have an R&D program? How is it structured – as a percentage of time, designated days, another format? How else are employees encouraged to innovate?

by Chris Campbell

This post describes feature flipping, an approach to development that helps solve some of the issues associated with risk management and quality assurance when a fast moving development team expands.

Continuous deployment in large teams

Developers at 99designs use agile and lean startup methodologies, but as our development team gets bigger, deployments happen more frequently. This volume of change brings increased risk for the stability of our site. It can also be quite a challenge to measure the success of a single new feature on a rapidly changing website with multiple new features operating at any given time. In our earlier days we’d demo new features on a staging server, but as 99designs has grown, using several staging servers to demo all our new features at once has become clunky. So how do we solve these issues? Feature flipping.

What is feature flipping?

Feature flipping enables the ability to turn site features on or off on a per-user basis. You’ve probably encountered it before with companies like Google or Facebook when they’re rolling out major changes. A few examples include the recent UI changes to Google Docs and Google Mail, and Facebook’s new Timeline. Our approach was inspired by our friends at Learnable, and slots in well with the Lean Startup methodology of releasing minimum viable products, measuring, and adapting through fast feedback.

Rolling out features incrementally gives companies the ability to ensure the appropriateness and stability of a feature. Some companies make this visible to users via an opt in/out approach, but in our case we use this internally as an improved form of A/B testing. Traditionally we would only A/B test on landing pages and static content, but feature flipping also allows us to experiment with pervasive site functionality.

How it works at 99

Before running through our approach, let’s first examine it from a developers perspective and from a site admin’s perspective.

Developers

As a developer, you create a new feature simply by registering it in a specific module in our codebase.

1

features.add(name='raptors', description='Unleashes raptors over the site')

Now that we’ve defined our feature, it’s available for use in a conditional statement, which you use to guard feature-specific code.

1
2

if feature('raptors'):
    unleash_raptors()

Boom! Our raptors feature can now be toggled on or off for any user who interacts with our site.

To kick off an incremental roll out, a commit is required with a defined set of eligibility criteria a user must meet before the feature is enabled for them. For example, the user must be a designer and needs to have submitted a minimum of 5 entries.

1

feature('raptors').rollout(role='designer', min_entries=5)

Of course, we make sure to test the new behaviour in addition to the old.

Admin

99designs staff get a nice interface where they can turn on and off features for themselves or a specific user. Here they can also view the progress of any experiments hanging off this feature. Many of our stakeholders are located remotely; by giving them the power to enable features themselves, they can try these out in production and provide feedback. This removes the need for a dedicated staging environment for demoing.

Under the hood

When we first considered how best to implement features, we recognized that every feature we introduced would require us to perform an additional user-feature check. This would clearly result in rapid growth of queries as we began checking features for every user on every page of our site.

For logged out users, we decide to to go with a cookie based solution. However, for logged-in users we wanted our user features to be persistent. We thought about using MySQL, but based on Redis benchmarks and its inbuilt sets data structure, going this route seemed an ideal solution. We decided to go with a combined cookie and Redis approach to minimise the number of database lookups without impacting the performance of our site.

These two methods work well together. When a user logs into 99designs, cookie based features are synched up with Redis by taking a simple union of features enabled and persisting the resulting set. This allows us to determine exactly who has what features as well as when a user obtained those features.

Challenges

Developing with a multi-version mindset is probably one of the most challenging parts of feature flipping. What was once a simple deployment requiring a migration now entails more thought to allow the old and new versions to function independently of one other.

Deprecated code can start to accumulate when you start leaving code wrapped in feature checks. We try to curtail this as much as possible by coming back to clean up unused code paths once a feature has been fully rolled out. Also, we generally have a grace period even for features we’re decided on, so that we can roll back if the need arises.

Unit and integration tests become more difficult when you’re working with an exponential number of feature combinations at any given time. This can lead to an outsized volume of test cases, a problem we’re still solving.

What now?

Feature flipping is our approach to solving several problems we face as 99designs expands. Managing features gives us fine-grained control over exactly what our users see, and is paving the way for us to experiment and adapt to our users much faster than was previously possible.

Big companies often execute this approach to development seamlessly, while there’s more of a learning curve for small to mid-sized companies. In our case, we’re still learning but we’re far enough along to be reaping the benefits.

Where to go from here? We suggest you explore a few open source solutions that deal with features:

• flip - Declare and manage features.
• rollout - Conditionally roll out features with redis.
• degrade - Keeps track of features, degrades functionality if errors are too high.

by Richard Bone

Although our user facing site is the most visible part of what we do, it’s only half the story. The other half takes part in asychronous queues, which work overtime behind the scenes, and process hundreds of millions of tasks each year. In this post I’ll explain a bit about why we use queueing at 99designs, and how it all works.

A Bit of Background

If you’ve never heard of asynchronous task queues before, the idea behind them is pretty simple. Say you had a task you needed to do such as buying some milk but you didn’t have the time to take care of it yourself, so instead you leave a note for a friend/spouse/roommate asking them to do it for you when they have a chance. Congratulations, you’ve just implemented an asynchronous task queue.

Why We Use Queues

Now obviously our web apps aren’t busy ordering milk, more common uses for a queue are things like talking to third party API’s, sending emails, or performing computationally expensive tasks like image resizing. But why do we need a queue at all? Wouldn’t it be easier to just do the work a user requires immediately? Well, there’s a few reasons:

The first reason is speed: When we’re talking to a third party API we have to face reality; unless that third party is physically located next to our infrastructure, there’s going to be latency involved. All it would take is the addition of a few API calls and we could easily end up doubling or tripling our response time, leading to a sluggish site and unhappy users. However if we push these API calls into our queue instead, we can return a response to our users immediately while our queues take as long as they like to talk to the API.

The second reason is reliability: We don’t live in a world of 100% uptime, services do go down, and when they do it’s important that our users aren’t the ones that suffer. If we were to make our API calls directly in the users requests we wouldn’t have any good options in the event of a failure. We could retry the call right away in the hope that it was just a momentary glitch, but more than likely we’ll either have to show the user an error, or silently discard whatever we were trying to do. Queues neatly get around this problem since they can happily continue retrying over and over in the background, and all the while our users never need to know anything is wrong.

The final reason to use a queue is for scalability. If we had a surge in requests that involved something CPU intensive like resizing images, we might have a problem if all of our apps were responsible for this. Not only would the increased CPU load slow down other image resize requests, it could very well slow down requests across the entire site. What we need to do is isolate this workload from the user’s experience, so that it doesn’t matter if it happens quickly or slowly. This is where queues shine. Even if our queues become overloaded, the rest of the site will remain responsive.

How We Implement It At 99designs

Understanding why we use queues is one thing, actually implementing it is quite another. In the case of 99designs we chose beanstalk as our queue, it’s key features being performance, reliability and simplicity. Rather than having just one centralized queue which our servers push tasks to, we instead have a separate queue on every app server. Queuing a task locally is always going to be very quick, and having multiple queues gives us a healthy dose of redundancy in the event of a beanstalk failure — something I’ve yet to see.

Of course on its own a queue doesn’t do anything; simply pushing tasks onto it doesn’t magically make them happen. For that you need a worker. Since a lot of what 99designs does is PHP-based we had to develop our own custom worker daemon, as there weren’t any open source solutions for beanstalk. Each worker node maintains a pool of worker processes to connect to each queue and listen for new tasks. In the event that our queues start to back up, we can easily launch a new worker instance thanks to AWS and have it processing tasks within minutes.

The final part that ties all of it together is how we deal with failures. In the event that a task fails it doesn’t simply disappear into the ether. Instead we’ll release the task with a delay, essentially putting it back onto the queue after a certain amount of time has passed. If a task continues to fail we use an exponential backoff strategy to prevent failing tasks from clogging up our queues. If after multiple releases the task still has not finished successfully then we “bury” it. Buried tasks are then manually inspected at a later date, and a decision is made to either fix the problem (if possible), or delete the task if it is deemed unrecoverable.

Work in progress

There’s still more that we’d like to do with our queueing systems. If too many tasks fail, and are buried, we get alerts which wake us in the night. Sometimes, this happens because an external service we use goes down, something we can’t do much about anyway. Unfortunately, our tasks don’t yet coordinate their back off in any way; each assumes it’s the only one failing. Ideally, we’d track failures against external APIs, and simply delay groups of tasks for longer, giving third party services time to recover.

In closing

Asynchronous tasks are a crucial, and often overlooked, part of how any large site operates. We hope this post gives some insight into why queues are important, and how they can improve your site’s performance and reliability. Whilst there are many different queuing architectures possible, we also hope our recipe serves as a useful pattern for other sites.