This post describes feature flipping, an approach to development that helps solve some of the issues associated with risk management and quality assurance when a fast moving development team expands.
Continuous deployment in large teams
Developers at 99designs use agile and lean startup methodologies, but as our development team gets bigger, deployments happen more frequently. This volume of change brings increased risk for the stability of our site. It can also be quite a challenge to measure the success of a single new feature on a rapidly changing website with multiple new features operating at any given time. In our earlier days we’d demo new features on a staging server, but as 99designs has grown, using several staging servers to demo all our new features at once has become clunky. So how do we solve these issues? Feature flipping.
What is feature flipping?
Feature flipping enables the ability to turn site features on or off on a per-user basis. You've probably encountered it before with companies like Google or Facebook when they're rolling out major changes. A few examples include the recent UI changes to Google Docs and Google Mail, and Facebook's new Timeline. Our approach was inspired by our friends at Learnable, and slots in well with the Lean Startup methodology of releasing minimum viable products, measuring, and adapting through fast feedback.
Rolling out features incrementally gives companies the ability to ensure the appropriateness and stability of a feature. Some companies make this visible to users via an opt in/out approach, but in our case we use this internally as an improved form of A/B testing. Traditionally we would only A/B test on landing pages and static content, but feature flipping also allows us to experiment with pervasive site functionality.
How it works at 99
Before running through our approach, let's first examine it from a developers perspective and from a site admin's perspective.
As a developer, you create a new feature simply by registering it in a specific module in our codebase.
features.add(name='raptors', description='Unleashes raptors over the site')
Now that we've defined our feature, it's available for use in a conditional statement, which you use to guard feature-specific code.
if feature('raptors'): unleash_raptors()
Boom! Our raptors feature can now be toggled on or off for any user who interacts with our site.
Do you like raptors? We like raptors!
To kick off an incremental roll out, a commit is required with a defined set of eligibility criteria a user must meet before the feature is enabled for them. For example, the user must be a designer and needs to have submitted a minimum of 5 entries.
Of course, we make sure to test the new behaviour in addition to the old.
99designs staff get a nice interface where they can turn on and off features for themselves or a specific user. Here they can also view the progress of any experiments hanging off this feature. Many of our stakeholders are located remotely; by giving them the power to enable features themselves, they can try these out in production and provide feedback. This removes the need for a dedicated staging environment for demoing.
Enabling or disabling a feature is just a button click
Under the hood
When we first considered how best to implement features, we recognized that every feature we introduced would require us to perform an additional user-feature check. This would clearly result in rapid growth of queries as we began checking features for every user on every page of our site.
For logged out users, we decide to to go with a cookie based solution. However, for logged-in users we wanted our user features to be persistent. We thought about using MySQL, but based on Redis benchmarks and its inbuilt sets data structure, going this route seemed an ideal solution. We decided to go with a combined cookie and Redis approach to minimise the number of database lookups without impacting the performance of our site.
These two methods work well together. When a user logs into 99designs, cookie based features are synched up with Redis by taking a simple union of features enabled and persisting the resulting set. This allows us to determine exactly who has what features as well as when a user obtained those features.
Developing with a multi-version mindset is probably one of the most challenging parts of feature flipping. What was once a simple deployment requiring a migration now entails more thought to allow the old and new versions to function independently of one other.
Deprecated code can start to accumulate when you start leaving code wrapped in feature checks. We try to curtail this as much as possible by coming back to clean up unused code paths once a feature has been fully rolled out. Also, we generally have a grace period even for features we're decided on, so that we can roll back if the need arises.
Unit and integration tests become more difficult when you’re working with an exponential number of feature combinations at any given time. This can lead to an outsized volume of test cases, a problem we're still solving.
Feature flipping is our approach to solving several problems we face as 99designs expands. Managing features gives us fine-grained control over exactly what our users see, and is paving the way for us to experiment and adapt to our users much faster than was previously possible.
Big companies often execute this approach to development seamlessly, while there’s more of a learning curve for small to mid-sized companies. In our case, we’re still learning but we're far enough along to be reaping the benefits.
Where to go from here? We suggest you explore a few open source solutions that deal with features:
- flip - Declare and manage features.
- rollout - Conditionally roll out features with redis.
- degrade - Keeps track of features, degrades functionality if errors are too high.