Speed Wins: Experiment and Learn

This is part 2 of the 4 part series: Growing eCommerce revenue $0 to $200MM at TripAdvisor

Lesson 2: Speed of experimentation and learning leads to sustainable success 

After establishing a foundational pattern that works for your product, the team’s focus shifted to breaking down what was working about this first product experience and finding ways to tweak and optimize the parts to drive up our conversion rate. To accelerate our ability to learn and grow revenue per shopper, we divided traffic & usage on each of our platforms (DW + TW, MW, Native Apps)  into 3 spaces - a baseline, a control, and a test space. The baseline is the version of the site that we started out with, control is the most current version of the site, and the test space was an area to perform A/B testing throughout the quarter. These three spaces enabled us to do two main things

1. Single factor test and optimize each component of the product experience. (Test slices vs. Control)
2. Clearly quantify the team’s impact to the business over time. (Control vs. Baseline)

We divided our resources into autonomous “two pizza” teams - the size that could be fed by 2 pizzas at lunch. Generally, this meant a ratio of 1 PM to 6 Engineers, along with a Designer and a user research resource as well. Then we’d allow these teams to go off and chase their own hypotheses, build their own A/B tests, do their own user studies, and keep track of their own wins. If you’re in an environment like that, here are four important things we learned while in the test-driven optimization phase of our product launch.

1. Invest in AAA measurement. An organization built of multiple teams optimizing parts of a single experience with minimal central control requires belief in data driven decision making. If there is any doubt about the accuracy of the data that supports the underlying decision making and actions of a team, that teams’ autonomy is weakened, cycle times slow down, and decisions are more influenced by personality and committee than an objective analysis of what is best for the business. To avoid this, invest resources to build and care for a centralized measurement and data platform that is actionable (teams know how to adjust roadmap and projects based on clearly reported impacts to KPIs that matter), accessible (timely, clearly visualized, easy to use), and auditable (anyone can easily check the math). 
2. Nominate a “test wrangler.” Depending on the amount of traffic you have to test with, you  may have so many tests running at any given moment that individual tests might conflict with one another. This happens when 2 or more tests are changing the same exact part of the experience. Minimize test conflicts with a little bit of up front planning and coordination from a person who oversees and schedules all the test spaces. This can be done on a rotation so that multiple people get hands on experience on how the test platform works.
3. 1+1 rarely adds up to 2. Here’s the scenario - Team A completes an A/B test and confirms a 1% lift in their target KPI. Soon afterward, Team B completes an A/B test and confirms an additional 1% lift in that same KPI by optimizing a different part of the experience. Everyone celebrates and pushes both features out to to Control . . . only to observe a mere 1% lift relative to the Baseline sustained over multiple weeks. You’re left with questions -  What happened? Where did the other 1% go? And importantly for the team members - which optimization made which team resulted in the 1% lift we’re now seeing in the measurement? The answer lies in the margin of error within each of the original tests, as well as the margin of error of the current Baseline vs. Control. Digging deeper, you’ll uncover a less than 1% MoE within each of the original tests. Probably with the current Baseline measurement. All that adds up to Noise upon Noise upon Noise.  You’ll pull your hair out and lose valuable days (weeks?) trying to derive precision that’s extremely hard to come by when you could be testing new features.