Design is about solving problems. When it comes to user experience (UX), design matters a lot. Like promoting the brand, getting users to take action, providing a fast response — several factors need to be considered to make sure users keep coming back for more. Even after optimizing all these factors, certain user interfaces fail to achieve the expected reception. This is why companies leverage experimentation platforms to test and optimize all the aspects of a product before making it available to the users. Even after enough optimizations, companies do not immediately launch it universally. Instead, they prefer a staged rollout.
A staged rollout is a process of rolling out a feature first to a smaller crowd and gradually scaling up to reach 100 percent of the users. This way, the key metrics are monitored and optimized at every stage. Staged rollouts help in understanding the pros and cons and streamline the areas that require improvement before letting them affect the entire customer base. Staged rollouts aim to make the new features as stable as possible by controlling user exposure. Besides enabling engineers to fix things at the early stages, the experimentation platforms also ensure that they do not cause any regression to the key metrics.
Like staged rollouts, there are a lot of other interesting techniques that companies use to improve their user experience. In this article, we will be discussing how Uber, Twitter, and Skyscanner leveraged experimentation to do exactly that.
Uber XP
For a business with a huge customer base, it is important to learn from the customers’ user experience insights before launching a new feature. This is why Uber leverages an experimentation platform that drives their decision making. Experimentation is at the core of Uber’s customer experience improvement and is a critical part of the product lifecycle.
Uber’s experimentation platform, commonly known as the Uber XP, aims to roll out all the new features successfully. More than 1,000 experiments run on the platform at any given time. Be it a feature, a marketing campaign, or even a new idea, Uber XP enables the team to precisely monitor and measure the effects. This way, the platform returns actionable analysis. The platform works on a real-time basis and helps the company make immediate improvements.
Uber XP’s biggest strength is the continuous monitoring that it makes on all its apps — driver, rider, restaurant, eater, and delivery-partner. When it comes to new features, Uber always opts for staged rollouts.
As the number of metrics used by the XP analytics grows it becomes more challenging for users to evaluate the experiment. The recommendation engine facilitates the discovery of metrics. The engine recommends metrics based on a score. This score is calculated by monitoring the correlation of two metrics. The engine calculates this score for all the metrics and recommends the one with the highest score.
Tweet-length analysis
According to the feedback received from the users of Twitter, most English tweeters had to edit their tweets a lot of times to make their thoughts fit into the 140 character limit. Some even had to remove certain important words. Twitter planned to extend the character limit to 280 characters. This would reduce the frustration caused by cramming and give enough room for people to express their thoughts. Though the team was confident about this move, they took a staged approach. Initially, this extension was made available only for a smaller crowd. Only after witnessing positive results the company launched it to everyone.
Also, the limit extension was launched only for certain languages. Because, based on the experiments, the 140 character tweet limit is not always suitable for all the languages. Different languages use different quantity of words to convey ideas. For example, English tweets tend to quickly run into the character limit whereas Japanese Iku tweets are left with some room to spare even after sharing enough thoughts. In fact, most of the Japanese tweets just have 15 characters. Languages like Japanese, Chinese, and Korean can express more thoughts in fewer characters. But Portuguese, Spanish, French, and English require more characters.
Twitter always uses enough tests and experiments for all its launches even if the team is confident about the results. Twitter values their community and takes feedback seriously. This is the reason why the company’s products get off the ground as soon as they hit the market.
Twitter Lite is another example of Twitter’s success through experimentation. Twitter Lite minimized data usage and loaded quicker even on slow connections. Twitter tested it initially with a small group of customers before a universal roll out. As a result, the tweets from Twitter Lite increased more than 50 percent. Twitter also continuously engages with users, experiments and keeps adding useful features like GIF search, ranked home feed etc.
Skyscanner’s Dr. Jekyll
Travel-fare aggregator Skyscanner is known for building an experimentation culture, not just with the customers but employees too. The team incorporates experimentation in everything they do. They embrace testing as a part of their daily routine. This is how they were able to get 60 million monthly users with their site available in 30 languages. They provide out-of-the-box features like a news section that updates users with the happenings in the travel industry and also provides useful travel tips — a nice way to improve the user experience.
They have an in-house experimentation platform called Dr. Jekyll tailored for the internal users. They aim to democratize experimentation using a user-friendly UI to explain the experimentation cycle. They have developed a “Hypothesis Kit” to guide people with no (or limited) experience to start running experiments quickly. Besides customer feedback, they also give a lot of importance to peer review. The experimentation platform has a dedicated slack channel, especially for this purpose. They also invite expert speakers and encourage colleagues to attend conferences for external inspiration. More than inspiration, these conferences help them get new perspectives and think out of the box.
Their “design like you are right and test like you are wrong” approach helps experimenters put themselves in the shoes of the end user. When they don’t achieve the results that they expect, they use a fail forward approach. In this approach, every failure is valid as a success. According to them, a failure can provide insights that hypotheses can’t. They incorporate the insights into the next experiment and use every failure to inch closer to success.
Improve user experience with data analysis and experimentation
Decision making has become data-driven and there is no place for assumptions today. Even hypotheses are made with proven experiments rather than hunches. Digital transformation is increasing the number of competitors day by day for almost all kinds of businesses. Today it is about identifying opportunities and issues to solve. The best way to achieve that is through experimentation. Companies like Skyscanner test with their internal users, some like Uber and Twitter test with a selective group of customers, and some companies take experimentation to the next level by allowing everybody to experiment. For example, when Snapchat’s augmented reality lenses became popular, they rolled out the World Lens platform. Using this platform anyone can design the lenses. Big Data and data analysis tools help data scientists to make hypothesis and predictions but then, only experimentation can help them understand the actual impact of an idea. Data analysis and experimentation are becoming popular among small businesses too. Experimentation will be democratized and become an inevitable component of every business in the future.
Featured image: Shutterstock
