Skip links

Designing Microsoft’s Experimentation platform

Leading the end to end design experience and introducing design into the software development process.

July 2016- July 2018

Reading time: 10 minutes

Introducing the Experimentation Platform (EXP)

The Experimentation and Analysis team in Microsoft is responsible for building the platform for A\B testing across Microsoft. The platform was established in 2009 and is an internal product used by teams within Microsoft to author and run their A/B tests (also known as experiments).

The platform is composed of four products namely


Microsoft’s internal score carding tool

Control Tower

The A/B experiment authoring tool


The platforms metrics authoring tool


The admin experience for control tower

I was the first ever full time UX designer hired to the team and was responsible for all the UX activities across the platform.

This was also an interesting challenge since the team structure was as follows




Data scientists




Program managers

Getting Started

When I joined the team, I was surprised to find that a significant number of the Program managers, developers and data scientists had never worked with a designer and no idea how I fit into the development life cycle.

To address that, my initial primary objectives were as follows.

Meeting with the developers, PM’s and data scientists to identify what their expectations from design

Making a list of all the product and feature areas that needed design guidance along with the priority

Analyzing the product with user studies to identify key improvements

Based upon my conversations with the team and going through the platform, I realized that while the team had strong opinions about the product (especially xCard), none of the feedback or concerns had been cataloged anywhere. I spent two weeks talking to 5 data scientists and one development manager to get their views on the platform.

Based upon the conversations, I identified three main areas of improvement where we could look at short term gains.

Filtering and Navigation

Reducing the number of filters and options available.

Metrics slice list

The list of all the metrics in xCard and how it was surfaced.

Metric details

Details about the metrics performance and variances.

These three areas were determined by where the most feedback was as shown in the feedback summary below.

[su_image_carousel source=”media: 581,582,583″ slides_style=”photo” controls_style=”light” crop=”none” align=”center” arrows=”no” dots=”no” image_size=”full”]

Setting up a Style guide

Taking this feedback to leadership, I started to look at how I could address some of the issues. On looking at xCard closely, I noticed inconsistency in colors, fonts, button and form field styles and general UI consistency.

On discussing with the development team, I found that since the team never had any design help, the development team often implemented them as they wanted leading to inconsistent experiences.

As one of my first tasks, I set out to create a style guide which would serve as a baseline for me for my designs and also the framework for a future component library to ensure design consistency for the development team

I also realized that while there was a general idea of how the entire product flowed together, there was no clear defined flow with regards to how everything tied in. Hence one of my tasks was to define an end-to-end scenario map which showed all the integration points between the different tools.

Download PDF version

Starting with xCard

Since xCard had the most critical feedback and was the tool used by all the data scientists in the team and the internal customers, taking the feedback from the user interviews conducted before, I created multiple wireframes and presented them to the larger team and leadership team in our off-sites.

The feedback taken from the meetings was then used to define the final screens which were handed over to the developers.

Post development, I also conducted several usability tests with users to evaluate if the changes proposed worked for them and the improvements we could continue to make.

Study template

Introducing ProgressionsExperiment authoring for Microsoft Office

Progressions was a new feature introduced inside control tower on the behest of Microsoft office where they wanted the experiment to run through a series of progressions or stages. The stages would generally involve internal groups before moving on trusted outsiders and then worldwide.

Taking on Sauron

Sauron, interestingly named because of how complex and scary the actual experience was. Based upon discussions with the data scientists in the team, I created a new information architecture which was based upon key tasks namely

  1. Setting up the tenant
  2. Setting up the experiment
  3. Setting up the analysis details
  4. Advanced settings or other options

I then printed out the designs as large posters and, in the team offsite, collected feedback by asking people to provide feedback via post-it notes

Download user study posters

The feedback from the designs lead to the final updated screens as shown below.

Experiment authoring

I also worked on simplifying our experiment authoring experience and used the off-sites to get feedback and iterate and simplify the experience

Helping the EXP platform move to the Azure portal

There was a strong push from leadership at the CVP level to stop having the experimentation platform be an internal only product but also an external product which would be available in the Azure portal. I helped design the proof-of-concept screens and scenarios for that.

Reorgs and priority shifts

Unfortunately, due to multiple re-orgs and priority shifts, this potential move to make it an external experience never took place, and my skip (Manager’s manager) asked me to move to the Azure ML team to help them with their design needs.

View AzureML Case study