Building an experiment dashboard

This article will explain a cool way of monitoring and presenting your experiments that will save you time and make you look (more) awesome! Prerequisites are you're familiar with; Adobe Target, Analytics/Omniture, Report Builder, Excel and you're using the Adobe Target plugin ...that being said you can probably repurpose this for other analytic and testing tool environments.

What problem does this solve?
Building an experiment dashboard gives you a single view of all your experiment metrics without having to create multiple reports spread across browser screens. Moreover it can also become your experimentation documentation, and the dashboard format will copy & paste easily into presentations, so ultimately you'll be killing many birds with a single stone (sorry birds).

How to do it
For saving time in the long run we want to make our dashboard reusable so this means creating templates per experiment type. The 'types' are defined by KPI and number of variants. So if you're running an A/B experiment and your main KPI is orders - this would be a template, likewise so is an A/B/C experiment with orders as the KPI. You may have another A/B template where registrations/installs are your main KPI ....you get the idea. Once you've created the template types for your common experiments you'll only need to change a couple of parameters in your excel sheet and hey presto everything works! So let's get started with an example; here are the steps to create a dashboard for an A/B experiment where orders are the main KPI.

Step 1)
Plan the dashboard. When running an A/B experiment one of my favourite reports is viewing the conversion/order rate daily performance versus the control. This simply shows whether it won or lost and by how much and is a great way to see if there were any anomalies to investigate. Additionally we want to see the running totals so our first 2 charts look like this:
The next 2 are very similar but this time we're looking at revenue per visitor:
And the final 2 we're adding are for distribution (how many order and visitors have been part of the experiment) and average order value:
Step 2)
Next we fire up Report Builder and add our data blocks, we need to use the "Campaign > Recipe" dimension which allows us to select the experiment recipes/variants.
It's advisable to have a "settings" sheet in the Excel document where we can add; the running date of the experiment and the "Campaign > Recipe" values. The data blocks will then reference the cell locations (as it's doing in the screen captures above and below). For future experiments it means we just change a few Excel cell values and everything works opposed to editing a load of data blocks - which isn't fun.

Campaign > Recipe values:
Step 3)
Add calculated fields, conditional formatting, charts and make it look pretty. I'm not going to go into detail as I'll assume you know Excel and the template used in this example can be downloaded below.

Step 4)
Optionally you may want to add a section for describing the experiment and documenting the outcome. In a separate sheet you can also add screen captures of the control and variant content.

And once done the final output will look something like the below. It's worth noting that we should try not to go overboard with charts, our goal is to have a good overview. We can dig deeper if needed by going to our Analytics tool. That being said adding metrics such as visitors, orders and revenue trended seems perfectly valid.

Hopefully this was useful - the Excel template used in the above example can be downloaded here. If you have any comments, questions or feedback please leave them below. And you can follow new posts from this blog on Twitter, Email or RSS.