Conversion ranges and confidence intervals
On the 24th June, the UK nation woke to the news that the country had voted in favour of leaving the EU. This was mostly a surprise. The financial markets and sterling had rallied the night before, confident that the UK would stay. But why the confidence, what did they know? The polls had always reported a close outcome. Part of the reason for the surprise was the media (no different to a digital marketer) misleading their audience by communicating stats without a margin of error or confidence interval (which is a range of outcomes). A YouGov poll published a day before the vote showed 51% of those surveyed supported staying in the EU while 49% supported leaving. The margin of error was 3 percentage points (not mentioned by the media). Those +/- 3 percentage points means the result could equally end up (as it did):
- Leave (49 + 3) = 52%
- Remain (51 - 3) = 48%
A common question when working in Digital Marketing is: "How's the test doing?" And maybe you reply: "It's up 20%". And maybe you present your results in a similar format:
If you're communicating absolute conversion numbers - here's the problem: it's misleading. Although I've never run such a survey, I'm sure a good proportion of people would read the proceeding stats as "we're 95% confident the variant outperforms the control by 20%". Of course, we know that's not true, right?? However, it's easy to understand why people are misled. Instead, we should communicate a conversion range and confidence interval. So now we'd say "the confidence interval for the variant is: 8.3% to 9.1%'" or "a range of likely values for the population mean is 8.3% to 9.1%, with a confidence level of 95%". Using the Control and Variant confidence interval we then calculate the conversion rate delta range. VWO (Visual Website Optimiser) reports this but I've struggled to find an online tool that does this, however you can use Ewan Millar's Chi-Squared Test tool: http://www.evanmiller.org/ab-testing/chi-squared.html for calculating the confidence interval between 2 samples and then calculate the delta between the upper and lower range, so using the stats from the preceding table we end up with:
- upper range: (9.1 - 6.9) / 6.9 = 33%
- lower range: (8.3 - 7.6) / 7.6 = 9%
- which gives us a conversion range: 9 to 33%
So there you have it, start being less specific with conversion rate while adding absolute values for context. Alternatively, report on the mean value but add the margin of error. It may confuse some to start with but it's more accurate and makes more sense.
Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.
