Improve your experiments with a better control

Client-side testing tools render content and URL redirects on the client (browser), which will cause a small deterioration in user experience. This deterioration will vary but as marketers, we accept this based on the advantages we gain from rapid execution compared to working with a less agile server-side implementation. Client side testing tools generally have 2 methods for displaying content:

1. Overlaying
Your original or control content is loaded and overwritten with new content using JavaScript. Depending on the testing tool some use CSS to hide the original content before the new content is added to prevent a flickering effect, such as:

"visibility:hidden;" or "display:none;"

2. Redirects
Your testing software makes a client side JavaScript redirect using a command such as:

 window.location = "http://www.yoururl.com/variant-a.html"; 

Redirects are useful when your variation code exceeds 200 lines of code or the layout you're testing is radically different, it could be you're testing a completely different website on a separate domain.

So what's the problem?
Both of the methods mentioned above could negatively impact your test result leading to an invalid result, this is how:

• Your variant page takes too much time to load due to the amount of additional code being output.
• Tracking fires less consistently with a redirect or overlay.
• Page flicker caused by an overlay or redirect (probably the biggest issue to look out for, especially with redirects).

All these scenarios give the control an advantage and invalidate your experiment. Of course you could argue that all of these issues should be picked up in QA but intermittent and browser specific problems can be hard to detect. Another solution could be to run regular A/A tests but that aside it should still be a best practice to create a more 'apples to apples' environment.

Solution
The fix is simple. When using an overlay also do the same on the control but of course you overlay with the original content - so nothing changes other than we now mirror the rendering mechanism of the variants we're testing. The same applies for redirects, so we redirect the original URL back on itself - you need to be careful that you don't create a redirect loop, I wrote a quick post on how to prevent this with Adobe Target here. We now have a fairer comparison with a more consistent user experience.

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.