Google’s guide to A/B and multivariate experiments (Techie Guide) is actually only about 25 very readable pages and includes such phrases as “full-on hairy scenario,” which I love. In reading through the Techie Guide, the one thing I found amusing/typical is the “how to test” portion is a single sentence: Review the data, test the pages using the preview function, and then launch your
test!
But, it does show a screenshot of the test from within Google Website Optimizer (GWO) interface, which is much like the Google Analytics site that we already know and love. So, the GWO is one place to QA the scripts, in addition to actually visiting the site you are testing, validating the user experience and GWO report, clear cookies, repeat.
In a nutshell, there are four scripts for standard multivariate (MVT) and three for A/B.
- The Control Script at the top of the page (this loads the JavaScript functions used to replace section content for MVT or redirect the user for A/B). The Control Script makes a request from Google’s servers for an additional script, siteopt.js, appending the experiment key and GWO cookie values, like this:
http://www.google-analytics.com/siteopt.js?v=1&utmxkey=<experiment key>&utmx=<__utmx cookie value>&utmxx=<__utmxx cookie value>&utmxtime=<request timestamp>
- The Section Script (utmx_section) right before a section with a </noscript> at the end (for MVT only)
- The Tracking Script JavaScript (typical GA code)
- The Conversion Script JavaScript (e.g. on your “thank you” page)
Notes for testing:
- Of course, understand what you are testing. Are users shown different experiences 50/50, 20/20/60 or other. What are the dependencies, or is it entirely random?
- As usual, opt to display a notification about every script error within your browser settings.
- GWO cookies have a two-year lifespan.
- Unique visitors are counted only once for visits and conversions, regardless how many times (s)he visits test or conversion pages.
- Conversions are only counted if user came through test page executing the control & tracking scripts. If user happens to reach the conversion page by via other route, they won’t be counted as conversion.
- Be aware that some A/B and MVTs are more complicated than others (think dynamic content, session variables and more).
P.S. Interesting related articles, including some actual A/B tests & results:
Marketing: A/B Split Testing Explained
A/B Testing: The difference one line can make
ABtests.com
25 Google Optimizer Tips