Google’s guide to A/B and multivariate experiments (Techie Guide) is actually only about 25 very readable pages and includes such phrases as “full-on hairy scenario,” which I love. In reading through the Techie Guide, the one thing I found amusing/typical is the “how to test” portion is a single sentence: Review the data, test the pages using the preview function, and then launch your test!

But, it does show a screenshot of the test from within Google Website Optimizer (GWO) interface, which is much like the Google Analytics site that we already know and love. So, the GWO is one place to QA the scripts, in addition to actually visiting the site you are testing, validating the user experience and GWO report, clear cookies, repeat.
In a nutshell, there are four scripts for standard multivariate (MVT) and three for A/B.

1. The Control Script at the top of the page (this loads the JavaScript functions used to replace section content for MVT or redirect the user for A/B). The Control Script makes a request from Google’s servers for an additional script, siteopt.js, appending the experiment key and GWO cookie values, like this:
http://www.google-analytics.com/siteopt.js?v=1&utmxkey=<experiment key>&utmx=<__utmx cookie value>&utmxx=<__utmxx cookie value>&utmxtime=<request timestamp>
2. The Section Script (utmx_section) right before a section with a </noscript> at the end (for MVT only)
3. The Tracking Script JavaScript (typical GA code)
4. The Conversion Script JavaScript (e.g. on your “thank you” page)

Notes for testing:

1. Of course, understand what you are testing. Are users shown different experiences 50/50, 20/20/60 or other. What are the dependencies, or is it entirely random?
2. As usual, opt to display a notification about every script error within your browser settings.
3. GWO cookies have a two-year lifespan.
4. Unique visitors are counted only once for visits and conversions, regardless how many times (s)he visits test or conversion pages.
5. Conversions are only counted if user came through test page executing the control & tracking scripts. If user happens to reach the conversion page by via other route, they won’t be counted as conversion.
6. Be aware that some A/B and MVTs are more complicated than others (think dynamic content, session variables and more).

P.S. Interesting related articles, including some actual A/B tests & results:

Marketing: A/B Split Testing Explained
A/B Testing: The difference one line can make
ABtests.com
25 Google Optimizer Tips

In building mobile sites, you simply cannot optimize for every device/browser/OS out there. So before you can think about a test plan, have your client pick their top 5 browsers/OS configurations and screen sizes — based on their scope, product, audience and locale — and shoot to optimize for those. Just like regular browsers, there will still be minor differences here and there.

Help your client narrow down mobile browser/OS configs:

1. Start by viewing their mobile analytics, of course.

2. Determine the scope of their mobile offerings, as well as their audience and locale.

3. Check out DeviceAtlas. Navigate deeper to see specs on each device, such as operating system, screen size, color depth, etc.

4. What are the top mobile configurations, you ask? Well it largely depends on locale, what’s new, etc. etc. They change frequently, but here’s a relatively current list of Top 10 selling smart phones spanning more than six continents and 50 countries.

The testing side:

Note that a good bit of testing can be done without an emulator or mobile device. Depending on the scope of the site, of course, it is possible to do most of your functionality testing on a full-size browser. Then move to the mobile to focus on design elements/rendering, formatting, usability. Don’t forget to check both portrait and landscape modes.

1.  Before writing your test plan, familiarize yourself with W3C Mobile Web Best Practices and Mobile Web App Best Practices. These are looooong, I know. But at least skim them to get a feel for some of the high-level best-practices. (At some point, I want to take a stab at whittling those docs down for a quick-hit view.)

2. On a full-size browser:

• As far as running automated tests, such as checking links, you can run your normal link validator. I use Web Link Validator and InSite.
• As usual, enable your JavaScript error notifications.
• Use your regular tools for validating analytics as well as dev and design elements (though not necessarily browser formatting): dev toolbars, Fiddler, Firebug, etc.
• Test your site for mobile readiness: Visit mobiReady or W3C and plug in your URL or file (on W3C) and go. They list errors in order of severity, error location and best practices around each error. Both are very user friendly. mobiReady even spits out estimated speed on WiFi, 3G, GPRS (what about EDGE?).

3.  Emulators: I recommend the actual hardware if you can get your hands on it, even if you have to ask people around the office to use their mobile for a while. If you must use an emulator, I’d start by simply searching for emulators on the product maker’s website. i.e. Go to Apple and search for iPhone emulators. Here are some links to emulators by manufacturer. As a last resort, there are generic emulators aplenty out there.

I haven’t done this yet, so I can’t really speak to it … but it looks interesting, and I intend on giving it a go. Test mobile sites using Firefox extensions.

Finally, remember that mobile sites are often a very pared down version of the full site, for good reason. Utilize user agent sniffers to determine if the user should be auto-directed to the mobile site, but always give them the option to view the complete site in case that’s what they prefer.

 

For some, simply looking at a site in a foreign language is daunting, never mind the thought of testing it. So, let’s say your client wants to use a single template to create the same site but in various global regions/languages. Here, I’ve put together some pointers to help guide the way.

To start, I’m using both REL Software Web Link Validator and Inspyder InSite, as they have some different abilities and in certain instances one is just easier or faster than the other. With any luck, the English site will be complete, so you can use it as a point of reference. Otherwise, open your regional site in one browser for testing and another to utilize Google translate.

Now for the fun stuff:

1. Links function, but are they going to the correct destination?
To an extent, testing this obviously requires human intervention. But, let’s say I’m testing a German site with a .de extension, and I need to validate that all internal links stay on the German site rather than going off to .com (or the English version).  

a. Simply run the link report excluding external URLs, elect to view all pages processed, do a search for .com (or something like us-en — whatever directory or extension is being used to distinguish regions). If both regions are sharing a CSS and imagery, then exclude the directories like */images/* and */themes/*. It will probably require a little finagling to match your specific scenario, but you get the idea.

b. Mailto. View all mailto protocols in a single step to confirm email addresses are updated accordingly.

2. Regional languages
My link checkers also have a spell checkers that enable you to select from several languages. So my goal in this exercise is to make sure all text has been translated from English to German, including the copy within Flash, meta data, javascript and buttons.  

a. The first thing I’d do is set up the project by excluding proper names.

b. Run your test, then throw out all the German words that came back as misspelled, as I wouldn’t know if they were really misspelled anyway (that’s what the translation partner is for). Go ahead and add them to the custom, project-specific dictionary, in case you have to run the same test again later.

c. Next, consider that some German product and business names may be in English and not recognized as a proper name by the tool. We don’t want to go translating things like business names, so add those to your custom dictionary, too. 

d.    Be aware:

i. Unless you have a character recognition software in your bag of tricks, you will have to manually check the copy on images like .jpg, .gif and .png. With these tools though, you can select to view specific file types, so at least you don’t have to click through each page trying to ascertain both what’s an image and which images have copy on them.

ii. Copy that overruns its boundaries: With translated sites (especially German) you will almost always find text that runs outside of its container. I’ve found it most effective to scour each page manually for this. Same deal for Flash, btw.

iii. Copy that’s easy to miss: dropdowns and success/error messaging. For whatever reason, translators like to overlook copy within dropdowns as well as webform success/error messaging. The spell checker should catch the dropdowns. But the validation messaging can be a little more complicated, and this is another aspect that I’ve found most effective to test manually.

e. Compile your results, and back to the translator!

There’s a lot more to do, but hopefully this will get you rolling in the right direction. And to think these tools are called “link checkers” ….

 

Traditional News: News from professional journalists without a personal or political slant (in theory). They gather the facts, think things through thoroughly, pass through editing and get multiple stamps of approval before delivering the message to the masses. Oh, and they make a living doing it.

Bionic News: It’s content crowdsourcing or sometimes disparagingly called content farming — news from both pros and non-professionals (er, “freelancers”) and sometimes with a slant. They are regular folks who are in the throes of what they are reporting, by choice: Often, they don’t get paid or get paid a menial sum and have to pay their own way into the very events they are covering. Why? Well, it’s exciting and you get your name out there while chalking up a little bit of experience.  Plus there’s the cred that comes with walking around at cool events looking like you are part of it. Okay, to be fair, some folks are simply passionate about reporting, whether it’s for a salary or for fun. But, with this so-called “bionic” news you get the grit of the story and LOTS of it quickly from all sorts of angles. 

Take AOL’s Seed for example, introduced to the masses at 2010 SXSW. They recruited writers who “can follow directions … and that was the criteria.” The goal is to supply mountains of relevant content fast and cheap. Now, submissions do get passed through AOL’s managing editor before making the final cut. So, it’s not to be discounted as gibberish by any stretch. It’s the real deal, just different — kind of like blogging but with an assignment and probably a few rules. Plus, it has been known to lead to higher profile, paid assignments —so maybe a foot in the door at a major corporation.

Another major player is Demand Media, which produces on average about 5K pieces of content a day via its Demand Studios. Demand is a little different in that they also foster crowdsourcing for third-party organizations. Even so, basically Seed and Demand are to content what Amazon’s Mechanical Turk is to IT developers. 

On the complete opposite end of the spectrum is a recent Reuters’ policy that prohibits their journalists from breaking news on Twitter, in an effort to avoid personal or political slants in the name of news. It must first be broken on the wire, and even then they should seek higher-up approval before tweeting — which btw, has to be on a professional account, separate from their personal account. While they recognize the inherent value of social media and immediate news, they also have a “hard-earned reputation” to maintain and understandably so.

Traditional and crowdsourced content are like night and day, really. I say there’s an absolute place for them both. For content providers, it's partly dependant on your objective, subject matter, audience and medium. And for the reader, well, do you want the bay full of oysters or just a pearl?  

 

DG FastChannel, Springbox’s parent company, served up the commercials that played during the 2010 Super Bowl. More specifically, DG’s role was to digitally transmit the commercials to the individual broadcasters. That got me wondering about this, that and the other but mostly about ROI and web traffic stats on those top-dollar spots.

For just under a few million dollars and 30 seconds of airtime, you can jumpstart next month’s sales by 11%. That’s 250 times greater than that of a regular TV ad. Admittedly, that’s the high end, and those numbers are variable of course, based on a brand’s industry, strategy, maturity and other factors.

Automotive ads are particularly effective, with an ROI that’s almost twice that of an average TV spot. Speaking of auto ads, Honda touted its Accord Crosstour during the Super Bowl and pointed to a website that Springbox had a hand in: You can read more about our rockin’ Flash Developer Matt Skoog and the technology behind our Crosstour Flash module here.

I was, however, mostly interested in how the commercials affect brands’ web traffic. As it turns out, the Crosstour page views increased by 14% from one 30-second slot. Not too shabby, but surprisingly it was actually at the low end of the auto ad/pageview ratio. (CarsDirect suggests that may be because the actual vehicle only made a brief appearance and at the tail end of the commercial.) But, Hyundai’s Sonata traffic increased by a whopping 166%, from three 30-second ads. That looks like a win, but time will tell how that translates to actual vehicle sales.

Most sources agree that Super Bowl advertisers primarily strive to drive viewers to their websites these days over physical locations. There are even big players (Pepsi) who decided to forego the Super Bowl scene altogether this year and instead put that chunk of change toward internet campaigns. Double score for the interactive industry!