I love split testing - Visual Website Optimizer Blog

How to Build a Strong Multivariate Testing Plan That Gets Results

Posted: 13 Sep 2016 07:03 AM PDT

"Multivariate tests are complicated."
"Multivariate tests give inconclusive results."
"Multivariate tests are same as A/B tests but with multiple variations."

Time and again, such statements may have stopped you from performing a multivariate test on your website. But how much of these statements is actually true?

Let's begin with understanding the basics:

Difference Between A/B Testing and Multivariate Testing?

While A/B testing allows you to compare two or more versions of a webpage by splitting traffic equally among them, Multivariate testing (MVT) allows changes on more than one element on a page and helps you find the best converting combination.

For instance, say we have three variations each for the two elements, headline and type of call to action (CTA). There would be a total of nine combinations of tests (as shown in the image) that need to be run. The total number of combinations can be calculated by the number of variations to the power of number of elements (three to the power of two, in case of the given example) that you want to test.

Brian Gatti, partner at Inspire Business Concepts puts the difference between the two in a nutshell,

“Honestly, the advantage to the MVT over traditional A/B testing is that I’m able to isolate winning variations faster and get to the new champion variation. Once we’ve mapped out the variables, it saves me the trouble of running a bunch of separate, smaller A/B tests. I also get the advantage of understanding how the impact of two variables changes when they occur together. (When I change the button color and image in an MVT, sometimes the synergy of the two variables changed together creates a unique result I’d not see in an A/B test)."

As Brian mentions, multivariate tests are about measuring the interaction effect among independent elements to see which combination works best. The interaction effect measures simultaneous influence of two or more variables on conversions.

A/B Test or Multivariate Test—Should One be Chosen Over Another?

A/B testing and multivariate testing serve different and specific functions.

A/B tests will help you answer questions such as: "Which part of the page should I add a video item to gather maximum engagement?" multivariate tests answer questions such as "Do visitors respond better to a video item next to a contact form?" or "…to a webpage with just a contact form and no video item?" and so on.

Simply put, A/B tests measure individual impact, while multivariate tests measure collective impact. Using both as complementary tests can help you gain maximum out of your optimization program.

Next, let's look at what are the prerequisites for creating a successful multivariate test.

Running a Multivariate Test

Chart Out a Plan for Your Multivariate Test

You should begin by weighing the pros and cons of conducting a multivariate test and then take a call judiciously after considering your available resources. Chris Goward chalks out the advantages of MVT rightly:

Easy isolation of many small page elements and understanding of their individual effects on the conversion rate
Allowing measurement of interaction effects between independent elements to find compound effects
Following a more conservative path of incremental conversion rate improvement
Facilitating interesting statistical analysis of interaction effects

Next, you could chart a out a calendar or roadmap for your multivariate tests. The calendar could include a repository of potential elements and variations to test, along with timelines for the tests. A systematic approach to conversion rate optimization requires:

You should remove the totally improbable/bizarre combination at the planning stage itself.

For example, while running an MVT on your eCommerce website, one of the variants may have a headline offering a "$15 off on watches" and a CTA claiming say "Free subscription" to your newsletter, instead of something like "buy now." These two messages are incompatible. Detect and remove incompatibilities at the preview stage itself.

Determine the Amount of Traffic You Would Need

As multivariate tests work on combinations, these create a larger number of variants compared to A/B tests. This means that your overall traffic would have to be divided among all the many variants.

For instance, if you have 1,000 visitors landing on your website and you decide to run a multivariate test with three elements and two variations, each variation would get 125 (1,000/8) visitors each.

Therefore, it is best to estimate the amount of traffic you would need beforehand and calibrate the number of variations (or number of elements) accordingly. You could use a significance calculator for this task.

Make Sure Your Testing Tool Accounts for Errors

Each test is accompanied by an acceptance of a certain probability of error in the test results.

You should also keep in mind that increasing the number of variations increases the chances of false positives.

For instance, let's assume the the probability of error is say 5% for each variant and you have 20 combinations in the test. The probability of a false positive in at least one combination is 64% [= 100* (1 – (95/100)^20)]. This problem is also nicely illustrated in the XKCD comic Significant.

Your A/B testing should account for these multiple comparison problems. We at VWO now employ the Sidak Correction to deal with this problem. You can now have confidence that the probability of having even one false positive, across all the combinations, is less than 5%, considering that all combinations are equal.

Gather Insights from all your Multivariate Tests

With the results of an MVT, you first check which variant has achieved the highest (and significant) uplift and gather insights. But what if you received inconclusive or negative results?

Regardless of whether improved versions of your website are found or not, every test ends up being a good learning opportunity.

For failed multivariate tests, you can find out hidden winners by applying post-test segmentation.

The parameters for segmentation could be anything. They can range from gender of the visitor to the demography they are from or the operating system they use. To help you get started, here is what Avinash Kaushik suggests:

By Source: Separate your organic traffic from your paid traffic; traffic that comes from social sites and referrals.
By Behavior: New versus Returning visitors; visitors who are from the US versus visitors from Europe. Different visitors have different purposes for coming to your website. It is important that you realize what they want and make sure they get it.
By Outcome: Segregate people on the basis of the products they purchase, the size of purchase, or who filled a form versus who made a purchase.

If you run into an inconclusive MVT result, you could still find into a hidden winner or visitor behavior analysis to derive meaningful insights.

Over to You

Do you run multivariate tests? I'd be curious to hear from you about your ideas and experience on multivariate testing. Please leave a comment.

The post How to Build a Strong Multivariate Testing Plan That Gets Results appeared first on VWO Blog.

Search This Blog

Internet Marketing Reviews