I love split testing - Visual Website Optimizer Blog

The Why and How of Prioritizing A/B Testing Hypothesis

Posted: 02 Aug 2016 06:48 AM PDT

Building a strong hypothesis is crucial for testing—no doubt. But what should you do when you have a backlog of potential hypotheses to test? How should you go about deciding which one to test first?

Often, organizations do not have a structured approach to conversion rate optimization (CRO) and arbitrarily pick out a hypothesis from the pool of options they have.

However, organizations following a structured approach to CRO realize the need for a robust prioritization framework. Let's first look at why your optimization process needs a prioritization framework in the first place.

Prioritization Framework for A/B Testing Hypothesis – Why?

Analogous to a calendar, a prioritization framework provides a clear direction to your optimization program and prevents you from doing aimless testing. This framework enables you to maintain a dedicated schedule for all your CRO activities.

Michal Parizek, Senior eCommerce & Optimization Specialist at Avast, in his interview with VWO, points out the importance of keeping a testing calendar: "A test calendar helps to keep focus on important tests being launched on time. It is also vital for resource planning and for bringing all stakeholders in a loop. We usually do a quarterly overview of what tests we'd like to run and then we specify and add details on a monthly basis."

Prioritization helps you answer, "here's what we'll test, in this order, and here's why." A good prioritization framework offers the following advantages:

Brings in transparency to the Optimization Process

Whether you have an internal team or an outsourced agency for your CRO activities, prioritization ensures that each test you perform is chosen on the basis of predefined objective factors and not subjective individual choices. Everyone in the team will have a clear idea as to why they are performing this test first.

All in all, the hypothesis would be picked and tested on the basis of its worth and not by the HiPPO. As Pauline Marol, Lead Product Manager at Hotwire puts it, "prioritization removes the emotion from A/B testing."

Sets the Right Expectations

You prioritize a particular hypothesis on the basis of certain predefined criteria, say, time of implementation, budget required, or potential of improvement.

As you have full control over the criteria you choose, you know what to expect from the results.

For instance, if you are prioritizing on the basis of time of implementation, you can expect early test results. If your criteria is the potential of improvement, you can expect early wins.

Setting the right expectations not not only helps the team, but also the top management to have greater confidence on your CRO plan.

Reduces the Opportunity Cost

For every test you choose to run, there would always be an opportunity cost, that is, the cost incurred by not choosing an alternative (and potentially more yielding) test. A good prioritization program ensures that the best opportunities are taken up first, thereby minimizing the opportunity cost.

Now that we know why we need a prioritization program, let's look at how to objectively set up a prioritization framework for your CRO plan:

Setting up your Prioritization Framework

You would find a plethora of frameworks for prioritizing your testing hypothesis. Some of these are listed below:

Widderfunnel's PIE framework for Prioritization: A prioritization model by Chris Goward that fosters scoring (on a scale of 10) on three factors: Potential, Importance, and Ease.

Bryan Eisenberg's T.I.R Model: Prioritization calculated by multiplying the individual scores of Time, Impact, and Resources.

Montate Model: Prioritization based on the level of difficulty of the test, creative effort required, and potential of the test.
Hotwire's Points Model: Binary scoring on various questions such as those based on reach, lift, strategic fit, and design.

So which of these should you decide to apply to your CRO plan? On what factors would you base your prioritization framework?

Karl E. Wiegers, in his paper on Prioritizing Requirements states this simple rule for prioritization: "When setting priorities, you need to balance the business benefit that each function provides against its cost and any implications it has for the product's architectural foundation future evolution."

Based on this definition and on analyzing different prioritization frameworks, you can customize your prioritization framework by following these two steps:

Determining the Potential Impact

Depending on the research you have put in to build your hypothesis, you can estimate the impact that it might yield. The more insight-driven your hypothesis is, the more confidence you can have on its success. For example, your heatmap analysis could indicate that a large chunk of your visitors are not able to locate your CTA, making you confident that changes made to it will yield positive results. Some of the other factors to estimate the impact of the test are:

Alignment with Business Objectives

For each hypothesis you want to test, there would be a target metric that you will track. It is imperative that this key metric is in alignment with your business objectives. ConversionXL explains the linear correlation between the business objectives and target metrics here.

Prioritization eventually becomes a matter of answering these questions:

Which business goals are we trying to improve at this moment?
Which features or pages are closely associated with the business goal?

Location of the Test

The page on which the test is performed is also critical to understand its potential. You can look for:

Most critical pages: In terms of the business goals they are seeking to fulfill. For instance, a pricing page would be more critical to a SaaS business than say the "About" page.
Most visited pages: Pages with high traffic volume are more likely to yield quicker results. The graph below indicates how higher the traffic, the lesser time it takes for the test to reach a conclusive result.

Pages with expensive visits: When choosing between two pages with similar traffic and conversion rates, pick the one with a higher cost of traffic (through paid advertisements) for a better split-testing ROI.

Other than this, you can look at Avinash Kaushik's post on data analysis strategies on Google Analytics.

Past tests

You can also gain confidence on the potential impact of your hypothesis by analyzing the tests performed in the past.

For instance, if you had a winning test in the past, you could have greater confidence on a similar test hypothesis. If you had an inconclusive or negative test, there still are a number of things you could do to redeem the situation and gain insights from it.

However, while considering these, make sure you don't end up with Confirmation Bias (looking at data that supports the hypothesis) or HARKing (Hypothesising After the Results are Known).

Determining Potential Ease

To estimate the effort required to test your hypothesis, consider the following factors:

Resources

A full-fledged CRO program would require help from the following members:

Strategist: Focuses on managing the program and deciding the goals. Usually knows the most about conversion journeys, personas, and persuasion design. Owns the KPIs.
Analyst: Looks at the data before and after an A/B test, connects it with other important data sources, and helps everyone understand the test outcomes.
Conversion Centered Designer: Focuses on conversion-centered design.
Copywriter: Someone who's great with the written word and can write to reduce anxieties, ease friction, and persuade and delight visitors.
Developer: To help you run tests with optimized front-end code and send events or record goals in your analytics software.

By looking at the bandwidth and strengths of your team/members, you can estimate which particular test should be conducted first.

Cost

The ease with which the test can be conducted is also dependent on the cost of conducting the test. The cost would typically include the cost of resources and tools used in the optimization process.

According to the 2016 Conversion Optimization Report by ConversionXL, 53% of organizations do not have a documented budget for CRO.

Duration of the test

To conclude, the potential ease of conducting the test can also be estimated by the time it would take to finish the test. You can calculate for how long should you conduct a test by using VWO test duration calculator.

Tests that take longer than a couple of months are not recommended because several external factors, such as cookie churn, start becoming relevant and impacting the experiment.

Bringing it all together

A cumulative assessment of the two factors could give you a simple framework for prioritizing. For instance, you would prioritize tests with high impact and low effort more than the ones with say, low impact and high effort.

A/B Testing Hypothesis Prioritization — Intercom’s 2×2 matrix for prioritization

Your Thoughts

Do you weigh your hypothesis before testing? What are the other factors that you have used? Share with us in the comments below.

The post The Why and How of Prioritizing A/B Testing Hypothesis appeared first on VWO Blog.

Search This Blog

Internet Marketing Reviews