When and How to Run A/B and Multivariate Testing for Your Website

A/B and multivariate testing (MVT) are used to help website owners determine the most effective marketing strategies for their products. Changes to business websites are made based on quantitativetesting results, rather than gut-feelings or design bias. Hypotheses can be tested on real customers resulting in actual numbers to prove or disprove an idea.

What is A/B testing?

The simplest of the tests, A/B testing sends half of the website traffic to a “control” version and half to a test version. Any number of elements can be tested for efficacy; a landing page, a site process or a call-to-action. The name A/B is misleading in that the control version can be tested against as many as nine different versions.

What is multivariate testing?

When a number of variables are tested at one time, such as testing the color of the cart button, the thumbnail image and size of each, it is called a multivariate test. The number of variables and the number of versions of each variable determines the number of test versions. If every conceivable combination is utilized the test is called “full-factorial” testing. If not, the design of the test is referred to as “fractional factorial”. Although this form of multivariate testing saves time, it is also less reliable because you may be excluding the best performing factor.

Deciding which testing method to apply can be determined by using the following examples.

When should you use A/B split testing?

You should use A/B split testing when:

• You are inexperienced with testing. Because A/B testing is the simplest form, tests can be completed quickly. After having a few successes, you can focus on the more involved experiments.
• You are only interested in the impact of one variable. For example, you may want to determine if placing testimonials on your landing page helps with conversion. This can be tested easily by testing with the testimonials on the page against the absence of the testimonials. You may believe that visitors will be more likely to convert if there is a security seal on your page. In addition, you could test the positioning of the security seal.
• You need design freedom. This could mean comparing a new website design to the older control version of the website. Testing big changes before moving to multivariate testing saves time in the end.
• You do not have access to a multivariate tool. Some e-commerce platforms do not provide a full testing suite.
• You do not have enough results to reach statistical significance. You may have the traffic, but if there are not enough conversions, you cannot get an accurate test result. The test will have to run too long which will likely skew the results.
• You have limited understanding of coding. The JavaScript coding necessary to do A/B testing is simple and easily added to a website.
• You need fast results. Because A/B tests measure extreme changes between fewer combinations, the time required to run the test is much shorter than MVT.

When should you use MVT testing?

You should use MVT testing when:

• You are fine-tuning. You have already performed extensive A/B testing and have found your best performing version. Then it is time to fine-tune for maximum effectiveness.
• You need to know a combination of variations for each test variable. You could use an A/B split test tool to test two variables with two variations, but it is ultimately a multivariate test. By using a multivariate tool, you will be able to access an aggregate report of the best performing variations.
• You want to know what variables are influencing each other. If you need to test more than what a single A/B test can handle (more than 10 variables at once), use the multivariate test. Testing all versions during the same time period with the same traffic will produce less testing bias and provide faster results.
• You have enough conversions and traffic. For multivariate testing, there is not a specific number to qualify. However, online test duration estimators may help. Although they are helpful, they are not perfect.
• You have technological or branding and design restrictions. In this case, you are only able to test specific variables. MVT will be a much more efficient strategy than trying to use the A/B testing method.

A/B testing vs MVT testing

Each testing technique has its own unique advantages and disadvantages. Essentially, the techniques go hand-in-hand. The A/B test finds the optimal website design and MVT would be used to optimize the elements of the site. Any organization must keep in mind, testing is not a one-time effort. Ongoing testing must be performed since markets and trends change rapidly.

Don`t forget to like & share this post if you like it.