Who Needs A/B Tests and Why?

A/B testing is one of the best methods for finding optimal ways of improving conversion, economic indicators, and behavioral factors. There are some other methods, but they are more complex and expensive. The key benefits of A/B tests are their affordability and suitability for businesses of any scale.

A/B testing can be called one of the most meaningful ways of searching for solutions and making decisions in business; those solutions and decisions that a company’s profit and development of various products both depend on. The tests give an opportunity to make decisions based not only on theories and hypotheses but also on practical knowledge of how specific changes modify the clients’ interaction with the network.

It is important to keep in mind that everything in retail should be tested: marketing campaigns, SMS marketing, tests of SMS/email marketing, placement of products on shelves and shelves in a store. As for an online store, what needs to be tested is the placement of elements, the design, captions, and texts.

A/B tests are the instrument that helps enterprises, for example, a retail company, to remain viable, notice changes in a timely manner, and evolve. This allows businesses to be as efficient as possible, maximizing their profits.

What are the nuances of this method?

The main thing is that there should be a goal or problem the testing will be based on. Here is an example: the problem - the small number of clients at a point of sale or an online store; the goal - to increase the influx of clients; the hypothesis - if the product cards in an online store are bigger and the photos are brighter, the number of purchases will increase. Next, A/B testing is carried out, and the result of it is an evaluation of the changes. After the results of all tests are obtained, one can start creating a plan of action for changing the website.

It’s not recommended to conduct tests with overlapping processes; otherwise, the results will be more difficult to evaluate. The first tests should be those on top-priority goals and clearly formulated hypotheses.

A test should last for quite a long time for its results to be considered reliable. How much time exactly a test should last depends on the test itself. For instance, the traffic of most online stores increases not long before Christmas. If changes to the design of an online store were made before this, a short-term test would show that everything was fine, the changes were successful, the traffic was growing. But no, the traffic will always increase before holidays, no matter what you do. The test shouldn’t stop before the New Year or immediately after it; it should be long enough to reveal all the correlations.

The correct connection between the goal and the metric is very important. For example, having changed the design of the online store website, the company observes an increase in the number of visitors or clients and satisfies its mind with that. But in fact, the average purchase amount may be smaller than usual, so the total income will become even lower. This can’t be called a positive result. In this situation, the problem is that the company didn’t simultaneously check the following combination: an increase in visitors — an increase in the number of purchases — the dynamics of the average purchase amount.

Is testing carried out only for online stores?

Not at all. In offline retail, there is a popular method of a full pipeline implementation to test hypotheses offline. This method involves building a process in which the risks of incorrect selection of groups for the experiment are reduced, and the optimal ratio of the number of stores, pilot time, and the size of the estimated effect is selected. Additionally, it involves reuse and continuous improvement of methodologies for the post-analysis of effects.

The method is needed to reduce the likelihood of errors of false acceptance and omission of the effect, as well as to increase sensitivity as, on the scale of a large business, even a small effect is of great importance. Therefore, one should be able to identify even the smallest changes and minimize risks, including incorrect conclusions about the results of the experiment.

Retail and Big Data

The data obtained during peak days of product consumption allows predicting demand more accurately, taking into account event-related factors. An accurate forecast makes it possible to anticipate customer expectations.

During the testing process, the following two methods can be used:

1. Bayesian structural models of time series with cumulative difference estimates;

2. Time series analysis in paired samples.

There are a good many methods and technologies; right off the bat, they are:

1. Demand forecasting;

2. Assortment optimization;

3. Computer vision for detecting empty places on shelves and an emerging queue;

4. Promo forecasting;

Retail/eCommerce is one of the key industries for Andersen. We have built software for Media Markt and other major players in the field. Also, we are actively developing a Big Data line. Now we can help our customers not only in app development but also in the processing and analysis of data, including data obtained from A/B testing.