Control Group: What It Is and Why It’s Needed in Marketing

The effectiveness of an advertising campaign isn’t always assessed solely by direct metrics like conversions or clicks. This is because not every marketing action directly influences the outcome. It may turn out that the purchase was not influenced by the channel recorded in the analytical report. For example, if a company used attribution models and chose last-click attribution, giving credit to the last channel before the desired action occurred.

Such a narrow view of results can be misleading, leading marketers to draw incorrect conclusions for planning future campaigns: if it worked this time, the same tactics should be repeated next time. However, it turns out that the old approach inexplicably doesn’t work.

To examine marketing effectiveness from different angles, the method of control groups is used. In this article, we’ll explain what it is, why this approach is necessary for businesses, and how research is conducted.

What is a control group

A control group is a method for assessing the effectiveness of marketing campaigns (email, push, SMS notifications) using individuals who have not engaged in marketing communications with the company or have received a limited number of notifications (e.g., only transactional ones). The method originated in science, where it was used to demonstrate the relationship between one phenomenon and another.

Through control groups, the actions of customers who have not received advertising messages from the brand, except for transactional ones (global group), or who have received a limited number of messages, are studied. Then, their activity is compared with that of users who received the messages. Ultimately, the aim is to evaluate whether marketing efforts have influenced the purchase decision or if the customer was ready to make a purchase without additional reminders.

The control group assesses the number of products/services sold or requested, the average order value, views, subscriptions, website visits, and other metrics.

For example, an email campaign promoting a shoe sale with two variations of text and design (A/B testing) resulted in sales doubling. Initially, the campaign seemed successful, with one text and design combination, A, appearing more effective. However, analysis with control groups revealed that a significant percentage of purchases were made by users unaware of the promotion, as they did not receive notifications.

Test and control group: differences

These two concepts may seem similar in meaning, but the principles of working with them differ.

It’s important that when conducting A/B testing, all results are taken into account: both the control and test groups. After the study, negative control is also evaluated — how the absence of any changes affected the users.

The control group confirms or refutes the results obtained from the test group.

The role of control groups

Utilizing a control group:

• Enhances the effectiveness of marketing campaigns by providing a broader analysis of results, demonstrating the true impact of marketing on conversions or other metrics. Irrelevant offers are excluded, leading to the creation of more desirable offers for customers.
• Saves your resources. Research identifies poorly performing or unnecessary channels where you allocate budgets. After their exclusion, time and money are freed up for other promotion methods.
• Indicates changes in customer preferences and market situations that may have gone unnoticed. Over time, the audience becomes fatigued, and even more attractive interfaces from competitors can lure users away. The control group will identify which incentive will re-engage customers with the brand.
• Tests hypotheses for new advertising campaigns. For example, you want to test whether the number of purchases in an online store will increase if push notifications about an additional discount on abandoned cart items are sent.

Platforms for working with control groups

The method is flexible and adaptable for hypothesis testing across various digital resources:

• Email and other distribution channels: here, the impact on user activity through email, push notifications, SMS, or messengers is explored. For example, the influence of a discount coupon for the test group and its absence for the control group is examined.
• Landing page, where different user samples during A/B testing are shown variants that differ from each other. Texts, images, and headlines are changed for the test groups, while the control group always sees the same variant without alterations.
• Advertising: various variables such as creatives, offers, texts, and formats are possible for the test group here.

When not to use control groups

Forming a control group is not necessary for every marketing campaign. The method will not yield results if the customer database is insufficient. For an audience of a few hundred people, the study will be statistically insignificant (too small a sample size to make a global conclusion).

Additionally, part of the revenue will have to be forfeited. For example, the global control group will be excluded from mailings and may not make any purchases if they do not receive marketing messages. To minimize losses, some companies refrain from using the control group method during peak sales seasons.

This research should also be avoided if it is not possible to accurately track all customer interactions with the company across all online sources, if there are several of them.

Disadvantages of control groups

The conclusions of the study may not always be applicable to a broader audience due to non-representativeness. The control group represents a specific percentage of customers, which may not always reflect the characteristics of all users. If more men were chosen for the test, data on their behavior may be irrelevant to the female audience.

Control groups are usually small compared to all users. Because a company will not risk revenue for research purposes and allocate more than 20% of customers for it. Such a quantity may be statistically insignificant, resulting in inaccurate results.

The company cannot account for all external factors that influence the decisions of the control group. For example, a brand conducted mailings for testing and checked them against the control group. Suppose the result was weak in terms of conversion quantity. That is, the advertising campaign was ineffective. But if it is known that at that time a competitor held a closed sale with significant discounts, such a conclusion can be challenged.

How to conduct research with a control group

Start by defining the goal: what hypothesis do you want to test through control groups and what answer do you expect to receive. Then you need to:

1. Identify the audience segment for the study. Common characteristics can be shared among users. For example, customers who have made purchases totaling N over the past six months. It is important for the control group to reflect those to whom advertising messages were sent — be similar in demographics and other characteristics. Otherwise, the study will be unreliable.
2. Choose the appropriate number of users that affects the significance of the result. Here, the rule applies: the larger the audience size for targeting, the fewer users should be used for research. And vice versa: a smaller quantity requires more people for the sample. For example, for a control group of 12,000, it may be 5%, while for 2,000 customers, it is advisable to choose 10–20%. If the research results show that the company is performing well, the control group will decrease in subsequent tests. Conversely, if the advertising is ineffective, the number of users in the sample should increase with each subsequent check.
3. Set the duration of the study. It can be short-term for a single campaign, but a longer one over several months, taking into account external factors, will provide more ideas for improving effectiveness.
4. Make a mailing in the context of the advertising campaign with carefully crafted content for this purpose. Testing of both a single marketing activity and a set of actions is possible. Then you can assess the effectiveness of the entire strategy. Choose the option that meets the business’s needs.
5. Analyze the results after completing the study. For example, if 15% from the control group made a purchase, and 30% received a promotion email and also made a purchase, it means the campaign worked. However, if the percentages in both groups are almost identical, the advertising strategy is likely not effective enough. To assess the significance of the results, use one of the A/B testing calculators.
6. Adjust campaigns and strategies to improve performance. At this stage, you may realize that you need to abandon certain channels that are not yielding results.

Summary

A control group is a method for testing marketing campaigns in which a certain number of users are excluded from all or some channels of brand communication. This is important to assess the significance of advertising activities in achieving business results, save budget, time, test hypotheses, and identify market changes.

There are two types of control groups: global (customers receive only transactional messages) and local (some marketing communications remain). Such research is necessary for conducting A/B tests to understand the effect of no changes in campaigns.

The choice of a control group depends on the size of the audience for advertising: the smaller the number of people in the sample, the more needed for tests. Advertising is launched to this audience, and the result is analyzed. Afterwards, marketers draw conclusions about the effectiveness of strategies and adjust them.

The article was originally published here.