Imagine a small business concurrently running two sales. One sale trumpets: “Buy one, get one 50% off!” A second sale offers, “Get 25% off when you buy two!” These two sales represent the same amount of savings. A customer purchasing two units would end up saving 25% on each unit. No matter which offer a potential customer chooses, the business’s bottom line remains the same.
However, when the business tests these two promotions, it notes a trend. The first sales pitch, which mentions getting a product for 50% off, is drawing more customers than the second pitch, which mentions 25% off two products. The company chooses to ditch the second marketing campaign and go all-in on the first one. Effectively, the company has just run a marketing experiment.
What are marketing experiments?
Marketing experiments are a form of market research whereby businesses test different material and/or means of communication (such as sales pitches, calls to action, social media posts, and email marketing campaigns) to see which ones yield the best results.
For instance, a company might run two concurrent ad campaigns: one on Instagram and one on TikTok. The creative is the same, but the platforms are different. The company wants to gain insight into which platform drives more traffic to the company’s website. If the company sees a difference with statistical significance, it may choose to fully commit more budget to the platform that produced a higher yield.
This marketing experiment, where two outreach approaches are compared side by side, combines active marketing campaigns with market research. Insights gleaned from the research will inform marketing strategies for future campaigns.
How to conduct a successful marketing experiment
A typical marketing experiment involves five steps that span planning, execution, and analysis. Here are the steps, as they apply to a hypothetical shoe company running an A/B email marketing campaign. A/B tests are where two (or more) different versions of a messaging or UI element (for example, the placement of an “add to cart” button on a page, or using “buy now” instead of “add to cart”) are sent to two randomized groups of users to see which version performs better.
- Determine what you want to learn.Our hypothetical shoe company has decided it wants to test different email marketing messages about a promotion to see which version has the highest open rate.
- Establish your experiment parameters.Now the shoe company’s marketing team gets into specifics. It will conduct an A/B marketing test based on email subject lines. Half of its mailing list will get a message with a subject line that reads “Surprise! Our Biggest Sale of the Year!” The other half of its mailing list will get a message with the subject line “Exclusive Limited Sale: 48 Hours Only!” Recipients will be divided into two groups: those who have bought from the brand before, and those who haven’t. Half of each group will receive one message; half will receive the other. The contents of the email will be the same, and use both “biggest” and “limited time only” language.
- Deploy the experiment.Our shoe brand sends these emails out the day the sale launches.
- Collect data.As the marketing experiment progresses, the marketing team monitors the open rates among the recipient groups: previous buyers who received the “biggest” email, previous buyers who received the “limited” email, prospective customers who received the “biggest” email, and prospective customers who received the “limited” email.
- Analyze the results.Once the sale concludes, the brand evaluates its data and learns that customers who were informed about an expiring sale (“48 Hours Only!”) opened the email at a 7% higher rate than those who received the “biggest” message. The company decides this is a statistically significant result, and it resolves to mention “expiring” sales more often in its future marketing campaigns.
Common marketing experiments
Companies use marketing experiments to study nearly all components of their marketing strategies. These experiments include:
- Ad copy.Companies invested in digital marketing may run marketing experiments with different advertising content. They experiment with headlines and calls to action to see what drives more clicks.
- Websitelanding pages.Companies sometimes show different websitelanding pagesto different customers, studying whetherlanding pagecontent can affect customerconversion rates.
- Social media influencers.The role of influencers has surged on social media. Companies can conduct a simple marketing test by paying multiple influencers to offer a referral code or exclusive link to their audiences. The influencer who drives more traffic to the website (and more sales) can prove they are worthy of an ongoing partnership with the company.
- Automation as part of the customer experience.Many of today’s online sales platforms offer automated chatbots that can handle basic customer requests. Companies can conduct a marketing experiment by letting some website visitors interact with these chatbots while other website visitors interact with human representatives. They can analyze if using cost-saving automation impacts sales outcomes.
- Email subject lines and preview text.A basicsubject linetest focuses on email open rates: Companies want to see what types ofsubject linesinspire more people to open messages. These tests can go further by linking subject lines and preview text to click-through rates and, ultimately,conversion rates.
- User experience.Ecommerce vendors often experiment with the user experience (UX) of the checkout flow on their website. Shopify offers many resources in this department, handling the checkout UX so that small business owners can focus on other areas of their company.
Marketing Experiment FAQ
What are the two types of experiments used in market research?
- Controlled Experiments: Controlled experiments are experiments in which the researcher has full control over the assignment of the participants in the experiment to the different conditions.
- Observational Experiments: Observational experiments are experiments in which the researcher observes the behavior of the participants in the experiment without actively manipulating the conditions of the experiment.