How to Understand & Calculate Statistical Significance [Example]

By jkopecky@hubspot.com (Juliette Kopecky)

New Data: Instagram Engagement Report [2021 Version]

Have you ever presented results from a marketing campaign and been asked, “But are these results statistically significant?” As data-driven marketers, we’re not only asked to measure the results of our marketing campaigns but also to demonstrate the validity of the data — exactly what statistical significance is.

While there are several free tools out there to calculate statistical significance for you (HubSpot even has one here), it’s helpful to understand what they’re calculating and what it all means. Below, we’ll geek out on the numbers using a specific example of statistical significance to help you understand why it’s crucial for marketing success.

In marketing, you want your results to be statistically significant because it means that you’re not wasting money on campaigns that won’t bring desired results. Marketers often run statistical significance tests before launching campaigns to test if specific variables are more successful at bringing results than others.

Statistical Significance Example

Say you’re going to be running an ad campaign on Facebook, but you want to ensure you use an ad that’s most likely to bring desired results. So, you run an A/B test for 48 hours with ad A as the control variable, and B as the variation. These are the results I get:

Ad	Impressions	Conversions
Ad A	6,000	430
Ad B	5869	560

Even though we can see based on the numbers that ad B received more conversions, you want to be confident that the difference in conversions is significant, and not due to random chance. If I plug these numbers into a chi-squared test calculator (more on that later), my p-value is 0.0, meaning that my results are significant, and there is a difference in performance between ad A and ad B that is not due to chance.

When I run my actual campaign, I would want to use ad B.

If you’re anything like me, you need more explanation as to what p-value and 0.0 mean, so we’ll go through an in-depth example below.