What Is Statistical Significance?
A measure of how likely a survey result reflects a real difference in the population, rather than random chance in who happened to respond.
When two survey results differ — Version A scored higher than Version B, or this quarter’s NPS beat last quarter’s — statistical significance asks whether that gap is likely real, or just noise from sampling a different set of people each time.
The standard convention is a p-value below 0.05: roughly, less than a 5% chance the observed difference happened by chance alone if there were actually no real difference. A result below that threshold is typically called “statistically significant.” Larger sample sizes make it easier to detect smaller real differences as significant — with a tiny sample, even a meaningful gap can look like noise.
Statistical significance is not the same as practical significance. A large, well-powered survey can find a “statistically significant” 0.3-point difference in a satisfaction score that’s too small to change any actual decision. Significance tells you a difference is probably real; it doesn’t tell you the difference is big enough to matter.
Example
Feature A scores 4.3/5, Feature B scores 4.1/5, p = 0.02. The gap is statistically significant (unlikely due to chance) — but whether a 0.2-point gap is worth acting on is a separate, practical judgment call.
How to Use Statistical Significance in SurveyRock
Larger, well-designed samples make real differences easier to detect. SurveyRock's sample size calculator helps you plan a sample that's actually big enough to catch the effect size you care about before you launch.
Use the sample size calculator