What Is Statistical Significance?
A measure of how likely a survey result reflects a real difference in the population, rather than random chance in who happened to respond.
When two survey results differ — Version A scored higher than Version B, or this quarter’s NPS beat last quarter’s — statistical significance asks whether that gap is likely real, or just noise from sampling a different set of people each time.
The standard convention is a p-value below 0.05: roughly, less than a 5% chance the observed difference happened by chance alone if there were actually no real difference. A result below that threshold is typically called “statistically significant.” Larger sample sizes make it easier to detect smaller real differences as significant — with a tiny sample, even a meaningful gap can look like noise.
Statistical significance is not the same as practical significance. A large, well-powered survey can find a “statistically significant” 0.3-point difference in a satisfaction score that’s too small to change any actual decision. Significance tells you a difference is probably real; it doesn’t tell you the difference is big enough to matter.
Example
| Feature | Average score |
|---|---|
| Feature A | 4.3 |
| Feature B | 4.1 |
- The gap
- 4.3 − 4.1=0.2 points
- Is the gap likely to be chance?
- 0.02 < 0.05=SignificantIf there were really no difference, a gap this big would turn up about 2 times in 100. The usual cut-off is 0.05.
Statistically significant is not the same as worth acting on. Whether a 0.2-point gap matters is a separate, practical judgment call.
How to Use Statistical Significance in SurveyRock
Larger, well-designed samples make real differences easier to detect. SurveyRock's sample size calculator helps you plan a sample that's actually big enough to catch the effect size you care about before you launch.