A/B Test Significance Calculator
Changed your app store screenshots, title, or description? Enter your impressions and installs for each variant to find out whether the difference is statistically significant — or just noise.
A Variant A (Control)
B Variant B (Test)
Results
Conversion Rate A
—
Conversion Rate B
—
Relative Improvement
—
Z-Score
—
P-Value
—
Confidence Level
—
Statistically Significant?
Enter your data above
How the significance test works
Two-proportion z-test
This calculator uses a two-proportion z-test. It compares the conversion rates of your two variants by computing a pooled proportion and standard error, then calculates how many standard deviations apart the two rates are (the z-score).
What 95% confidence means
When the confidence level reaches 95% or higher (p-value below 0.05), there is less than a 5% probability that the observed difference occurred by chance. This is the standard threshold used in experimentation to declare a result statistically significant.
The formula
Z = (pB - pA) / SE, where SE = sqrt(pPool * (1 - pPool) * (1/nA + 1/nB)) and pPool is the combined conversion rate across both variants. The p-value is then derived from the normal distribution using a two-tailed test.
Two-tailed vs one-tailed
This calculator uses a two-tailed test, meaning it checks whether Variant B is significantly different in either direction (better or worse). This is the more conservative and widely recommended approach for A/B testing.
Tips for running better A/B tests
Run tests long enough
Don't stop a test the moment one variant looks ahead. Short tests are prone to false positives. Aim for at least 7 days to account for daily and weekly traffic patterns in the app stores.
Test one variable at a time
If you change your icon, screenshots, and title all at once, you won't know which change caused the uplift. Isolate a single variable per test so you can attribute results with confidence.
You need enough sample size
Small differences require large samples to detect reliably. If your app gets fewer than 1,000 impressions per week, you may need to run tests for several weeks before drawing conclusions.
Focus on conversion rate, not just downloads
Raw download numbers can be misleading if traffic fluctuates. Conversion rate (installs / impressions) controls for traffic volume, giving you a fairer comparison between variants.
More free ASO tools
- ASO Score Checker — Get an instant audit of your app listing
- App Store Character Counter — Check your title, subtitle, and description fit within store limits
- Keyword Density Checker — Analyse keyword frequency in your app listing
- ASO Checklist — Step-by-step checklist for optimising your store listing
Test smarter. Optimise faster.
Get your ASO score and see exactly what to test first — free.
Get your free ASO scoreNo credit card required