A/B Test Significance Calculator

Calculate statistical significance for A/B tests. Determine if test results are statistically valid or due to chance.

Control VisitorsVariant A visitors

Control ConversionsVariant A conversions

Variant VisitorsVariant B visitors

Variant ConversionsVariant B conversions

Control Conversion Rate

controlRate%

Variant A performance

Variant Conversion Rate

variantRate%

Variant B performance

Relative Uplift

relativeUplift%

% improvement over control

Absolute Uplift

absoluteUplift%

Percentage point difference

💡 Statistical Significance Insights

• 95% confidence (Z ≥ 1.96) is industry standard

• Minimum 100 conversions per variant recommended

• Run tests for at least 1-2 full business cycles

• Small sample sizes can show false positives

• Don't stop test early even if winning

A/B Test Statistical Significance

Statistical significance determines if your A/B test results are valid or just random variation.

Key Metrics

Relative Uplift: ((Variant - Control) / Control) × 100

Absolute Uplift: Variant Rate - Control Rate

Z-Score: Measures standard deviations from expected

Interpreting Results

Z-Score ≥ 1.96: Statistically significant at 95% confidence. Safe to declare winner and implement variant.

Z-Score 1.65-1.96: Marginal significance (90% confidence). Consider running longer or use directional insight only.

Z-Score less than 1.65: Not statistically significant. Results could be random chance. Don't implement variant yet.

Example Calculation

Landing page A/B test:

Control: 10,000 visitors, 200 conversions = 2.0% rate
Variant: 10,000 visitors, 250 conversions = 2.5% rate
Relative Uplift: ((2.5 - 2.0) / 2.0) × 100 = 25%
Absolute Uplift: 2.5% - 2.0% = 0.5 percentage points
Z-Score: ~2.5 (statistically significant at 95%)
Decision: Implement variant, expect 25% conversion improvement

Common Mistakes

Stopping too early: Seeing early uplift and declaring winner before significance = high risk of false positive.

Not accounting for variance: Conversion rates fluctuate. Tuesday might differ from Saturday. Run full weeks.

P-hacking: Running multiple tests and only reporting winners leads to false discoveries.

Too small sample: 10 conversions per variant is meaningless. Need 100+ minimum, ideally 250+.

Best Practices

Set target sample size and confidence level before starting
Run for minimum 1-2 weeks regardless of early results
Test one variable at a time for clear attribution
Use 95% confidence as standard (99% for critical changes)
Document everything: hypothesis, variants, results, decision

Frequently Asked Questions

What is statistical significance in A/B testing?

Statistical significance measures if test results are real or due to random chance. Standard: 95% confidence means less than 5% chance results are random. Example: Control 2% conversion, Variant 2.5% conversion, 10k visitors each. Z-score 1.96+ = statistically significant. Below 1.96 = inconclusive (need more data). Never trust results without statistical significance.

How long should I run an A/B test?

Run tests until: 1) Reach statistical significance (95% confidence), 2) Minimum 100-250 conversions per variant, 3) At least 1-2 full weeks (captures day-of-week variance), 4) Cover business cycles (month-end, paydays). Stopping early risks false positives. Example: Seeing 20% uplift after 2 days with 50 conversions = meaningless. Wait for 95% confidence + sufficient sample.

What sample size do I need for A/B testing?

Sample size depends on baseline rate and minimum detectable effect (MDE). Lower conversion rates and smaller MDEs need larger samples. Rule of thumb: 350-1000 conversions per variant for 95% confidence. Use calculator: 2% baseline, detect 10% relative lift (2% to 2.2%), need ~40k visitors per variant. 10% baseline, same lift = ~8k visitors. Higher baseline = smaller sample needed.

Product

Solutions

Resources