Determining Sample Size and Test Duration

What You’ll Learn

You’ll learn how to calculate the minimum sample size needed for statistical validity and determine how long your split test must run to achieve reliable results. Understanding these calculations ensures your test reaches statistical significance rather than producing false conclusions based on insufficient data.

Key Concepts

Sample size and test duration are interdependent factors that determine whether your split test results are trustworthy. The larger your sample size, the shorter your test can run; conversely, smaller daily traffic volumes require longer testing periods. Statistical power, baseline conversion rate, and minimum detectable effect size all influence these calculations. Running a test too briefly with insufficient traffic can lead to underpowered tests that fail to detect real differences between variants.

• Statistical Power and Significance: A properly sized test requires 80% statistical power (ability to detect true differences) and 95% confidence level (only 5% chance of false positives). These industry standards mean you need enough visitors to reliably distinguish real performance differences from random variation.
• Baseline Conversion Rate Impact: Your existing conversion rate directly affects required sample size—tests on pages with 2% conversion rates need different sample sizes than pages with 20% conversion rates. Calculate baseline rate from your analytics before designing the test to determine appropriate sample requirements.
• Minimum Detectable Effect: Define the smallest improvement you care about detecting, such as a 10% relative increase in conversions or a 2% absolute lift. Chasing tiny 1% improvements requires exponentially larger sample sizes, so establishing a meaningful effect threshold helps you plan realistic test durations.
• Sample Size Calculator Application: Use tools like Evan Miller’s A/B test calculator or your platform’s built-in calculator by inputting baseline conversion rate, desired statistical power, significance level, and minimum detectable effect. These calculators provide the exact number of visitors needed per variant to achieve statistical validity.

Practical Application

Calculate your test’s required sample size using your website’s baseline conversion rate and expected minimum improvement you’ll consider actionable for your business. Then divide the required total sample size by your daily visitor count to determine how many days your test must run, and document this projection in your test plan.