← back

Calculate Statistical Power for Experiment Design

#296 · Statistics · Medium

⊣ Solve on deep-ml.com

Problem

Calculate the statistical power for an A/B test experiment design. Given a significance level, effect size, and sample size, determine the probability of detecting a true effect (1 - beta).

Solution

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import numpy as np

def normal_cdf(x: float) -> float:
    """Approximate CDF of standard normal distribution."""
    return 0.5 * (1 + np.math.erf(x / np.sqrt(2)))

def normal_ppf(p: float) -> float:
    """Approximate inverse CDF (percent point function) using Newton's method."""
    # Rational approximation for initial guess
    if p <= 0:
        return -10.0
    if p >= 1:
        return 10.0
    t = np.sqrt(-2 * np.log(min(p, 1 - p)))
    c0, c1, c2 = 2.515517, 0.802853, 0.010328
    d1, d2, d3 = 1.432788, 0.189269, 0.001308
    x = t - (c0 + c1 * t + c2 * t * t) / (1 + d1 * t + d2 * t * t + d3 * t * t * t)
    if p < 0.5:
        return -x
    return x

def statistical_power(alpha: float, effect_size: float, n: int) -> float:
    z_alpha = normal_ppf(1 - alpha / 2)
    z_beta = effect_size * np.sqrt(n) - z_alpha
    power = normal_cdf(z_beta)
    return round(float(power), 4)

def required_sample_size(alpha: float, power: float, effect_size: float) -> int:
    z_alpha = normal_ppf(1 - alpha / 2)
    z_beta = normal_ppf(power)
    n = ((z_alpha + z_beta) / effect_size) ** 2
    return int(np.ceil(n))

Explanation

  1. Statistical power = P(reject H0 | H1 is true) = 1 - P(Type II error).
  2. Under the alternative hypothesis, the test statistic is shifted by effect_size * sqrt(n).
  3. Power = CDF(effect_size * sqrt(n) - z_{alpha/2}) where z_{alpha/2} is the critical value.
  4. The required_sample_size function inverts this relationship: n = ((z_alpha + z_beta) / effect_size)^2.

Complexity

  • Time: O(1)
  • Space: O(1)
← #295#297 →