Skip to content

Lesson 21 ยท Sequential and Bayesian Tests

Group Sequential Tests and Alpha Spending

Learn how group sequential designs let you stop an A/B test early without inflating errors: Pocock and O'Brien-Fleming boundaries and Lan-DeMets alpha spending, with runnable code.

The peeking problem says that looking many times costs you error rate. A group sequential design accepts that cost openly: you plan a small number of interim looks (say 4 or 5) and give each one a stricter threshold than 1.96, so the total false positive rate across all looks stays at 5%.

The error rate is a budget. You have 5% to spend (this is called alpha) and you decide in advance how much to spend at each look. Spend little early, when the data is thin and noisy; spend the rest at the end.

Boundaries: the two classic shapes

At each look you compute the usual z-statistic and compare it with a boundary. If it crosses, you may stop.

  • Pocock uses the same boundary at every look (for 5 looks it is about 2.41 instead of 1.96). Easy to understand, but it makes the final analysis harder to pass, and it spends a lot of alpha early.
  • O'Brien-Fleming uses very strict early boundaries (above 4 at the first of five looks) that relax toward roughly 2.0 at the end. Early stopping is rare and needs a huge effect, but the final test is almost as sensitive as a fixed-horizon test. Most experimentation teams prefer this shape.

Two ways to spend alpha

Pocock Same bar at every look
O'Brien-Fleming Very strict early, relaxed late

Both keep total error at 5%

1 / 2

Alpha spending functions

Fixing the number of looks in advance is awkward in practice: the third look may happen on day 9 instead of day 7. Lan-DeMets alpha spending removes that problem. You pick a function that says how much cumulative alpha may be spent by information fraction t (the share of the planned sample collected so far, from 0 to 1). The looks can then happen whenever you like.

Two common functions:

  • O'Brien-Fleming-like: alpha(t) = 2 - 2 * Phi(z_{alpha/2} / sqrt(t))
  • Pocock-like: alpha(t) = alpha * ln(1 + (e - 1) * t)
Python Cumulative alpha spent at each fraction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the first rows: at 20% of the data the O'Brien-Fleming-like function has spent almost nothing, the Pocock-like one already about 30% of the budget. Both end at exactly 0.05 at t = 1.

Turning a spending plan into boundaries

At look k you want the probability of first crossing at that look to equal the alpha newly spent: alpha(t_k) - alpha(t_{k-1}). Exact values come from numerical integration (software such as the R package gsDesign does this). We can get close by simulating many no-effect experiments as random walks and picking each boundary from the paths that have not yet stopped.

Python Simulate boundaries for 5 equally spaced looks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The boundaries start high and fall toward a value near 2.0, and the overall false positive rate across all five looks comes out close to 0.05. (They are Monte Carlo estimates, so the first boundary is noisy; production tools compute them exactly.)

Using the plan during the experiment

Python Check a result against the boundary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that a z-statistic near 2.5 at look 2 (which would be "significant" in a fixed test) does not stop the test, because the bar there is 3.23. This is the price of looking.

Things the boundaries do not fix

  • Futility stopping. You may also want to stop when the effect is clearly absent. Futility bounds are separate and are usually non-binding (you may ignore them without hurting alpha).
  • Point estimates. A test stopped early at a crossing still overestimates the effect. Adjusted estimates and intervals exist and are part of proper software.
  • Pre-registration. The looks and the spending function must be chosen before launch. Choosing them after seeing data brings the peeking problem back.
  • Metric timing. Interim looks need metrics that are mature at each look; a 14-day retention metric cannot be evaluated on day 3.

If your team wants only one early-stop safety net, use an O'Brien-Fleming-like boundary with 3-5 looks plus a guardrail check for serious harm. It protects you from disasters at almost no cost in final power.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main idea of a group sequential test?

    Alpha is a budget split across the looks, so each look uses a stricter boundary than 1.96.
  2. 2. Which boundary shape is very strict early and close to 1.96 at the end?

    O'Brien-Fleming boundaries start high and relax, costing almost no power at the final analysis.
  3. 3. What does a Lan-DeMets spending function allow?

    The function depends on the information fraction, so looks can happen whenever you like. It must still be chosen in advance.
  4. 4. At look 2 of 5 your z is 2.5 and the O'Brien-Fleming-type boundary is 3.23. What do you do?

    2.5 is below 3.23, so the boundary is not crossed. Continue to the next planned look.

Summary

  • Group sequential tests give each interim look a stricter boundary so the overall false positive rate stays at 5%.
  • Pocock boundaries are flat; O'Brien-Fleming boundaries are strict early and near 1.96 at the end.
  • Lan-DeMets alpha spending lets looks happen at flexible times, using a pre-chosen spending function.
  • Boundaries can be estimated by simulation; real tools compute them exactly.
  • Stopped tests still overestimate effects, and the plan must be fixed before launch.

Next: Bayesian A/B Testing with the Beta-Binomial Model