Your eval gate says 5%, and that number is true only once. It holds when you read the score once, at the end of a run whose case count you fixed in advance. Stop the run at the first result that clears the bar, and the same gate breaks. On two identical arms it found a “significant” difference 26.1% of the time [2]. Streaming evals and live canary dashboards have made that broken gate the default one.

At a glance

  • The shift: Evals are watched while they run, so a gate gets consulted many times instead of once.
  • The number: 26.1% on arms that were the same, once the run could stop early [2].
  • The risk: An interval built for one look means nothing at the second.

Why the fifth look differs

Evan Miller worked this out in April 2010. Dashboards refreshed after every order then, and teams stopped tests as soon as the number looked good. Adrián Molas asked the same question with a Bayesian simulation on 30 October 2025, and found the shortcut no safer. In August 2026, a UK AI Security Institute preprint reversed the question. Instead of asking how many looks spoil a threshold, it asked when a run may be stopped honestly [1].

The mechanism is the sampling plan, not the software. A false-positive rate — how often a gate calls a difference that is not there — holds only when the case count is fixed first. Every extra look is another chance to cross the bar, and one chance changes the odds for the next. A test read after every case is not the test that was designed.

Miller’s table shows how large the effect is: with ten looks at a running test, a dashboard has to read 1.0% for the real error rate to be 5% [2]. I read that row three times before I believed it. Nobody sets a gate to 1.0%, so the ten-look gate keeps its label and loses its meaning.

A Bayesian posterior probability looks like an escape. It stays readable at any sample size, a real plus, but it does not make stopping on success safe. Molas ran a simulation with two arms at the same conversion rate, checked the posterior every hundred cases, and stopped whenever it passed 0.95. The false-positive rate reached 80% [3]. His conclusion is narrower than that. The posterior stays readable under continuous monitoring, but a fixed threshold does not control false positives when you stop on success [3].

An eval harness adds speed to that problem: a few hundred cases finish in minutes, so the score is legible long before the run is complete. The sums behind how many cases a 95% gate actually needs assume exactly one look.

By the numbers

  • 80% false positives: Molas’s Bayesian gate at a 0.95 threshold, checked every 100 cases on two arms that behave the same [3].
  • 57% to 97% of planned trials: what a precision-based stopping rule removed across nine validation settings [1].
  • 16 times the variance: Miller’s rule of thumb for the sample a fixed-horizon gate needs [2].
Your 5% Gate Breaks at 26.1% Figures as stated in this article's own numbers section (verified figures, %) False positives 80% ([3]) Of planned trials 57% ([1])

Where this lands in a harness

Harness teams have two fixes, and they are not interchangeable. Some stop looking: they fix the case count in advance and report the interval that belongs to that one look. Mid-run readings become progress indicators rather than decisions. Miller’s fixed-sample rule gives that count directly; for a pass rate, the cases needed grow fast as the smallest difference worth detecting narrows [2].

Other teams keep looking and change the instrument. An interval that stays valid at any stopping time is a different object from one computed at a fixed sample size. Johari, Pekelis and Walsh call that property always valid; it lets a team act on data as it comes in without losing the promise [4]. The 2026 preprint works differently. It treats an eval as a sequence of measurements, samples where uncertainty is high, and stops where the estimate is tight [1].

I would watch the safeguard more than the stopping rule. It samples more cautiously as measured performance nears zero, because a suite near the floor turns on a handful of successes [1]. The open question is whether that safeguard holds on a real suite whose pass rate is near zero.

The catch

The savings range in that paper belongs to one example design: a 200-item set run to ten epochs, reported in a single-author preprint [1]. The paper says the savings depend on how the eval is built. A suite of independent, costly cases may not save as much as a fixed plan that repeats cheap items once the estimate is tight.

I think the width is the actual product, not a drawback. Any interval that stays honest across every look is wider than one made at the end. When a team adopts it, some of the detail its old dashboard appeared to have disappears, and part of that detail was noise. Miller’s 26.1% is likewise one setting: a test after every observation with a 150-observation cap, which he calls close to a worst case [2].

The question to ask of any number a gate prints is plain: how many times was it allowed to move before somebody acted on it? A rate that describes a model cannot be separated from the number of looks that produced it. The number of looks is the part a team controls.

Related reading

Sources