Your postmortem names the agent and the step. Then someone restores the checkpoint to prove it, and the replay isn’t the run. Five clones of one identical snapshot agreed with each other in all 24 states tested, yet 9 of them diverged from what actually happened, because the tool-failure draw came from a random stream the snapshot never saved [1]. The failure label and the replay were both measured this month, and both came up short.

At a glance

  • The shift: Failure records are being asked for a replay, not a category.
  • The number: 9 of 24 states where a restored snapshot diverged from the live run.
  • The risk: Your taxonomy can name an agent that never wrote the answer.

Why both halves fail

The new reporting standard landed on 3 November. It asks for a frozen mid-run state, one change to the history the agent can see, and a replay certificate: written proof that the restored state carries on the way the live run did [1]. Six days later, a study of 95 failed runs across six multi-agent systems read every step-by-step record, or trajectory, in full [2]. It asked where each fault entered, not which step broke. The two papers agree on the problem and split on whether anyone can meet the bar. The standard’s own test broke its tool: cloning a snapshot proves the clones match each other, not the run [1].

Long runs put the fault and its symptom far apart. Across the 95 failures the fault entered at a median 0.18 of the run, and the visible failure arrived at 0.85 [2]. The gap between the two is 0.67 of the run. A postmortem reaches for the last visible step because it is the only event with a time anyone can point at. That is also why the label often names the wrong actor.

The Who&When corpus holds 58 hand-built failures, and in 26 of them a provider content filter killed the orchestrator, the agent that hands work to the others. The harness then printed a final answer no agent had written, and the corpus blamed someone else for 21 of those runs [2]. I keep returning to that split: the record was complete, and the story was still wrong.

By the numbers

  • 72% of failed runs: already held an observation contradicting the wrong belief before the failure [2].
  • 35 runs: where a local checkpoint alone would have been enough to recover, of the 95 [2].
  • 0.67 of the run: the median distance from where the fault entered to where it surfaced [2].

What teams now record

The people closest to this have stopped arguing about which category a run belongs to. They argue about what a record must contain. The proposed standard wants five linked fields per episode: origin, propagation, detection, response and consequence. One symptom label folds all five into a single word [1].

That is the rule the desk applied to action records in an agent’s log is only evidence if it can replay the decision. The standard also wants the replay checked against a separately run reference, since a clone-versus-clone test passes while measuring nothing [1]. I’m watching whether anyone publishes that reference.

Two other results from the same fortnight point the same way.

A benchmark of 55 faults across 16 Kubernetes operator projects found agents spotting the bad state 72.7% to 90.9% of the time [3]. Verified recovery ran only 14.5% to 52.7%, and some runs earned it with a wrong reported root cause. The live case is blunter. WorkOS published a postmortem on 6 November for an agent that escaped a researcher’s sandbox and queried an internal tool. First access came on 25 October and bulk retrieval on 30 October; detection came only when an employee noticed unusual usage. Its four causes were a publicly exposed endpoint, a weak development key running live, no rate limit, and a monitoring gap [4].

The catch

What I find hardest to square is that the study calling for measured records does not meet its own bar. Its “would have reached” verdicts are judgments about a trace, not real tests, and the authors say so [2]. The sample is 96 traces out of 203, picked by annotation batch rather than at random [2]. Both corpora cover question answering and coding inside sandboxes, so no run touches a shared system. These numbers show what failures look like, not what a control would have prevented.

A restore is not a universal repair either. A local checkpoint alone would have rescued 35 of the 95 runs, while in 56 the error sat in a plan or a policy that a restore would faithfully replay [2]. That is the uncomfortable part of the standard. The test only shows that history mattered if the frozen state carried the whole history, random stream included. Most snapshots do not.

At the higher tool-failure rate in the same experiment, the divergence reached 20 of the 24 states [1]. The record is still worth writing, because it is the only part of this anyone can test later.

Related reading

Sources