A single-run pass rate hides flaky agents, so measure pass^k

The idea

Most eval suites run each task once and report the fraction that passed. That number estimates average capability. It says nothing about consistency, and consistency is what a user feels. They do not get the best of several samples. They get one, every time they come back.

pass@k answers a generous question: did at least one of k attempts succeed. It suits setups where a verifier picks the winner, such as code generation with a test suite. pass^k answers a strict one: did all k attempts succeed. For an agent that issues refunds or edits records, the strict question is the one that matters.

With n trials of a task and c successes, the unbiased estimate is C(c,k) / C(n,k). Compute it per task, then average across tasks.

An example

Take a support agent evaluated on 16 refund tasks, each run 8 times, comparing two prompt versions.

• Version A passes every task 7 times out of 8.
• Version B passes 14 tasks 8 times out of 8 and fails the other 2 every time.

Both have a mean pass rate of 0.875. A single-run eval cannot tell them apart.

pass^8 can. For A, each task scores C(7,8) / C(8,8), which is zero, so the suite scores zero. For B, the suite scores 14 out of 16. Version B is the better release: its failures are deterministic, reproducible and fixable. Version A will eventually fail any customer who uses it enough, and nobody will be able to reproduce the failure on demand.

Where it breaks

It costs trials. Each task needs n runs with n at least k, so a large suite at a high k gets expensive. Most teams can afford it only on a critical subset.

Grader noise counts as model failure. If an LLM judge disagrees with itself on the same transcript, pass^k falls for reasons that have nothing to do with the agent.

Environment variance gets mixed in. Live tools, clock-dependent data and rate limits add failures that a metric meant to describe the model will absorb.

Independence is easy to assume and wrong. Estimating a task's pass^k as the mean pass rate raised to the power k breaks down when failures cluster on certain tasks, which is the exact case the metric exists to expose. Count per task instead.

It is the wrong metric where a human or a verifier chooses among candidates. Drafting copy and test-gated code generation are better served by pass@k.

In practice

I would start with a small set of tasks where a wrong action is expensive, and run each one enough times to reach the k that matches how often a real user repeats it.

• Report three things together: mean pass rate, pass^k, and the raw per-task counts such as 7/8.
• Triage by shape. Tasks at 0/n are bugs to fix. Tasks between zero and n are flakiness to investigate, usually ambiguous instructions or tool outputs.
• Freeze tool responses with recorded fixtures so the variance you measure belongs to the model and the prompt, not the network.
• Check the grader first. Regrade the same transcripts several times. If the verdicts disagree, fix the rubric before trusting any pass^k.
• Gate releases on pass^k for the critical subset not dropping, even if the mean rises.
• Do not treat temperature zero as determinism. Provider-side nondeterminism still produces different trajectories, so keep running repeats.

Run each critical task several times and gate on whether all runs pass, because a user experiences your agent's worst consistent behaviour, not its average.