caching

Solve AUC@2

Skill score: full credit for a first-try solve, half for a retry solve. Not the solve rate.

Formula: AUC@2 = (pass@1 + solve@2) / 2

Use as the headline ranking metric. Rewards first-try correctness over fail-then-repair without ignoring the two-attempt protocol. De-saturates the headline that pass_at_n compresses. Significance via paired bootstrap (tier bands), not Wilson.

Avg cost / task

Average LLM cost of running one benchmark task once, in USD.

Formula: SUM(cost_usd) / COUNT(DISTINCT (run_id, task_id)) across all the model's results in scope.

Use to compare operating cost across models with similar pass rates. Does not account for quality. Combine with $/Pass for a cost-efficiency view.

Latency p95

95th-percentile per-task wall time. Captures tail latency.

Formula: 95th percentile of per-task duration_ms across all tasks in all runs.

Use p95 to understand worst-case latency. A low p95 means the model rarely stalls, relevant for automated pipelines with timeouts. Shown as N/A for batch runs: a batch request waits in the provider queue, so no per-request model timing is recorded and the only duration left would be our own compile-and-test time, which describes the harness rather than the model.