CentralGauge

Benchmark for LLMs on Microsoft Dynamics 365 Business Central AL code.

Updated 12d ago 232 tasks 2 attempts/model 95% paired-bootstrap CI Solve AUC@2 = (pass@1 + solve@2) / 2

Best overall

GPT-6.1 Sol gpt gpt-6.1-sol

· 86.4

Tier 1

Best value · AUC ≥ 75

GPT-6.1 Sol gpt gpt-6.1-sol

86.4 AUC · $0.02/solved

Fastest ≥ 75 AUC

GPT-6.1 Sol gpt gpt-6.1-sol

p95 140.2s · 86.4 AUC

Best open-weight

—

Leaderboard
#	Model	Solve AUC@2 Skill score: full credit for a first-try solve, half for a retry solve. Not the solve rate. Formula: `AUC@2 = (pass@1 + solve@2) / 2` Use as the headline ranking metric. Rewards first-try correctness over fail-then-repair without ignoring the two-attempt protocol. De-saturates the headline that pass_at_n compresses. Significance via paired bootstrap (tier bands), not Wilson.	CI Pass Rate 95% CI 95% Wilson confidence interval on the pass rate. Formula: `Wilson score interval: center ± half-width, where n = strict denominator (task_set_size or category/difficulty-scoped count when taskSetHash is provided; falls back to tasks_attempted_distinct for legacy callers).` Use to judge whether a lead over another model is statistically meaningful. Wide CIs indicate too few tasks to draw firm conclusions. Wilson score interval (Wikipedia) ↗	Avg cost / task Average LLM cost of running one benchmark task once, in USD. Formula: `SUM(cost_usd) / COUNT(DISTINCT (run_id, task_id)) across all the model's results in scope.` Use to compare operating cost across models with similar pass rates. Does not account for quality. Combine with $/Pass for a cost-efficiency view.	Latency p95 95th-percentile per-task wall time. Captures tail latency. Formula: `95th percentile of per-task duration_ms across all tasks in all runs.` Use p95 to understand worst-case latency. A low p95 means the model rarely stalls, relevant for automated pipelines with timeouts. Shown as N/A for batch runs: a batch request waits in the provider queue, so no per-request model timing is recorded and the only duration left would be our own compile-and-test time, which describes the harness rather than the model.	Details
1	GPT-6.1 Sol gpt gpt-6.1-sol	86.4	±3.2	$0.02	140.2s

Showing 1 of 1