Pass rate
Tasks solved / tasks in scope, up to 2 attempts (strict per-set denominator).
Formula: (tasks_passed_attempt_1 + tasks_passed_attempt_2_only) / task_set_size, where both numerator terms are means across the runs.
Includes unattempted tasks as failures. Scope-aware; reflects active filters (set, category, difficulty). A model benched several times scores the average of its runs, so run count does not inflate it. Final assisted solve rate with up to 2 attempts; drill-down companion to Solve AUC@2.
Avg cost / task
Average LLM cost of running one benchmark task once, in USD.
Formula: SUM(cost_usd) / COUNT(DISTINCT (run_id, task_id)) across all the model's results in scope.
Use to compare operating cost across models with similar pass rates. Does not account for quality. Combine with $/Pass for a cost-efficiency view.
Latency p50
Median per-task wall time (LLM call + compile + test), in milliseconds.
Formula: 50th percentile of per-task duration_ms: LLM latency + compile time + test time.
Use p50 for a typical-case latency expectation. Unaffected by outlier slow tasks. Shown as N/A for batch runs: a batch request waits in the provider queue, so no per-request model timing is recorded and the only duration left would be our own compile-and-test time, which describes the harness rather than the model.
Avg attempt score
Mean per-attempt score on a 0–100 point scale (partial credit). Drill-down only.
Formula: Mean of attempt scores across all results rows: SUM(score) / COUNT(*) over the results table. Each attempt earns 0–100 points based on compile + test outcomes.
Drill-down companion to pass_at_n. Rewards partial credit but not directly comparable to pass rate; use for within-model analysis.
All-runs pass rate
Fraction of tasks the model solved in every single run (strict consistency, also written pass^n).
Formula: tasks where ALL runs produced a passing result / tasks_attempted_distinct
Measures reliability under repetition. High value means the model is unlikely to regress on a re-run, important for CI and production use. Formal name in the literature: pass^n.
$/Pass
Average USD cost per solved task (any-attempt pass).
Formula: SUM(cost_usd) / number of passed (run, task) cells across all runs.
Best single cost-efficiency metric. Penalises expensive models that pass few tasks and rewards cheap models with high pass rates.
Latency p95
95th-percentile per-task wall time. Captures tail latency.
Formula: 95th percentile of per-task duration_ms across all tasks in all runs.
Use p95 to understand worst-case latency. A low p95 means the model rarely stalls, relevant for automated pipelines with timeouts. Shown as N/A for batch runs: a batch request waits in the provider queue, so no per-request model timing is recorded and the only duration left would be our own compile-and-test time, which describes the harness rather than the model.
Overview
Claude Sonnet 5 has run on 3 occasions, attempting 232 tasks with an average score of 72.1 / 100.
Settings
Generation parameters used across this model's runs. "varies" indicates the value differed between runs.
- Temperature
- varies
- Thinking budget
- varies
- Avg tokens / run (input + output)
- 2,921,473
- Consistency
- 71.5%
History
Cost
Failure modes
- AL0104 188 Syntax error, ')' expected view all →
- AL0000 181 App generation failed view all →
- AL0132 98 'FieldType' does not contain a definition for 'Enum' view all →
- AL0107 66 Syntax error, identifier expected. Provide a valid name (letters, digits, and underscores only). view all →
- AL0111 57 Semicolon expected. Add a semicolon (;) to terminate the statement. view all →
- AL0105 56 Syntax error, identifier expected; 'key' is a keyword view all →
- AL0126 38 No overload for method 'Clear' takes 1 arguments. Candidates: 'Clear()' defined in Codeunit 'CG H054 Cache' by the extension CentralGauge_CG-AL-H054_2 by CentralGauge (1.0.0.0) view all →
- AL0135 27 There is no argument given that corresponds to the required formal parameter 'ConflictMessage' of 'SyncContact(Code[20], Code[20], Text[100], Text[20], Text[100], Text[20], var Text)' view all →
- AL0198 25 Expected one of the application object keywords (table, tableextension, page, pageextension, pagecustomization, profile, profileextension, codeunit, report, reportextension, xmlport, query, controladdin, dotnet, enum, enumextension, interface, permissionset, permissionsetextension, entitlement) view all →
- AL0118 23 The name 'CreateSequentialGuid' does not exist in the current context. view all →
Shortcomings
AL concepts Claude Sonnet 5 struggles with. Click a row for description, correct pattern, and observed error codes.
Shortcomings analysis queued
Queued for analysis. This section will populate once the run is processed.
Recent runs
| Started | Run | Model | Tasks | Score | Cost | Duration | Status |
|---|---|---|---|---|---|---|---|
| 8d ago | a1d5ecf0-e65… | 186/231 | 71.8 / 100 | $19.86 | 57m 49s | completed | |
| 8d ago | b447ecd3-d6b… | 187/232 | 73.0 / 100 | $20.30 | 1h 1m | completed | |
| 8d ago | ae125a12-0b6… | 183/231 | 71.6 / 100 | $20.39 | 1h 3m | completed | |
| 9d ago | 145a7708-c39… Excluded | 138/208 | 41.9 / 100 | $29.17 | 40m 29s | completed | |
| 9d ago | c7a94f8b-2bc… Excluded | 169/232 | 54.2 / 100 | $27.93 | 52m 11s | completed | |
| 9d ago | 4c095313-8a6… Excluded | 165/232 | 64.2 / 100 | $20.20 | 51m 11s | completed |
Methodology
Pass rate counts unattempted tasks as failures (strict denominator). Avg score is the mean of every attempt the model produced — failed first tries that triggered a retry contribute one observation each, pulling the mean down. See the about page for the full breakdown and unit conventions.