← Back to Benchmark Results

anthropic/claude-opus-4-6

78.6%
Pass Rate
44/56
Tasks Passed
3
Runs
77.4%
pass@1
78.6%
pass@3
98.2%
Consistency
0.1
Temperature
-
Thinking
686,026
Tokens
$11.89
Cost
1st: 1072nd: 23Failed: 1244/56 passed

Known Shortcomings (8)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 reserved-keyword-as-parameter-name al-syntax-reserved-words 1 CG-AL-H014

Description: The model used 'Key' as a parameter name in the SafeGetText procedure. In AL, 'key' is a reserved keyword and cannot be used as an identifier. The model should have used a non-reserved name like 'KeyName', 'PropertyKey', or quoted it with double quotes. The task prompt specified 'Text named Key' which is itself problematic since 'Key' is reserved, but the model should have known to escape or rename it.

Correct Pattern:
procedure SafeGetText(Obj: JsonObject; "Key": Text; DefaultValue: Text): Text
// Or use a non-reserved name like KeyName:
procedure SafeGetText(Obj: JsonObject; KeyName: Text; DefaultValue: Text): Text
Incorrect Pattern:
procedure SafeGetText(Obj: JsonObject; Key: Text; DefaultValue: Text): Text

Error Codes: AL0105

2 cross-join-dataitem-link query-dataitem-joins 1 CG-AL-H017

Description: The model incorrectly added a DataItemLink on the 'DimensionSetEntry' dataitem that references a field from the 'Department' dataitem (DataItemLink = "Dimension Code" = Department."Dimension Code"). In AL queries, a Column or Filter source must reference a field defined on the table of its parent DataItem. With CrossJoin between Department and Project, the DimensionSetEntry is nested under Project, so its DataItemLink can only reference fields from its direct parent (Project) or use fields from its own table. The reference to Department."Dimension Code" is invalid because 'Dimension Code' on the Dimension Value table (Department) is not a field on the Dimension Set Entry table's parent chain in the way the compiler expects. Additionally, for a LeftOuterJoin, the DataItemLink must reference the immediate parent dataitem's fields. The model should have either linked DimensionSetEntry to Project's fields or used an appropriate link structure that the compiler accepts.

Correct Pattern:
dataitem(DimensionSetEntry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Project."Dimension Code", "Dimension Value Code" = Project."Code";

    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}
Incorrect Pattern:
dataitem(DimensionSetEntry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Department."Dimension Code";

    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}

Error Codes: AL0345

3 incomplete-procedure-body code-generation-completeness 1 CG-AL-M009

Description: The model generated a truncated/incomplete procedure body for the 'ContainsDigit' local procedure. The code cuts off at 'var\n i: Integer;\n Current' without completing the variable declaration, procedure body, or closing the codeunit. This caused cascading syntax errors. The model failed to produce complete code, likely due to output token limits or generation issues, resulting in an unterminated procedure and codeunit block.

Correct Pattern:
local procedure ContainsDigit(InputText: Text): Boolean
    var
        i: Integer;
        CurrentChar: Char;
    begin
        for i := 1 to StrLen(InputText) do begin
            CurrentChar := InputText[i];
            if (CurrentChar >= '0') and (CurrentChar <= '9') then
                exit(true);
        end;
        exit(false);
    end;
Incorrect Pattern:
local procedure ContainsDigit(InputText: Text): Boolean
    var
        i: Integer;
        Current

Error Codes: AL0104

4 flowfield-calcfields-requirement flowfield 1 CG-AL-H003

Description: The Item.Inventory field is a FlowField in Business Central. FlowFields are not automatically calculated when reading records - you must explicitly call CalcFields before accessing their value. The generated code reads Item.Inventory directly without calling Item.CalcFields(Inventory), so the value is always 0, resulting in 0% discount for all items. The test finds items with Inventory >= 100 (using SetFilter which evaluates at SQL level) and expects them to appear with 15% discount, but they don't because the codeunit sees Inventory as 0.

Correct Pattern:
Item.CalcFields(Inventory);
DiscountPercent := CalculateDiscount(Item.Inventory);
Incorrect Pattern:
DiscountPercent := CalculateDiscount(Item.Inventory);
5 parse-failure unknown 1 CG-AL-M005

Description: Failed to parse LLM analysis response: { "outcome": "model_shortcoming", "category": "model_knowledge_gap", "concept": "multi-file-output-formatting", "alConcept": "code-generation-output-format", "description": "The model includ

Correct Pattern:
Incorrect Pattern:
6 report-labels-block-closure report-structure 1 CG-AL-M007

Description: The model failed to properly close the report object. The generated code has an incomplete 'labels' block at the end of the report - it starts the labels section but never closes it with proper closing braces. Additionally, the model concatenated two separate AL files into one file, placing the codeunit 80097 code directly after the incomplete report definition. The labels block is missing closing braces for the labels section, the report object itself, and then the second file (the mock calculator codeunit) is appended without proper file separation. This results in unexpected characters (backticks from markdown formatting) at line 453 and missing structural elements. The model also failed to include several helper procedures referenced in the report (BuildCustomerRankings, BuildTopProductsList, CalcInvoiceAmounts, CalcInvoiceQuantity, CalcCustomerSalesAmount, CalcCustomerSalesQty, CalcCustomerOrderCount, CalcRegionTotals, IsItemTopProduct, GetCustomerRank) and the required variable declarations.

Correct Pattern:
The labels block should be properly closed with '}', followed by variable declarations (var section) with all referenced variables, all helper procedure implementations, and then the report closing '}'. The mock calculator codeunit should be in a separate file. Example:

    labels
    {
        CustomerNoLbl = 'Customer No.';
        // ... more labels
    }

    var
        StartDate: Date;
        EndDate: Date;
        // ... all other variables

    // ... all helper procedures
}
Incorrect Pattern:
    labels
    {
        CustomerNoLbl = 'Customer No.';
        CustomerNameLbl = 'Customer Name';
        RegionLbl = 'Region';
        CategoryLbl = 'Category';
        TotalSalesLbl = 'Total Sales';
        OrderCountLbl = 'Order Count';
        AvgOrderValueLbl = 'Avg. Order Value';
        YoYGrowthLbl = 'YoY Growth %';

// File: CG-AL-M007.MockCalculator.al
codeunit 80097 "CG-AL-M007 Mock Calculator"

Error Codes: AL0183

7 query-filter-element-syntax query-definition 1 CG-AL-H011

Description: The model used 'const(Order)' in the ColumnFilter property of a filter element, but 'Order' is an enum value that is not a simple identifier — it likely needs to be quoted or the syntax is slightly different. Looking at line 26, the error is at the ColumnFilter = const(Order) line. In AL query objects, the ColumnFilter for enum values requires the value to be a string literal. The correct syntax is ColumnFilter = const("Order") with proper quoting. The AL compiler sees 'Order' as an unexpected token because enum values in const() filters need to be string-quoted.

Correct Pattern:
filter(Document_Type; "Document Type")
{
    ColumnFilter = const("Order");
}
Incorrect Pattern:
filter(Document_Type; "Document Type")
{
    ColumnFilter = const(Order);
}

Error Codes: AL0107

8 code-block-markers-in-output output-formatting 1 CG-AL-M009

Description: The model produced a truncated first version of the code (the GetShipmentStatus case statement was cut off mid-line with 'StatusText := CopyStr'), then emitted a 'BEGIN-CODE' marker and started a second complete version of the code. This resulted in the generated file containing two overlapping copies of the interface and codeunit definitions, causing syntax errors. The first copy is incomplete (truncated in the middle of a case statement), and the second copy starts without the first being properly closed. The compilation errors at lines 394-397 correspond to the truncation point where the first copy abruptly ends, and errors at line 798-799 correspond to structural issues from the duplicated/overlapping code.

Correct Pattern:
The model should have produced a single, complete AL file without any BEGIN-CODE markers or duplicate definitions. The GetShipmentStatus procedure should have been completed properly, e.g.:
            else
                StatusText := CopyStr(StrSubstNo('Shipment %1: Status unknown', TrackingNumber), 1, 100);
        end;
        exit(StatusText);
    end;
Incorrect Pattern:
            else
                StatusText := CopyStr
BEGIN-CODE
interface "Shipping Provider"
{

Error Codes: AL0104