← Back to Benchmark Results

anthropic/claude-sonnet-4-6

76.8%
Pass Rate
43/56
Tasks Passed
3
Runs
70.8%
pass@1
76.8%
pass@3
91.1%
Consistency
0.1
Temperature
-
Thinking
784,969
Tokens
$8.23
Cost
1st: 992nd: 20Failed: 1343/56 passed

Known Shortcomings (5)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 event-subscriber-event-name event-subscriber 1 CG-AL-E010

Description: The model used 'OnAfterInsert' as the event name in the EventSubscriber attribute, but the correct event name for the Item table's after-insert event in Business Central is 'OnAfterInsertEvent'. The Item table (and other BC tables) expose integration events with the 'Event' suffix. The compiler error AL0280 confirms that 'OnAfterInsert' is not found on the Item table. Additionally, the procedure parameter signature should use 'RunTrigger: Boolean' as the second parameter for OnAfterInsertEvent.

Correct Pattern:
[EventSubscriber(ObjectType::Table, Database::Item, OnAfterInsertEvent, '', false, false)]
local procedure OnAfterInsertItem(var Rec: Record Item; RunTrigger: Boolean)
begin
    Message('New item has been created: %1 - %2', Rec."No.", Rec.Description);
end;
Incorrect Pattern:
[EventSubscriber(ObjectType::Table, Database::Item, 'OnAfterInsert', '', false, false)]
local procedure OnAfterInsertItem(var Rec: Record Item)

Error Codes: AL0280

2 count-method-column-syntax query-aggregation-columns 1 CG-AL-H017

Description: The model generated a column with Method = Count but did not provide a valid data source field. In AL queries, a column with Method = Count still requires a source field reference (e.g., a field from the dataitem). The model wrote 'column(MatchCount)' without specifying a source field, which triggers AL0353: 'A Column must have a valid data source or have the Method property set to Count'. Despite the Method being set to Count, the AL compiler still requires a source field to be specified in the column declaration syntax, e.g., 'column(MatchCount; "Dimension Code")' with Method = Count.

Correct Pattern:
column(MatchCount; "Dimension Code")
{
    Caption = 'Match Count';
    Method = Count;
}
Incorrect Pattern:
column(MatchCount)
{
    Caption = 'Match Count';
    Method = Count;
}

Error Codes: AL0353

3 code-truncation-incomplete-output complete-code-generation 1 CG-AL-M009

Description: The model generated code that was truncated/incomplete. The output contains multiple `BEGIN-CODE` markers, indicating the model attempted to regenerate the code multiple times but each attempt was cut off before completion. The final version of the main codeunit (CG-AL-M009.al) is truncated mid-procedure - the `ValidateZipCodeFormat` procedure ends with 'if CountryUpper = 'AU' then begin if ZipLen <> 4 then exit(false); IsNum' which is incomplete. Additionally, the `LogAuditEntry` and `ValidateStateForCountry` local procedures are never defined, and the codeunit's closing `end;` and `}` are missing. The compilation errors at line 370 correspond to the file comment `// File: CG-AL-M009.MockShippingProvider.al` being parsed as AL code within the incomplete first file, since the codeunit was never properly closed.

Correct Pattern:
The model should have generated a complete, non-truncated codeunit with all local procedures (LogAuditEntry, ValidateStateForCountry, ValidateZipCodeFormat fully completed) and proper closing syntax. The code should fit within the output token limit, or the model should simplify the implementation to avoid truncation.
Incorrect Pattern:
if CountryUpper = 'AU' then begin
            if ZipLen <> 4 then
                exit(false);
            IsNum

// File: CG-AL-M009.MockShippingProvider.al

Error Codes: AL0224

4 jsonvalue-type-checking-api json-handling 1 CG-AL-M020

Description: The model attempted to use non-existent methods on JsonValue such as IsBoolean(), IsDecimal(), IsInteger(), and GetType(). The AL JsonValue type does not have these methods. The compilation errors reference lines in the ParseConfigSettings procedure where the model tried to check value types using methods that don't exist on JsonValue. The correct approach is to simply call AsText() on any JsonValue to get its string representation, since AsText() works for all JSON value types (boolean, number, string). The model's generated code has a nonsensical case statement comparing JsonVal.GetType() with JsonToken.GetType(), and the actual compiled code (which differs slightly from what's shown - the errors reference IsBoolean, IsDecimal, IsInteger at line 75/78/79) likely had an earlier version or the compilation errors come from a different attempt. Regardless, the model doesn't understand that JsonValue in AL doesn't expose type-checking methods and that AsText() is the universal way to convert any JSON value to text.

Correct Pattern:
foreach KeyName in KeyNames do begin
    ConfigJson.Get(KeyName, JsonTok);
    JsonVal := JsonTok.AsValue();
    ResultDict.Add(KeyName, JsonVal.AsText());
end;
Incorrect Pattern:
if JsonVal.IsNull() then
    ResultDict.Add(KeyName, '')
else begin
    case true of
        (JsonVal.GetType() = JsonToken.GetType()):
            ResultDict.Add(KeyName, JsonVal.AsText());
        else
            ResultDict.Add(KeyName, JsonVal.AsText());
    end;
end;

Error Codes: AL0132

5 flowfield-calcsums-restriction flowfield 1 CG-AL-M006

Description: The model used CalcSums on 'Remaining Amount' which is a FlowField on the Cust. Ledger Entry table. In AL, CalcSums can only be used on Normal fields, not FlowFields. The model should have iterated through the records and used CalcFields to get each entry's Remaining Amount, or used a different approach to sum outstanding balances.

Correct Pattern:
Instead of CalcSums on a FlowField, iterate through records and use CalcFields for each entry:

if CustLedgerEntry.FindSet() then
    repeat
        CustLedgerEntry.CalcFields("Remaining Amount");
        OutstandingBalance += CustLedgerEntry."Remaining Amount";
    until CustLedgerEntry.Next() = 0;
Incorrect Pattern:
CustLedgerEntry.CalcSums("Remaining Amount");

Error Codes: AL0827