← Back to Benchmark Results

anthropic/claude-sonnet-4-5-20250929

67.9%
Pass Rate
38/56
Tasks Passed
3
Runs
66.7%
pass@1
67.9%
pass@3
96.4%
Consistency
0.1
Temperature
-
Thinking
435,408
Tokens
$4.12
Cost
1st: 892nd: 23Failed: 1838/56 passed

Known Shortcomings (14)

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 multiline-string-literals text-literal-syntax 1 CG-AL-E050

Description: The model attempted to use raw multiline string literals by embedding newlines directly within single-quoted strings. AL does not support multiline string literals in this way — a text literal must be terminated on the same line. The task description says 'Use a multiline string literal, not string concatenation', which likely refers to AL's newer multiline text literal syntax (available in newer AL versions) or requires a specific approach. In classic AL, multiline strings must be constructed using concatenation with CR/LF characters or using the newer AL multiline string literal syntax (triple single quotes or similar). The model incorrectly assumed that simply breaking a single-quoted string across multiple lines would work, which causes AL0360 'Text literal was not properly terminated' errors.

Correct Pattern:
In AL, multiline text can be built using concatenation with newline characters, e.g.:

var
    CrLf: Text[2];
begin
    CrLf[1] := 13;
    CrLf[2] := 10;
    SqlQuery := 'SELECT CustomerNo, Name, Balance' + CrLf + 'FROM Customer' + CrLf + 'WHERE Active = true' + CrLf + 'ORDER BY Name';

Note: Despite the task saying 'use a multiline string literal, not string concatenation', AL does not support raw multiline string literals in the way the model attempted. The task description may be misleading, but the model should know AL's actual string literal rules.
Incorrect Pattern:
SqlQuery := 'SELECT CustomerNo, Name, Balance
FROM Customer
WHERE Active = true
ORDER BY Name';

Error Codes: AL0360

2 query-filter-element-syntax query-definition 1 CG-AL-H011

Description: The model incorrectly specified the ColumnFilter property on a filter element. In AL query objects, when using a 'filter' element, the ColumnFilter should reference the filter element's name (not the source field name with quotes). The correct syntax is 'ColumnFilter = Document_Type = const(Order)' rather than 'ColumnFilter = "Document Type" = const(Order)'. The filter element name 'Document_Type' is the identifier used in the ColumnFilter, not the source field name 'Document Type'. The compiler error AL0186 indicates that the reference '"Document Type"' does not exist in the context of the application object 'Document_Type'.

Correct Pattern:
filter(Document_Type; "Document Type")
{
    ColumnFilter = Document_Type = const(Order);
}
Incorrect Pattern:
filter(Document_Type; "Document Type")
{
    ColumnFilter = "Document Type" = const(Order);
}

Error Codes: AL0186

3 jsonobject-get-method-signature json-api-usage 1 CG-AL-H014

Description: The model incorrectly used JsonObject.Get('key') as a single-parameter function that returns a JsonToken directly. In AL, JsonObject.Get() requires two parameters: Get(Key: Text, var Result: JsonToken) and returns Boolean. The model also incorrectly called JsonValue.Get() which does not exist — JsonValue has AsText(), AsInteger(), AsDecimal(), AsBoolean() methods instead. The model confused the AL JSON API with other language JSON APIs.

Correct Pattern:
var NameToken: JsonToken;
if CustomerJson.Get('name', NameToken) then
    Name := NameToken.AsValue().AsText();
if CustomerJson.Get('age', AgeToken) then
    Age := AgeToken.AsValue().AsInteger();
if CustomerJson.Get('active', ActiveToken) then
    Active := ActiveToken.AsValue().AsBoolean();
// For decimals: Amount := AmountToken.AsValue().AsDecimal();
Incorrect Pattern:
CustomerJson.Get('name').AsValue().Get(Name);
ItemObject.Get('quantity').AsValue().Get(Quantity)
ResultToken.AsValue().Get(ResultValue)
AmountToken.AsValue().Get(Amount)

Error Codes: AL0135

4 cross-join-dataitem-link-constraints query-dataitem-linking 1 CG-AL-H017

Description: The model incorrectly specified a DataItemLink on the 'Dimension_Set_Entry' dataitem that references a field from the 'Department' dataitem (the grandparent), but in AL queries, a Column or Filter source must reference a field defined on the table of its parent DataItem. The DataItemLink on 'Dimension_Set_Entry' uses 'Department."Dimension Code"' and 'Department.Code', but the parent DataItem of 'Dimension_Set_Entry' is 'Project' (which references 'Dimension Value'). Additionally, with CrossJoin between Department and Project, the linking semantics are complex. The model should have either linked to the parent 'Project' dataitem's fields or used a different linking strategy that is valid for the LeftOuterJoin on 'Dimension Set Entry'. For a LeftOuterJoin, the DataItemLink must reference fields from the immediate parent dataitem's table.

Correct Pattern:
The DataItemLink on 'Dimension_Set_Entry' must only reference fields from its immediate parent dataitem 'Project'. For example:
DataItemLink = "Dimension Code" = Project."Dimension Code", "Dimension Value Code" = Project.Code;
Alternatively, if the intent is to link to both Department and Project, the query structure needs to be redesigned since AL query DataItemLink can only reference the parent dataitem's fields.
Incorrect Pattern:
dataitem(Dimension_Set_Entry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Department."Dimension Code",
                   "Dimension Value Code" = Department.Code;
    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}

Error Codes: AL0345

5 reserved-keyword-as-variable-name al-reserved-words 1 CG-AL-H020

Description: The model used 'Key' as a variable name in the MergeDictionaries and GetKeys procedures. In AL, 'Key' is a reserved word (it's used in table definitions for key groups), so it cannot be used as a variable identifier. The compiler raises AL0519 ('Key' is not valid value in this context) because it interprets 'Key' as a keyword rather than a variable name. The model should have used a different variable name like 'DictKey' or 'KeyValue'.

Correct Pattern:
var
    DictKey: Text;
...
foreach DictKey in Keys do
Incorrect Pattern:
var
    Key: Text;
...
foreach Key in Keys do

Error Codes: AL0519

6 reserved-keyword-as-variable-name al-reserved-keywords 1 CG-AL-M021

Description: The model used 'Key' as a variable name in AL, but 'Key' is a reserved keyword in AL. This causes compilation errors AL0519 and AL0105. The variable declarations on line 84 ('Key: Text;') and usage on line 139 ('Key: Text') fail because 'Key' cannot be used as an identifier. The model should have used a different variable name like 'KeyName', 'YamlKey', or 'PropertyKey' to avoid the reserved keyword conflict.

Correct Pattern:
YamlKey: Text;
...
Value := CopyStr(Line, ColonPos + 1).Trim();
...
local procedure GetJsonValue(JsonObj: JsonObject; KeyName: Text): Text
Incorrect Pattern:
Key: Text;
...
Value := CopyStr(Line, ColonPos + 1).Trim();
...
local procedure GetJsonValue(JsonObj: JsonObject; Key: Text): Text

Error Codes: AL0519

7 http-headers-get-pattern http-client-api 1 CG-AL-M005

Description: The model incorrectly called GetHeaders() as a function returning a value (fluent style), but in AL, HttpContent.GetHeaders() requires a var HttpHeaders parameter. The correct pattern is to declare an HttpHeaders variable, call GetHeaders(Headers) to populate it, then use Headers.Clear() and Headers.Add(). The same error occurs on RequestMessage.GetHeaders(). This is a common AL HTTP API pattern the model failed to use correctly.

Correct Pattern:
var
    ContentHeaders: HttpHeaders;
    RequestHeaders: HttpHeaders;
begin
    Content.GetHeaders(ContentHeaders);
    ContentHeaders.Clear();
    ContentHeaders.Add('Content-Type', 'application/json');
    ...
    RequestMessage.GetHeaders(RequestHeaders);
    RequestHeaders.Add('Authorization', AuthTokenTxt);
Incorrect Pattern:
Content.GetHeaders().Clear();
Content.GetHeaders().Add('Content-Type', 'application/json');
...
RequestMessage.GetHeaders().Add('Authorization', AuthTokenTxt);

Error Codes: AL0135

8 report-rendering-layout-syntax report-definition 1 CG-AL-M007

Description: The model generated a report with a 'rendering' section that references an RDLC layout file ('SalesPerformanceAnalysis.rdl') that doesn't exist. More critically, the compilation errors at lines 140 and 168 indicate structural syntax issues in the report definition. The AL0104 errors ('Syntax error, } expected') at lines 140 and 168 point to problems with the 'rendering' block syntax. In AL for Business Central, the 'rendering' section with 'layout' keyword may not be supported in the version being compiled, or the syntax used is incorrect. The model should have either used the older 'RDLCLayout' property at the report level, or ensured the rendering block syntax matches the compiler version. Additionally, the referenced .rdl layout file doesn't exist, which would cause further issues. The AL0198 error at line 202 is a cascading error from the earlier syntax failures, where the compiler sees the codeunit definition as unexpected because it couldn't properly parse the report object.

Correct Pattern:
Either use the report-level property 'RDLCLayout = './SalesPerformanceAnalysis.rdl';' (and provide the file), or use 'DefaultLayout = RDLC;' with a proper rendering section compatible with the target BC version. For compilation without an actual layout file, remove the rendering section and the DefaultRenderingLayout property entirely, using 'ProcessingOnly = true;' if no layout is needed.
Incorrect Pattern:
    rendering
    {
        layout(SalesPerformanceLayout)
        {
            Type = RDLC;
            LayoutFile = 'SalesPerformanceAnalysis.rdl';
        }
    }

Error Codes: AL0104

9 code-fence-stripping output-format-compliance 1 CG-AL-H003

Description: The model included '<BEGIN-CODE>' and potentially other markers in its output that were not stripped before being written to the .al file. Line 1 of the file starts with a non-AL token (the '<BEGIN-CODE>' marker or residual text), causing the compiler to expect an application object keyword (table, codeunit, etc.) but instead finding unexpected content. The actual AL code (table and codeunit definitions) is correct, but the output wrapper/markers were included in the file, making it unparseable.

Correct Pattern:
table 70203 "CG Discount Result"
{
    DataClassification = CustomerContent;
    ...
}

codeunit 70202 "CG Temp Table Processor"
{
    Access = Public;
    ...
}
Incorrect Pattern:
<BEGIN-CODE>
table 70203 "CG Discount Result"
{

Error Codes: AL0198

10 cross-join-dataitem-link-constraint query-cross-join-left-outer-join 1 CG-AL-H017

Description: The model made an error on the DataItemLink for the LeftOuterJoin dataitem 'Dimension_Set_Entry'. Error AL0345 states 'The source of a Column or Filter must be a field defined on the table referenced by its parent DataItem'. The issue is that in the DataItemLink clause, the model referenced 'Department."Dimension Code"' and 'Department.Code' — linking to a non-parent dataitem (Department) from a child of Project. In AL queries, a DataItemLink can only reference the direct parent dataitem's fields. Additionally, with CrossJoin, the link semantics are tricky. The correct approach would be to either link Dimension_Set_Entry to its direct parent (Project) or use appropriate fields. Furthermore, the task description says to use CrossJoin for the first dataitem as well, but the model only applied CrossJoin to the second dataitem (Project). The root dataitem cannot have SqlJoinType since it's the first dataitem, so the task description point #1 saying 'Uses SqlJoinType = CrossJoin to create a Cartesian product between two dataitems' refers to the second dataitem having CrossJoin, which the model did correctly. The core error is the DataItemLink referencing a non-parent dataitem's fields.

Correct Pattern:
dataitem(Dimension_Set_Entry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Project."Dimension Code", "Dimension Value Code" = Project.Code;
    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}
Incorrect Pattern:
dataitem(Dimension_Set_Entry; "Dimension Set Entry")
{
    SqlJoinType = LeftOuterJoin;
    DataItemLink = "Dimension Code" = Department."Dimension Code", "Dimension Value Code" = Department.Code;
    column(MatchCount; "Entry No.")
    {
        Method = Count;
    }
}

Error Codes: AL0345

11 reserved-keyword-as-variable-name al-variable-naming 1 CG-AL-H020

Description: The model used 'Key' as a variable name (visible at lines 58 and 102 of the compiled file), but 'Key' is a reserved keyword in AL and cannot be used as an identifier. The generated code shown uses 'DictKey' as the variable name, but the actual compiled file apparently differs - the compilation errors at lines 58 and 102 reference 'Key' being used as a variable name. This suggests the actual file that was compiled used 'Key' as a variable name in the MergeDictionaries and GetKeys procedures, which is invalid because 'Key' is a reserved word in AL. The model should have used a non-reserved identifier like 'DictKey' or 'KeyValue'.

Correct Pattern:
DictKey: Text;
...
foreach DictKey in Dict1.Keys do
Incorrect Pattern:
Key: Text;
...
foreach Key in Dict1.Keys do

Error Codes: AL0519

12 recordref-open-return-type recordref-api 1 CG-AL-H022

Description: The model incorrectly assumed that RecordRef.Open() returns a Boolean that can be used with 'if not'. In AL, RecordRef.Open() is a void procedure (returns nothing). It throws a runtime error if the table ID is invalid. To handle invalid table IDs, the model should use TryFunction or wrap the call in a helper that uses [TryFunction] attribute, or check validity before calling Open. The correct pattern is to use a local [TryFunction] procedure to attempt the Open call and catch any errors.

Correct Pattern:
Use a [TryFunction] helper procedure to safely attempt RecordRef.Open(), e.g.:

[TryFunction]
local procedure TryOpenTable(var RecRef: RecordRef; TableId: Integer)
begin
    RecRef.Open(TableId);
end;

Then call: if not TryOpenTable(RecRef, TableId) then exit('');
Incorrect Pattern:
if not RecRef.Open(TableId) then
    exit('');

Error Codes: AL0173

13 text-split-char-parameter al-text-methods 1 CG-AL-M021

Description: The model attempted to use Text.Split() with a Char variable as the separator. In AL, the Split method expects a Text parameter, not a Char. The line `Lines := YamlString.Split(NewLineChar);` where NewLineChar is declared as Char fails because Split doesn't accept Char directly. The model needed to convert the Char to Text first (e.g., by using Format() or assigning to a Text variable). The compilation error at line 88 about 'Key' not being valid is a cascading error from the Split issue or from the use of `.Trim('"')` which is also not valid AL syntax - the Trim method in AL doesn't accept a character parameter like in C#. The actual root errors are about AL's Text API differences from C#/.NET string methods.

Correct Pattern:
Use Format(NewLineChar) to convert Char to Text for Split, and use DelChr or manual string manipulation instead of Trim with a character parameter. For example:
var Lf: Text[1];
Lf[1] := 10;
Lines := YamlString.Split(Lf);

And instead of ValueText.Trim('"'), use: ValueText := DelChr(ValueText, '=', '"');
Incorrect Pattern:
NewLineChar := 10;
Lines := YamlString.Split(NewLineChar);
...
ValueText := ValueText.Trim('"');

Error Codes: AL0519

14 multiline-string-literals string-operations 1 CG-AL-E050

Description: The task explicitly requires using multiline string literals (a feature available in newer AL versions). The model instead used string concatenation with a helper function NewLine() that returns a backslash character '\' rather than an actual line break. AL supports multiline text constants using the syntax with line breaks directly in the source code, or the model could have at minimum used proper line break characters (e.g., via TypeHelper or character codes 13/10). The model's NewLine() procedure returns a literal backslash, which is not a newline character, causing the multiline tests to fail.

Correct Pattern:
Use proper newline characters. For example:

local procedure NewLine(): Text
var
    LF: Char;
begin
    LF := 10;
    exit(Format(LF));
end;

Or use AL multiline string literals if supported by the runtime version.
Incorrect Pattern:
local procedure NewLine(): Text
    begin
        exit('\');
    end;