← Back to Benchmark Results

openai/gpt-5.2-2025-12-11

Pass Rate:51.8%
Tasks Passed:29/56
Total Shortcomings:22

All Known Shortcomings

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 json-typed-getter-methods json-handling 4 CG-AL-H014, CG-AL-M020, CG-AL-M021, CG-AL-M005

Description: The model failed to generate any code (generated code not found), and the compilation error AL0133 indicates that when attempting to use JSON getter methods, the model incorrectly passed a Text argument where a Boolean was expected. This suggests the model doesn't understand the correct signatures for AL's typed JSON getter methods like AsText(), AsInteger(), AsBoolean() on JsonValue, or the Get() method patterns for JsonObject. The task requires using methods like GetText('name'), GetInteger('age'), GetBoolean('active') which don't exist as direct methods on JsonObject - instead you must use Get() to retrieve a JsonToken, then convert it appropriately.

Correct Pattern:
procedure ParseCustomerData(CustomerJson: JsonObject): Text
var
    JToken: JsonToken;
    JValue: JsonValue;
    Name: Text;
    Age: Integer;
    Active: Boolean;
begin
    if CustomerJson.Get('name', JToken) then begin
        JValue := JToken.AsValue();
        Name := JValue.AsText();
    end;
    if CustomerJson.Get('age', JToken) then begin
        JValue := JToken.AsValue();
        Age := JValue.AsInteger();
    end;
    if CustomerJson.Get('active', JToken) then begin
        JValue := JToken.AsValue();
        Active := JValue.AsBoolean();
    end;
    exit(StrSubstNo('Name: %1, Age: %2, Active: %3', Name, Age, Active));
end;
Incorrect Pattern:
// Generated code not found - but error at line 38:35 suggests incorrect parameter type usage

Error Codes: AL0133, AL0132, AL0134

2 table-field-property-context table-definition 2 CG-AL-H003, CG-AL-H005

Description: The model generated code that uses the 'Caption' property in an invalid context. The AL0124 error indicates that the Caption property was placed somewhere it cannot be used - likely on a field or object where it's not allowed, or with incorrect syntax. The model failed to generate valid AL code for the table definition, specifically around field property declarations.

Correct Pattern:
In AL table definitions, Caption property should be declared as: Caption = 'Field Caption'; within the field definition block. For example:
field(1; "Line No."; Integer)
{
    Caption = 'Line No.';
}
The Caption property must be inside the field's curly braces, not outside or in other contexts.
Incorrect Pattern:
Unable to see exact generated code, but error at line 47 indicates Caption property used incorrectly in table/field definition

Error Codes: AL0124

3 query-crossjoin-syntax query-definition 2 CG-AL-H017, CG-AL-H011

Description: The model generated AL code with syntax errors around lines 27-28, likely related to the CrossJoin query structure. The errors indicate missing commas, semicolons, and braces, suggesting the model doesn't properly understand the AL query syntax for CrossJoin dataitems or column definitions. CrossJoin queries in AL require specific syntax for linking dataitems without a DataItemLink property, and the model appears to have malformed the dataitem or column declarations.

Correct Pattern:
query 70017 "CG Dimension Matrix"
{
    QueryType = Normal;

    elements
    {
        dataitem(DepartmentDim; "Dimension Value")
        {
            SqlJoinType = CrossJoin;
            DataItemTableFilter = "Dimension Code" = const('DEPARTMENT');
            column(DepartmentCode; Code) { }
            column(DepartmentName; Name) { }

            dataitem(ProjectDim; "Dimension Value")
            {
                SqlJoinType = CrossJoin;
                DataItemTableFilter = "Dimension Code" = const('PROJECT');
                column(ProjectCode; Code) { }
                column(ProjectName; Name) { }

                dataitem(DimSetEntry; "Dimension Set Entry")
                {
                    SqlJoinType = LeftOuterJoin;
                    column(MatchCount; "Dimension Set ID")
                    {
                        Method = Count;
                    }
                }
            }
        }
    }
}
Incorrect Pattern:
Code around lines 27-28 has syntax errors - exact code not available but errors indicate malformed dataitem/column declarations in a CrossJoin query structure

Error Codes: AL0104, AL0107

4 report-syntax-structure report-definition 2 CG-AL-M007, CG-AL-E007

Description: The model generated a complex report with syntax errors - missing closing braces at multiple points in the report structure. The compilation errors show AL0104 (Syntax error, '}' expected) at lines 85 and 101, and AL0198 at line 122 indicating the parser expected a new object declaration but found something else. This indicates the model failed to properly close nested structures within the report definition (likely dataitem blocks, column definitions, or trigger blocks). The task required a complex report with multiple data items, triggers, and request page - the model appears to have lost track of the nesting structure.

Correct Pattern:
report 70001 "Sales Performance Analysis"
{
    UsageCategory = ReportsAndAnalysis;
    ApplicationArea = All;
    
    dataset
    {
        dataitem(Customer; Customer)
        {
            column(CustomerNo; "No.") { }
            
            dataitem(SalesHeader; "Sales Header")
            {
                DataItemLink = "Sell-to Customer No." = field("No.");
                
                trigger OnAfterGetRecord()
                begin
                    // calculations
                end;
            } // properly closed
        } // properly closed
    } // properly closed
    
    requestpage
    {
        layout
        {
            area(Content)
            {
                // fields
            }
        }
    }
} // report closed
Incorrect Pattern:
// Generated code not found - but errors indicate missing closing braces at lines 85, 101, and malformed structure at line 122

Error Codes: AL0104, Report 70000 does not have a valid layout. Valid layout types are: Word, RDLC, Excel, and Custom.

5 empty-or-missing-code-generation interface-definition 2 CG-AL-M009, CG-AL-E008

Description: The model failed to generate any code at all. The generated code shows '// Generated code not found' which means the model either returned empty output, returned only comments, or failed to produce valid AL code. The task required creating an interface 'Shipping Provider' and a codeunit 'Standard Shipping Provider' (ID 70004) that implements it. The compilation errors (AL0107, AL0104, AL0198) at position 1:11 indicate the file starts with invalid content - likely empty or malformed output that doesn't begin with a valid AL object keyword.

Correct Pattern:
interface "Shipping Provider"
{
    procedure CalculateShippingCost(Weight: Decimal; FromCountry: Text; ToCountry: Text): Decimal;
    procedure EstimateDeliveryTime(FromCountry: Text; ToCountry: Text; ServiceType: Text): Integer;
    procedure CreateShipment(OrderNumber: Text; FromAddress: Text; ToAddress: Text; Weight: Decimal): Text[50];
    procedure TrackShipment(TrackingNumber: Text): Text[100];
    procedure ValidateAddress(Street: Text; City: Text; State: Text; ZipCode: Text; Country: Text): Boolean;
}

codeunit 70004 "Standard Shipping Provider" implements "Shipping Provider"
{
    // Implementation of all interface procedures
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0107

6 fluent-api-self-reference codeunit-self-reference 1 CG-AL-H018

Description: The model attempted to use 'Self' keyword to return the current codeunit instance for fluent API chaining, but AL does not have a 'Self' keyword. In AL, to implement a fluent API pattern where methods return the current codeunit instance, you must declare a local variable of the codeunit type and use 'CurrCodeunit' or pass the codeunit by VAR parameter. The correct pattern is to declare a variable like 'var Builder: Codeunit "CG Request Builder"' and then use 'Builder := this;' or simply 'exit(this);' in newer AL versions, or use the Rec pattern with 'exit(Rec);'.

Correct Pattern:
In AL, to return the current codeunit instance for fluent chaining, you should use: 'var ReturnValue: Codeunit "CG Request Builder"; begin ... exit(ReturnValue); end;' where ReturnValue is assigned to the current instance, or in modern AL you can use 'exit(this);' if supported by the runtime version.
Incorrect Pattern:
exit(Self);

Error Codes: AL0118

7 code-generation-failure basic-codeunit-generation 1 CG-AL-H022

Description: The model failed to generate valid AL code for the codeunit. The compilation errors show 'Semicolon expected' at multiple locations (lines 14, 32, 104), indicating the model produced syntactically invalid AL code. The generated code section shows 'Generated code not found', suggesting the model either produced no code or produced code with fundamental structural issues that couldn't even be captured. The task and test are valid - they test RecordRef and FieldRef usage which are standard AL patterns. The model simply failed to produce syntactically correct AL code.

Correct Pattern:
codeunit 70224 "CG Dynamic Record Handler"
{
    Access = Public;

    procedure GetTableName(TableId: Integer): Text
    var
        RecRef: RecordRef;
    begin
        if not RecRef.Open(TableId, false) then
            exit('');
        exit(RecRef.Name);
    end;
    // ... additional procedures
}
Incorrect Pattern:
// Generated code not found - model produced invalid or no code

Error Codes: AL0111

8 empty-file-generation code-generation-basics 1 CG-AL-H023

Description: The model failed to generate any code at all. The generated code shows '// Generated code not found' which means the model produced an empty or invalid response. The compilation errors (AL0107, AL0104, AL0198) all point to line 1:11 with 'identifier expected' and 'application object keywords expected', indicating the file is essentially empty or contains only whitespace/comments. This is a complex task requiring a codeunit with 10 procedures plus an interface, and the model completely failed to produce any valid AL code.

Correct Pattern:
codeunit 70226 "CG Record Introspector"
{
    Access = Public;

    procedure SerializeToJson(SourceRecord: Variant): JsonObject
    var
        RecRef: RecordRef;
        FldRef: FieldRef;
        JsonResult: JsonObject;
        MetadataObj: JsonObject;
        i: Integer;
    begin
        RecRef.GetTable(SourceRecord);
        // ... implementation
    end;

    // ... other procedures
}

interface "IFieldTransformer"
{
    procedure Transform(var FieldRef: FieldRef): Variant;
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0107

9 workflow-record-change-fields bc-standard-table-schema 1 CG-AL-M008

Description: The model generated code that references incorrect field names on the 'Workflow - Record Change' table and uses incorrect enum values. The errors show: 1) Using 'Document Type'::"Purchase Header" which is not a valid enum value (should be Order, Invoice, etc.), 2) Passing Text arguments where Boolean is expected in function calls, 3) Referencing non-existent fields on 'Workflow - Record Change' table like 'Table ID', 'User ID', 'Date and Time', and 'Description'. The model lacks knowledge of the actual schema of BC standard workflow tables and proper enum usage.

Correct Pattern:
The 'Workflow - Record Change' table has different field names. The model should use the actual field names from the BC schema such as 'Workflow Step Instance ID', 'Record ID', 'Field No.', 'Old Value', 'New Value', etc. For Document Type enum, use valid values like 'Document Type'::Order, not 'Document Type'::"Purchase Header".
Incorrect Pattern:
WorkflowRecordChange."Table ID" := ...; WorkflowRecordChange."User ID" := ...; WorkflowRecordChange."Date and Time" := ...; WorkflowRecordChange."Description" := ...

Error Codes: AL0132

10 code-generation-failure codeunit-definition 1 CG-AL-E005

Description: The model failed to generate any code at all. The generated code shows 'Generated code not found' which means the model did not produce the required AL codeunit. The compilation error AL0104 'Syntax error, ')' expected' at line 72 indicates the model produced malformed or incomplete AL code that couldn't even be parsed. The task and test are valid - they simply require a Text Utilities codeunit with three procedures (CapitalizeFirstLetter, CountWords, IsValidEmail), but the model failed to generate proper AL syntax.

Correct Pattern:
codeunit 70000 "Text Utilities"
{
    Access = Public;

    procedure CapitalizeFirstLetter(InputText: Text): Text
    begin
        if InputText = '' then
            exit('');
        exit(UpperCase(CopyStr(InputText, 1, 1)) + CopyStr(InputText, 2));
    end;

    procedure CountWords(InputText: Text): Integer
    var
        WordCount: Integer;
        InWord: Boolean;
        i: Integer;
    begin
        if InputText = '' then
            exit(0);
        for i := 1 to StrLen(InputText) do begin
            if InputText[i] <> ' ' then begin
                if not InWord then begin
                    WordCount += 1;
                    InWord := true;
                end;
            end else
                InWord := false;
        end;
        exit(WordCount);
    end;

    procedure IsValidEmail(EmailAddress: Text): Boolean
    begin
        exit((StrPos(EmailAddress, '@') > 0) and (StrPos(EmailAddress, '.') > 0));
    end;
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0104

11 test-codeunit-assert-reference test-codeunit-structure 1 CG-AL-H002

Description: The model generated code that includes test procedures using 'Assert' but failed to properly reference the Assert codeunit. In AL test codeunits, the Assert codeunit must be declared as a variable (typically 'Assert: Codeunit Assert;') to use assertion methods like Assert.AreEqual. The generated code appears to have test procedures that call Assert methods without the proper codeunit variable declaration, resulting in AL0118 'The name Assert does not exist in the current context' errors.

Correct Pattern:
var
    Assert: Codeunit Assert;

[Test]
procedure TestSomething()
begin
    Assert.AreEqual(expected, actual, 'Error message');
end;
Incorrect Pattern:
The model generated test code that calls Assert.AreEqual() without declaring the Assert codeunit variable

Error Codes: AL0118

12 secrettext-unwrap-scope SecretText-handling 1 CG-AL-H016

Description: The model used the SecretText.Unwrap() method which has scope 'OnPrem' and cannot be used in Extension development (SaaS/Cloud). For cloud-compatible extensions, SecretText values must be handled differently - they can be passed to system APIs that accept SecretText directly, or converted using specific patterns that don't require Unwrap(). The task requires working with SecretText in a way that's compatible with Extension development scope.

Correct Pattern:
For Extension development, instead of using Unwrap(), use Format() function on SecretText which returns the unwrapped value as Text in cloud-compatible way: Format(ApiKey) instead of ApiKey.Unwrap(). Alternatively, use SecretText with APIs that natively accept SecretText parameters without unwrapping.
Incorrect Pattern:
ApiKey.Unwrap() // Used in BuildAuthHeader, ValidateCredentials, and MaskSecret procedures

Error Codes: AL0296

13 list-dictionary-clear-method collection-types 1 CG-AL-H021

Description: The model attempted to use a 'Clear' method on List and Dictionary types containing interfaces. In AL, List and Dictionary types do not have a 'Clear' method. To clear these collections, you need to reinitialize them by assigning a new empty collection or use other approaches like iterating and removing items. The model incorrectly assumed these collection types have a Clear() method similar to other programming languages.

Correct Pattern:
To clear a List or Dictionary in AL, you should reinitialize the variable. For example:

var
    ChannelList: List of [Interface "INotificationChannel"];
    ChannelDict: Dictionary of [Text, Interface "INotificationChannel"];
begin
    // To clear, simply declare new empty collections or use Clear() on the variable itself
    Clear(ChannelList);
    Clear(ChannelDict);
    // Or reinitialize:
    // ChannelList := List of [Interface "INotificationChannel"];
end;
Incorrect Pattern:
ChannelList.Clear();
ChannelDict.Clear();

Error Codes: AL0132

14 json-object-add-overload-resolution json-manipulation 1 CG-AL-H023

Description: The model generated code that uses JsonObject.Add() with JsonValue parameters in a way that creates ambiguous method calls. In AL, when adding values to a JsonObject, you need to explicitly convert values to avoid ambiguity between Add(Text, JsonToken) and Add(Text, JsonObject) overloads. The model also incorrectly used FieldType.Enum (which doesn't exist), JsonValue.IsNumber, JsonValue.IsString, and JsonValue.IsBoolean methods that don't exist in AL's JSON API.

Correct Pattern:
// For adding to JsonObject, use explicit JsonValue creation:
var JValue: JsonValue;
JValue.SetValue(SomeValue);
JsonObject.Add('key', JValue);

// For field type checking, use valid FieldType values like FieldType::Option
// For JSON value type checking, use TryAs* methods:
var DecVal: Decimal;
if JsonValue.AsValue().TryGetValue(DecVal) then // Check if it's a number
Incorrect Pattern:
JsonObject.Add('key', SomeJsonValue); // Ambiguous call
if FieldRef.Type = FieldType::Enum then // Enum not valid
if JsonValue.IsNumber() then // Method doesn't exist

Error Codes: AL0196

15 padstr-function-signature built-in-function-usage 1 CG-AL-M003

Description: The model generated code using PadStr with 4 arguments, but the AL built-in function PadStr only accepts 2 or 3 arguments: PadStr(Text, Integer, [Text]). The model likely tried to use a pattern like PadStr(Text, Integer, Text, Boolean) or similar 4-argument variant that doesn't exist in AL. This is typically used when generating auto-incrementing contract numbers, where the model attempted to pad a number string but used incorrect function syntax.

Correct Pattern:
PadStr(Text, Integer) or PadStr(Text, Integer, Text) - for example: PadStr('', 20 - StrLen(Format(NextNo)), '0') + Format(NextNo) to create a zero-padded number
Incorrect Pattern:
PadStr(..., ..., ..., ...) // 4 arguments at line 126:21

Error Codes: AL0126

16 table-extension-field-definition table-extension 1 CG-AL-M006

Description: The model failed to generate any code at all ('Generated code not found'). The task required creating a table extension with ID 70001 that extends the Customer table with fields including 'Preferred Payment Method' (Code[10] with TableRelation to Payment Method table). The compilation errors show that the test file references fields like 'Preferred Payment Method' that don't exist because the model didn't produce the required table extension code. This is a complete failure to generate the requested AL object.

Correct Pattern:
tableextension 70001 "Advanced Customer Extension" extends Customer
{
    fields
    {
        field(50000; "Credit Score"; Integer)
        {
            DataClassification = CustomerContent;
            MinValue = 300;
            MaxValue = 850;
            trigger OnValidate()
            begin
                if ("Credit Score" < 300) or ("Credit Score" > 850) then
                    Error('Credit Score must be between 300 and 850');
            end;
        }
        field(50001; "Risk Level"; Option)
        {
            OptionMembers = Low,Medium,High,Critical;
        }
        field(50002; "Last Risk Assessment Date"; Date) { }
        field(50003; "Payment History Rating"; Decimal) { }
        field(50004; "Preferred Payment Method"; Code[10])
        {
            TableRelation = "Payment Method";
        }
    }
    procedure UpdateRiskLevel() ... 
    procedure CalculatePaymentHistoryRating(): Decimal ...
    procedure GetCreditLimit(): Decimal ...
    procedure ValidateNewOrder(OrderAmount: Decimal): Boolean ...
    procedure TriggerRiskAssessment() ...
}
Incorrect Pattern:
// Generated code not found

Error Codes: AL0118

17 purchase-header-status-enum-values enum-field-values 1 CG-AL-M008

Description: The model attempted to use 'Pending Approval' as a status value for Purchase Header, but this value does not exist in the standard Purchase Header Status enum. The compilation error AL0118 indicates that the name 'Pending Approval' does not exist in the current context. The model lacks knowledge of the actual enum values available for Purchase Header status fields in Business Central.

Correct Pattern:
The Purchase Header Status field uses enum values like 'Open', 'Released', 'Pending Prepayment'. For approval workflow status tracking, the model should either use the standard 'Approval Status' field which has values like 'Open', 'Pending Approval', 'Approved', or create custom tracking using a separate table/field. Example: PurchaseHeader.Status := PurchaseHeader.Status::Open; or use the Approval Entry table for workflow tracking.
Incorrect Pattern:
Status := "Pending Approval"

Error Codes: AL0118

18 onvalidate-trigger-placement table-field-triggers 1 CG-AL-M010

Description: The model generated code with an 'OnValidate' trigger in an invalid location. In AL, OnValidate is a field-level trigger that must be placed inside a field definition block, not at the table level or elsewhere. The error 'OnValidate is not a valid trigger' indicates the model placed this trigger outside of a field definition context. The generated code file was not captured but the compilation error clearly shows the model misunderstood where OnValidate triggers can be declared in AL table objects.

Correct Pattern:
field(1; "Project Code"; Code[20])
{
    trigger OnValidate()
    begin
        // validation logic here
    end;
}
Incorrect Pattern:
OnValidate trigger placed at line 197 outside of a field definition (exact code not captured but error indicates invalid placement)

Error Codes: AL0162

19 dictionary-clear-method dictionary-operations 1 CG-AL-M020

Description: The model generated code that attempts to call a 'Clear' method on a Dictionary of [Text, Text] type. In AL, the Dictionary type does not have a Clear method. To clear a dictionary, you need to reassign it to a new empty dictionary or use other approaches. The model lacks knowledge of the available methods on the AL Dictionary type.

Correct Pattern:
To clear a dictionary in AL, reassign it: Result := Dictionary of [Text, Text]; or simply declare a new local variable. Alternatively, iterate and remove keys individually, though reassignment is cleaner.
Incorrect Pattern:
Dictionary.Clear()

Error Codes: AL0132

20 flowfield-calcfields-requirement flowfield 1 CG-AL-E009

Description: The test expects the XMLport to export the Inventory field value (50) in the XML output. However, Item.Inventory is a FlowField in Business Central that requires CalcFields to be called before its value can be read. The model's XMLport implementation likely exports the Inventory field directly without calling CalcFields, resulting in the field value being 0 or empty instead of the expected 50. The test sets Item.Inventory := 50 but this doesn't actually set the FlowField value - FlowFields are calculated from related records (Item Ledger Entries). The XMLport needs to either use a trigger to call CalcFields before exporting, or the test setup is incorrect for testing FlowField export.

Correct Pattern:
In the XMLport, add an OnBeforeExportRecord trigger that calls Item.CalcFields(Inventory) before exporting, OR use a different approach such as exporting from a query or using a temporary table with the calculated value. Example: trigger OnBeforeExportRecord() begin Rec.CalcFields(Inventory); end;
Incorrect Pattern:
XMLport exports Inventory field directly without CalcFields
21 ondelete-trigger-error-handling table-triggers 1 CG-AL-E031

Description: The model failed to implement the OnDelete trigger correctly. The test expects that when deleting an active subscription plan, an error should be raised with the exact text 'Cannot delete active subscription plan'. The test uses 'asserterror Plan.Delete()' which expects an error to be thrown, but the error 'An error was expected inside an ASSERTERROR statement' indicates that no error was raised during the Delete operation. This means the OnDelete trigger either wasn't implemented or doesn't properly check the Active field and raise an error when Active is true.

Correct Pattern:
trigger OnDelete()
begin
    if Rec.Active then
        Error('Cannot delete active subscription plan');
end;
Incorrect Pattern:
// OnDelete trigger implementation missing or incorrect - likely missing the Active check and Error statement
22 parse-failure unknown 1 CG-AL-M112

Description: Failed to parse LLM analysis response: {"outcome":"test_logic_bug","category":"test_logic_bug","description":"The test fails because the CG Time Entry table has AutoIncrement = true on the Entry No. field, but the test doesn't account for

Correct Pattern:
Incorrect Pattern: