← Back to Benchmark Results

gemini/gemini-3-pro-preview

Pass Rate:60.7%
Tasks Passed:34/56
Total Shortcomings:6

All Known Shortcomings

Sorted by occurrence count (most frequent first)

# Concept AL Concept Count Affected Tasks
1 event-subscriber-syntax event-subscription 1 CG-AL-E010

Description: The model generated code that subscribes to 'OnAfterInsert' event, but the correct event name in Business Central for the Item table is 'OnAfterInsertEvent'. The AL compiler error AL0280 indicates the event name is not found because BC table events follow the pattern 'OnAfter<Action>Event' (e.g., OnAfterInsertEvent, OnAfterModifyEvent, OnAfterDeleteEvent), not 'OnAfter<Action>'.

Correct Pattern:
[EventSubscriber(ObjectType::Table, Database::Item, 'OnAfterInsertEvent', '', false, false)]
local procedure OnAfterInsertItem(var Rec: Record Item)
begin
    // Handle the event
    Message('Item %1 was created', Rec."No.");
end;
Incorrect Pattern:
[EventSubscriber(ObjectType::Table, Database::Item, 'OnAfterInsert', '', false, false)]

Error Codes: AL0280

2 json-value-extraction-methods json-handling 1 CG-AL-H014

Description: The model failed to generate any code (empty generated code), and when it did attempt compilation, it incorrectly used JsonValue.AsText() with a boolean parameter instead of the correct pattern. The task requires using typed JSON getter methods like GetText(), GetInteger(), GetBoolean() on JsonObject, but the model doesn't understand the correct AL JSON API patterns. The error 'cannot convert from Text to Boolean' at line 63:33 suggests the model tried to pass a Text value where a Boolean was expected, likely confusing the JsonValue extraction methods.

Correct Pattern:
procedure ParseCustomerData(CustomerJson: JsonObject): Text
var
    NameToken, AgeToken, ActiveToken: JsonToken;
    NameValue, AgeValue, ActiveValue: JsonValue;
    CustomerName: Text;
    CustomerAge: Integer;
    IsActive: Boolean;
begin
    CustomerJson.Get('name', NameToken);
    NameValue := NameToken.AsValue();
    CustomerName := NameValue.AsText();
    
    CustomerJson.Get('age', AgeToken);
    AgeValue := AgeToken.AsValue();
    CustomerAge := AgeValue.AsInteger();
    
    CustomerJson.Get('active', ActiveToken);
    ActiveValue := ActiveToken.AsValue();
    IsActive := ActiveValue.AsBoolean();
    
    exit(StrSubstNo('Name: %1, Age: %2, Active: %3', CustomerName, CustomerAge, IsActive));
end;
Incorrect Pattern:
// Generated code not found - model failed to produce valid AL code

Error Codes: AL0133

3 empty-or-malformed-output code-generation-basics 1 CG-AL-H020

Description: The model failed to generate any valid AL code. The generated code shows 'Generated code not found' and the compilation errors indicate the file contains markdown formatting characters (backticks) and no valid AL object declaration. The model either returned empty output, wrapped the code in markdown code blocks that weren't properly stripped, or completely failed to understand the task. Error AL0198 'Expected one of the application object keywords' at position 1:1 indicates the file doesn't start with a valid AL object type, and the multiple AL0183 'Unexpected character backtick' errors suggest markdown formatting was included in the output.

Correct Pattern:
codeunit 70020 "CG Collection Processor"
{
    procedure JoinTexts(Items: List of [Text]; Separator: Text): Text
    var
        Item: Text;
        Result: Text;
        IsFirst: Boolean;
    begin
        IsFirst := true;
        foreach Item in Items do begin
            if not IsFirst then
                Result += Separator;
            Result += Item;
            IsFirst := false;
        end;
        exit(Result);
    end;
    // ... other procedures
}
Incorrect Pattern:
// Generated code not found (file contains backticks and no valid AL structure)

Error Codes: AL0198

4 variant-type-conversion variant-handling 1 CG-AL-H023

Description: The model incorrectly tried to call AsInteger(), AsDecimal(), and AsBoolean() methods directly on a Variant type. In AL, Variant is a 'Joker' type that doesn't have these conversion methods. To extract typed values from a Variant, you need to assign the Variant to a variable of the target type, or use Format() to get a text representation and then evaluate it.

Correct Pattern:
To convert Variant to specific types in AL, assign directly to typed variable: 'IntValue := VariantValue;' or 'DecValue := VariantValue;' - AL will perform implicit conversion if the Variant contains a compatible value. For explicit checking, use VariantValue.IsInteger, VariantValue.IsDecimal, etc. before assignment.
Incorrect Pattern:
Variant.AsInteger(), Variant.AsDecimal(), Variant.AsBoolean()

Error Codes: AL0132

5 purchase-header-event-names event-subscriber-binding 1 CG-AL-M008

Description: The model attempted to subscribe to events 'OnBeforeRelease' and 'OnAfterReopen' on the Purchase Header table, but these events do not exist on that table. In Business Central, release and reopen operations for Purchase Header are handled through the 'Release Purchase Document' codeunit (Codeunit 415), not directly on the table. The correct events would be found in that codeunit, such as 'OnBeforeReleasePurchaseDoc' and 'OnAfterReopenPurchaseDoc', or the model should use different event patterns available on the Purchase Header table itself.

Correct Pattern:
[EventSubscriber(ObjectType::Codeunit, Codeunit::"Release Purchase Document", 'OnBeforeReleasePurchaseDoc', '', false, false)]
local procedure OnBeforeReleasePurchDoc(var PurchaseHeader: Record "Purchase Header")
begin
    // Handle before release
end;

[EventSubscriber(ObjectType::Codeunit, Codeunit::"Release Purchase Document", 'OnAfterReopenPurchaseDoc', '', false, false)]
local procedure OnAfterReopenPurchDoc(var PurchaseHeader: Record "Purchase Header")
begin
    // Handle after reopen
end;
Incorrect Pattern:
[EventSubscriber(ObjectType::Table, Database::"Purchase Header", 'OnBeforeRelease', '', false, false)]
...
[EventSubscriber(ObjectType::Table, Database::"Purchase Header", 'OnAfterReopen', '', false, false)]

Error Codes: AL0280

6 yaml-parsing-implementation text-parsing-algorithms 1 CG-AL-M021

Description: The model failed to generate valid AL code for a YAML handler codeunit. The compilation errors indicate syntax issues at line 140 with missing 'end', ';', and '}' tokens, suggesting the model produced incomplete or malformed AL code. The task requires implementing custom YAML parsing logic (since AL doesn't have native YAML support), which involves complex string manipulation, line-by-line parsing, and conversion between YAML and JSON formats. The model likely struggled with the complexity of implementing a proper YAML parser in AL and produced syntactically invalid code with unclosed blocks or malformed procedure definitions.

Correct Pattern:
codeunit 70121 "CG YAML Handler"
{
    Access = Public;

    procedure ParseYamlConfig(YamlContent: Text): Text
    var
        JsonObj: JsonObject;
        Lines: List of [Text];
        Line: Text;
        Key: Text;
        Value: Text;
        ColonPos: Integer;
    begin
        // Parse YAML lines into JsonObject
        Lines := YamlContent.Split(GetLf());
        foreach Line in Lines do begin
            ColonPos := Line.IndexOf(':');
            if ColonPos > 0 then begin
                Key := Line.Substring(1, ColonPos - 1).Trim();
                Value := Line.Substring(ColonPos + 1).Trim();
                JsonObj.Add(Key, Value);
            end;
        end;
        // Extract and format result
        exit(StrSubstNo('App: %1 v%2, Debug: %3, Port: %4', ...));
    end;
    // ... other procedures with proper end statements
}
Incorrect Pattern:
// Generated code not found - but errors at line 140 indicate incomplete/malformed code structure with missing end statements and syntax errors

Error Codes: AL0104