locodellm.bench.prompt_test

Defines test cases for evaluating generated code from prompts.

class locodellm.bench.prompt_test.ExpectedResult(args: tuple[Any, ...], expected: Any)

A single expected input/output pair for a generated function.

args

Positional arguments to pass to the function.

Type:

tuple[Any, …]

expected

The expected return value.

Type:

Any

classmethod from_dict(data: dict[str, Any]) → ExpectedResult

Creates an instance from a dictionary.

to_dict() → dict[str, Any]

Converts the instance to a JSON-serializable dictionary.

class locodellm.bench.prompt_test.PromptTest(prompt: str, expected: list[ExpectedResult] = <factory>)

A test case pairing a prompt with expected input/output pairs.

prompt

The natural-language prompt sent to the model to generate a function.

Type:

str

expected

A list of ExpectedResult instances describing how the generated function should behave.

Type:

list[locodellm.bench.prompt_test.ExpectedResult]

classmethod from_json(text: str) → PromptTest

Deserializes a PromptTest from a JSON string.

to_json() → str

Serializes the instance to a JSON string.

locodellm.bench.prompt_test.dump_prompt_tests(tests: list[PromptTest], path: str) → None

Writes a list of PromptTest to a JSON Lines file.

Each line in the output file is a self-contained JSON object representing one PromptTest.

Parameters:

• tests – The list of prompt tests to serialize.

• path – File path to write to.

locodellm.bench.prompt_test.load_prompt_tests(path: str) → list[PromptTest]

Reads a list of PromptTest from a JSON Lines file.

Parameters:

path – File path to read from.

Returns:

The deserialized list of prompt tests.