locodellm.bench#

Benchmarking utilities for generated code.

class locodellm.bench.BenchPromptTest(tests: list[PromptTest], description: str = '', max_length: int = 200)#

Benchmarks a model by running prompt tests and comparing results.

tests#: The list of PromptTest to evaluate.

description#: A short description of the benchmark.

max_length#: Maximum token length for generation.

run(session: Any, verbose: int = 0, json_output: str | None = None, **search_options: Any) → BenchResult#

Runs all prompt tests against the given session.

For each PromptTest, the session is restarted, the prompt is submitted, the generated code is extracted and executed with the specified arguments, and the results are compared to expected values.

Parameters:

session – A SessionState instance.
verbose – Verbosity level. When >= 1, displays a progress bar.
json_output – If set, writes results incrementally to this JSON file path after each prompt test completes.
**search_options – Extra options forwarded to generate().

Returns:

A BenchResult with outcomes for all tests.

class locodellm.bench.BenchResult(results: list[PromptTestResult] = <factory>)#

Aggregated results from a BenchPromptTest run.

results#

Individual results for each prompt test.

Type:: list[locodellm.bench.bench_result.PromptTestResult]

property failed: int#: Returns the number of prompt tests with at least one failure.

property passed: int#: Returns the number of prompt tests where all assertions passed.

to_dataframe() → DataFrame#

Exports the results as a pandas DataFrame.

Each row represents one expected result assertion. Columns are:

prompt: the prompt text
duration: time in seconds spent generating the answer
compiled: whether the generated code compiled
ran: whether the generated code ran without error
input_index: index of the input set
passed: whether expected matched actual

Returns:: A pandas.DataFrame with one row per assertion.

to_json() → list[dict[str, Any]]#

Exports the results as a JSON-serializable list.

Each entry contains the prompt, the generated code, and the detailed results for each input set.

Returns:: A list of dictionaries, one per prompt test.

property total: int#: Returns the total number of prompt tests evaluated.

class locodellm.bench.ExpectedResult(args: tuple[Any, ...], expected: Any)#

A single expected input/output pair for a generated function.

args#

Positional arguments to pass to the function.

Type:: tuple[Any, …]

expected#

The expected return value.

Type:: Any

classmethod from_dict(data: dict[str, Any]) → ExpectedResult#: Creates an instance from a dictionary.

to_dict() → dict[str, Any]#: Converts the instance to a JSON-serializable dictionary.

class locodellm.bench.PromptTest(prompt: str, expected: list[ExpectedResult] = <factory>)#

A test case pairing a prompt with expected input/output pairs.

prompt#

The natural-language prompt sent to the model to generate a function.

Type:: str

expected#

A list of ExpectedResult instances describing how the generated function should behave.

Type:: list[locodellm.bench.prompt_test.ExpectedResult]

classmethod from_json(text: str) → PromptTest#: Deserializes a PromptTest from a JSON string.

to_json() → str#: Serializes the instance to a JSON string.

class locodellm.bench.PromptTestResult(prompt_test: PromptTest, generated_code: str, run_status: RunStatus, results: list[tuple[~locodellm.bench.prompt_test.ExpectedResult, ~typing.Any, bool]]=<factory>, duration: float = 0.0, token_count: int = 0)#

Result of running a single PromptTest.

prompt_test#

The original prompt test that was evaluated.

Type:: locodellm.bench.prompt_test.PromptTest

generated_code#

The code extracted from the model output.

Type:: str

run_status#

The RunStatus from compiling/running the code with undefined arguments.

Type:: locodellm.bench.run_code.RunStatus

results#

A list of tuples (expected, actual, passed) for each ExpectedResult entry.

Type:: list[tuple[locodellm.bench.prompt_test.ExpectedResult, Any, bool]]

duration#

Time in seconds spent generating the answer.

Type:: float

property all_passed: bool#: Returns True if every expected result matched.

property tokens_per_second: float#: Returns the token generation speed.

class locodellm.bench.RunStatus(compiled: bool = False, compile_error: BaseException | None = None, ran: bool = False, run_error: BaseException | None = None, result: Any = None, _function: Any = None)#

Result of compiling and running a Python function string.

compiled#

Whether the code compiled successfully.

Type:: bool

compile_error#

The exception raised during compilation, if any.

Type:: BaseException | None

ran#

Whether the function executed without error.

Type:: bool

run_error#

The exception raised during execution, if any.

Type:: BaseException | None

result#

The return value of the function, or None if it did not run or raised.

Type:: Any

property success: bool#: Returns True if the code both compiled and ran without error.

locodellm.bench.dump_prompt_tests(tests: list[PromptTest], path: str) → None#

Writes a list of PromptTest to a JSON Lines file.

Each line in the output file is a self-contained JSON object representing one PromptTest.

Parameters:

tests – The list of prompt tests to serialize.
path – File path to write to.

locodellm.bench.get_available_benchmarks() → dict[str, str]#

Returns a dictionary of available benchmark names and descriptions.

Returns:: A mapping from benchmark name to a short description.

locodellm.bench.load_prompt_tests(path: str) → list[PromptTest]#

Reads a list of PromptTest from a JSON Lines file.

Parameters:: path – File path to read from.
Returns:: The deserialized list of prompt tests.

locodellm.bench.run_function(source: str) → RunStatus#

Compiles and executes a Python function defined in source.

The function is called with UNDEFINED for each of its positional parameters and as the default for keyword arguments.

Parameters:: source – Python source code defining exactly one function.
Returns:: A RunStatus describing the outcome.