locodellm.bench.bench_result#

Result classes for benchmark runs.

class locodellm.bench.bench_result.BenchResult(results: list[PromptTestResult] = <factory>)#

Aggregated results from a BenchPromptTest run.

results#

Individual results for each prompt test.

Type:

list[locodellm.bench.bench_result.PromptTestResult]

property failed: int#

Returns the number of prompt tests with at least one failure.

property passed: int#

Returns the number of prompt tests where all assertions passed.

to_dataframe() → DataFrame#

Exports the results as a pandas DataFrame.

Each row represents one expected result assertion. Columns are:

  • prompt: the prompt text

  • duration: time in seconds spent generating the answer

  • compiled: whether the generated code compiled

  • ran: whether the generated code ran without error

  • input_index: index of the input set

  • passed: whether expected matched actual

Returns:

A pandas.DataFrame with one row per assertion.

to_json() → list[dict[str, Any]]#

Exports the results as a JSON-serializable list.

Each entry contains the prompt, the generated code, and the detailed results for each input set.

Returns:

A list of dictionaries, one per prompt test.

property total: int#

Returns the total number of prompt tests evaluated.

class locodellm.bench.bench_result.PromptTestResult(prompt_test: PromptTest, generated_code: str, run_status: RunStatus, results: list[tuple[~locodellm.bench.prompt_test.ExpectedResult, ~typing.Any, bool]]=<factory>, duration: float = 0.0, token_count: int = 0)#

Result of running a single PromptTest.

prompt_test#

The original prompt test that was evaluated.

Type:

locodellm.bench.prompt_test.PromptTest

generated_code#

The code extracted from the model output.

Type:

str

run_status#

The RunStatus from compiling/running the code with undefined arguments.

Type:

locodellm.bench.run_code.RunStatus

results#

A list of tuples (expected, actual, passed) for each ExpectedResult entry.

Type:

list[tuple[locodellm.bench.prompt_test.ExpectedResult, Any, bool]]

duration#

Time in seconds spent generating the answer.

Type:

float

property all_passed: bool#

Returns True if every expected result matched.

property tokens_per_second: float#

Returns the token generation speed.