Note
Go to the end to download the full example code.
Run a benchmark on a local LLM#
This example loads a model and runs the basic benchmark against it.
Each prompt in the benchmark asks the model to write a Python function.
The generated code is compiled, executed with test inputs, and compared
to the expected results.
The model, precision, and execution provider can be changed from the command line:
python docs/examples/plot_bench.py \
--model Qwen/Qwen2.5-Coder-0.5B-Instruct --precision fp32 --provider cpu
The equivalent CLI command is:
python -m locodellm bench Qwen/Qwen2.5-Coder-0.5B-Instruct basic \
--chat-template chatml --precision fp32 --verbose 1
Configuration#
Default values can be overridden with --model, --precision, and
--provider when running the script directly.
Under UNITTEST_GOING=1 (used during CI and documentation builds),
the mock model is used instead of a real HuggingFace model.
import argparse
import os
import sys
UNITTEST_GOING = os.environ.get("UNITTEST_GOING") == "1"
if "__file__" in dir():
_project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
if _project_root not in sys.path:
sys.path.insert(0, _project_root)
_defaults = dict(
model="Qwen/Qwen2.5-Coder-0.5B-Instruct",
precision="fp32",
provider="cpu",
verbose=1,
chat_template="chatml",
)
if "__file__" in dir():
_parser = argparse.ArgumentParser(description="Run a benchmark on a local LLM.")
_parser.add_argument("--model", default=_defaults["model"], help="HuggingFace model id.")
_parser.add_argument(
"--precision",
default=_defaults["precision"],
help="Conversion precision (fp32, fp16, int4).",
)
_parser.add_argument(
"--provider", default=_defaults["provider"], help="Execution provider (cpu, cuda)."
)
_parser.add_argument(
"--verbose", type=int, default=_defaults["verbose"], help="Verbosity level (0=silent)."
)
_parser.add_argument(
"--chat-template",
default=_defaults["chat_template"],
help="Chat template (chatml, or empty for none).",
)
_args = _parser.parse_args()
MODEL_ID = _args.model
PRECISION = _args.precision
PROVIDER = _args.provider
VERBOSE = _args.verbose
CHAT_TEMPLATE = _args.chat_template or None
else:
MODEL_ID = _defaults["model"]
PRECISION = _defaults["precision"]
PROVIDER = _defaults["provider"]
VERBOSE = _defaults["verbose"]
CHAT_TEMPLATE = _defaults["chat_template"]
print(
f"MODEL_ID={MODEL_ID}, PRECISION={PRECISION}, PROVIDER={PROVIDER}, "
f"VERBOSE={VERBOSE}, CHAT_TEMPLATE={CHAT_TEMPLATE}"
)
MODEL_ID=Qwen/Qwen2.5-Coder-0.5B-Instruct, PRECISION=fp32, PROVIDER=cpu, VERBOSE=1, CHAT_TEMPLATE=chatml
Load the model#
We use get_session() to
download, convert, and load the model. Under UNITTEST_GOING=1, the
mock model mock/generate is used instead.
from locodellm.generate.generate_from_model import get_session # noqa: E402
if UNITTEST_GOING:
MODEL_ID = "mock/generate"
session = get_session(
model_id=MODEL_ID,
precision=PRECISION,
chat_template=CHAT_TEMPLATE,
verbose=max(VERBOSE - 1, 0),
)
print(f"Model loaded: {MODEL_ID}")
Model loaded: mock/generate
Run the benchmark#
We load the basic benchmark and run it against the session.
The benchmark contains 10 Python function prompts with growing
difficulty.
from locodellm.bench import load_benchmark # noqa: E402
benchmark = load_benchmark("basic")
print(f"Benchmark: {benchmark.description}")
print(f"Number of prompts: {len(benchmark.tests)}")
result = benchmark.run(session, verbose=VERBOSE)
Benchmark: 10 Python function prompts with growing difficulty, from returning a constant string to computing an edit distance.
Number of prompts: 10
[███░░░░░░░░░░░░░░░░░░░░░░░░░░░] 1/10
[██████░░░░░░░░░░░░░░░░░░░░░░░░] 2/10
[█████████░░░░░░░░░░░░░░░░░░░░░] 3/10
[████████████░░░░░░░░░░░░░░░░░░] 4/10
[███████████████░░░░░░░░░░░░░░░] 5/10
[██████████████████░░░░░░░░░░░░] 6/10
[█████████████████████░░░░░░░░░] 7/10
[████████████████████████░░░░░░] 8/10
[███████████████████████████░░░] 9/10
[██████████████████████████████] 10/10
Results table#
Each row shows one input set for a prompt: whether the generated code compiled, ran, and produced the expected result.
| prompt | duration | token_count | tokens_per_second | compiled | ran | input_index | passed |
|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------:|--------------:|--------------------:|:-----------|:------|--------------:|:---------|
| write a python function called hello that returns the string "hello" | 0.0142096 | 35 | 2463.12 | True | True | 0 | True |
| write a python function called hello that returns the string "hello" | 0.0142096 | 35 | 2463.12 | True | True | 1 | True |
| write a python function called add that takes two numbers and returns their sum | 0.0133592 | 34 | 2545.07 | True | True | 0 | False |
| write a python function called add that takes two numbers and returns their sum | 0.0133592 | 34 | 2545.07 | True | True | 1 | False |
| write a python function called add that takes two numbers and returns their sum | 0.0133592 | 34 | 2545.07 | True | True | 2 | False |
| write a python function called reverse_string that takes a string and returns it reversed | 0.0129969 | 33 | 2539.06 | True | True | 0 | False |
| write a python function called reverse_string that takes a string and returns it reversed | 0.0129969 | 33 | 2539.06 | True | True | 1 | False |
| write a python function called reverse_string that takes a string and returns it reversed | 0.0129969 | 33 | 2539.06 | True | True | 2 | False |
| write a python function called find_max that takes a list of numbers and returns the maximum value | 0.0119982 | 30 | 2500.37 | True | True | 0 | False |
| write a python function called find_max that takes a list of numbers and returns the maximum value | 0.0119982 | 30 | 2500.37 | True | True | 1 | False |
| write a python function called find_max that takes a list of numbers and returns the maximum value | 0.0119982 | 30 | 2500.37 | True | True | 2 | False |
| write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 0 | False |
| write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 1 | False |
| write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 2 | False |
| write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 3 | False |
| write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 0 | False |
| write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 1 | False |
| write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 2 | False |
| write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 3 | False |
| write a python function called char_count that takes a string and returns a dictionary mapping each character to its count | 0.0109835 | 27 | 2458.23 | False | False | 0 | False |
| write a python function called char_count that takes a string and returns a dictionary mapping each character to its count | 0.0109835 | 27 | 2458.23 | False | False | 1 | False |
| write a python function called char_count that takes a string and returns a dictionary mapping each character to its count | 0.0109835 | 27 | 2458.23 | False | False | 2 | False |
| write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 0 | False |
| write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 1 | False |
| write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 2 | False |
| write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 3 | False |
| write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 0 | False |
| write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 1 | False |
| write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 2 | False |
| write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 3 | False |
| write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 0 | False |
| write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 1 | False |
| write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 2 | False |
| write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 3 | False |
Per-case statistics#
One row per prompt showing how many input sets passed.
stats = df.groupby("prompt", sort=False)
rows = []
for prompt, group in stats:
total = len(group)
passed = int(group["passed"].sum())
rows.append(
{
"prompt": prompt[:60] + "..." if len(prompt) > 60 else prompt,
"duration": round(float(group["duration"].iloc[0]), 3),
"tokens_per_second": round(float(group["tokens_per_second"].iloc[0]), 1),
"compiled": bool(group["compiled"].iloc[0]),
"ran": bool(group["ran"].iloc[0]),
"passed": f"{passed}/{total}",
"score": round(passed / total, 2) if total > 0 else 0.0,
}
)
import pandas # noqa: E402
stats_df = pandas.DataFrame(rows)
print(stats_df.to_markdown(index=False))
| prompt | duration | tokens_per_second | compiled | ran | passed | score |
|:----------------------------------------------------------------|-----------:|--------------------:|:-----------|:------|:---------|--------:|
| write a python function called hello that returns the string... | 0.014 | 2463.1 | True | True | 2/2 | 1 |
| write a python function called add that takes two numbers an... | 0.013 | 2545.1 | True | True | 0/3 | 0 |
| write a python function called reverse_string that takes a s... | 0.013 | 2539.1 | True | True | 0/3 | 0 |
| write a python function called find_max that takes a list of... | 0.012 | 2500.4 | True | True | 0/3 | 0 |
| write a python function called is_prime that takes an intege... | 0.011 | 2436 | False | False | 0/4 | 0 |
| write a python function called factorial that takes a non-ne... | 0.012 | 2520.3 | True | True | 0/4 | 0 |
| write a python function called char_count that takes a strin... | 0.011 | 2458.2 | False | False | 0/3 | 0 |
| write a python function called is_palindrome that takes a st... | 0.009 | 2303.4 | False | False | 0/4 | 0 |
| write a python function called fibonacci that takes a non-ne... | 0.008 | 2205.1 | False | False | 0/4 | 0 |
| write a python function called edit_distance that takes two ... | 0.007 | 1255.2 | False | False | 0/4 | 0 |
Summary#
Overall benchmark score.
Total prompts: 10
Passed (all inputs correct): 1/10
Failed: 9/10
Overall score: 10%
Total running time of the script: (0 minutes 1.083 seconds)