.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/plot_bench.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_plot_bench.py: Run a benchmark on a local LLM =============================== This example loads a model and runs the ``basic`` benchmark against it. Each prompt in the benchmark asks the model to write a Python function. The generated code is compiled, executed with test inputs, and compared to the expected results. The model, precision, and execution provider can be changed from the command line:: python docs/examples/plot_bench.py \ --model Qwen/Qwen2.5-Coder-0.5B-Instruct --precision fp32 --provider cpu The equivalent CLI command is:: python -m locodellm bench Qwen/Qwen2.5-Coder-0.5B-Instruct basic \ --chat-template chatml --precision fp32 --verbose 1 .. GENERATED FROM PYTHON SOURCE LINES 23-31 Configuration ------------- Default values can be overridden with ``--model``, ``--precision``, and ``--provider`` when running the script directly. Under ``UNITTEST_GOING=1`` (used during CI and documentation builds), the mock model is used instead of a real HuggingFace model. .. GENERATED FROM PYTHON SOURCE LINES 31-88 .. code-block:: Python import argparse import os import sys UNITTEST_GOING = os.environ.get("UNITTEST_GOING") == "1" if "__file__" in dir(): _project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) if _project_root not in sys.path: sys.path.insert(0, _project_root) _defaults = dict( model="Qwen/Qwen2.5-Coder-0.5B-Instruct", precision="fp32", provider="cpu", verbose=1, chat_template="chatml", ) if "__file__" in dir(): _parser = argparse.ArgumentParser(description="Run a benchmark on a local LLM.") _parser.add_argument("--model", default=_defaults["model"], help="HuggingFace model id.") _parser.add_argument( "--precision", default=_defaults["precision"], help="Conversion precision (fp32, fp16, int4).", ) _parser.add_argument( "--provider", default=_defaults["provider"], help="Execution provider (cpu, cuda)." ) _parser.add_argument( "--verbose", type=int, default=_defaults["verbose"], help="Verbosity level (0=silent)." ) _parser.add_argument( "--chat-template", default=_defaults["chat_template"], help="Chat template (chatml, or empty for none).", ) _args = _parser.parse_args() MODEL_ID = _args.model PRECISION = _args.precision PROVIDER = _args.provider VERBOSE = _args.verbose CHAT_TEMPLATE = _args.chat_template or None else: MODEL_ID = _defaults["model"] PRECISION = _defaults["precision"] PROVIDER = _defaults["provider"] VERBOSE = _defaults["verbose"] CHAT_TEMPLATE = _defaults["chat_template"] print( f"MODEL_ID={MODEL_ID}, PRECISION={PRECISION}, PROVIDER={PROVIDER}, " f"VERBOSE={VERBOSE}, CHAT_TEMPLATE={CHAT_TEMPLATE}" ) .. rst-class:: sphx-glr-script-out .. code-block:: none MODEL_ID=Qwen/Qwen2.5-Coder-0.5B-Instruct, PRECISION=fp32, PROVIDER=cpu, VERBOSE=1, CHAT_TEMPLATE=chatml .. GENERATED FROM PYTHON SOURCE LINES 89-95 Load the model --------------- We use :func:`~locodellm.generate.generate_from_model.get_session` to download, convert, and load the model. Under ``UNITTEST_GOING=1``, the mock model ``mock/generate`` is used instead. .. GENERATED FROM PYTHON SOURCE LINES 95-110 .. code-block:: Python from locodellm.generate.generate_from_model import get_session # noqa: E402 if UNITTEST_GOING: MODEL_ID = "mock/generate" session = get_session( model_id=MODEL_ID, precision=PRECISION, chat_template=CHAT_TEMPLATE, verbose=max(VERBOSE - 1, 0), ) print(f"Model loaded: {MODEL_ID}") .. rst-class:: sphx-glr-script-out .. code-block:: none Model loaded: mock/generate .. GENERATED FROM PYTHON SOURCE LINES 111-117 Run the benchmark ------------------ We load the ``basic`` benchmark and run it against the session. The benchmark contains 10 Python function prompts with growing difficulty. .. GENERATED FROM PYTHON SOURCE LINES 117-126 .. code-block:: Python from locodellm.bench import load_benchmark # noqa: E402 benchmark = load_benchmark("basic") print(f"Benchmark: {benchmark.description}") print(f"Number of prompts: {len(benchmark.tests)}") result = benchmark.run(session, verbose=VERBOSE) .. rst-class:: sphx-glr-script-out .. code-block:: none Benchmark: 10 Python function prompts with growing difficulty, from returning a constant string to computing an edit distance. Number of prompts: 10 [███░░░░░░░░░░░░░░░░░░░░░░░░░░░] 1/10 [██████░░░░░░░░░░░░░░░░░░░░░░░░] 2/10 [█████████░░░░░░░░░░░░░░░░░░░░░] 3/10 [████████████░░░░░░░░░░░░░░░░░░] 4/10 [███████████████░░░░░░░░░░░░░░░] 5/10 [██████████████████░░░░░░░░░░░░] 6/10 [█████████████████████░░░░░░░░░] 7/10 [████████████████████████░░░░░░] 8/10 [███████████████████████████░░░] 9/10 [██████████████████████████████] 10/10 .. GENERATED FROM PYTHON SOURCE LINES 127-132 Results table ------------- Each row shows one input set for a prompt: whether the generated code compiled, ran, and produced the expected result. .. GENERATED FROM PYTHON SOURCE LINES 132-146 .. code-block:: Python df = result.to_dataframe() columns = [ "prompt", "duration", "token_count", "tokens_per_second", "compiled", "ran", "input_index", "passed", ] print(df[columns].to_markdown(index=False)) .. rst-class:: sphx-glr-script-out .. code-block:: none | prompt | duration | token_count | tokens_per_second | compiled | ran | input_index | passed | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------:|--------------:|--------------------:|:-----------|:------|--------------:|:---------| | write a python function called hello that returns the string "hello" | 0.0142096 | 35 | 2463.12 | True | True | 0 | True | | write a python function called hello that returns the string "hello" | 0.0142096 | 35 | 2463.12 | True | True | 1 | True | | write a python function called add that takes two numbers and returns their sum | 0.0133592 | 34 | 2545.07 | True | True | 0 | False | | write a python function called add that takes two numbers and returns their sum | 0.0133592 | 34 | 2545.07 | True | True | 1 | False | | write a python function called add that takes two numbers and returns their sum | 0.0133592 | 34 | 2545.07 | True | True | 2 | False | | write a python function called reverse_string that takes a string and returns it reversed | 0.0129969 | 33 | 2539.06 | True | True | 0 | False | | write a python function called reverse_string that takes a string and returns it reversed | 0.0129969 | 33 | 2539.06 | True | True | 1 | False | | write a python function called reverse_string that takes a string and returns it reversed | 0.0129969 | 33 | 2539.06 | True | True | 2 | False | | write a python function called find_max that takes a list of numbers and returns the maximum value | 0.0119982 | 30 | 2500.37 | True | True | 0 | False | | write a python function called find_max that takes a list of numbers and returns the maximum value | 0.0119982 | 30 | 2500.37 | True | True | 1 | False | | write a python function called find_max that takes a list of numbers and returns the maximum value | 0.0119982 | 30 | 2500.37 | True | True | 2 | False | | write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 0 | False | | write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 1 | False | | write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 2 | False | | write a python function called is_prime that takes an integer and returns True if it is prime, False otherwise | 0.0110838 | 27 | 2435.98 | False | False | 3 | False | | write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 0 | False | | write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 1 | False | | write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 2 | False | | write a python function called factorial that takes a non-negative integer n and returns n! | 0.0123001 | 31 | 2520.31 | True | True | 3 | False | | write a python function called char_count that takes a string and returns a dictionary mapping each character to its count | 0.0109835 | 27 | 2458.23 | False | False | 0 | False | | write a python function called char_count that takes a string and returns a dictionary mapping each character to its count | 0.0109835 | 27 | 2458.23 | False | False | 1 | False | | write a python function called char_count that takes a string and returns a dictionary mapping each character to its count | 0.0109835 | 27 | 2458.23 | False | False | 2 | False | | write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 0 | False | | write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 1 | False | | write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 2 | False | | write a python function called is_palindrome that takes a string and returns True if it is a palindrome ignoring case and spaces, False otherwise | 0.00911696 | 21 | 2303.4 | False | False | 3 | False | | write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 0 | False | | write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 1 | False | | write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 2 | False | | write a python function called fibonacci that takes a non-negative integer n and returns the nth Fibonacci number where fibonacci(0)=0 and fibonacci(1)=1 | 0.00770924 | 17 | 2205.14 | False | False | 3 | False | | write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 0 | False | | write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 1 | False | | write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 2 | False | | write a python function called edit_distance that takes two strings a and b and returns the minimum number of single-character edits (insertions, deletions, or substitutions) needed to transform a into b | 0.00716989 | 9 | 1255.25 | False | False | 3 | False | .. GENERATED FROM PYTHON SOURCE LINES 147-151 Per-case statistics -------------------- One row per prompt showing how many input sets passed. .. GENERATED FROM PYTHON SOURCE LINES 151-174 .. code-block:: Python stats = df.groupby("prompt", sort=False) rows = [] for prompt, group in stats: total = len(group) passed = int(group["passed"].sum()) rows.append( { "prompt": prompt[:60] + "..." if len(prompt) > 60 else prompt, "duration": round(float(group["duration"].iloc[0]), 3), "tokens_per_second": round(float(group["tokens_per_second"].iloc[0]), 1), "compiled": bool(group["compiled"].iloc[0]), "ran": bool(group["ran"].iloc[0]), "passed": f"{passed}/{total}", "score": round(passed / total, 2) if total > 0 else 0.0, } ) import pandas # noqa: E402 stats_df = pandas.DataFrame(rows) print(stats_df.to_markdown(index=False)) .. rst-class:: sphx-glr-script-out .. code-block:: none | prompt | duration | tokens_per_second | compiled | ran | passed | score | |:----------------------------------------------------------------|-----------:|--------------------:|:-----------|:------|:---------|--------:| | write a python function called hello that returns the string... | 0.014 | 2463.1 | True | True | 2/2 | 1 | | write a python function called add that takes two numbers an... | 0.013 | 2545.1 | True | True | 0/3 | 0 | | write a python function called reverse_string that takes a s... | 0.013 | 2539.1 | True | True | 0/3 | 0 | | write a python function called find_max that takes a list of... | 0.012 | 2500.4 | True | True | 0/3 | 0 | | write a python function called is_prime that takes an intege... | 0.011 | 2436 | False | False | 0/4 | 0 | | write a python function called factorial that takes a non-ne... | 0.012 | 2520.3 | True | True | 0/4 | 0 | | write a python function called char_count that takes a strin... | 0.011 | 2458.2 | False | False | 0/3 | 0 | | write a python function called is_palindrome that takes a st... | 0.009 | 2303.4 | False | False | 0/4 | 0 | | write a python function called fibonacci that takes a non-ne... | 0.008 | 2205.1 | False | False | 0/4 | 0 | | write a python function called edit_distance that takes two ... | 0.007 | 1255.2 | False | False | 0/4 | 0 | .. GENERATED FROM PYTHON SOURCE LINES 175-179 Summary ------- Overall benchmark score. .. GENERATED FROM PYTHON SOURCE LINES 179-184 .. code-block:: Python print(f"Total prompts: {result.total}") print(f"Passed (all inputs correct): {result.passed}/{result.total}") print(f"Failed: {result.failed}/{result.total}") print(f"Overall score: {result.passed / result.total:.0%}") .. rst-class:: sphx-glr-script-out .. code-block:: none Total prompts: 10 Passed (all inputs correct): 1/10 Failed: 9/10 Overall score: 10% .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 1.083 seconds) .. _sphx_glr_download_auto_examples_plot_bench.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_bench.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_bench.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_bench.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_