Available Benchmarks

Use get_available_benchmarks() to list the built-in benchmarks and load_benchmark() to load one by name.

from locodellm.bench import get_available_benchmarks, load_benchmark

# List available benchmarks
for name, description in get_available_benchmarks().items():
    print(f"{name}: {description}")

# Load a benchmark
bench = load_benchmark("basic")

Built-in Benchmarks

basic

<<<

from locodellm.bench import load_benchmark

bench = load_benchmark("basic")
print(bench.description)
print()
print(f"**{len(bench.tests)} prompt tests:**")
print()
print(".. list-table::")
print("   :header-rows: 1")
print("   :widths: 5 60 10")
print()
print("   * - #")
print("     - Task")
print("     - Expected results")
for i, test in enumerate(bench.tests, 1):
    prompt = test.prompt
    if len(prompt) > 80:
        prompt = prompt[:77] + "..."
    print(f"   * - {i}")
    print(f"     - {prompt}")
    print(f"     - {len(test.expected)}")

>>>

10 Python function prompts with growing difficulty, from returning a constant string to computing an edit distance.

10 prompt tests:

#	Task	Expected results
1	write a python function called hello that returns the string “hello”	2
2	write a python function called add that takes two numbers and returns their sum	3
3	write a python function called reverse_string that takes a string and returns…	3
4	write a python function called find_max that takes a list of numbers and retu…	3
5	write a python function called is_prime that takes an integer and returns Tru…	4
6	write a python function called factorial that takes a non-negative integer n …	4
7	write a python function called char_count that takes a string and returns a d…	3
8	write a python function called is_palindrome that takes a string and returns …	4
9	write a python function called fibonacci that takes a non-negative integer n …	4
10	write a python function called edit_distance that takes two strings a and b a…	4