generate_from_model#

High-level generation from a model id.

locodellm.generate.generate_from_model.generate_from_model(model_id: str, prompt: str, precision: str | None = None, provider: str | None = None, max_length: int = 200, cache_dir: str | None = None, chat_template: str | None = None, verbose: int = 0, **search_options: Any) → SessionState#

Generates text from a prompt using the specified model.

Models are loaded once and cached for subsequent calls. For mock models (ids starting with mock/), the ONNX conversion is performed only on the first call and reused afterwards.

Parameters:

  • model_id – The model identifier. Use mock/generate or mock/tiny for test models, a HuggingFace id like Qwen/Qwen2.5-Coder-0.5B-Instruct, or a filesystem path for pre-converted models.

  • prompt – The text prompt to send to the model.

  • precision – Optional precision qualifier (e.g. "fp16", "int4"). Currently used only for cache keying.

  • provider – Execution provider name (e.g. "CUDAExecutionProvider"). When None, the default provider is used.

  • max_length – Maximum number of tokens to generate.

  • cache_dir – Directory for storing converted models. Defaults to ~/.cache/locodellm/models.

  • chat_template – Chat template to wrap prompts with (e.g. "chatml").

  • verbose – Verbosity level (0 = silent, 1+ = print progress).

  • **search_options – Extra search options forwarded to generation (e.g. temperature, top_k, top_p).

Returns:

A SessionState with the generation results. Access .text for the generated text.

locodellm.generate.generate_from_model.get_session(model_id: str, precision: str | None = None, provider: str | None = None, provider_options: dict[str, str] | None = None, session_options: dict[str, Any] | None = None, cache_dir: str | None = None, chat_template: str | None = None, verbose: int = 0) → SessionState#

Returns a cached session for the given model, creating it on first access.

This loads the model without generating any text. Useful when you need the session object for benchmarking or multiple prompts.

Parameters:

  • model_id – The model identifier (mock id, HuggingFace id, or path).

  • precision – Optional precision qualifier (e.g. "fp16").

  • provider – Execution provider name.

  • provider_options – Options applied to the execution provider.

  • session_options – ONNX Runtime session options.

  • cache_dir – Directory for storing converted models.

  • chat_template – Chat template name (e.g. "chatml").

  • verbose – Verbosity level.

Returns:

A SessionState ready for generation.