locodellm.session.session_state#

Holds the state of a generation session.

class locodellm.session.session_state.SessionState(model: Any, tokenizer: Any, verbose: int = 0, model_path: str | None = None, chat_template: str | None = None)#

Holds the state of a generation session.

Use create_session() to build an instance from a model path or an already-loaded onnxruntime_genai.Model.

model#

The loaded onnxruntime_genai.Model.

tokenizer#

The tokenizer bound to model.

tokens#

All token ids accumulated so far (prompt + generated).

text#

The text generated during the last generate() call.

verbose#

Verbosity level (0 = silent, 1+ = print progress).

generate(prompt: str, max_length: int = 200, **search_options: Any) → SessionState#

Generate text from prompt, appending to the conversation history.

The full token history (previous turns + prompt) is replayed so the model sees the complete context.

Parameters:

  • prompt – The text prompt to send to the model.

  • max_length – Maximum number of tokens (including all tokens accumulated across turns).

  • **search_options – Extra search options forwarded to GeneratorParams.set_search_options (e.g. temperature, top_k, top_p).

Returns:

self, with tokens and text updated.

locodellm.session.session_state.create_session(model: str | Any, providers: list[str] | None = None, verbose: int = 0, chat_template: str | None = None) → SessionState#

Create a SessionState from a model path or loaded model.

Parameters:

  • model – A path (str) to the model directory or an already-loaded onnxruntime_genai.Model instance.

  • providers – Ordered list of execution providers, e.g. ["CUDAExecutionProvider", "CPUExecutionProvider"]. When None, onnxruntime-genai picks its default provider. Ignored when model is not a path.

  • verbose – Verbosity level (0 = silent, 1+ = print progress messages during model loading and generation).

  • chat_template – Chat template to apply around prompts. Use "chatml" for Qwen and other ChatML-based instruct models. When None, prompts are sent as-is.

Returns:

A new SessionState ready for SessionState.generate().