runtime_session.h#

Declares :cpp:class:RuntimeSession, a reusable execution session that separates kernel initialization from execution.

:cpp:class:RuntimeSession binds a precomputed :cpp:class:ExecutionPlan (which already carries the node list it drives) and, on its first :cpp:func:Run, resolves every executed node’s kernel once against the supplied :cpp:class:RuntimeContext; subsequent runs reuse the cached kernels. Every entry point that runs a node list — model callers (via :cpp:func:RegisterModelFunctions followed by their own :cpp:class:RuntimeSession), :cpp:class:SubgraphSession, the model-local function call helper, and the If / Loop / Scan control-flow kernels — constructs one of these sessions (over the graph’s or function’s cached :cpp:class:ExecutionPlan) and calls :cpp:func:Run a single time.

namespace onnx_light

Alias that makes onnx-light headers compatible with code that references ONNX_LIGHT_NAMESPACE (the macro used in the standard onnx package).

Set to ONNX_LIGHT_NAMESPACE so both names resolve to the same namespace.

Symbol-visibility attribute for the public onnx-light C++ API.

Maps the upstream compatibility macro to onnx-light’s explicit proto ABI annotation. This keeps declarations from vendored ONNX headers visible when lib_onnx_proto uses hidden visibility by default.

Namespace alias so that ONNX C++ code (and consumers such as onnxruntime) that refers to the literal onnx namespace — rather than the ONNX_NAMESPACE macro — resolves to the onnx-light namespace. The standard onnx package lives in namespace onnx; onnx-light uses onnx_light (via ONNX_LIGHT_NAMESPACE), so this alias keeps onnx-light a true drop-in. It is only introduced when the onnx-light namespace differs from onnx.

namespace core

namespace runtime

struct KernelTuningResolutionStatistics#

#include <runtime_session.h>

Reports the one-time kernel tuning work performed by a runtime session.

Public Functions

inline uint64_t TotalDurationNs() const noexcept#: Returns the measured cold tuning duration.

bool operator==(const KernelTuningResolutionStatistics&) const = default#

Public Members

uint64_t snapshot_duration_ns = 0#: Time spent capturing the immutable registry generation.

uint64_t resolution_duration_ns = 0#: Time spent resolving execution-specific profiles from that generation.

size_t tunable_kernels = 0#: Number of kernels that exposed a defined tuning key.

size_t resolved_profiles = 0#: Number of tunable kernels for which registered parameters were found.

class RuntimeSession#

#include <runtime_session.h>

A reusable execution session that binds a precomputed :cpp:class:ExecutionPlan and separates the runtime lifecycle into three explicit phases:

Construction — the session records the plan it will drive. The node list is recovered from :cpp:func:ExecutionPlan::nodes, so the session no longer needs the nodes (nor a :cpp:class:RuntimeContext) passed separately.
Kernel initialization — the first :cpp:func:Run resolves the kernel for every node the plan executes once (against the model-local function registry, the control-flow handlers, the user custom kernels and the static :cpp:func:KernelDispatchTable of the supplied :cpp:class:RuntimeContext), builds the resulting per-node kernel instance, and caches it. Any unsupported operator is rejected here rather than mid-run. It also records the external inputs the scheduled nodes read (:cpp:func:required_inputs) so :cpp:func:Run can verify they are supplied before executing.
Execution — :cpp:func:Run replays the plan, invoking each pre-resolved kernel instance and releasing intermediates as scheduled. It may be called more than once (e.g. to re-run the same graph with fresh inputs) without redoing the per-node dispatch lookup or re-constructing the concrete per-node kernel objects.

This mirrors how an inference runtime prepares an executable graph once and then runs it repeatedly. Every caller that needs to run a node list builds one of these sessions over the list’s :cpp:class:ExecutionPlan and calls :cpp:func:Run on it.

Subclassed by onnx_light::core::runtime::SubgraphSession

Public Functions

explicit RuntimeSession(const ModelProto &model, int verbose = 0)#

Builds a session over an :cpp:class:ExecutionPlan the session owns, constructed from model’s graph (:cpp:func:ModelProto::graph). Use this when no precomputed plan is available: the session builds and owns the plan itself, so a caller can create a runnable session from a model alone (without first building an :cpp:class:ExecutionPlan). Kernel resolution is still deferred to the first :cpp:func:Run.

Parameters:

model – Model whose graph drives execution. The model (and the graph it owns) must outlive the session, since the built plan holds non-owning pointers into the graph’s nodes.
verbose – Verbosity level used by :cpp:func:Run for its progress lines. 0 (the default) leaves verbosity to the :cpp:class:RuntimeContext.

RuntimeSession(const ModelProto &model, RuntimeSessionOptions options)#

explicit RuntimeSession(const ExecutionPlan &plan, int verbose = 0)#

Builds a session over plan. Kernel resolution is deferred to the first :cpp:func:Run (which supplies the :cpp:class:RuntimeContext the kernels are resolved against).

Parameters:

plan – Precomputed execution / release schedule. Its node list (:cpp:func:ExecutionPlan::nodes) drives execution. The plan (and the graph / function it was built from) must outlive the session.
verbose – Verbosity level used by :cpp:func:Run for its progress lines. 0 (the default) leaves verbosity to the :cpp:class:RuntimeContext.

RuntimeSession(const ExecutionPlan &plan, RuntimeSessionOptions options)#

RuntimeSession(const RuntimeSession&) = delete#

RuntimeSession &operator=(const RuntimeSession&) = delete#

void Run(RuntimeContext &rt)#

Executes the plan once against rt: on the first call it resolves and caches the kernel for every scheduled node (rejecting unsupported operators) and records the external inputs those nodes read; every call then verifies rt supplies each of those required inputs and runs each scheduled node using its resolved kernel. When :cpp:func:RuntimeContext::release_intermediates is enabled on rt, it additionally frees each intermediate whose last reference has been reached, as scheduled by the plan; when disabled, every intermediate the plan would have released instead stays observable in rt after Run returns. Safe to call repeatedly on the same session.

The allocator attached to rt (:cpp:func:RuntimeContext::allocator) is captured once, on the first call, as the session’s unique allocator. After each scheduled node executes, every tensor it produced that is allocator-backed (:cpp:func:Tensor::has_allocation) is verified to be owned by that same allocator, catching kernels that allocate their output from the wrong allocator (or from none at all). This verification is skipped when :cpp:func:allow_external_output_allocators is enabled, so a kernel may legitimately return an output allocated outside the common allocator.

Parameters:: rt – In/out runtime context used both to resolve the kernels (function registry / custom kernels) and to exchange tensors.
Throws:: std::invalid_argument – if the plan references an out-of-range node index, if any executed node cannot be dispatched, if rt does not define one of the plan’s required external inputs, or if a node produces an output tensor backed by an allocator other than the session’s.

inline const RuntimeParameters &parameters() const noexcept#: Model-independent execution parameters (e.g. the requested degree of parallelism, :cpp:var:RuntimeParameters::num_threads) applied to the nodes this session runs.

inline const CpuExecutionPolicy &cpu_execution_policy() const noexcept#: Returns the CPU execution policy this session requests, either the one supplied at construction or the one derived from :cpp:func:parameters.

const std::shared_ptr<CpuExecutor> &cpu_executor()#

Returns the shared CPU executor this session leases, acquiring it from :cpp:func:GlobalCpuExecutorRegistry on first use. Sessions whose resolved policies are compatible share one executor; incompatible policies get distinct pools. :cpp:func:Run installs it on the calling thread so every parallel region a kernel launches uses exactly these participants. Like the rest of the session state, the first acquisition is not synchronized: a session is prepared and run by one thread at a time, while distinct sessions may run concurrently on a shared executor.

Returns: The leased executor, never null.

inline bool has_explicit_cpu_execution_policy() const noexcept#: Returns whether the CPU execution policy was supplied by the caller rather than derived from :cpp:func:parameters. A session with a derived policy that runs inside another session’s run (a subgraph, a model-local function, or any nested session) reuses the executor already installed on the :cpp:class:RuntimeContext instead of leasing a second pool.

inline int verbose() const noexcept#: Verbosity level requested for :cpp:func:Run. When non-zero, it overrides :cpp:func:RuntimeContext::verbose for this session’s progress lines without mutating the context itself.

inline const std::vector<std::string> &required_inputs() const noexcept#: Returns the external input names the scheduled nodes read (the inputs that must be present in the :cpp:class:RuntimeContext before :cpp:func:Run). Populated during kernel initialization; empty until the first :cpp:func:Run.

std::vector<std::string> used_kernels() const#: Returns the normalized "<domain>:<op_type>" identifiers of the kernels resolved by this session, in execution order. Repeated operators are preserved because each node owns a distinct kernel instance. The list is empty until the first :cpp:func:Run initializes the kernels.

inline uint64_t tuning_generation() const noexcept#: Returns the immutable tuning-registry generation captured while kernels were initialized, or 0 before the first :cpp:func:Run.

inline const KernelTuningResolutionStatistics &tuning_resolution_statistics() const noexcept#: Returns the cold tuning work recorded during first-run kernel initialization.

inline bool check_shapes() const noexcept#: Enables or disables concrete-shape validation. When enabled, :cpp:func:Run checks that the concrete shape of every tensor carrying a declared (possibly symbolic) shape — the graph inputs, outputs and value_info recorded by :cpp:func:SetDeclaredShapes — is consistent with that declaration: a concrete dim_value must match exactly, and every symbolic dim_param must resolve to the same concrete value everywhere it appears during a single :cpp:func:Run. Disabled by default so the hot path stays free of the extra checks. Declared shapes are populated automatically when the session is built from a :cpp:class:ModelProto; a session built from an :cpp:class:ExecutionPlan alone must call :cpp:func:SetDeclaredShapes for the check to have anything to validate.

inline bool allow_external_output_allocators() const noexcept#: When true, :cpp:func:Run does not require a node’s allocator-backed outputs to be owned by the session’s unique allocator, so a kernel may return an output allocated outside the common allocator. When false (the default), such an output is rejected. See :cpp:member:RuntimeSessionOptions::allow_external_output_allocators.

inline const std::shared_ptr<ParallelRegionCollector> &parallel_region_collector() const noexcept#: Returns the optional collector retained by this session.

void SetDeclaredShapes(const GraphProto &graph)#: Records the declared (possibly symbolic) shapes carried by graph’s inputs, outputs and value_info so that, when :cpp:func:check_shapes is enabled, :cpp:func:Run can validate concrete tensor shapes against them. Only tensor-typed values whose type carries a shape are recorded; values without a shape (unknown rank) are ignored. Calling this replaces any previously recorded declarations for the listed names. graph is read but not retained, so it need not outlive the session.

void SetInitializers(const GraphProto &graph)#: Materializes graph initializers into the session-owned cache. A session built from a model or graph calls this automatically; callers constructing a session from an :cpp:class:ExecutionPlan may call it once before the first :cpp:func:Run.

Public Static Functions

static std::vector<std::string> CollectExternalInputs(const utils::RepeatedProtoField<NodeProto> &nodes)#: Returns the list of input names referenced by nodes that are not produced as outputs by any node in the same list — i.e. the external dependencies of the node set. Subgraph attributes (GRAPH / GRAPHS) are inspected recursively. The returned list preserves first-seen order and contains no duplicates; empty input names are skipped.

static std::vector<std::string> CollectNodeInputs(const NodeProto &node)#: Returns the full list of tensor / sequence names a single node depends on at runtime: the names referenced by node.input() (skipping empty optional-input slots) plus every external input of the subgraph attributes (GRAPH / GRAPHS) attached to node. The returned list preserves first-seen order and contains no duplicates.

Protected Functions

explicit RuntimeSession(const GraphProto &graph, int verbose = 0)#: Constructs a session over a bare :cpp:class:GraphProto, owning the resulting :cpp:class:ExecutionPlan in default_plan_. Used by :cpp:class:SubgraphSession so a control-flow subgraph can be a :cpp:class:RuntimeSession with the same default resolution behavior as a top-level graph session.

std::unique_ptr<KernelBase> ResolveNodeKernel(const NodeProto &node, RuntimeContext &rt, const std::string &domain, const std::string &op_type) const#: Default node-kernel resolution used during :cpp:func:InitializeKernels, so :cpp:class:RuntimeSession and derived sessions that reuse the base initialization path share the same default resolution behavior. Builds and returns the ready-to-invoke kernel instance for node.

Private Functions

void InitializeKernels(RuntimeContext &rt)#: Resolves and builds the kernel instance for every node the plan executes, resolving against rt, and records the external inputs those nodes read in :cpp:member:required_inputs_.

void SeedInitializers(RuntimeContext &rt) const#: Seeds missing initializer names with payload views into the session store.

void VerifyOutputAllocators(const NodeProto &node, RuntimeContext &rt) const#: Normalizes every raw tensor output of node into the arena implied by that output slot’s role: a declared graph output (a name present in :cpp:member:output_names_set_) is normalized into :cpp:member:session_io_allocator_ when an I/O allocator is attached, every other (intermediate) output into :cpp:member:session_allocator_. This is output-slot routing: a node producing both a declared output and an intermediate keeps each value in its own arena. Called after a node’s kernel has run, once :cpp:member:session_allocator_ has been captured.

bool ProducesDeclaredOutput(const NodeProto &node) const#: Returns whether node produces at least one declared graph output (a name present in :cpp:member:output_names_). Used by :cpp:func:Run to route that node’s kernel invocation through the I/O allocator instead of the execution allocator.

void VerifyDeclaredShape(const std::string &name, const RuntimeContext &rt, core::shapes::ShapesContext &bindings) const#: Verifies, when :cpp:func:check_shapes is enabled, that the concrete shape of the tensor stored under name in rt (if any) matches the declared :cpp:class:core::symbolic::SymShape recorded in :cpp:member:declared_shapes_. Concrete dimensions must match exactly; symbolic dimensions are resolved against bindings (a :cpp:class:core::shapes::ShapesContext binding each symbolic expression to the concrete value it first resolved to during the current :cpp:func:Run), so an inconsistent reuse of the same symbol is rejected. Names without a recorded declaration, or not currently present as a tensor, are ignored.

void MaterializeBorrowedOutputs(RuntimeContext &rt) const#: Detaches every graph output present in rt from any external memory it borrows: for each name in :cpp:member:output_names_ whose tensor is a borrowed view (:cpp:func:Tensor::is_borrowed), the entry is replaced with an owned deep copy (:cpp:func:Tensor::ToOwned). A graph output can borrow into the model (e.g. a Constant reading its value’s raw_data or an initializer passed straight through), which would dangle once the model is released; owning the bytes keeps the output valid independently of the model’s lifetime. Called at the end of :cpp:func:Run. Only runs when :cpp:member:output_names_ is populated (sessions built from a :cpp:class:ModelProto / :cpp:class:GraphProto).

Private Members

ExecutionPlan default_plan_#: Plan owned by the session, referenced by :cpp:member:plan_ when the session is constructed from a :cpp:class:ModelProto (no external plan supplied). Built from the model’s graph. Left empty (and unused) when a plan is passed in through the plan-taking constructor.

const ExecutionPlan &plan_#

std::vector<PreparedKernel> kernels_#

std::vector<Tensor> initializers_#

std::optional<KernelTuningRegistrySnapshot> tuning_snapshot_#: One immutable registry generation shared by every kernel in this session. Kernels copy resolved values during initialization; retaining the snapshot also makes the generation available for diagnostics.

KernelTuningResolutionStatistics tuning_resolution_statistics_#

std::vector<std::string> required_inputs_#

std::unordered_map<std::string, core::symbolic::SymShape> declared_shapes_#: Declared (possibly symbolic) shapes keyed by tensor name, populated by :cpp:func:SetDeclaredShapes and consulted by :cpp:func:Run when :cpp:member:check_shapes_ is enabled.

std::vector<std::string> output_names_#: Names of the graph’s declared outputs, populated from the :cpp:class:ModelProto / :cpp:class:GraphProto the session is built from (empty for a session built from a bare :cpp:class:ExecutionPlan). Consulted by :cpp:func:MaterializeBorrowedOutputs so borrowed graph outputs are detached from the model before :cpp:func:Run returns.

std::unordered_set<std::string> output_names_set_#: Same names as :cpp:member:output_names_, indexed for O(1) membership checks by :cpp:func:ProducesDeclaredOutput and :cpp:func:VerifyOutputAllocators.

bool kernels_initialized_ = false#

bool check_shapes_ = false#: When true, :cpp:func:Run validates concrete tensor shapes against the declarations in :cpp:member:declared_shapes_.

bool allow_external_output_allocators_ = false#: When true, :cpp:func:Run skips the per-node output allocator check (see :cpp:func:VerifyOutputAllocators), allowing outputs allocated outside the session’s common allocator.

RuntimeParameters parameters_#

CpuExecutionPolicy cpu_execution_#: Requested CPU execution policy, derived from :cpp:member:parameters_ when the caller did not supply one.

bool cpu_execution_explicit_ = false#: Whether :cpp:member:cpu_execution_ was requested explicitly. A session with a derived policy nested inside another session’s run reuses the enclosing executor instead of leasing a second pool.

bool cpu_execution_counters_ = false#: Whether this session enables optional executor dispatch counters.

std::shared_ptr<ParallelRegionCollector> parallel_region_collector_#: Optional bounded profiling collector installed for each run.

std::shared_ptr<CpuExecutor> cpu_executor_#: Lease on the shared executor, acquired on first use by :cpp:func:cpu_executor.

int verbose_ = 0#: Verbosity level used by :cpp:func:Run when non-zero.

RawBufferAllocator *session_allocator_ = nullptr#: Allocator observed on rt the first time :cpp:func:Run executes; every output tensor produced by a scheduled node is verified to be backed by this same allocator (see :cpp:func:Run).

RawBufferAllocator *session_io_allocator_ = nullptr#: I/O allocator observed on rt (:cpp:func:RuntimeContext::io_allocator) the first time :cpp:func:Run executes, alongside :cpp:member:session_allocator_. When non-null, :cpp:func:Run routes the kernel invocation of any node that produces a declared graph output (see :cpp:member:output_names_set_) through this allocator instead of :cpp:member:session_allocator_.

bool session_allocator_captured_ = false#

struct PreparedKernel#

A node’s kernel instance built once during :cpp:func:InitializeKernels, together with the normalised domain and op_type fused into a single "<domain>:<op_type>" key (the same format used to look the kernel up in the :cpp:func:KernelDispatchTable) so :cpp:func:Run never has to recompute or re-store them separately.

Public Members

std::string key#

std::unique_ptr<KernelBase> instance#

struct RuntimeSessionOptions#

#include <runtime_session.h>

Construction-time settings for :cpp:class:RuntimeSession.

Sessions are intended to be built once and reused, so these values stay fixed for the session’s lifetime.

Public Members

RuntimeParameters parameters = {}#

std::optional<CpuExecutionPolicy> cpu_execution = std::nullopt#: Requested CPU execution policy. When empty, the session derives its policy from :cpp:var:RuntimeParameters::num_threads and leaves worker placement to the operating system (:cpp:enumerator:CpuAffinityPolicy::kNone), which preserves the behavior of sessions built before the policy existed. A caller that wants pinned workers, another spin policy, or nested parallelism supplies the policy explicitly.

bool cpu_execution_counters = false#: Enables optional executor dispatch counters for inspection.

std::shared_ptr<ParallelRegionCollector> parallel_region_collector = nullptr#: Optional bounded collector retained and installed for every session run.

int verbose = 0#

bool check_shapes = false#

bool allow_external_output_allocators = false#: When false (the default), :cpp:func:RuntimeSession::Run verifies that every allocator-backed output a node produces is owned by the session’s unique allocator, rejecting a kernel that allocates its output from a different allocator (or from none at all). When true, that check is skipped so a kernel may legitimately return an output allocated outside the common allocator.

namespace shapes#