cpu_executor.h

Shared CPU executor and compatible-pool registry.

namespace onnx_light

Alias that makes onnx-light headers compatible with code that references ONNX_LIGHT_NAMESPACE (the macro used in the standard onnx package).

Set to ONNX_LIGHT_NAMESPACE so both names resolve to the same namespace.

Symbol-visibility attribute for the public onnx-light C++ API.

Maps the upstream compatibility macro to onnx-light’s explicit proto ABI annotation. This keeps declarations from vendored ONNX headers visible when lib_onnx_proto uses hidden visibility by default.

Namespace alias so that ONNX C++ code (and consumers such as onnxruntime) that refers to the literal onnx namespace — rather than the ONNX_NAMESPACE macro — resolves to the onnx-light namespace. The standard onnx package lives in namespace onnx; onnx-light uses onnx_light (via ONNX_LIGHT_NAMESPACE), so this alias keeps onnx-light a true drop-in. It is only introduced when the onnx-light namespace differs from onnx.

namespace core

namespace runtime

Typedefs

using ParallelRangeFn = void (*)(void*, int64_t, int64_t)

Type-erased range callable: function(context, begin, end).

Functions

CpuExecutorKey MakeCpuExecutorKey(const ResolvedCpuExecutionPolicy &policy)

Returns the immutable sharing key for a resolved policy.

Returns: The behavior-only executor key.

Parameters:

policy – The resolved CPU policy.

CpuExecutor *CurrentCpuExecutor() noexcept

Returns the executor installed on the calling thread.

A session installs its leased executor for the duration of a run so portable helpers dispatch through it instead of a hidden process-wide pool.

Returns: The installed executor, or nullptr when the thread runs outside any executor scope.

CpuExecutorRegistry &GlobalCpuExecutorRegistry()

Returns the process-wide executor registry.

Returns: The bounded process-owned registry.

Variables

constexpr size_t kDefaultCpuExecutorRegistryCapacity = 8

Default maximum number of simultaneously live executor pools.

class CpuExecutor

#include <cpu_executor.h>

Executes range work on one caller and a persistent set of workers.

Instances are obtained from :cpp:class:CpuExecutorRegistry. Concurrent regions sharing an executor serialize dispatch metadata while their surrounding inference calls remain independent. Nested regions execute inline to avoid deadlock and oversubscription.

Public Functions

CpuExecutor(const CpuExecutor&) = delete

CpuExecutor &operator=(const CpuExecutor&) = delete

~CpuExecutor()

uint32_t effective_threads() const noexcept

Returns the effective participant count, including the caller.

const ResolvedCpuExecutionPolicy &policy() const noexcept

Returns the immutable resolved policy.

const CpuExecutorKey &key() const noexcept

Returns the immutable registry-sharing key.

uint64_t instance_id() const noexcept

Returns the process-local identity of this executor instance.

Compatible sessions that share one lease observe the same identifier. The identifier is diagnostic only and must not be persisted as a tuning or cache key.

void EnableCounters()

Enables optional dispatch counters. Repeated calls preserve existing counts.

bool counters_enabled() const noexcept

Returns whether optional dispatch counters are enabled.

CpuExecutorCounters counters() const noexcept

Returns a consistent snapshot of the optional dispatch counters.

void ParallelFor(int64_t total, int64_t grain, void *context, ParallelRangeFn function, uint32_t maximum_participants = 0, ParallelRegionCollector *collector = nullptr, std::string_view label = {}, std::source_location location = std::source_location::current())

Executes contiguous ranges covering [0, total).

maximum_participants == 0 uses the session limit. A positive value may lower but never raise that limit. Work below grain runs inline. Executors inherited across fork are rejected.

Parameters:

• total – Number of iterations. Values <= 0 are a no-op.

• grain – Minimum iterations per parallel range. Must be positive.

• context – Opaque context passed to function.

• function – Range callback, which must not throw.

• maximum_participants – Optional kernel-specific participant limit.

Private Functions

explicit CpuExecutor(ResolvedCpuExecutionPolicy policy)

Private Members

std::unique_ptr<Impl> impl_

Friends

friend class CpuExecutorRegistry

struct CpuExecutorCounters

#include <cpu_executor.h>

Snapshot of optional executor dispatch counters.

Public Functions

bool operator==(const CpuExecutorCounters&) const = default

Public Members

uint64_t dispatches = 0

Number of ParallelFor dispatches, including inline dispatches.

uint64_t nested_inline_dispatches = 0

Number of dispatches that ran inline because they were nested.

uint64_t limited_inline_dispatches = 0

Number of dispatches that ran inline for size or participant limits.

struct CpuExecutorKey

#include <cpu_executor.h>

Identifies every resolved property that changes executor behavior.

Request spelling and diagnostics are intentionally excluded. The key records whether no affinity was explicitly requested so that a successful no-affinity policy is not confused with an unsupported affinity fallback.

Public Functions

bool operator==(const CpuExecutorKey&) const = default

Public Members

uint32_t effective_threads = 1

Effective participants, including the caller.

std::optional<CpuLogicalProcessor> caller_processor

Optional caller placement.

std::vector<CpuLogicalProcessor> worker_processors

Exact worker placement, or an empty vector for unpinned workers.

ResolvedSpinPolicy spin

Resolved spin-before-park behavior.

bool explicit_no_affinity = false

Whether the request explicitly selected no affinity.

bool allow_nested_parallelism = false

Whether nested parallelism was requested.

class CpuExecutorRegistry

#include <cpu_executor.h>

Leases compatible shared executors from a process-owned bounded registry.

The registry stores weak references. Compatible callers receive the same executor while at least one lease remains. Releasing the final lease stops and destroys its workers immediately. Capacity limits simultaneously live incompatible pools; expired entries never consume capacity.

Public Functions

explicit CpuExecutorRegistry(size_t capacity)

Creates a registry with a strict live-pool capacity.

CpuExecutorRegistry(const CpuExecutorRegistry&) = delete

CpuExecutorRegistry &operator=(const CpuExecutorRegistry&) = delete

std::shared_ptr<CpuExecutor> Acquire(const CpuExecutionPolicy &request)

Resolves and acquires an executor for request.

std::shared_ptr<CpuExecutor> Acquire(const ResolvedCpuExecutionPolicy &policy)

Acquires an executor for an already resolved policy.

inline size_t capacity() const noexcept

Returns the configured maximum number of simultaneously live pools.

size_t live_pool_count()

Returns the number of live pools currently tracked.

Private Functions

void ResetAfterForkLocked(uint64_t process_id)

void RemoveExpiredLocked()

Private Members

size_t capacity_

uint64_t process_id_

std::mutex mutex_

std::vector<Entry> entries_

struct Entry

Public Members

CpuExecutorKey key

std::weak_ptr<CpuExecutor> executor

class CpuExecutorScope

#include <cpu_executor.h>

Installs a non-owning executor view on the calling thread.

The previous view is restored when the scope ends, so nested scopes and subgraph executions compose. The scope does not extend the executor lifetime: the installer must keep its lease alive.

Public Functions

explicit CpuExecutorScope(CpuExecutor *executor) noexcept

Installs executor (possibly nullptr) on the calling thread.

CpuExecutorScope(const CpuExecutorScope&) = delete

CpuExecutorScope &operator=(const CpuExecutorScope&) = delete

~CpuExecutorScope()

Private Members

CpuExecutor *previous_