Parallel kernel execution#

onnx-light-cpu does not own worker threads. A kernel divides independent work into contiguous ranges and submits those ranges to the onnx-light session’s CpuExecutor. Consequently, a session has one executor policy for both its built-in and CPU-extension kernels. This page describes the shared mechanism; Kernels explains how individual kernel families choose their work units.

Execution flow#

  1. A kernel turns its independent work into a one-dimensional total: for example, elements for unary operations, output tiles for matrix multiplication, or rows for attention.

  2. The kernel calls ExecuteRanges or ExecuteCostedRanges from impl/execution.h. Its schedule supplies minimum work and block sizes, a participant limit, and, where useful, a preferred participant count. Costed calls additionally describe bytes read, bytes written, and estimated compute cycles.

  3. The helper keeps small jobs serial. For a scheduled call it limits the number of blocks by the available threads, requested maximum, and work that can meet the minimum block size. For a costed call, the executor’s PlanParallelFor selects its grain size and participants. Block boundaries are rounded to a requested multiple, such as a SIMD lane count, and each callback receives its [begin, end) range.

  4. When more than one block is useful, the helper calls the executor’s ParallelFor bridge. Workers execute the block callbacks; each callback writes only its assigned range. The caller waits for the bridge to finish, then the kernel returns its output.

Obtaining the executor#

During a normal onnx-light model run, the runtime places the session’s CpuExecutor on the calling thread. CurrentExecutionExecutor() adapts that executor into an ExecutionExecutorView and exposes its effective thread count, range runner, and optional cost planner. The adapter is implemented in impl/execution.cc. If a low-level kernel is invoked directly rather than through a session, it can install an explicit ExecutionExecutorScope; otherwise there is no current executor and execution is serial.

The public Registering kernels guide covers the equivalent onnx-light CpuExecutorScope requirement for direct KernelBase use.

Nested work stays serial#

Every submitted range callback enters an execution-region scope. If code inside that callback invokes ExecuteRanges again, the nonzero thread-local execution depth makes the inner call use one block. This avoids recursive thread-pool submissions and oversubscribing the session’s workers. The same rule applies to an outer kernel that has already chosen serial execution: its callback has a region scope, so nested helpers remain direct.

Thus parallelism is selected at one level of the active kernel plan, while small jobs, direct low-level calls without an executor, and nested helpers all take the same serial range path.