Kernel classes#

Public kernel implementations declared in onnx_light_cpu/kernels.

class AbsKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

SIMD-accelerated onnx-light kernel for the ONNX Abs operator.

AbsKernel derives from onnx-light’s :cpp:class:onnx_light::core::runtime::KernelBase so it plugs into the runtime exactly like a built-in kernel: the dispatch table constructs it once per node and calls :cpp:func:Run on every execution. The computation uses private SIMD dispatch for every supported numeric type. Float16 and bfloat16 share the exact sign bit clearing path, while signed integers use width-specific SIMD absolute value operations.

Public Functions

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s single input, computes the elementwise absolute value and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh output tensor and writes |x| into it.

When rt is non-null its allocator backs the output buffer.

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Writes |x| into the caller-supplied output tensor, whose data_type, shape and buffer size must already match x.

Public Static Attributes

static constexpr const char *kName = "onnx_light_cpu::Abs"#: Library-qualified name identifying this kernel, recorded through :cpp:func:RuntimeContext::RecordKernelUsage on every :cpp:func:Run when enabled so callers can tell this kernel apart from onnx-light’s built-in Abs.

class AttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

CPU kernel for the ai.onnx::Attention operator (opset 23 through 25).

Supports rank-3/rank-4 FP32, FP16, and BF16 MHA/GQA/MQA. Streaming invocations may consume tensor past inputs and v24 nonpad_kv_seqlen directly; observable present and qk_matmul_output outputs use a materialized path and preserve the Q/K/V element type.

Public Functions

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::NodeProto &node, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &q, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &k, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &v, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *mask, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *past_k = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *past_v = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *nonpad_kv_seqlen = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *head_sink = nullptr) const#: Direct invocation used by backend test cases.

class BatchNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class BiasGeluKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Scalar onnx-light-cpu kernel for the com.microsoft BiasGelu contrib operator.

BiasGeluKernel derives from onnx-light’s :cpp:class:onnx_light::core::runtime::KernelBase so it plugs into the runtime exactly like a built-in kernel: the dispatch table constructs it once per node and calls :cpp:func:Run on every execution. Given an input A of any positive rank and a rank-1 bias B whose length matches A’s last dimension, it broadcasts B over that last dimension and applies the exact (erf-based) Gaussian Error Linear Unit Gelu(z) = 0.5 * z * (1 + erf(z / sqrt(2))) to A + B. The computation is delegated to the BiasGelu* kernels declared in ml_cpu_compute/com_microsoft/bias_gelu.h and scheduled row-by-row. Float32 uses runtime-dispatched AVX2/FMA or AVX-512 kernels when available; float16 and bfloat16 values are decoded to float32 scalar-by-scalar and re-encoded without temporary allocations.

Public Functions

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s A and B inputs, computes Gelu(A + B) and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &a, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &b, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh output tensor with A’s shape and writes Gelu(A + B) into it, broadcasting b over a’s last dimension.

When rt is non-null its allocator backs the output buffer.

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &a, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &b, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Writes Gelu(A + B) into the caller-supplied output tensor, whose data_type, shape and buffer size must already match a.

class BinaryElementwiseKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class CDistKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

onnx-light-cpu kernel for the com.microsoft CDist contrib operator.

CDistKernel derives from onnx-light’s :cpp:class:onnx_light::core::runtime::KernelBase so it plugs into the runtime exactly like a built-in kernel: the dispatch table constructs it once per node and calls :cpp:func:Run on every execution. Given a rank-2 A (M x N) and a rank-2 B (K x N), it computes the M x K pairwise distance matrix using a direct sum of squared differences (see :cpp:func:CDistFloat32) for either the sqeuclidean (default) or euclidean metric attribute; every other scipy.spatial.distance metric name accepted by the ONNX Runtime contract is rejected since onnx-light-cpu only implements these two. The computation is delegated to the cache-friendly CDist* kernels declared in ml_cpu_compute/com_microsoft/cdist.h, scheduled row-by-row and runtime-dispatched to AVX2/FMA or AVX-512 when available.

Public Functions

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s A and B inputs together with the metric attribute, computes the pairwise distance matrix and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &a, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &b, const std::string &metric = "sqeuclidean", ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh M x K output tensor and writes the pairwise distance matrix between a’s and b’s rows into it.

metric must be "sqeuclidean" (the default) or "euclidean". When rt is non-null its allocator backs the output buffer.

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &a, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &b, const std::string &metric, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Writes the pairwise distance matrix into the caller-supplied output tensor, whose data_type and shape must already match (M, K).

class CastKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Shape-preserving numeric Cast, with concrete builtin compatibility for STRING, float8 and packed sub-byte formats (no global dispatch recursion).

Public Functions

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &data, int32_t to, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Validates dtype, shape, storage and non-overlap before any output writes.

class CausalConvWithStateKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class ConcatKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Standard variadic ONNX Concat for every fixed-byte dtype supported by onnx-light.

Public Functions

void operator()(std::span<const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor> inputs, int64_t axis, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Output must match the inferred dtype, shape and byte size, and not overlap any input.

class ExpKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

SIMD-accelerated onnx-light kernel for the ONNX Exp operator.

ExpKernel derives from onnx-light’s :cpp:class:onnx_light::core::runtime::KernelBase so it plugs into the runtime exactly like a built-in kernel: the dispatch table constructs it once per node and calls :cpp:func:Run on every execution. The computation uses private SIMD dispatch for float32/float64 directly; float16 and bfloat16 use vector conversion blocks around the same float32 approximation.

Public Functions

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s single input, computes the elementwise exponential and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh output tensor and writes exp(x) into it.

When rt is non-null its allocator backs the output buffer.

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Writes exp(x) into the caller-supplied output tensor, whose data_type, shape and buffer size must already match x.

Public Static Attributes

static constexpr const char *kName = "onnx_light_cpu::Exp"#: Library-qualified name identifying this kernel, recorded through :cpp:func:RuntimeContext::RecordKernelUsage on every :cpp:func:Run when enabled so callers can tell this kernel apart from onnx-light’s built-in Exp.

class GatherBlockQuantizedKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class GatherKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Standard ONNX Gather for every fixed-byte type supported by onnx-light.

Public Functions

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &data, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &indices, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Validates output dtype, shape, size and non-overlap before writing.

class GemmKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

SIMD-accelerated onnx-light kernel for the ONNX Gemm operator.

GemmKernel derives from onnx-light’s :cpp:class:onnx_light::core::runtime::KernelBase so it plugs into the runtime exactly like a built-in kernel: the dispatch table constructs it once per node and calls :cpp:func:Run on every execution. The computation

Y = alpha * op(A) @ op(B) + beta * C

(where op(X) transposes X when the corresponding transA / transB attribute is set) uses private register-blocked SIMD dispatch for float32 and float64. float16 and bfloat16 inputs are accumulated in float32 and rounded back down for the output, matching common fp16/bf16 GEMM backend conventions. The optional bias C is consumed directly as a scalar, row, column, or matrix view without materializing an expanded M x N tensor. FP16/BF16 narrowing is combined with the bias epilogue.

Public Functions

explicit GemmKernel(const ONNX_LIGHT_NAMESPACE::core::runtime::KernelContext &ctx)#: Constructs the kernel and eagerly allocates the immutable-plan cache so no per-run lazy initialization (and its data race) is needed.

~GemmKernel() override#: Declared so the std::unique_ptr to the incomplete plan cache can be destroyed where the cache type is complete (in the translation unit).

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s A, B and optional C inputs together with the alpha, beta, transA and transB attributes, computes the general matrix multiplication and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &a, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &b, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &c, float alpha, float beta, bool trans_a, bool trans_b, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh output tensor and writes alpha * op(A) @ op(B) + beta * C into it, broadcasting the bias c to the M x N output shape.

When rt is non-null its allocator backs the output buffer.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &a, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &b, float alpha, bool trans_a, bool trans_b, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#: Same as above but for a Gemm node without a bias input: allocates a fresh output tensor and writes alpha * op(A) @ op(B) into it.

Public Static Functions

static std::int64_t ActiveInstanceCountForTesting() noexcept#

Returns the number of live instances in this library.

Intended for lifecycle regression tests that verify lazy backend-test registration.

Public Static Attributes

static constexpr const char *kName = "onnx_light_cpu::Gemm"#: Library-qualified name identifying this kernel, recorded through :cpp:func:RuntimeContext::RecordKernelUsage on every :cpp:func:Run when enabled so callers can tell this kernel apart from onnx-light’s built-in Gemm.

class GroupNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class GroupQueryAttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

CPU kernel for com.microsoft::GroupQueryAttention.

Implements the exact contract used by KV-cached decoder-only LLM export graphs (e.g. Qwen3): rank-3 query/key/value or packed QKV in query with empty key/value inputs, an optional rank-4 (batch, kv_num_heads, past_sequence_length, head_size) past_key/past_value tensor cache, do_rotary/ split-half or interleaved RoPE applied to query/key at the absolute position derived from seqlens_k (or an explicit position_ids), and observable present_key/present_value outputs holding the concatenation of the past cache with the (rotated) current step. local_window_size is -1 (full attention) or a positive INT32 token count including the current token, and requires causal attention. It restricts attended keys without truncating the cache. Optional head_sink supplies one virtual softmax logit per query head; it contributes to normalization but not to the value reduction. The attention score/softmax/value reduction itself is delegated to :cpp:class:AttentionKernel (this kernel never duplicates that math); only the RoPE application and the present cache materialization are implemented here.

Public Functions

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::NodeProto &node, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &query, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &key, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &value, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &seqlens_k, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &total_sequence_length, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *past_key = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *past_value = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *cos_cache = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *sin_cache = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *position_ids = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *attention_bias = nullptr, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *present_key = nullptr, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *present_value = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *head_sink = nullptr) const#

Direct invocation used by backend test cases and unit tests.

node carries the operator attributes and is also used to decide, from its declared inputs/outputs, which optional tensors participate (matching :cpp:func:Run’s reading of rt.tensors()): a non-null optional tensor pointer is only honored when the corresponding node input is wired (non-empty name), and present_key/present_value are only populated when node declares the corresponding output.

class InstanceNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class LayerNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class LinearAttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class LogKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

SIMD-accelerated onnx-light kernel for the ONNX Log operator.

LogKernel mirrors :cpp:class:ExpKernel, using private SIMD dispatch for float32/float64, a scalar exact FP16 path, and vector conversion blocks for bfloat16.

Public Functions

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s single input, computes the elementwise natural logarithm and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh output tensor and writes log(x) into it.

When rt is non-null its allocator backs the output buffer.

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Writes log(x) into the caller-supplied output tensor, whose data_type, shape and buffer size must already match x.

Public Static Attributes

static constexpr const char *kName = "onnx_light_cpu::Log"#: Library-qualified name identifying this kernel, recorded through :cpp:func:RuntimeContext::RecordKernelUsage on every :cpp:func:Run when enabled so callers can tell this kernel apart from onnx-light’s built-in Log.

class LpNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class MatMulIntegerKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class MatMulKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: SIMD-accelerated kernel for the ONNX MatMul operator.

class MatMulNBitsKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class MeanVarianceNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class MicrosoftLinearAttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class MoEKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class MultiHeadAttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class NaiveBiasGeluKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: Readable scalar correctness oracle for com.microsoft::BiasGelu.

class NaiveCDistKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: Readable scalar correctness oracle for com.microsoft::CDist.

class NaiveCausalConvWithStateKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class NaiveGroupQueryAttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Independent, scalar reference CPU kernel for com.microsoft::GroupQueryAttention.

Implements the exact same contract as :cpp:class:GroupQueryAttentionKernel (rank-3 query/key/value, an optional rank-4 (batch, kv_num_heads, past_sequence_length, head_size) past_key/past_value tensor cache, do_rotary/ split-half or interleaved RoPE applied to query/key at the absolute position derived from seqlens_k (or an explicit position_ids), an optional attention_bias additive mask, a per-query-head head_sink virtual softmax logit, and observable present_key/present_value outputs) but computes every step – RoPE, the score/softmax/value reduction, and the KV-cache concatenation – with plain, readable scalar loops. It never calls :cpp:class:AttentionKernel or any other optimized attention compute helper, so it is a self-contained reference implementation intended for differential testing against the optimized kernel.

Public Functions

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::NodeProto &node, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &query, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &key, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &value, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &seqlens_k, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &total_sequence_length, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *past_key = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *past_value = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *cos_cache = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *sin_cache = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *position_ids = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *attention_bias = nullptr, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *present_key = nullptr, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *present_value = nullptr, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *head_sink = nullptr) const#

Direct invocation used by unit tests.

node carries the operator attributes and is also used to decide, from its declared inputs/outputs, which optional tensors participate (matching :cpp:func:Run’s reading of rt.tensors()): a non-null optional tensor pointer is only honored when the corresponding node input is wired (non-empty name), and present_key/present_value are only populated when node declares the corresponding output.

class NaiveMicrosoftLinearAttentionKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: Test-only Microsoft adapter to the concrete onnx-light reference kernel.

class NaiveSkipSimplifiedLayerNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: Provides an independent scalar implementation of the fused residual RMS normalization.

class NonZeroKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: BOOL NonZero with a count-sized INT64 output in row-major index order.

class NotKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

SIMD-accelerated onnx-light kernel for the ONNX Not operator.

NotKernel derives from onnx-light’s :cpp:class:onnx_light::core::runtime::KernelBase so it plugs into the runtime exactly like a built-in kernel: the dispatch table constructs it once per node and calls :cpp:func:Run on every execution. The computation uses private SIMD dispatch. ONNX Not only accepts bool inputs, so the kernel is a full drop-in replacement for the built-in one.

Public Functions

void Run(ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext &rt) override#: Reads the node’s single input, computes the elementwise logical negation and stores the single output back into rt.

ONNX_LIGHT_NAMESPACE::core::runtime::Tensor operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Allocates a fresh output tensor and writes !x into it.

When rt is non-null its allocator backs the output buffer.

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &x, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Writes !x into the caller-supplied output tensor, whose data_type, shape and buffer size must already match x.

Public Static Attributes

static constexpr const char *kName = "onnx_light_cpu::Not"#: Library-qualified name identifying this kernel, recorded through :cpp:func:RuntimeContext::RecordKernelUsage on every :cpp:func:Run when enabled so callers can tell this kernel apart from onnx-light’s built-in Not.

class QLinearMatMulKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class QMoEKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class ReduceSumKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: CPU ReduceSum implementation for INT64, preserving onnx-light’s other types.

class RmsNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class RotaryEmbeddingKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class ScatterNDKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

ScatterND replacement (opset 11+), with INT32 indices as an extension.

Duplicate destinations are updated in tuple order (last tuple wins).

Public Functions

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &data, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &indices, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &updates, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Output must not overlap any input. Validation precedes all writes.

class SigmoidKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class SimplifiedLayerNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: CPU implementation of the experimental ai.onnx operator used by Qwen exports.

class SkipLayerNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: Computes fused residual addition and last-axis layer normalization.

class SkipSimplifiedLayerNormalizationKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Computes fused residual addition and RMS normalization over the last dimension.

Accepts FLOAT, FLOAT16, and BFLOAT16 tensors with rank 2 or 3. Arithmetic uses FLOAT until the final output conversion, including residual addition for the narrow types. Optional outputs retain their ONNX slots: mean (1, zero), inverse RMS (2), and sum (3).

Public Functions

SkipSimplifiedLayerNormalizationResult operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &input, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &skip, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &gamma, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *bias = nullptr, float epsilon = 1.0e-12F, bool output_sum = false, bool output_mean = false, bool output_inv_std_var = false, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#: Returns the normalized output and any requested FLOAT statistics or typed residual sum.

class SliceKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

ONNX Slice with modern tensor inputs and legacy opset 1-9 node attributes.

Public Functions

void operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &data, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &starts, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &ends, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *axes, const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor *steps, ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &output) const#: Output must have the expected dtype, shape, byte size and not overlap any input.

class SoftmaxKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class SplitKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

Standard ONNX Split for all supported fixed-byte tensor types.

Public Functions

std::vector<ONNX_LIGHT_NAMESPACE::core::runtime::Tensor> operator()(const ONNX_LIGHT_NAMESPACE::core::runtime::Tensor &data, int64_t axis, std::span<const int64_t> split, int64_t num_outputs, ONNX_LIGHT_NAMESPACE::core::runtime::RuntimeContext *rt = nullptr) const#

Exactly one of nonempty split sizes or positive num_outputs is required.

num_outputs uses Split-18 ceil partitioning with a smaller final chunk. Outputs always own storage and preserve the input’s bytes verbatim.

class SwiGLUKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class TanhKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

class TreeEnsembleKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#: Runtime kernel adapter for ai.onnx.ml::TreeEnsemble version 5.

class VariadicElementwiseKernel : public ONNX_LIGHT_NAMESPACE::core::runtime::KernelBase#

struct BatchNormalizationResult#

struct LayerNormalizationResult#

struct LinearAttentionResult#

struct SimplifiedLayerNormalizationResult#

struct SkipLayerNormalizationResult#

struct SkipSimplifiedLayerNormalizationResult#

Kernel implementation notes#

GroupQueryAttention#

Both optimized and naive com.microsoft::GroupQueryAttention kernels support full and partial rotary embeddings in split-half and interleaved layouts for FP32, FP16, and BF16. Matching rank-2 cosine/sine caches determine rotary_dim = 2 * cache.shape[1], which must be positive and no larger than the head size. Only this even-sized prefix of Q/K is rotated; the suffix and previously cached keys remain bitwise unchanged, including during decode.

MatMulNBits#

The com.microsoft::MatMulNBits-1 CPU implementation targets the Qwen2/Qwen3 weight layout with matching FLOAT, FLOAT16, or BFLOAT16 activations, scales, optional bias, and output. Packed UINT8 storage holds 2-bit, 4-bit, or 8-bit weights with power-of-two block_size in [16, UINT32_MAX], including 128-element ternary groups. The implicit midpoint zero point is 2, 8, or 128 respectively; explicit packed UINT8 or matching floating-point zero points are also supported. Scales and zero points are indexed by quantization group, independently of the 32-element compute panels. It consumes the packed weights directly without allocating or expanding a floating-point weight matrix. accuracy_level 0 and 4 are accepted. g_idx, prepacked provider-specific layouts, invalid block sizes, mixed floating-point types, and DOUBLE are rejected. The separate encoded INT3 layout requires block size 32. INT4/INT8 optimized preparation is used only for block size 32 with implicit zero points; all other supported groups use the bounded-panel path, including K tails and optional bias.

With onnx-light’s declarative preparation API, MatMulNBits declares its immutable sources without reading their payloads. The session schedules packing after those sources become available and accounts for source, temporary and prepared bytes before execution. RuntimeSession::Prepare can perform this work before inference. This declarative preparation API is required. Tuning uses the activation input A’s type, not the packed UINT8 weights. If A has no declared type and is not yet supplied, tuning and preparation wait until A becomes available. Preparation metadata identifies the immutable sources and packing geometry, independently of node output names. Nodes sharing the same weights and scales reuse the prepared storage.

Gather#

Gather copies fixed-width elements without numerical conversion, accepts INT32 or INT64 indices, and handles scalar/multidimensional indices, negative axes and indices, and empty outputs. Outputs below 192 KiB remain serial. Between 192 KiB and 1 MiB, gathers use the runtime executor with at least 64 KiB per work block and at most eight participants, bounded by the available threads. Larger outputs retain 256 KiB work blocks, and nested calls remain serial. In particular, tail-sized outputs just below 1 MiB no longer fall back to a single worker. As in onnx-light’s built-in Gather, strings, complex values, and packed sub-byte types are not supported.

NonZero#

NonZero supports BOOL inputs from opset 9 onwards, including the Equal -> NonZero -> Transpose token-position pattern. It produces INT64 coordinates in deterministic row-major order with shape [rank, count]. The count stays symbolic during shape inference and is recomputed for each invocation before checked output allocation, without worst-case index scratch storage. Empty dimensions and all-zero inputs produce zero columns. Other input types retain onnx-light’s built-in NonZero implementation.

Scalar inputs follow the ONNX specification: [0, 0] for false and [0, 1] for true. ONNX Runtime 1.30 instead returns [1, count] for scalars; ranked BOOL outputs match ONNX Runtime.

ScatterND#

ScatterND implements replacement semantics from opset 11 onwards. The reduction attribute (introduced in opset 16) must be absent or "none"; add, mul, min, max, and unknown reductions are explicitly rejected, including for empty updates. Payloads use the same fixed-width types as Gather, including FP32, FP16, and BF16, without numerical conversion. INT64 indices follow ONNX; INT32 indices are an onnx-light-cpu extension and must be cast to INT64 for portable ONNX graphs.

Index tuples address scalars or contiguous trailing slices. Negative coordinates are normalized per dimension, and all shapes, buffer sizes, and bounds are validated before writing. The input is copied exactly once into independent output storage, then only selected blocks are replaced. Empty updates still produce an independent copy. Preallocated outputs must not overlap any input, even partially; inputs may share storage with each other.

ONNX’s replacement contract says indices should not contain duplicate destinations, because update order is unspecified. This implementation processes tuples in row-major order, with the last tuple winning (including equivalent positive and negative indices). Do not depend on this ordering when moving graphs between runtimes; duplicate parity tests use identical updates. The initial copy can use the runtime executor, but replacement writes are serial to avoid duplicate-index races.

The embedding fixture combines Gather -> Equal -> NonZero -> Transpose -> ScatterND with dynamic [visual_tokens, 1] positions and [visual_tokens, 6656] vision features, including zero-image inputs.

AVX-512CD conflict-detection evaluation (2026-10-03)#

ScatterND is the only registered indexed-destination writer; Gather reads indices but writes contiguous output, and there is no histogram or ScatterElements kernel. For reduction="none", ONNX says duplicate destinations should not occur and does not define their update order. The local last-tuple-wins guarantee above nevertheless permits a narrow experiment: for each group of eight normalized, four-byte scalar destinations, reverse the offsets, use VPCONFLICTQ to retain only each destination’s final lane, and mask-scatter the retained values. Groups remain ordered, and the scalar tail runs afterwards. Distinct destination bytes do not overlap; trailing slices wider than four bytes still use the ordered memcpy loop. This does not implement reduction modes.

The standalone, non-dispatched prototype is tools/scatter_nd_conflict_probe.cc. On an Intel Xeon Platinum 8573C with AVX-512CD, GCC 13.3, one pinned CPU and the flags shown below, it checked bitwise parity before measuring. Each trial copies 65,536 INT32 elements and writes either 8,192 or 65,536 INT32 updates using already prepared byte offsets. Nine alternating-path runs of 100 trials each give these median seconds per copy plus update (one thread):

Updates	Destinations	Ordered scalar (s)	AVX-512CD (s)	AVX-512CD / scalar
8,192	unique	0.000008138	0.000011916	1.46
8,192	clustered (16)	0.000008570	0.000012376	1.44
8,192	all one destination	0.000008667	0.000012556	1.45
65,536	unique	0.000030537	0.000063837	2.09
65,536	clustered (16)	0.000029877	0.000063727	2.13
65,536	all one destination	0.000030799	0.000065988	2.14

Reproduce on an AVX-512CD host (choose an allowed CPU for affinity):

g++ -O3 -std=c++20 -mavx512f -mavx512cd -mavx2 \
    tools/scatter_nd_conflict_probe.cc -o /tmp/scatter_cd
taskset -c <allowed-core> /tmp/scatter_cd

C++ parity coverage in test_onnx_light_scatter_nd_kernel.cc also exercises unique, clustered and repeated destinations, negative aliases, both index widths, and counts around eight-lane boundaries against the built-in kernel and the local last-wins contract. Because every measured case regressed, no AVX-512CD compiler or runtime dispatch is added; the existing scalar replacement path remains.

Cast#

Cast preserves the input shape while converting its element type. Common numeric conversions use typed loops and the runtime executor for large tensors; integer-to-integer casts do not pass through floating point. This includes the INT64 to INT32 sequence-length conversions used by Qwen3. Outputs own their storage, including identity casts. String, float8 and packed low-precision conversions retain the built-in onnx-light compatibility path, including the saturate attribute. The built-in restrictions on extended conversion pairs still apply. For floating-to-integer values outside the defined ONNX conversion range, the numeric path provides deterministic behavior: NaN becomes zero and values outside the integer intermediate range clamp to the destination bounds. Representable INT64 intermediates retain modular narrowing; UINT64 destinations clamp to their own range.

Slice#

Slice supports tensor parameters from opset 10 onwards and the legacy attribute form. It handles optional axes and steps, clipped bounds, negative steps, empty outputs, and the same fixed-width data types as Gather. Contiguous trailing dimensions are copied together; strided copies use bounded-rank coordinates and the runtime executor for large outputs.

Concat#

Concat accepts one or more equal-rank tensors, including empty inputs and negative axes. It preserves fixed-width element bytes and validates matching non-axis dimensions, output sizes and non-overlap. Large concatenations use runtime-owned byte tiles, including concatenations along axis zero.

Split#

Split supports explicit sizes (legacy attributes or an INT64 tensor), implicit equal partitions, and the opset-18 num_outputs form with a smaller final partition. Negative axes and zero-length partitions are supported for the same fixed-width data types as Gather. Outputs own their storage, including last-axis QKV partitions across multiple tokens. Large copies use the runtime executor; small splits remain serial.

RotaryEmbedding#

ai.onnx::RotaryEmbedding (opset 23+) accepts rank-3 or rank-4 input and either per-token rank-3 cosine/sine caches or rank-2 caches with INT64 position indices. com.microsoft::RotaryEmbedding (opset 1+) uses the order input, position_ids, cos_cache, sin_cache and accepts a scalar start position. Both support split-half/interleaved and partial rotation for FLOAT, FLOAT16, and BFLOAT16. Packed batching and dynamic cache extension are not supported. The Microsoft form also accepts full-head-width caches for partial rotation.

SimplifiedLayerNormalization#

The experimental default-domain operator normalizes the suffix beginning at axis and broadcasts Scale to the entire input. Input and Scale may have independent floating-point types; Y uses Scale’s type. Optional inverse statistics use stash_type (FLOAT or DOUBLE). Unlike RMSNormalization, low-precision values are rounded only after scaling, and DOUBLE inputs retain double arithmetic. The FP32 suffix path reuses the SIMD RMS engine and can save statistics without a second reduction.