Operator kernels and the C++ backend tests#

Date:

2026-06

complete

Objective#

onnx ships a large corpus of backend tests: one reference model per operator variant together with the input and expected output tensors. Upstream they live in Python (onnx/backend/test/case) and are consumed by a Python runtime. onnx-light needs the same coverage, but from C++ and without a Python dependency, so that the native runtime can be validated in every build (including builds where Python is disabled).

The objective was twofold:

  • provide a native execution engine — a kernel per operator plus a dispatch table that resolves a NodeProto to the code that runs it;

  • provide a native backend-test corpus so those kernels are exercised against known-good input/output pairs directly from the C++ test binary.

Post-mortem#

The two concerns were kept in separate libraries so the runtime core never depends on any particular kernel, and so the test corpus can be built or skipped independently.

Two libraries, one dispatch table#

The runtime core (lib_onnx_core) owns the mechanism — the core::runtime::KernelDispatchTable and the KernelBase interface — but knows nothing about individual operators. The operator implementations live in lib_onnx_kernels (onnx_light/onnx_extensions/kernels). A single entry point, onnx_kernels::RegisterKernelFunctions(), populates the dispatch table with a factory for every built-in operator, keyed by "<domain>:<op>" (for example "ai.onnx:Add"). Registration is idempotent, so linking the kernels does not force them to be registered until the caller asks for it.

Each kernel derives from KernelBase and follows the same shape:

class Add : public KernelBase {
  void Run(RuntimeContext &rt) override;      // dispatch entry point
  Tensor operator()(const Tensor &x, const Tensor &y, ...) const;
  void operator()(const Tensor &x, const Tensor &y, Tensor &out) const;
  static constexpr bool CanRunInPlace() noexcept { return true; }
};

Run reads the node inputs from the RuntimeContext, calls the typed operator(), and writes the outputs back. The two operator() overloads — one that allocates and returns, one that fills a pre-allocated tensor — let the same code serve both the allocating path used by tests and the in-place path the execution plan prefers when CanRunInPlace() allows an output to alias an input. Kernels are grouped by domain family under kernels/kernels/{math,nn,tensor,reduction,logical,...}, each with an include_*.h declaration header and one kernel_*.cc per operator. Shared attribute parsing and tensor helpers live in kernel_run_helpers.h to avoid per-kernel duplication.

The backend-test corpus#

The C++ corpus mirrors the upstream Python cases but stays lazy. A core::backend_test::TestCase stores metadata (name, tag, tolerances, declared input/output element counts) plus a std::function that builds the single-node ModelProto and its data sets only when the case is actually run. Laziness matters because the benchmark variants declare millions of elements; materializing every case up front would be wasteful. The Expect helper captures node, inputs and outputs into that lambda and appends the case to a per-category registry.

Cases are organized to match the kernels: backend_test/cases/{math,nn,tensor,...} hold the per-operator numeric cases, while cases_for_shapes (constant, empty-shape, inference, in-place, peak-memory, release, shape-tag), cases_numerical (NaN/Inf) and cases_runtime (model-local functions) cover the structural and edge-case behavior that is not tied to a single operator. collect_test_cases.cc registers every category collector through a static initializer and exposes CollectTestCases / CollectTestCasesByName / GetTestCaseByName plus a TestMode (TEST vs BENCHMARK) so the same definitions serve both correctness and performance runs.

What worked#

  • Keeping the dispatch mechanism in the runtime core and the kernels in a separate library meant the core could be built, tested, and reused without pulling in hundreds of operator implementations.

  • The lazy TestCase made a very large corpus cheap to enumerate: filtering by operator or by the _big_ benchmark flag never materializes the models it skips.

  • Sharing one node definition between the allocating and in-place operator() overloads meant the numeric cases and the memory-reuse execution plan validated the same kernel code.

  • Reusing upstream ONNX’s reference outputs (transcribed into C++ cases) kept the native runtime honest against a well-known baseline.

Kernels on and off in CI#

ONNX_LIGHT_BUILD_KERNELS gates both lib_onnx_kernels and lib_onnx_backend_test. .github/workflows/ci_core.yml builds both variants: a fast KERNELS=OFF preflight that exercises shape inference and the runtime core without any operator code, and a full KERNELS=ON job that links the real kernels and runs the backend corpus through the ctest / test_onnx_light binary. Tests that assert a kernel is absent therefore have to GTEST_SKIP when the kernels are linked in.

What remains#

Coverage tracks the operators onnx-light implements, not the entire ONNX opset; unimplemented operators have neither a kernel nor a case. Benchmark cases share the correctness definitions but are only run on demand, so performance regressions are not part of the default matrix.

See also#