Next Steps

Performance roadmaps for the kernels that onnx-light-cpu plans to optimize next.

Started

Plan	Contribution
AVX2 performance	Uses the explicit AVX2 SIMD ceiling to measure and rank the remaining gaps below the completed AVX-512 paths before optimizing matrix, Attention, activation, normalization, unary, and binary workloads. The isolated-runtime diagnostic baseline identifies matrix and Attention priorities and a thread-pool interference bias in the existing runner. The matrix kernel follow-up delivers measured FP16 packing and FP16/FP32 single-row improvements; full ORT parity remains open.
AVX2 activation and normalization gap closure	Extends the #604 AVX2 activation work (AVX2 PR04a): measured Sigmoid, Softmax, BiasGelu, and RMSNormalization against ONNX Runtime and closed the one found gap, a serialized Float16 RMSNormalization reduction; the full onnx-light backend-corpus acceptance run remains.
AVX2 Gemm and MatMul gap closure	Measures the AVX2-ceiling FP32/FP64 Gemm/MatMul corpus from #633 (AVX2 PR02a); dedicated AVX2-only hardware and an onnx-light checkout remain required for a genuine AVX2-vs-AVX2 ONNX Runtime parity gate.

Planned

Plan	Contribution
Qwen3 inference	Freezes the audited end-to-end model benchmark, then builds the canonical batch-1 INT4 and persistent-decode path on top of the delivered GQA, RMSNormalization, and Sigmoid primitives.
Qwen3 missing kernels	Splits the remaining INT4 projections, input/layout kernels, and normalization adapters into independent PRs, with runtime metadata and persistent-cache ownership kept in onnx-light.
Qwen3 non-MatMulNBits operators	Implements the metadata, input, layout, activation, and normalization slice required by the audited Qwen3 graph independently of packed INT4 projections; Sigmoid and the shared RMS engine are already available.

Completed

2026

Plan	Contribution
Exp and Log parity	Brings the transcendental kernels to their published ONNX Runtime parity and numerical gates.
Gemm and MatMul	Delivers the shared matrix engine, typed and compact paths, scheduling, packing, AVX-512 tuning, and the corrective work through #605 and #608.
Attention	Delivers materialized and bounded-memory streaming Attention, including the AVX-512 optimization pass and the AVX2 decode and scheduling foundations through #599, #605, and #608.
Binary elementwise performance	Delivers prepared traversal, typed bulk execution, AVX-512 arithmetic, low-precision comparison, integer validation, and dispatch optimizations through #599.
Binary elementwise	Supplies the shared prepared broadcast engine, generated correctness corpus, and parity gate for all 19 registered binary operators.
TreeEnsemble	Provides the typed in-place v5 engine, compact prepared storage, and final correctness, parity, and memory gates, with the large-batch row-parallel follow-up tracked in #580.
Runtime execution controls	Supplies the typed session policy and shared executor used by every registered kernel.
Processor performance profile	Measures cache and RAM bandwidth and latency plus sustained arithmetic throughput through one explicit Python API.
Registered kernel documentation	Derives the Python inventory and generated reference directly from the actual C++ registrations.
Recent kernel performance improvements	Consolidates the August sprint and the August 30–September 2 follow-up across unary, binary, matrix, Attention, normalization, com.microsoft, and TreeEnsemble kernels.
com.microsoft domain support	Introduces CDist and BiasGelu end to end, including schemas, runtime kernels, gradients, fusion patterns, tests, documentation, and the latency follow-ups in #562 and #564.

Discussion

Plan	Contribution
Conv	Defines prepared convolution plans and specialized algorithms that avoid a universal materialized im2col path.
Unary elementwise	Unifies scalar, SIMD, traversal, and scheduling implementations across unary operators.
Processor-aware tuning	Defines calibration and persistence of processor-specific elementwise thresholds.
SVM	Defines prepared SVM classification and regression kernels.
Persistent KV cache	Defines mutable, paged, and optionally quantized cache storage that avoids copying complete past K/V tensors during decode.