Recent Kernel Performance Improvements#

Date:

2026-08

Updated:

2026-09-02

complete (August 25–September 1 optimization sprint)

Objective#

The objective was to improve the kernels exercised by the expanded backend benchmark corpus before adding more operators. The work targeted measured hot paths across unary and binary elementwise operations, matrix multiplication, Attention, normalization, and TreeEnsemble while preserving the existing ONNX semantics and runtime registration contract.

This completed step records the cross-cutting optimization sprint. It does not replace the dedicated roadmaps or declare every final ONNX Runtime parity gate complete; remaining measured gaps continue in their operator-specific plans.

Delivered work#

Area

Contribution

Pull requests

Unary kernels

Reworked Abs, Exp, and Log execution and tuning, added vectorized paths and tails, refined thresholds from backend measurements, and added the direct ONNX Runtime differential matrix.

#442, #459, #460, #556

Binary kernels

Specialized broadcast plans and arithmetic dispatch, expanded bulk execution and benchmark coverage, and optimized FP16/BF16 Pow for contiguous, scalar-broadcast, and mixed-exponent signatures. Later passes unrolled AVX-512 arithmetic and removed repeated integer Div validation over broadcast expansions.

#443– #446, #475, #563, #577

GEMM and MatMul

Improved blocking, packing, scheduling, and benchmark coverage; added specialized square and large-K float64 paths plus native half and skinny integer MatMul execution. The follow-up added medium AVX-512 tiles, cached MatMul plans, fused bias, productive scheduling, and worker-local dynamic-B packing.

#447, #467, #468, #473, #560, #561, #566, #567, #575

Higher-level kernels

Applied shared execution improvements across registered kernels, streamlined TreeEnsemble traversal, shared SIMD normalization primitives, and widened FP16 Attention into the optimized streaming and tiled paths. The latest merged passes add realistic Qwen3.6 normalization and Attention cases.

#451, #463, #466, #472, #558, #559, #569, #578, #579

com.microsoft kernels

Reduced optimized BiasGelu and CDist latency and added focused ONNX Runtime parity benchmarks without changing their operator contracts.

#562, #564

Validation and completion#

Each optimization retained a portable fallback and added focused coverage for the affected data types, vector tails, broadcast or matrix shapes, and runtime dispatch. The backend benchmarks supplied reproducible filters, warmups, repeats, raw timings, and ONNX Runtime comparisons for the changed paths.

The step is complete because all listed implementations and their correctness coverage are merged. Dedicated parity roadmaps remain authoritative where a kernel family still has an explicit minimum-speedup or cross-machine gate.