Gemm and MatMul Performance Roadmap#
- Date:
2026-08
- Updated:
2026-09-05
complete
PR #307 delivered the shared MatMul kernel and the original parity corpus. The correctness foundations are complete, including the FP16/BF16, integer, compact-format, and tuning paths. The default-policy corrective sequence is also complete through #575, with the later AVX2 scheduling and masked-tail follow-ups delivered by #605 and #608. Further AVX2 parity work is tracked by the AVX2 performance follow-up.
Current status#
The original roadmap did not validate the primary end-to-end configuration.
tools/benchmark_gemm_parity.py defaulted to one thread and passed the same
explicit count to both runtimes. Those controlled measurements remain useful
diagnostics, but they did not test the autonomous execution policies used by
the published dashboard. The larger corpus revealed that float32 is close to
ONNX Runtime on one thread but scales substantially less on representative
multi-core shapes.
Gate |
Delivered scope |
Status |
|---|---|---|
Default-policy float32 profiling, scaling correction, parity-runner semantics, and this roadmap correction. |
Closed by the implementation sequence through #575; dedicated-machine parity remains optional validation. |
|
FP16/BF16 kernels, tuning, correctness coverage, and isolated benchmark reports. |
Closed; further dedicated x86/ARM measurements are optional validation. |
|
Integer and compact kernels, correctness coverage, tuning, and raw isolated benchmark reports. |
Closed; further dedicated x86/ARM measurements are optional validation. |
|
Complete Gemm/MatMul corpus and an |
Closed; its original controlled-thread gate is delivered, but #374 adds the missing default-policy gate. |
Objective#
The objective is performance parity with the ONNX Runtime CPU execution
provider for the important Gemm and MatMul workloads, for every
supported data type, without sacrificing ONNX correctness.
For GEMM, parity means a corpus median speed-up of at least 1.0x versus
ONNX Runtime and no priority shape below 0.9x. This is a catch-up effort
with standard ONNX tensors and semantics.
The current implementation is a correctness-first, register-blocked kernel
with AVX2/AVX-512 paths, K blocking, A/B packing, a task-aware M x N scheduler,
batch scheduling, packed SIMD split-K, and typed broadcast/fused epilogues; see
the current design. Roadmap PR01 closed
the scheduler under-utilization identified on multi-panel shapes, and Roadmap
PR02 removed expanded bias temporaries. The FP32 investigation in
onnx-light-cpu #162 shows that scalar
skinny-N, weak GEMV/skinny-M, an operator path that bypasses GemmPlan, and
untuned Zen/generic-x86 blocking still prevent parity. The remaining roadmap
work first closes those measured FP32/FP64 gaps, then covers low-precision
kernels and the final ONNX Runtime parity gates.
Default-policy profiling on the 96-core development host showed that square FP32 GEMM is limited by excessive participation rather than single-core compute: the 1024-square case reaches parity with a 32-thread executor but regresses when all 96 participants are admitted. General GEMM plans therefore size participation from the output tile count (about 32K output elements per participant); the existing wide-projection profile remains separate because its larger N dimension scales profitably to more workers.
The backend corpus also includes three-node GEMM graphs, not only isolated operators. Correctness cases cover rectangular, widen-then-narrow, and alternating dimensions. Benchmark cases cover square/projection, transformer-projection, and alternating large shapes so changes to executor pool reuse are measured across consecutive GEMMs with different participant limits.
Scope and type matrix#
Gemm and MatMul should share one matrix-multiplication engine.
The operator adapters retain distinct ONNX semantics:
Gemmhandles rank-2 inputs,alpha,beta, optional broadcast bias, andtransA/transB.MatMulhandles vectors, matrices, arbitrary leading batch dimensions, NumPy-style batch broadcasting, and output-rank squeezing.
The implementation must follow the type constraints of the selected ONNX
opset. Integer and quantized multiplication may be exposed through MatMul,
MatMulInteger or QLinearMatMul rather than forcing unsupported types
through Gemm.
Type |
Accumulation |
Preferred kernel |
Fallback |
|---|---|---|---|
|
|
AVX2+FMA, AVX-512F, NEON/SVE |
Portable blocked scalar kernel |
|
|
AVX2+FMA, AVX-512F, NEON/SVE |
Portable blocked scalar kernel |
|
Normally |
AVX-512FP16 or convert-and-FMA during packing |
F16C/NEON conversion into packed |
|
Normally |
AVX-512BF16, AMX-BF16, or ARM BF16 |
Convert during packing into |
|
|
Hardware-specific tensor/dot-product path |
Vectorized decode during packing |
|
|
AVX-VNNI/AVX-512VNNI, AMX-INT8, NEON dot product |
Widening integer micro-kernel |
|
Schema-defined integer result |
Vectorized integer multiply/add where profitable |
Portable exact-arithmetic path |
Packed |
|
Unpack-and-dot kernel or AMX/vendor extension |
Vectorized unpack into an |
Benchmark contract#
Optimization must begin with a reproducible benchmark. End-to-end runtime measurements and isolated kernel measurements answer different questions and must both be retained.
Use identical tensors, transposition flags, and correctness tolerances for MLAS and
onnx-light-cpu.The primary end-to-end parity run leaves thread selection and affinity to each runtime. Record resolved information only when the runtime exposes it; do not infer an ONNX Runtime thread count.
Explicit 1-, 2-, 4-, physical-core, and logical-core runs are controlled scaling diagnostics, not substitutes for the default-policy parity gate.
Isolate backend session lifetimes so an idle thread pool cannot perturb the other runtime. Warm up every candidate, alternate backend order between cases, and report median and dispersion rather than the best observation.
Run on an otherwise idle, pinned machine with a fixed power policy. Record CPU model, cache sizes, ISA features, compiler, and build flags.
Measure packing, the blocked multiplication, and low-precision conversion separately. The end-to-end number must still include every cost visible to a caller.
Cover tiny matrices, square matrices, skinny M, skinny N, large K, batched MatMul, broadcast batches, every transpose combination, and transformer projection shapes.
Separate dynamic-B from constant-B cases. Dynamic inputs can only be packed at execution time. An initializer may be prepacked during session preparation once the runtime has explicitly identified it as constant.
Compare controlled single-thread throughput and scaling at 2, 4, physical-core, and logical-core thread counts. Hybrid P/E-core machines need their own results.
Four committed instruments implement this contract. tools/benchmark_gemm_parity.py
is the end-to-end floating-point parity runner (PR06.0/PR10.3): it alternates
the registered operator against ONNX Runtime and reports GFLOP/s and speed-up
per priority shape. tools/gemm_throughput.cc is its isolated GemmPlan
counterpart, built with -DONNX_LIGHT_CPU_BUILD_BENCHMARKS=ON.
tools/benchmark_integer_gemm_parity.py applies the same alternating,
raw-sample contract to UINT8 x INT8 MatMulInteger. The opt-in
tools/compact_gemm_throughput.cc driver publishes isolated INT8, packed
INT4, E4M3, and E5M2 throughput for the same tiny, direct, square, skinny,
large-K, and transformer shape families. Together the end-to-end and isolated
numbers distinguish operator overhead from packing and micro-kernel limits.
Target computation algorithm#
Choosing an ISA-specific function once instead of once per call is useful, but it cannot explain or close a 10x throughput gap. The central change must be the matrix-multiplication algorithm: the order in which panels move through the cache hierarchy and are reused by the arithmetic micro-kernel.
The general dense path should follow the five-loop GotoBLAS/BLIS decomposition.
For C = A @ B, with MC, NC and KC sized for the cache hierarchy
and MR x NR sized for the vector registers:
for jc in range(0, N, NC): # L3-sized columns of C and B
for pc in range(0, K, KC): # reduction panel
Bc = pack(B[pc:pc+KC, jc:jc+NC]) # packed once
for ic in range(0, M, MC): # L2-sized rows of A and C
Ac = pack(A[ic:ic+MC, pc:pc+KC]) # reused across all NR panels
for jr in range(0, NC, NR):
for ir in range(0, MC, MR):
microkernel(Ac[ir:], Bc[:, jr:], C[ic+ir:, jc+jr:])
This order is important:
one packed B
KC x NCpanel is reused by everyMCrow block;one packed A
MC x KCpanel is reused by everyNRcolumn micro-panel;the
MR x NRC tile remains in registers for the completeKCreduction;the working sets deliberately move from L3 (
NC) to L2 (MC/KC), then L1 and registers, instead of relying on one fixed 64 x 256 tile;transposition is resolved while packing, so the arithmetic loop sees only contiguous canonical panels.
The current five-loop engine packs one B panel for a column/K block and shares it across its row panels. It then parallelizes either column panels or row panels, not their Cartesian product. Its task granularity is therefore tied to MC/NC: a large cache-derived NC can leave only one column task, while a large MC can leave only a few row tasks.
One algorithm is not optimal for every shape. The plan must choose among distinct computational algorithms:
Shape |
Algorithm |
|---|---|
General M x N x K |
Five-loop packed GEMM with hierarchical MC/NC/KC blocking. |
Tiny matrices |
Direct, un-packed micro-kernel; packing costs more than the arithmetic. |
|
GEMV/skinny-M kernel that streams B once and vectorizes across N. |
|
Dot/GEMV kernel that vectorizes the K reduction and avoids B-panel packing. |
Small K |
Outer-product or direct kernel with wide N tiles and no KC loop. |
Batched MatMul |
Batch outer loop for small independent products; merge batch with M/N task dimensions when one product cannot occupy all cores. |
Constant B |
Plan-owned B prepacked once when shape and layout are stable, with the original tensor representation retained only when required by a guarded fallback. |
Extremely large K with small M/N |
Split K only when M/N/batch parallelism is insufficient, then combine partial accumulators in a controlled reduction. |
Planning once, computing many times#
Kernel selection belongs in session preparation. A GemmPlan/MatMulPlan
should be created after shapes, types, constant inputs, CPU features, and thread
limits are known. It should contain:
the computational algorithm (general, direct, GEMV, batched, or split-K);
MC/NC/KC and MR/NR;
typed function pointers for packing, micro-kernel, and epilogue;
the parallel decomposition and useful thread count;
plan-owned constant B storage when B is an initializer.
The execution path then invokes the plan directly. This removes repeated selection, but its main benefit is enabling the correct algorithm and data layout to be prepared ahead of time; the branch removal itself is negligible relative to the multiplication.
Phase 1: implement the blocked float32 algorithm#
float32 is the first parity target because it exposes the quality of the
core algorithm without conversion overhead.
Implement the five-loop algorithm as a separate, testable engine.
Correct the loop ownership so each B panel is packed once and shared across its row blocks.
Add direct, GEMV, skinny-M, skinny-N, and small-K algorithms.
Derive MC/NC/KC from measured L1/L2/L3 capacity and associativity, with conservative defaults when cache discovery is unavailable. This is implemented for x86 deterministic CPUID cache descriptors; other platforms currently use the conservative defaults.
Build the immutable execution plan once and benchmark each algorithm both single-threaded and multi-threaded. The benchmark corpus includes a shape-forced case for every algorithm and selects participants through the
onnx-lightsession execution policy.
Phase 2: saturate the floating-point units#
Compile FMA micro-kernels in dedicated translation units and use them only after checking both AVX/AVX2 and FMA. The current default
-mavx2build does not define__FMA__. Dedicated FP32/FP64 AVX2+FMA micro-kernels are now compiled with-mavx2 -mfmaand selected only when CPUID reports FMA; the baseline AVX path remains available on CPUs without FMA.Keep distinct SSE2, AVX2+FMA, AVX-512F, and microarchitecture-specific variants. Zen, Skylake, Ice Lake, and hybrid Intel CPUs can require different MR/NR and cache blocks even when they expose the same ISA.
Generate several MR x NR micro-kernels instead of fixing
MR == 4. AVX2+FMA emits compile-timeMR=1..6variants and AVX-512 emitsMR=1..8, both for NR=1 and NR=2. The detected microarchitecture selects MR=4 for generic AVX2/SSE, MR=5 for modern Intel Core AVX2, MR=6 for AMD Zen AVX2, and MR=6 for AVX-512, and the choice is propagated through cache blocking, packing, algorithm selection, and execution. Per-model tuning within an ISA remains.Unroll K enough to maintain independent FMA chains without spilling accumulators. The AVX2+FMA and AVX-512 FP32/FP64 kernels now reduce four K rows per loop iteration and use a scalar-count remainder loop without adding accumulator registers.
Use aligned panel loads and software prefetch only where hardware-counter measurements show reduced stalls.
Specialize the arithmetic epilogue for
alpha == 1,beta == 0, scalar bias, row/column bias, and no bias. Unitalpha/beta, zerobeta, and no-bias cases now avoid redundant vector/scalar multiplication and bias reads in every x86 micro-kernel and scalar tail; scalar and row/column broadcast bias interfaces remain.ARM64 NEON FP32/FP64 kernels are implemented with six-row, two-vector register tiles and scalar sub-vector tails. SVE/SVE2 use four-row, two-scalable-vector tiles and predicated tails; runtime vector lengths below 256 bits deliberately retain NEON. Linux HWCAP detection and separate SVE compilation keep unsupported processors on the safe fallback.
Parallel execution#
Roadmap PR01 is implemented by onnx-light-cpu #155 and Roadmap PR02 by onnx-light-cpu #156, and Roadmap PR03 by onnx-light-cpu #157, and Roadmap PR04 by onnx-light-cpu #158, and Roadmap PR05 by onnx-light-cpu #159. Roadmap PR06.0 implemented the parity runner in onnx-light-cpu #160, but its measured gate does not pass. Roadmap PR06.1 diagnoses the gap in onnx-light-cpu #162. The remaining P4 implementation work is Roadmap PR06.2 through PR06.6 in the final table; the blocking dedicated-machine parity gate is deliberately last as PR10.5. Constant-B prepacking is now included in PR06.4 because #162 identifies its absence from the operator hot path as part of the measured gap. No performance work demonstrated necessary by the parity corpus may be deferred while the gate remains unmet.
PR06 does not restart P4 or invalidate PR01 through PR05: their correctness, dispatch, scheduling, and architecture tests remain required foundations. The measured fixes proceed in this order:
Vectorize the skinny-N K reduction, including
N == 1, and keep split-K disabled when its partition and reduction costs dominate the tiny output.Add a dedicated GEMV/skinny-M path that streams each B row once and reuses it across output columns.
Route ONNX
GemmandMatMulthrough immutableGemmPlanandMatMulPlaninstances so shape selection and plan-owned constant-B state are prepared once, including persistent packed-B panels for initializer weights.Tune Zen and generic-x86 MR/NR and cache blocking against the complete corpus so 1024³ and larger shapes sustain, rather than lose, throughput.
Rerun the complete FP32/FP64 Gemm, MatMul, and batched corpus on dedicated, frequency-stabilized machines. Publish raw results only when both dtype medians reach
1.0xand every priority case reaches0.9x.
The scheduler decomposes Y = A @ B into a Cartesian grid of row and column
panels:
B (K x N)
+--------+--------+--------+
| B0 | B1 | B2 | NC-wide column panels
+--------+--------+--------+
A (M x K) Y (M x N)
+--------+ +------+------+------+
| A0 |----------->| T00 | T01 | T02 |
+--------+ +------+------+------+
| A1 |----------->| T10 | T11 | T12 |
+--------+ +------+------+------+
| A2 |----------->| T20 | T21 | T22 |
+--------+ +------+------+------+
MC-high independent output zones
row panels
Task T(i,j) multiplies row panel Ai by column panel Bj and writes
only the corresponding, disjoint zone of Y. Column panels are processed in
bounded waves large enough to occupy the available threads. For example, with
six threads, three row panels, and three column panels:
wave 1: B0, B1 -> T00 T10 T20 T01 T11 T21
wave 2: B2 -> T02 T12 T22
For each K chunk, every B panel in the active wave is packed once and shared by
all its row-panel tasks. The tasks accumulate into their zones of Y before
the scheduler advances to the next K chunk. If the complete M x N task grid
still cannot occupy the pool, split-K partitions the reduction dimension:
K = [K0 | K1 | K2]
| | |
v v v
P0 P1 P2 -> Y = alpha * (P0 + P1 + P2) + beta * C
Each partial Pi uses the same packed SIMD micro-kernels. Independent
batches take priority over split-K: when a GEMM already runs inside a parallel
batch region, it executes its M x N grid directly instead of creating nested
K partitions.
Phase 3: native low-precision kernels#
The existing FP16/BF16 path widens complete tensors to float32, calls
GemmFloat32, then narrows the complete result. It is correct but performs
extra full-matrix memory passes.
For AVX2/F16C, load FP16 panels, convert vectors to FP32 while packing, and accumulate with FMA. Narrow only the final output. (Landed: every FP16/BF16 execution path – the general algorithm plus the skinny-M, skinny-N, direct small-K, and split-K paths – converts each element to float32 during packing or reduction, accumulates in float32, and narrows only in the epilogue with no full-tensor widening. ``GemmHalfPlan`` caches the selected algorithm and blocking. Vectorized F16C and AVX2 conversion for contiguous panels is isolated in Roadmap PR07.1; removing the remaining widening paths is Roadmap PR07.2.)
For AVX-512BF16 and AVX-512FP16, add native dot-product/multiply-accumulate kernels with FP32 accumulation where required by the ONNX numerical contract. (First landed: Roadmap PR07.3 adds a native AVX-512FP16 general kernel that keeps both operands in FLOAT16 to the register file, widens each 16-lane ``B`` vector with ``vcvtph2psx``, and accumulates in float32. It is dispatched by ``CpuSupportsAvx512Fp16()`` for non-transposed ``B`` and otherwise keeps the converting float32 path. Roadmap PR07.4 adds the sibling native AVX-512BF16 general kernel that reduces pairs of ``k`` iterations with the ``vdpbf16ps`` dot-product, accumulating in float32; it is dispatched by ``CpuSupportsAvx512Bf16()`` for non-transposed ``B`` and otherwise keeps the converting float32 path.)
Add AMX tile kernels behind OS-enabled tile-state detection. AMX must remain optional because enabling the ISA and configuring tiles have non-trivial per-thread costs. (First landed: Roadmap PR07.5 adds the AMX tile-state lifecycle – ``CpuSupportsAmxTile``/``AmxBf16``/``AmxInt8`` detection, the one-time Linux ``XTILEDATA`` permission request behind ``AmxTileStateAvailable``, the validating ``AmxTileConfig`` builder, and the per-worker ``AmxTileScope`` (``LDTILECFG``/``TILERELEASE``) with a safe no-op fallback – with no GEMM kernel yet; the AMX-BF16 kernel is PR07.6 and AMX-INT8 is PR09.4. Roadmap PR07.6 then adds the native AMX-BF16 GEMM kernel: a ``tdpbf16ps`` (``_tile_dpbf16ps``) tile micro-kernel that reuses the PR07.5 lifecycle, keeps both operands in BFLOAT16 with a VNNI-packed ``B`` tile, and is dispatched ahead of AVX-512BF16 for non-transposed ``B`` when ``CpuSupportsAmxBf16()`` and ``AmxTileStateAvailable()`` both report the ISA; it falls back to AVX-512BF16 or the converting float32 path otherwise.)
Implement equivalent ARM FP16/BF16 and dot-product paths. (First landed: Roadmap PR08.1 vectorizes the FP16/BF16 convert-while-packing panels on ARM with NEON – a ``vmovl_u16`` zero-extend plus 16-bit shift for BFLOAT16 and the ``vcvt_f32_f16`` (``FCVTL``) instruction for FLOAT16, both with an exact scalar tail matching the bit decode and a scalar fallback when the FP16 intrinsics are unavailable. Roadmap PR08.2 then adds the native NEON arithmetic kernels (``GemmMicroKernel_NEON_BF16`` always, ``GemmMicroKernel_NEON_FP16`` when the FP16 intrinsics compile): both keep the operands half-precision to the register file, widen each ``B`` vector on the fly (zero-extend/shift for BFLOAT16, ``FCVTL`` for FLOAT16) and accumulate in float32, and are dispatched from ``GemmHalfPlanned<kGeneral>`` for non-transposed ``B`` ahead of the converting float32 path, which stays the fallback. Roadmap PR08.3 then adds the native SVE arithmetic kernels (``GemmMicroKernel_SVE_BF16`` / ``GemmMicroKernel_SVE_FP16``): they reuse the same drivers, keep the operands half-precision to the register file, widen each ``B`` vector on the fly (``svld1uh_u32`` zero-extend/shift for BFLOAT16, the SVE ``FCVT`` ``svcvt_f32_f16`` for FLOAT16), accumulate in float32, drive the lane count from the runtime vector length and cover the column remainder with an ``svwhilelt`` predicated tail, and are dispatched ahead of NEON when the runtime profile selects SVE (a vector length of at least 256 bits); shorter vectors keep the better-unrolled NEON kernel.)
For INT8, fuse zero-point correction and requantization into packing and the epilogue. Accumulate in INT32 and define overflow behavior through the ONNX operator contract. (First landed: Roadmap PR09.2 adds the x86 VNNI INT8 kernel (native ``vpdpbusd`` path with a portable scalar sibling) behind the shared ``IntegerMatMul2D`` driver for the contiguous rank >= 2 ``MatMulInteger`` case. Roadmap PR09.3 then adds the native ARM NEON dot-product INT8 kernel ``GemmMatMulIntegerNeonDotProd``, dispatched from that same ``IntegerMatMul2D`` entry point. A single unsigned ``UDOT`` reduction serves every signedness combination by folding a signed operand’s ``+128`` bias into its effective zero point and recovering the raw products with per-row / per-column byte-sum corrections, so the INT32 accumulation matches the portable scalar fallback bit for bit modulo 2^32; it is gated on the ``+dotprod`` build flag and the runtime ``CpuSupportsNeonDotProd`` capability, keeping the scalar reduction as the fallback.)
Treat Float8 and packed 4-bit types as separate packing formats, not as branches in the FP32 inner loop. (First landed: Roadmap PR09.5 adds the four ONNX Float8 formats (``E4M3FN``, ``E4M3FNUZ``, ``E5M2``, ``E5M2FNUZ``) as separate packing formats: exact per-format decoders decode each one-byte pattern to float32 while packing – the contiguous copies gather from an exact 256-entry per-format table through an AVX2 ``vgatherdps`` helper with a scalar tail and fallback – reusing the tuned FP32 algorithms with float32 accumulation. Packed 4-bit types are Roadmap PR09.6.)
Phase 4: complete MatMul#
The MatMul adapter should lower every ONNX shape case into a sequence of core GEMM calls without materializing broadcast copies:
Normalize rank-1 inputs to temporary logical dimensions.
Compute broadcasted batch strides, using zero strides for broadcast axes.
Collapse contiguous batch dimensions where possible.
Dispatch independent batches through the same scheduler used by GEMM.
Select dedicated GEMV and dot-product kernels for
M == 1orN == 1.Restore the exact ONNX output rank without copying data.
The adapter and engine require tests for empty dimensions, scalar-like vectors, non-contiguous batch strides, asymmetric broadcasting, transposed packed weights, and every supported type.
How to exceed ONNX Runtime#
Matching MLAS with a generic dynamic GEMM is difficult. Beating it is more
realistic when onnx-light exploits model-level information. The estimates
below are relative to a tuned ONNX Runtime/MLAS run with the same thread count,
not relative to the current onnx-light-cpu implementation. They are targets
to verify, not guarantees.
Optimization |
Expected gain over MLAS |
Conditions and quantitative bound |
Estimated effort |
|---|---|---|---|
Full shape specialization |
2-10% normally; 10-20% on stable skinny or tail-heavy shapes. |
Instantiate loop order, MC/NC/KC, MR/NR, packing format, micro-kernel, and thread decomposition for one exact shape. Removing the dispatch branch alone is expected to save less than 1%; the gain comes from a better algorithm and eliminating generic tail work. |
5-10 days for a bounded shape family. |
Fused epilogues |
5-25% for a compute-heavy GEMM; 1.2-1.8x for small or bandwidth-bound GEMM chains. |
Fusing each following FP32 elementwise operator avoids approximately 8 x M x N bytes of traffic (one read and one write). The upper range requires an epilogue not already fused by ONNX Runtime, such as a project-specific bias + residual + activation + narrowing combination. |
3-7 days per epilogue family. |
Batch fusion |
1.1-1.5x for ordinary batches of small matrices; up to 2-3x for hundreds of tiny or irregular products. |
Useful when one product takes only a few microseconds and dispatch or thread synchronization is a significant fraction of its time. The gain tends to 0% once each individual GEMM already saturates the cores. |
5-10 days. |
Sparse weights |
1.3-2x around 70-80% sparsity; 2-4x around 90% structured sparsity. |
With nonzero density |
10-20 days per sparse format. |
Low-rank weights |
1.5-5x when an exact or accepted approximation has rank
|
Replacing an |
5-10 days for exact factors; model work is additional for approximation. |
Model-specific autotuning |
3-15% normally; up to 20-30% across heterogeneous CPUs or unusual shapes. |
Benchmark 5-20 safe candidates during session preparation and cache the winner by CPU, type, shape, transpose flags, and thread count. Limit tuning to roughly 1-50 ms per unique shape or load a persistent tuning cache; never tune in the inference path. |
5-10 days plus dedicated benchmark infrastructure. |
These gains are not additive. Shape specialization and autotuning often choose the same improvement. A credible target sequence is:
reach at least 1.0x ONNX Runtime median performance with the generic blocked algorithm and tuned scheduler;
reach 1.05-1.15x ONNX Runtime through shape specialization and tuning;
target 1.2-1.8x on fused or tiny-batch workloads;
reserve gains above 2x for workloads with exploitable sparsity, low-rank structure, or very large collections of tiny matrices.
External libraries#
OpenBLAS, BLIS, oneDNN, and vendor BLAS libraries are useful as performance oracles and optional large-matrix fallbacks. They do not remove the need for internal kernels: small inference shapes, FP16/BF16/quantized types, constant weights, and fused epilogues are precisely where a model-aware runtime can win. Any optional dependency must have a deterministic internal fallback and must not create a second competing thread pool.
Acceptance criteria#
Area |
Exit criterion |
|---|---|
Correctness |
ONNX backend and differential tests pass for every type, shape, transpose, broadcast, alpha/beta, bias, empty-dimension, and tail case. |
FP32/FP64 parity |
Median speed-up is at least 1.0x versus ONNX Runtime on the priority shape corpus, with no priority shape below 0.9x. |
Low precision parity |
FP16, BF16, and INT8 meet the same target on hardware with native support; fallback paths remain correct and avoid full-matrix conversion where panel conversion is possible. |
Scaling |
Throughput improves through the physical-core count without severe regressions on tiny or skinny shapes. |
Data movement |
Every dynamic A panel and constant B panel is packed no more often than required by the selected loop nest; low-precision paths avoid full-matrix conversion where panel conversion is possible. |
Exceeding MLAS |
Constant-weight or fused workloads demonstrate at least a repeatable 1.10x improvement on dedicated benchmark machines. |
Performance gates should run on dedicated, pinned hardware and store the raw samples and environment metadata. Shared CI machines can enforce correctness and detect catastrophic slowdowns, but they should not decide a 5-10% performance regression.
Implementation order and dependencies#
This roadmap and its dependency ordering were consolidated in onnx-light-cpu #137. The status below distinguishes implemented code from performance exit criteria that still require measurements on dedicated hardware.
Step |
Deliverable |
Exit criterion |
Dependency |
Status |
Pull requests |
|---|---|---|---|---|---|
P0 |
Reproducible MLAS cases in |
Stable medians and dispersion for the agreed shape/type corpus on pinned hardware. |
None. |
Corpus implemented; dedicated-hardware measurements pending. |
|
P1 |
|
Existing Gemm results remain correct with no material performance regression. |
P0. |
Implemented. |
|
P2 |
Complete MatMul shape/broadcast adapter. |
Differential tests pass for rank-1, batched, broadcast, transpose, and empty-dimension cases. |
P1. |
Implemented. |
|
P3 |
Five-loop FP32/FP64 engine and shape-specific algorithms. |
Generic dense path reaches at least 0.8x MLAS before assembly-level tuning. |
P1. |
Engine, algorithms, cache-derived blocking, and benchmark corpus implemented; 0.8x MLAS gate pending. |
onnx-light-cpu #136, onnx-light-cpu #139, onnx-light-cpu #140 |
P4 |
FMA/AVX2/AVX-512/ARM micro-kernels and tuned scheduler. |
Priority FP32/FP64 corpus reaches at least 1.0x ONNX Runtime median performance with no priority shape below 0.9x. |
P3. |
Scheduler PR01, epilogue PR02, x86 tuning PR03, thread runtime PR04, ARM kernels PR05, parity runner PR06.0, diagnosis PR06.1, vectorized skinny-N selection PR06.2, the dedicated GEMV/skinny-M kernel PR06.3, immutable operator plans PR06.4, and the measured Zen/Intel register tiles PR06.5 and shared-runner diagnostics PR06.6 are implemented. The blocking dedicated-machine ONNX Runtime gate is deferred to PR10.5, after every GEMM implementation and type-specific gate. |
onnx-light-cpu #133, onnx-light-cpu #141, onnx-light-cpu #142, onnx-light-cpu #143, onnx-light-cpu #145, onnx-light-cpu #146, onnx-light-cpu #147, onnx-light-cpu #149, onnx-light-cpu #155, onnx-light-cpu #156, onnx-light-cpu #157, onnx-light-cpu #158, onnx-light-cpu #159, onnx-light-cpu #160, onnx-light-cpu #162, onnx-light-cpu #167, onnx-light-cpu #176 |
P5 |
Native/panel-converted FP16, BF16, and integer paths. |
Low-precision corpus reaches at least 1.0x ONNX Runtime median performance with no priority shape below 0.9x where the type is supported. |
P3-P4. |
PR07.0, PR07.1, PR07.2, PR07.3, PR07.4, PR07.5, PR07.6, PR08.1, PR08.2, and PR08.3 are implemented. The remaining work is split by execution path, ISA, and type below; hardware-specific lanes may proceed in parallel after their shared semantic dependency. |
Roadmap PR07.0 through PR10.5 below. |
Remaining pull-request sequence#
The following table is the single source of truth for the sequence after
#149. Large phases use decimal sub-PRs so that each change has one type or
ISA, one measurable merge criterion, and no unrelated performance gate.
Hardware-specific lanes may run in parallel; only their shared semantics and
fallbacks are ordered. Completed rows remain visible so scope is not lost.
PR |
Scope |
Merge criterion |
Depends on |
Status |
|---|---|---|---|---|
Roadmap PR01 |
Scheduler, blocking, batch, and split-K. |
The five-loop engine schedules the full row-panel x column-panel grid,
packs each B panel once, consumes |
|
|
Roadmap PR02 |
Broadcast and fused epilogues. |
None, scalar, row, column, and full-matrix C layouts are consumed directly for every alpha/beta case; the expanded M x N bias temporary disappears. Priority bias, residual, activation, and output-conversion combinations use typed epilogues without intermediate tensors. |
PR01 |
|
Roadmap PR03 |
Complete x86 kernel tuning. |
AVX2 and AVX-512 candidate MR/NR profiles, aligned panels/loads, instruction ordering, and measured prefetch choices are benchmarked. CPUID family/model dispatch selects the winners; remaining gaps receive assembly kernels, with no priority-shape regression. |
PR02 |
|
Roadmap PR04 |
Complete thread runtime. |
The scheduler detects physical cores, SMT siblings, P-cores, and E-cores and applies tested Linux/Windows affinity. Bounded spin-before-park is configurable, and caller-owned pools run without nested workers or oversubscription. |
PR01 |
|
Roadmap PR05 |
ARM FP32/FP64 kernels. |
NEON packing, kernels, tails, and dispatch pass all GEMM/MatMul cases. Runtime vector-length-aware SVE/SVE2 profiles pass the ARM correctness and performance corpus with NEON fallback. |
PR02 |
|
Roadmap PR06.0 |
FP32/FP64 parity runner. |
The reproducible runner records raw alternating samples, dispersion, CPU affinity, SIMD level, and effective thread count for every priority Gemm shape. |
PR01 through PR05 |
Implemented in #160. An initial six-core diagnostic run on an i7-13800H under WSL reaches 0.317x FP32 and 0.347x FP64 median, with 0.064x and 0.036x minima. These diagnostic numbers are not final dedicated-machine evidence. |
Roadmap PR06.1 |
Isolate and explain the FP32 performance gaps. |
Isolated C++ driver measurements plus traced operator and plan paths identify the responsible algorithm, blocking, planning, and conversion costs before kernel changes are proposed. |
PR06.0 |
In progress in #162. The analysis identifies scalar skinny-N, weak GEMV/skinny-M, unused operator plans, and Zen/generic-x86 blocking as the next measured priorities. |
Roadmap PR06.2 |
Vectorized skinny-N and tiny-output selection. |
|
PR06.1 |
Implemented in #167. |
Roadmap PR06.3 |
Dedicated GEMV/skinny-M kernel. |
|
PR06.2 |
Implemented in #170. |
Roadmap PR06.4 |
Use immutable plans on operator paths. |
Registered ONNX |
PR06.3 |
Implemented in #176. The registered
|
Roadmap PR06.5 |
Sustain large-matrix throughput on Zen and generic x86. |
Measured MR/NR candidates and shape-constrained MC/NC/KC choices keep enough independent FMA chains and parallel panels active. The 1024³ and 2048³ priority cases no longer regress from 512³, and the complete corpus shows no priority-shape regression. |
PR06.4 |
Implemented in #180. AMD Zen AVX2+FMA
now selects the measured six-row register tile
( |
Roadmap PR06.6 |
Shared-runner FP32/FP64 diagnostics. |
Isolated measurements identify and verify fixes for skinny-N and large-matrix regressions without claiming ONNX Runtime parity. |
PR06.2 through PR06.5 |
Implemented. The new isolated |
Roadmap PR07.0 |
Panel-converted FP16/BF16 general path. |
The five-loop engine converts FP16/BF16 while packing, accumulates in FP32, and narrows in the epilogue without full A/B tensor widening. |
PR06.5 |
|
Roadmap PR07.1 |
Vectorized contiguous x86 conversion. |
Contiguous FP16 panels use F16C and contiguous BF16 panels use AVX2, with exact tails and runtime ISA fallback. No native dot-product kernel or unrelated execution path is included. |
PR07.0 |
|
Roadmap PR07.2 |
Remove remaining low-precision full-tensor widening. |
Skinny-M, skinny-N, direct, small-K, and split-K execute from typed inputs and FP32 accumulators. Immutable plans cover FP16/BF16, and tests prove that no priority algorithm allocates expanded A or B tensors. |
PR07.0 |
|
Roadmap PR07.3 |
Native AVX-512FP16 kernel. |
One FP16 micro-kernel family, its CPUID dispatch, tails, and differential tests land without BF16 or AMX changes. |
PR07.2 |
Implemented. A new |
Roadmap PR07.4 |
Native AVX-512BF16 kernel. |
One BF16 dot-product kernel family, its CPUID dispatch, tails, and differential tests land with the existing converted-panel fallback. |
PR07.2 |
Implemented. A new |
Roadmap PR07.5 |
AMX tile-state lifecycle. |
OS-enabled tile-state detection, per-worker tile configuration, and safe fallback pass focused lifecycle tests. No GEMM kernel is included. |
PR07.2 |
Implemented. New |
Roadmap PR07.6 |
AMX-BF16 kernel. |
The AMX-BF16 kernel reuses PR07.5, passes differential tests on native hardware, and falls back to AVX-512BF16. No INT8 work is included. |
PR07.4, PR07.5 |
Implemented. A dedicated |
Roadmap PR08.1 |
ARM FP16/BF16 panel conversion. |
NEON vectorizes conversion while packing with exact tails and scalar fallback; native arithmetic and SVE are excluded. |
PR07.2, PR05 |
Implemented. The NEON translation unit
( |
Roadmap PR08.2 |
Native ARM FP16/BF16 arithmetic. |
NEON FP16 and available BF16 dot-product kernels pass the complete differential corpus while retaining PR08.1 as fallback. |
PR08.1 |
Implemented. The NEON translation unit ( |
Roadmap PR08.3 |
SVE/SVE2 FP16/BF16 kernels. |
Runtime-vector-length-aware kernels and predicated tails pass under native hardware or QEMU, with NEON selected for short vector lengths. |
PR08.2 |
Implemented. The SVE translation unit ( |
Roadmap PR09.1 |
Portable integer semantics. |
INT8/UINT8/INT32/INT64 implement schema-defined zero points, overflow, accumulation, and requantization with exact scalar differential tests. No ISA-specific code is included. |
PR07.2 |
Implemented. |
Roadmap PR09.2 |
x86 VNNI INT8 kernel. |
Signed and unsigned VNNI paths have exact tails, runtime dispatch, and differential tests over the PR09.1 fallback. ARM and AMX are excluded. |
PR09.1 |
Implemented. |
Roadmap PR09.3 |
ARM dot-product INT8 kernel. |
Signed and unsigned NEON dot-product paths have exact tails, runtime dispatch, and differential tests over the PR09.1 fallback. |
PR09.1 |
Implemented. The contiguous rank >= 2 |
Roadmap PR09.4 |
AMX-INT8 kernel. |
The PR07.5 tile-state lifecycle is reused for signed and unsigned INT8, with exact tails and VNNI fallback. |
PR07.5, PR09.2 |
Implemented. The |
Roadmap PR09.5 |
Float8 packing formats. |
Each supported Float8 format has an explicit vectorized decode/packing path, exact tail handling, and differential tests. Integer and INT4 kernels are unchanged. |
PR07.2 |
Implemented. Each of the four ONNX Float8 formats ( |
Roadmap PR09.6 |
Packed INT4/UINT4 formats. |
Nibbles unpack into typed panels or a native dot-product path with exact odd-length tails and differential tests. Float8 is unchanged. |
PR09.1 |
Implemented. |
Roadmap PR10.1 |
FP16/BF16 correctness gate. |
The complete FP16/BF16 corpus passes on x86, ARM, and every fallback. This PR contains tests and fixes only, not performance tuning. |
PR07.2 through PR08.3 |
Implemented. |
Roadmap PR10.2 |
Integer and compact-format correctness gate. |
The complete integer, Float8, and packed-4-bit corpus passes on every available ISA and fallback. This PR contains tests and fixes only. |
PR09.1 through PR09.6 |
Implemented. |
Roadmap PR10.3 |
FP16/BF16 performance gate. |
Dedicated-machine x86 and ARM results reach at least 1.0x ONNX Runtime median with no priority case below 0.9x where the type is supported. The PR contains measurement-driven tuning only. |
PR10.1 |
Implemented; #341 is closed.
The earlier #333 was closed without the required dedicated-machine
parity evidence. The first tuning pass in #267 raises the 18-case
median from 0.341x to 0.479x ONNX Runtime and the minimum from 0.138x to
0.237x on a pinned AVX2/FMA/F16C i7-13800H thread. The second tuning
pass in #273
vectorizes FP16/BF16 output narrowing, adds an AVX2 BF16 direct
micro-kernel, and routes the operator workspace through the runtime
arena. The focused FLOAT16 direct/tiny corpus reaches a 1.178x median
and 1.046x minimum. The #337 pass wires
|
Roadmap PR10.4 |
Integer and compact-format performance gate. |
Dedicated-machine results are published per type and ISA. Types supported by ONNX Runtime reach at least 1.0x median with no priority case below 0.9x; unsupported types publish correctness and throughput. |
PR10.2 |
Implemented; #340 is closed.
The earlier #334 was closed without the required dedicated-machine
end-to-end parity evidence. The first tuning pass in #274 adds the integer
parity and compact throughput instruments plus an exact AVX2 UINT8 x
INT8 dot product used by both byte and packed-4-bit GEMM. On the
diagnostic AVX2 host, isolated square-512 throughput rises from 20.66 to
58.54 GOPS for INT8 and from 18.64 to 49.97 GOPS for INT4. The second
pass replaces the per-output AVX2 reduction with a 2x2 blocked output
micro-kernel that reuses loaded A and packed B vectors across outputs
while preserving the exact scalar output and reduction tails. The
third pass (#338)
fixes a measured INT4/UINT4 unpacking regression ( |
Roadmap PR10.5 |
Final GEMM parity tooling and regression gate. |
Raw dedicated-machine results cover Gemm, shared MatMul, batched paths, every priority platform, and every supported type. FP32, FP64, and each type supported by ONNX Runtime reach at least 1.0x median performance with no priority case below 0.9x. These historical certification targets remain available for dedicated-machine validation. The separate Attention roadmap may proceed independently. |
PR10.3, PR10.4 |
Implemented; #342 is closed. The complete certification command remains available for optional cross-machine validation. |
Roadmap PR10.5 delivered the original controlled-thread validation tooling. The later default-policy corrective work is recorded under the now-closed #374.
The reproducible gate command is tools/benchmark_gemm_parity.py
--operator all --dtype all --output gemm_matmul_parity_results.json.
It measures the registered CPU kernel and ONNX Runtime in separate session
lifetimes, alternates backend order between cases, records every raw sample and
environment field, and includes dynamic and constant Gemm, shared
MatMul, batched/broadcast, vector, transpose, bias, skinny, large-K,
split-K, and transformer cases. The primary command leaves execution policy
selection to each runtime. Explicit --threads runs remain available for
controlled scaling diagnostics; pass --enforce only when publishing a
completed dedicated-machine result.
Issue #374 corrective passes#
The corrective sequence is implemented by #560, #561, #566, #567, and #575. It adds medium AVX-512 tiles, cached and skinny-M MatMul execution, fused bias, productive participant scheduling, and worker-local dynamic-B packing.
The final pass keeps dynamic B packing in the invocation, but distributes native float32 panels of at least 131,072 elements across the session executor. This provisional threshold is deliberately left for a later dedicated-machine tuning pass. All micro-panels in one B-panel wave share one dispatch, and smaller panels stay inline. The multiplication phase assigns balanced tile intervals only when equal-size scheduler blocks would lose participants: for example, 12 tiles with 10 requested blocks previously collapsed to only 6 effective blocks. Otherwise the existing task distribution is retained.
On the local i7-13800H diagnostic, #575 reduces dynamic-B square-512 four-thread median latency from 2.345 ms to 0.855 ms and improves 1-to-4-thread scaling from 1.67x to 3.94x. These are diagnostic A/B results, not a cross-machine parity claim. Issue #374 is closed because the identified default-policy scheduling and packing defects are implemented; the reproducible dedicated-machine gate remains available for validation.
Roadmap PR10.5 final validation pass#
After #333 and #334 were closed without their required evidence, this pass ran
the regression checks available in its sandbox and confirmed none of the fixes
from PR10.3/PR10.4 regressed the shared GEMM implementation.
test_gemm_kernel (65 passed, 2
skipped for missing AVX-512FP16/AMX-BF16 hardware) and test_gemm_plan (31
passed) build and pass cleanly against a fresh AVX-512/Zen4 configuration.
The isolated gemm_throughput and compact_gemm_throughput drivers also
build and run without error, reproducing FP32/FP64/FP16/BF16/INT8/INT4/E4M3/
E5M2 throughput consistent with the numbers already recorded in the PR10.3
and PR10.4 tuning records above (for example skinny_n INT4 remains at
12.70 GOPS, matching the #338 fix), so no additional regression is exposed by
this run and no further kernel change is made.
This sandbox still has no onnxruntime or onnx-light install and no
access to the dedicated x86/ARM machines required for the pinned one-thread
and physical-core sweeps, so tools/benchmark_gemm_parity.py cannot be
executed here to produce gemm_matmul_parity_results.json. The tool
already covers the full required corpus (Gemm, shared MatMul, batched paths,
transpose, bias/epilogue, dynamic/constant B, skinny, large-K, split-K, and
transformer cases across FLOAT32/FLOAT64/FLOAT16) and needs no further
changes to run the gate once it is executed on a dedicated machine with
ONNX Runtime and onnx-light installed:
python tools/benchmark_gemm_parity.py --operator all --dtype all \
--output gemm_matmul_parity_results.json
Repeat with explicit one-thread and physical-core settings for controlled
diagnostics, then pass --enforce once every priority case reaches the 1.0x
median / 0.9x minimum gate.
Roadmap PR10.4 follow-up validation record#
Issue #340 was created
to revisit the integer/compact gate after #334 closed without dedicated-machine
pinned one-thread/physical-core sweeps or MatMulInteger/QLinearMatMul
ONNX Runtime parity reruns. This pass records exactly what its sandbox could
and could not verify.
What this sandbox reproduces: test_gemm_kernel (64 passed, 3 skipped for
missing AVX-512FP16/AVX-512BF16/AMX-BF16 hardware), test_gemm_plan (31
passed), and test_integer_gemm_vnni (8 passed) build and pass cleanly on
this AVX-512/AVX-512VNNI x86 host, confirming the #338 packing fix and the
2x2 blocked integer micro-kernel are not regressed. The isolated
compact_gemm_throughput driver, pinned to a single core with taskset,
also runs without error and produces INT8/INT4/E4M3/E5M2 GOPS across every
priority shape (for example square_512 reaches 93.17 INT8 GOPS and 69.77
INT4 GOPS on this host’s AVX-512VNNI path), with no shape collapsing to the
kind of sub-1-GOPS regression the #338 fix addressed.
What this sandbox cannot reproduce: it has no onnx-light install (the
package is not published and its source is not part of this checkout) and no
access to a dedicated, frequency-pinned x86 or ARM machine, so
tools/benchmark_integer_gemm_parity.py and
tools/benchmark_gemm_parity.py cannot be executed here against ONNX
Runtime, and no ARM sweep of any kind is possible. onnxruntime itself can
be installed from PyPI in this sandbox, but the parity tools require the
CPU kernel to be importable as onnx_light_cpu through an onnx-light
build (see onnx_light_cpu/kernels/math/gemm_kernel.cc registration and
tools/benchmark_integer_gemm_parity.py’s onnx_light/onnx_light_cpu
imports), which this environment cannot provide. The virtualized CPU used for
this run (cloud hypervisor, shared vCPUs, no pinned frequency) also does not
qualify as the “dedicated” machine the issue and its closure rule require
even where the parity tools could run.
#340 is now closed because the integer and compact implementation is complete.
Dedicated-machine MatMulInteger/QLinearMatMul ONNX Runtime sweeps and
INT4/UINT4/Float8 correctness-and-throughput reports from real x86 and ARM
machines remain optional external validation.
Roadmap PR10.3 tuning record#
The first PR10.3 pass adds alternating FLOAT16 measurements to
benchmark_gemm_parity.py and tunes the AVX2/F16C path without allocating
expanded FP32 operands. Contiguous A/B packing now widens eight values per
vcvtph2ps instruction. Dedicated skinny-M and skinny-N kernels reuse each
widened operand across vector FMAs. Small-K direct GEMM widens B in registers,
while larger general GEMM deliberately retains shared FP32 B panels: measuring
the native kernel on all general shapes showed that reconverting B for every
row tile regressed square_1024 and transformer projection. Tiny 2 x 2
split-K instead bypasses thread-pool partials and uses a two-column,
K-vectorized kernel; this changes its pinned one-thread result from 0.148x to
2.174x and its ten-thread result from 0.056x to 2.215x.
The second pass in #273 isolates the
32 x 128 x 16 direct shape and shows that its prepared kernel reaches only
3.30 GFLOP/s FP16 and 1.97 GFLOP/s BF16 before operator dispatch is included,
disproving the initial assumption that plan construction is the primary
bottleneck. The hot epilogue narrows every float32 accumulator through a scalar
software conversion. AVX2/F16C conversion now narrows eight FP16 outputs at
once, AVX2 integer rounding narrows eight BF16 outputs at once, and both retain
the scalar contract for NaNs and tails. The direct BF16 path also keeps its
compact inputs to the register file through an AVX2 micro-kernel instead of
using converted FP32 panels. Isolated direct throughput rises to 20.60 GFLOP/s
FP16 and 23.36 GFLOP/s BF16 on the same host. At the operator level, FLOAT16
direct falls from 26.54 to 7.45 microseconds and rises from 0.330x to 1.178x
ONNX Runtime; tiny_dynamic and tiny_constant remain above parity at
1.252x and 1.046x. The float32 workspace now comes from the reusable runtime
execution arena instead of a new std::vector allocation on every invocation.
The third pass replaces scalar transposed FP16/BF16 gathers on AVX2 with blocked
8 x 8 16-bit register transposes followed by eight-lane F16C or AVX2 BF16
widening. Both transposed A and B packing use the same kernel, while arbitrary
row and column tails retain the scalar conversion contract, including NaNs.
The compact throughput driver now reports isolated FP16 and BF16 throughput
alongside the other compact formats. Parity reports retain every timing sample
and dispersion and additionally record affinity policy, compiler, and NumPy,
onnx-light, onnx-light-cpu, and ONNX Runtime versions.
The fourth pass in #330 packs each native FP16/BF16 B panel once per K and column chunk, then shares it read-only across parallel row tasks. Prepared half plans retain separate blocking profiles for the compact native panels and the float32 fallback. The AVX2 general path now uses the compact driver, including a six-row micro-kernel dispatch for the Zen profile. Three one-thread diagnostic runs on the i7-13800H show FP16 ratios between 0.98x and 1.31x and BF16 ratios between 0.99x and 1.51x against the previous once-widened float32 path across square, transposed-A, and transformer projection cases.
The fifth pass (#333, following #332’s exposed blocking/compact_blocking/
parallel.maximum_threads tuning knobs) extends tools/compact_gemm_throughput
to build its FP16/BF16 measurements through the same tunable GemmHalfPlan
instead of the untuned convenience entry point, so --mc/--nc/--kc,
--compact-mc/--compact-nc/--compact-kc, and --participants select
blocking and participant values with the tuning tool instead of hard-coding a
machine threshold; --json publishes every raw sample plus CPU model,
affinity, compiler, and detected ISA. A one-thread and a two-physical-core
sweep of MC/NC/KC in {0, 64, 128, 256} x {0, 128, 256, 512} x {0,
128, 256} on the isolated Intel Xeon Platinum 8370C runner (the same host used
for the Exp/Log parity gate) shows every
setting within one standard deviation of the automatic (0) default for
square_512 and transformer (FP16 35.1-36.5 GFLOP/s, BF16 44.1-46.5
GFLOP/s); no configuration measurably regresses, so no blocking heuristic or
kernel change is made from this pass. The
one-thread raw JSON report and the
two-physical-core raw JSON report record every sample
alongside the metadata above.
The sandbox running this pass had no onnxruntime or onnx-light install
and no ARM hardware, so it could not reproduce the FLOAT16-vs-ONNX-Runtime
benchmark_gemm_parity.py median/minimum gate. The isolated instrument
remains available for optional dedicated x86 and ARM one-thread and
physical-core validation.
The published numbers above are diagnostic WSL measurements, not cross-machine certification. ONNX Runtime 1.28.0 does not implement CPU BFLOAT16 Gemm on this host, so BFLOAT16 remains an isolated throughput measurement rather than a parity ratio. The existing reports contain no ARM results.
Optional cross-machine validation is:
Run dedicated x86 and ARM sweeps and publish one-thread and physical-core raw samples, dispersion, FP16 parity, and isolated BF16 throughput per ISA.
Roadmap PR10.4 tuning record#
The first PR10.4 pass in #274 establishes two
reproducible instruments. The Python runner alternates registered UINT8 x INT8
MatMulInteger against ONNX Runtime, verifies exact INT32 output, and stores
every timing sample plus CPU, ISA, affinity, and thread metadata. The isolated
C++ driver reports INT8, packed INT4, E4M3, and E5M2 throughput across the
priority shape families.
The initial AVX2 measurement exposed a scalar fallback below AVX-512 VNNI:
isolated INT8 reached only 20.66 GOPS for square-512, 14.39 GOPS for large-K,
and 19.48 GOPS for transformer projection. The new AVX2 dot product splits
each UINT8 byte into low-seven-bit and high-bit terms before
vpmaddubsw. Each pair therefore remains inside the exact INT16 range,
including 255 x 127 and 255 x -128 adversarial inputs, before vpmaddwd
accumulates modulo 2^32. The shared dispatcher uses it for both INT8 and
expanded packed-4-bit panels when VNNI, AMX, or NEON dot product is unavailable.
The same isolated shapes reach 58.54, 40.53, and 46.58 GOPS respectively;
packed INT4 square-512 rises from 18.64 to 49.97 GOPS.
The one-thread end-to-end MatMulInteger median improves from 0.109x to 0.193x ONNX Runtime and its minimum from 0.063x to 0.097x on the diagnostic host. This does not close PR10.4: the current driver still packs complete matrices for every invocation.
The second pass replaces the per-output AVX2 reduction with a register-budgeted 2x2 micro-kernel for products with at least two rows, two columns, and one 32-byte reduction vector. On the same host with VNNI and AMX disabled, the isolated square-512, large-K, and transformer cases improve from 102.15, 57.46, and 80.33 GOPS to 105.12, 69.90, and 91.81 GOPS. Tiny and small-K products keep the single-output reduction. These are diagnostic results, not cross-machine certification.
The current pass keeps the shared integer panels reusable across row work by
partitioning contiguous matrix batches through ExecuteRanges under the
session executor. QLinearMatMul now routes contiguous rank-2+ products
through IntegerMatMul2D and applies a vectorized AVX2 requantization
epilogue for INT8/UINT8 outputs while retaining the scalar fallback and exact
tail behavior for non-AVX2 and rank-1 promotion paths.
The #338 pass measures
compact_gemm_throughput (which already reports INT8, packed INT4, E4M3,
and E5M2 alongside FP16/BF16) on the isolated sandbox host and finds a
measured INT4/UINT4 regression: IntegerMatMul4Bit2DWithDot unpacked every
4-bit element through a per-element ReadPacked4Bit call while building the
transposed per-row (A) and per-column (B) dot-product panels, recomputing the
nibble index shift and mask for every element instead of walking whole bytes.
Isolated profiling attributes effectively all of the unpack time to this
per-element call: on skinny_n (1024 x 1 x 1024) it costs 4.2 ms versus
0.03 ms for the vectorized dot product that consumes its output, an over
150x gap for work that is the same order of magnitude as the reduction itself.
The fix unpacks each whole operand once, in its natural contiguous order
(row * depth + inner for A, inner * cols + column for B), into a raw
signed byte buffer using a branchless byte-pair loop that extracts both
nibbles per iteration; the existing per-row/per-column panel-building loops
then read plain bytes from that buffer instead of calling ReadPacked4Bit,
with no change to the exact modulo-2^32 result, signedness handling, or
odd-tail behavior already covered by IntegerPacked4Bit.*. On the same
one-thread sandbox host, isolated INT4 throughput rises from 0.47 to 1.97
GOPS on skinny_m, from 0.49 to 12.70 GOPS on skinny_n, from 7.62 to
57.42 GOPS on large_k, from 41.41 to 74.63 GOPS on transformer, and
from 62.12 to 99.72 GOPS on square_512, closing most of the previous gap
to INT8 on the same shapes.
This pass also closes a JSON-publishing gap: compact_gemm_throughput
--json previously wrote only the FP16/BF16 medians and raw seconds samples
even though INT8, INT4, E4M3, and E5M2 were already measured and printed to
stdout. The tool now publishes the median and raw seconds samples for every
format it measures. The one-thread raw JSON report records the INT4 fix
above alongside INT8, E4M3, and E5M2 samples and the same hardware/compiler/
ISA metadata as the FP16/BF16 reports.
This pass remains diagnostic rather than dedicated-machine evidence. The
sandbox running it has no onnxruntime or onnx-light install, so
benchmark_integer_gemm_parity.py and
benchmark_gemm_parity.py --operator qlinearmatmul cannot be run to
reproduce the ONNX Runtime MatMulInteger/QLinearMatMul median/minimum
gate, and it has no ARM hardware. No further measured regression is evident
in the isolated INT8, INT4, E4M3, or E5M2 throughput on this host; tuning the
blocking/compact_blocking knobs is not applicable to the integer path,
which has no plan-owned blocking parameters (unlike GemmHalfPlan).
Optional PR10.4 follow-up validation can run pinned dedicated x86 and ARM
one-thread and physical-core
benchmark_integer_gemm_parity.py and benchmark_gemm_parity.py sweeps
and publish their raw samples, dispersion, and complete environment metadata.
Those results may evaluate the historical 1.0x median and 0.9x minimum parity
targets without changing the completed implementation status.