Qwen3 CPU Inference Critical Path

Date:

2026-08

Updated:

2026-09-05

planned (several prerequisite kernels are complete)

Objective

The objective is the shortest path from the current kernel set to useful batch-1 CPU inference for a dense, text-only Qwen3 model, with particular priority given to weight-only INT4. This is not a request to complete every generic operator roadmap first. Work is ordered by its effect on:

• model load time and resident weight memory;

• time to first token for prompt prefill;

• steady-state single-token decode latency;

• copied KV-cache bytes per generated token.

The first functional target is the concrete 28-layer INT4 artifact inventoried below because it keeps the graph contract fixed and exposes every immediate execution blocker. Qwen3-0.6B remains the short correctness reference and Qwen3-4B remains the first performance and memory target. The initial scope excludes Qwen3 MoE, VL, hybrid recurrent/attention variants, continuous batching, beam search, and speculative decoding. Those extensions must not delay the dense decoder.

The critical order is:

1. benchmark one frozen Qwen3 graph end to end;

2. implement constant weight-only INT4 MatMul, decode GEMV first;

3. complete the small set of Qwen block operators;

4. implement batch-1 causal GQA and rotary embedding;

5. eliminate full KV-cache copies during decode;

6. tune prefill, threads, and fusions only after decode is structurally sound.

The first useful milestone ends after step 5. Paged caches, INT4 KV caches, sampling inside the graph, generic attention masks, and broad operator parity come later.

Frozen model contract

Qwen PR01 pins immutable Hugging Face model and tokenizer commit hashes; model names alone are not reproducible inputs. The initial revisions must retain the official dense configurations:

Model	Layers	Hidden	Intermediate	Q heads	KV heads	Head size	Vocabulary
Qwen/Qwen3-0.6B	28	1,024	3,072	16	8	128	151,936
Qwen/Qwen3-4B	36	2,560	9,728	32	8	128	151,936

Both models use BF16 activations and weights in the unquantized artifact, Q/K RMS normalization, SwiGLU, tied token-embedding/LM-head weights, full half-split RoPE with theta 1,000,000, no rope scaling or sliding window, and a 40,960-token configured context. A revision that changes any of these values is a new benchmark artifact, not an update to an existing result.

The large context is an opt-in structure and memory check. Shared CI uses a deterministic synthetic two-layer fixture with the same head size, GQA ratio, Q/K normalization, tied-weight aliasing, cache boundary, and operator forms. The existing four-layer Qwen3-like fixture in onnx-light remains useful for shape and memory planning, but it is not an executable or numerical Qwen baseline and does not satisfy Qwen PR01.

Audited ORT INT4 artifact

The first executable target is the local artifact /home/xadupre/examples/models/qwen3-8b-cpu-int4/model.onnx. The absolute path is a development input, not a portable model identity. Qwen PR01 must copy the graph and external data into the external model cache under immutable artifact metadata before CI or published benchmarks use it.

The audited graph has SHA-256 f6745c77935bc5640751b3a961246e77a9bc515055dc8cba08d0f2ca5ba183a3. It uses IR version 10, ai.onnx opset 26 and com.microsoft opset 1, contains 519 nodes and 398 initializers, and references model.onnx.data with 1,016,070,144 physical bytes and SHA-256 43ea4a553800251cd2f74505a7a4140f4e19be78bad27e343425ab9dd420608c. Its graph dimensions identify 28 decoder layers, hidden size 1,024, intermediate size 3,072, 16 query heads, 8 KV heads, head size 128, and vocabulary size 151,936. These dimensions do not describe an 8B model despite the development directory name; model metadata and tokenizer revision must determine the final published artifact name.

The public contract is FP32: input_ids and attention_mask are INT64, 56 past K/V inputs and 56 present K/V outputs use FP32, and logits uses FP32. Rotary caches have shape [40960,64]. The embedding initializer is an unquantized FP32 [151936,1024] matrix; only the 141 projection nodes use INT4 MatMulNBits. This means the audited artifact is not equivalent to the fully quantized tied-embedding QDQ contract described below.

Exact kernel inventory

The table is exhaustive for this graph. Registered means a kernel is present in the current onnx-light-cpu registration inventory, not merely that an ONNX reference implementation may exist. Every row still marked Missing must receive either a CPU kernel or an explicit, tested onnx-light core execution path before the artifact can run without an untracked fallback.

Operator	Nodes	Current status	Exact required contract
ai.onnx::Mul	56	Registered	FP32 tensor multiplication for SiLU and gated MLP products.
ai.onnx::Sub	1	Registered	INT64 scalar/vector subtraction in the attention-mask prelude; verify opset-26 dispatch against the registered v14+ kernel.
ai.onnx::Constant	6	Missing CPU registration	Materialize the six INT64 shape/axis constants. This should normally be a prepared graph value in onnx-light core rather than a timed CPU compute kernel, but its execution ownership must be explicit.
ai.onnx::Cast	2	Missing	INT64 to INT32 conversion for seqlens_k and total_sequence_length consumed by GQA.
ai.onnx::Gather	2	Missing	FP32 embedding lookup with INT64 token ids on axis 0, plus one INT64 scalar gather from the attention-mask shape.
ai.onnx::ReduceSum	1	Missing	INT64 reduction over an axes tensor input with keepdims=0 for the attention-mask sequence length.
ai.onnx::Reshape	112	Missing	FP32 reshape from an INT64 shape tensor, including 0 and -1 semantics; use a metadata view whenever the input is contiguous.
ai.onnx::Shape	1	Missing	Produce the INT64 shape of the rank-2 attention_mask input.
ai.onnx::Sigmoid	28	Registered (#585)	FP32 sigmoid for x * sigmoid(x). #600 and #604 optimize the shared implementation; retain model-level coverage before adding a SiLU/gate fusion.
ai.onnx::SimplifiedLayerNormalization	57	Missing compatibility adapter	FP32 RMS-style normalization with axis=-1, epsilon=1e-6 and stash_type=1. Lower to the existing RMSNormalization engine only after proving identical output and accumulation semantics.
ai.onnx::Split	28	Missing	Split FP32 QKV projections on axis=-1 using the INT64 input [2048,1024,1024] and produce three views where legal.
com.microsoft::MatMulNBits v1	141	Missing and blocking	FP32 activation, packed UINT8 storage containing 4-bit weights, FP32 scales, block_size=32 and accuracy_level=4. Required shapes are 1024x3072 (56), 1024x4096 (28), 2048x1024 (28), 3072x1024 (28), and 1024x151936 (1). The implementation must keep weights compressed and cover GEMV decode plus GEMM prefill.
com.microsoft::GroupQueryAttention v1	28	Registered (#494, #507)	The delivered CPU adapter covers FP32 causal GQA with 16 query heads, 8 KV heads, head size 128, scale=1/sqrt(128), integrated half-split RoPE, external FP32 cos/sin caches, tensor past/present K/V, and three outputs through the shared Attention engine.
com.microsoft::SkipSimplifiedLayerNormalization v1	56	Missing and blocking	Fused FP32 residual addition and RMS-style normalization with epsilon=1e-6. Fifty-five nodes expose the normalized value and the residual through sparse optional outputs; the final node exposes only the normalized value. Lower to Add plus the shared normalization engine first, then add a measured fused traversal.

The minimum functional implementation order for this exact artifact is: prepared Constant/Shape plus Gather/Cast/ReduceSum for inputs; MatMulNBits; Split/Reshape; SimplifiedLayerNormalization; GroupQueryAttention; Sigmoid; and SkipSimplifiedLayerNormalization. Mul and Sub need model-level coverage but no new kernel. GroupQueryAttention already owns rotary embedding, softmax, mask handling, and tensor KV concatenation in this graph, so separate RotaryEmbedding, Softmax, Concat and Slice kernels are not execution prerequisites for this artifact.

Current implementation status

Several Qwen-critical primitives are already delivered even though the frozen model and end-to-end milestone remain pending:

• #494 and #507 provide com.microsoft::GroupQueryAttention, including the audited causal GQA geometry, fused half-split rotary embedding, masks, and tensor past/present inputs and outputs;

• #498 and #579 provide and optimize the shared RMSNormalization primitive;

• #585, #600, and #604 provide and optimize the FP32 Sigmoid path required by the audited MLP.

These implementations do not complete Qwen PR04 or PR05 by themselves. Standalone RotaryEmbedding and the two normalization compatibility adapters are still missing, and the delivered GQA tensor cache is invocation-local: it is not the request-owned persistent cache required by Qwen PR06a and PR06b.

Plans to execute first

The missing-kernel implementation plan splits the native portions of Qwen PR02-PR04 into isolated kernel PRs. Those PRs can start from deterministic fixtures without waiting for the complete model benchmark or persistent-cache API. The frozen graph remains required for model-level acceptance.

Do not execute the existing roadmaps from top to bottom. Use only their Qwen-critical slices in this order:

Order	Existing/new plan	Work to execute	Work deliberately deferred
1	This roadmap, Qwen PR01	Frozen graph, operator inventory, generation correctness, TTFT/decode benchmark, memory and per-node profile.	Model-level optimization until the baseline is reproducible; isolated missing-kernel development may use deterministic fixtures.
2	Completed Gemm/MatMul PR09.6 plus Qwen PR02-PR03	Direct MatMulNBits v1 adapter for the audited artifact, shared packed decode GEMV, small-M, then prefill GEMM; retain standard QDQ as the portable serialization.	Float8, full integer parity, float64, and generic PR10.5.
3	Completed ExpLog PR01–PR03 plus Qwen3 non-MatMulNBits operators (Qwen PR04)	The exact standard-operator inventory above: Gather, Cast, ReduceSum, Shape, Reshape, Split, Sigmoid, SimplifiedLayerNormalization, and the SkipSimplifiedLayerNormalization adapter.	Non-Qwen unary/binary operator matrices and graph fusions.
4	Completed #494/#507 plus Qwen PR05 integration	Integrate the delivered GroupQueryAttention adapter with the frozen graph and retain shared materialized and online Attention execution for batch-1 causal prefill and decode.	The generic Attention PR15 performance gate.
5	Corrected KV-cache C0-C2 plus Qwen PR06a-PR06b	Request-owned contiguous cache and cache-aware streaming decode.	Paging, beam/speculative operations, and KV quantization.
Completed prerequisite	Runtime PR01, PR02, and PR04	Reuse the delivered session executor and no-nested-pools contract.	No new runtime-executor integration in the Qwen sequence.
6	Qwen PR07-PR08	End-to-end scheduling, measured fusions, and parity gate.	Any optimization absent from the profile.

Export contract

Three graph contracts are retained. The audited native graph is executable first; the two standard-ONNX graphs provide portability and differential correctness:

qwen3-ort-int4

The audited opset-26 graph described above. It is the immediate execution target and uses MatMulNBits, GroupQueryAttention and SkipSimplifiedLayerNormalization from com.microsoft plus experimental ai.onnx::SimplifiedLayerNormalization. Each adapter must lower to the same internal compute plans as the standard contracts so native and portable support do not fork into separate engines.

qwen3-float

The primary correctness graph with ordinary constant BF16 weights and activations at ONNX opset 23. An FP32 diagnostic export is retained for numerical localization, but it is not a separate performance target.

qwen3-int4-qdq

A weight-only BF16 graph using one canonical encoding. Each non-embedding projection is a logical UINT4[K,N] initializer in ONNX row-major, low-nibble-first storage. DequantizeLinear uses axis=0, block_size=32, BF16 scales shaped [ceil(K/32),N], explicit UINT4 zero points of the same shape, and BF16 output feeding MatMul.

The tied embedding/LM-head weight is stored once as logical UINT4[vocabulary,hidden]. Its DequantizeLinear uses axis=1 and scale/zero-point shape [vocabulary,ceil(hidden/32)]. The standard graph feeds that dequantized value to Gather for embeddings and through Transpose to the LM-head MatMul. The optimized plan dequantizes only selected embedding rows and streams the same packed initializer in transposed access order for the LM-head; it does not clone the serialized initializer.

Qwen dimensions have no partial quantization block. Partial K blocks, odd N, and packed-byte tails remain mandatory synthetic tests. A transposed or differently blocked exporter output is a distinct artifact and must be normalized before matching.

The runtime recognizes only this exact constant pattern initially and passes packed weights, scales, and zero points directly to an internal BF16-activation weight-only plan. It never materializes a complete BF16 or float weight tensor. The portable DequantizeLinear path must accept BF16 scales/output before Qwen PR02 exits. Qwen PR01 may record that current gap but makes no production-kernel change.

Opset 23 is preferred because it provides the standard RMSNormalization and RotaryEmbedding schemas. An exporter that emits their primitive decompositions remains supported through graph matching, but those decompositions are not the preferred optimized representation.

Every benchmark artifact records the exact model revision, exporter revision, opsets, graph digest, quantization block size, symmetric/asymmetric encoding, weight packing order, tokenizer, prompt tokens, generated token ids, and runtime options. It also records initializer aliasing: the tied embedding and LM-head storage may have separate access plans but must not duplicate the serialized weight or an unbounded expanded representation. No result from one INT4 encoding is attributed to another.

Operator priority

Functional graph coverage and performance priority are different. Shape-only operators may be required to run the graph but do not deserve a dedicated optimization project before MatMul.

Priority	Standard ONNX operators	Role	Required optimization
P0	DequantizeLinear + MatMul	Q/K/V/O projections, gate/up/down MLP projections, and LM head.	Recognize constant blocked INT4 QDQ; packed weight-only GEMV/GEMM with fused scales and zero points.
P0	MatMul, Gemm	Float fallback, attention products, and projection reference.	Constant-B packing, M == 1/small-M kernels, then prefill GEMM.
P0	Gather	Token embedding lookup.	Recognize the tied quantized initializer and dequantize only selected rows; retain correct typed contiguous copy for float weights.
P1	RMSNormalization v23	Pre-attention, pre-MLP, final, and optional Q/K normalization.	One-pass or stable two-pass SIMD reduction with fused scale.
P1	RotaryEmbedding v23	RoPE on Q and K.	SIMD interleaved/half-split layouts without temporary tensors.
P1	Add, Mul, Sigmoid	Residuals and the SiLU-gated MLP.	SIMD; fuse Sigmoid(x) * x and the following gate multiply when the graph proves single use.
P1	Softmax	Materialized attention reference and sampling.	Stable row softmax; online softmax belongs to the Attention plan.
P1	Reshape, Transpose, Slice, Concat, Split	Head layouts and tensor past/present compatibility.	Metadata views where legal; avoid copies or fuse layout conversion into projection, rotary, and cache append.
P2	Pow, ReduceMean, Sqrt, Div	Decomposed RMSNorm fallback.	Recognize and lower to the same RMSNorm plan; do not optimize four independent materialized passes first.
P2	Sin, Cos, Gather, Range	Decomposed or dynamically generated RoPE tables.	Prefer constant tables or the standard RotaryEmbedding plan.
P3	TopK, ArgMax, Multinomial, sampling Softmax	Token selection.	Keep sampling outside the model initially; optimize after decoder latency is competitive.
Later	DynamicQuantizeLinear, MatMulInteger, QLinearMatMul	Activation-aware INT8 alternatives.	Benchmark only after the weight-only INT4 path works.

The graph inventory may add small correctness adapters such as Cast, Shape, Unsqueeze, Squeeze, Expand and Where. They are implemented or delegated to onnx-light fallback as required, but they do not change the priority order unless measured in the end-to-end profile.

INT4 MatMul is first

Model weights dominate Qwen3 memory traffic. During batch-1 decode, each projection has one or a few activation rows, so a generic square GEMM result does not predict token latency. The first optimized kernel is a float-activation by block-quantized INT4 constant-weight GEMV.

Prepared weight-only plan

Session preparation validates and captures:

• logical K and N, nibble order, signedness, block size, and tails;

• per-block or per-channel scales and optional zero points;

• transposition and exporter packing convention;

• CPU ISA and decode/prefill micro-kernel functions;

• packed constant storage aligned for the selected kernel;

• bounded workspace and useful thread count.

The implemented PR09.6 integer kernel supplies tested nibble decoding, panel packing, tail handling, and dot-product primitives. It does not implement BF16/FP32 activations multiplied by scaled INT4 weights and is therefore prior art, not the Qwen weight-only plan.

The plan keeps weights compressed. Decode kernels unpack into vector registers or small cache-resident panels, apply zero-point correction and scales, and accumulate into float32. They do not expand the full matrix to INT8 or float. Odd K/N tails, partial quantization blocks, and asymmetric zero points have scalar differential tests.

Decode and prefill are separate algorithms:

int4_gemv

Optimized first for M == 1 and short speculative blocks. Partition output columns across workers so each weight byte is streamed once.

int4_small_m

Reuses unpacked weight blocks across a few activation rows.

int4_gemm

Packs activation panels and reuses weight blocks for prompt prefill. It lands only after decode correctness and throughput are established.

The kernel corpus uses the exact Qwen3 dimensions for embedding, every Q, K, V, O, gate, up, down, and LM-head matrix, not only synthetic powers of two. It reports weight bytes read, effective memory bandwidth, unpack/scale time, first-use packing time, and steady-state throughput. Tests prove that the embedding row path and transposed LM-head path reference one packed initializer.

Phase Q0: freeze the executable baseline

Add an end-to-end generation driver before another generic kernel:

• Qwen3-0.6B correctness and Qwen3-4B performance models;

• float and standard QDQ INT4 graphs with identical public inputs/outputs;

• prompt lengths 1, 32, 128, 512, and 2,048;

• generated lengths 1, 32, and 128 at batch 1;

• context checkpoints at 128, 1,024, and 4,096, with 40,960 opt-in;

• greedy decoding outside the graph for the first milestone;

• per-node and per-phase profiling with warmup and raw samples;

• a backend manifest for every executed node, including explicit portable fallback outside timed priority regions.

Qwen PR01 also registers lazy backend cases through the standard onnx-light-cpu collector. Case names encode model, float/QDQ contract, prefill/decode, prompt/context/generated lengths, and thread policy. Shared CI executes the synthetic fixture in correctness and TestMode::BENCHMARK modes. Real Qwen3-0.6B and Qwen3-4B cases are opt-in, download pinned revisions into an external cache, and publish through the same benchmark runner and dashboard; model weights are not committed to this repository. Metadata tests verify unique names, lazy construction, graph digests, tensor types and sizes, exact projection shapes, and the opt-in large-context cases.

Report model-load latency, first-use preparation latency, peak and steady resident memory, time to first token, prefill tokens/second, median and tail decode latency, generated tokens/second, and bytes allocated/copied per token. Compare identical graphs under two separately labelled contracts:

standard-contract

The identical standard ONNX float or QDQ graph runs in both runtimes. This is the correctness and graph-coverage comparison. Qwen PR01 establishes the graph and reference outputs even if the current onnx-light runtime reports unsupported BF16 blocked dequantization; Qwen PR02 must make the QDQ graph executable before publishing standard-contract timings.

native-performance

The same quantized values, scales, zero points, prompts, and cache semantics are converted to ONNX Runtime MatMulNBits and GroupQueryAttention where required. This is the primary product-level performance comparison; graph conversion time and persistent converted bytes are reported. Results are never presented as identical-graph speed.

Qwen PR01 pins the reference CPU, OS, compiler, ONNX Runtime version, power mode, NUMA placement, and compact affinity. It publishes one-thread and physical-core policies; both runtimes receive the same admitted thread count, affinity, warmup, and alternating sample order. The priority performance matrix is Qwen3-4B with prompt lengths 32 and 512, context checkpoints 128, 1,024, and 4,096, and 128 generated tokens.

Correctness uses teacher-forced prefixes so one unstable greedy choice does not hide later numerical errors. PR01 freezes explicit per-type absolute and relative logit tolerances from the reference implementation. The selected token must be identical when the reference top-1 margin exceeds twice the measured logit error bound; near ties may select any token within that bound. Stable reference prompts additionally retain exact greedy-token regression sequences. INT4 quality is compared with the same quantized weights, scales, and zero points, not with the unquantized model alone.

Phase Q1: INT4 projections and LM head

Extend the completed packed-integer INT4 work in the Gemm and MatMul roadmap into a complete standard QDQ weight-only contract and a direct adapter for the audited graph:

1. com.microsoft::MatMulNBits v1 validation and lowering for the exact three-input, four-bit, block-32 contract;

2. constant DequantizeLinear -> MatMul recognition into the same plan;

3. int4_gemv for Qwen decode shapes;

4. plan-owned packed weights reused by every token;

5. tied quantized Gather and transposed LM-head access without duplication;

6. int4_small_m for short prompts/speculative blocks;

7. int4_gemm for prefill;

8. LM-head and vocabulary-tail specialization.

Do not wait for Float8, every integer operator, float64 parity, or the final generic MatMul PR10.5 gate. The Qwen path depends on existing float correctness, constant-B planning, and the new weight-only kernels only.

Q1 exits when the full INT4 graph loads without expanded BF16/float weights, every projection is dispatched to the packed plan, and batch-1 decode projection time is competitive with ONNX Runtime MatMulNBits on the frozen reference CPU. Persistent packed storage, including tied-weight plans, stays within 1.25x the serialized UINT4 weights plus serialized scales and zero points.

Phase Q2: one complete Qwen block

The implementation-ready sequence, ownership boundaries, tests and acceptance gates for this phase are maintained in Qwen3 non-MatMulNBits operators.

Implement the minimum standard-operator slice needed to run one decoder block:

• Gather for FP32 embeddings and the INT64 shape scalar;

• INT64 Shape, ReduceSum, Sub and INT64-to-INT32 Cast for the attention-mask prelude;

• SimplifiedLayerNormalization lowered to the RMSNormalization engine;

• zero-copy FP32 Reshape and Split views where legal;

• SIMD FP32 Mul/Sigmoid and fused SiLU-gate traversal;

• SkipSimplifiedLayerNormalization lowered to Add plus the shared normalization engine, including sparse optional outputs.

Standard RMSNormalization v23, RotaryEmbedding v23, stable Softmax, Slice, Transpose and Concat remain required by the portable graph contract, but they are not blockers for the audited native graph because GQA contains RoPE, attention softmax, and tensor-cache concatenation.

This phase reuses only the Qwen-relevant portions of the unary and binary elementwise roadmaps. It does not wait for their complete operator matrices. ExpLog PR01 through PR03 are complete and provide the corrected, fast Exp needed by sigmoid and online softmax. Qwen PR04 adds only missing Qwen-specific adapters and optimized traversals. Existing portable RMSNormalization, RotaryEmbedding, and DequantizeLinear kernels in onnx-light remain the differential fallback; their existence does not count as optimized CPU dispatch.

Q2 exits when one float and one INT4 decoder block match ONNX Runtime and no primitive RMSNorm or RoPE decomposition materializes avoidable intermediates. The standard RoPE nodes use rank-4 [batch,heads,sequence,128] inputs, interleaved=0, rotary_embedding_dim=128, and explicit position ids; Q and K use their respective frozen head counts.

Phase Q3: narrow causal GQA

Qwen PR05 does not create a second attention descriptor, planner, graph matcher, or compute engine. It integrates the delivered com.microsoft::GroupQueryAttention adapter and the portable standard graph with the shared AttentionDescriptor, per-invocation AttentionPlan, materialized fallback, and online engine. The Qwen priority subset is:

• batch 1;

• Qwen3 query-head/KV-head geometry;

• causal attention without arbitrary masks;

• BF16 for the frozen models, with FP32 as the diagnostic fallback;

• prompt prefill and Lq == 1 decode;

• standard tensor past_key/past_value and present_key/present_value for the correctness fallback.

The existing onnx-light graph optimizer recognizes the standard MatMul -> scale/mask -> Softmax -> MatMul attention pattern and emits standard ai.onnx::Attention. onnx-light-cpu dispatches that node to the shared engine. The native compatibility layer validates GroupQueryAttention attributes and inputs and builds the same descriptor; it is not a private GQA compute engine. GQA maps query-head groups onto shared K/V heads without physically repeating K or V.

The optimized path uses blocked online softmax and never materializes the complete attention-score matrix. Prefill uses query blocks; decode uses a single-query streaming kernel. General masks, arbitrary layouts, and broad type parity remain in the generic Attention roadmap.

Q3 exits when the narrow fused path and the standard materialized fallback produce equivalent outputs for the frozen models and improve end-to-end prefill or decode without increasing peak memory.

Phase Q4: persistent contiguous KV cache

Tensor Concat of all past K/V is acceptable only as a correctness baseline. It copies work proportional to context length at every generated token and prevents competitive long-context decode.

Execute C0 through C2 from the Persistent KV Cache roadmap:

1. one request-owned persistent runtime cache value;

2. contiguous K/V storage with capacity growth;

3. append of only the new rotated K/V token;

4. direct block iteration by the Q3 decode kernel;

5. explicit reset and session/request isolation;

6. import/export adapters only when standard tensor cache outputs are visible.

This cross-repository work is split at the ownership boundary. Qwen PR06a in onnx-light delivers the request-owned cache handle, persistent lifetime, reset/isolation semantics, execution-plan actions, safe tensor-cache rewrite, and backend-neutral import/export interfaces. Qwen PR06b in onnx-light-cpu delivers contiguous CPU allocation and growth, append/view/ export kernels, the block iterator, and direct consumption by shared streaming Attention. PR06b is the implementation vehicle for the Qwen-critical C0-C2 slice; the later paged and quantized parts of the broader KV roadmap remain discussed.

Do not block this milestone on paged storage, beam reorder, speculative truncate, sliding windows, INT8/INT4 KV compression, or observable cache format APIs. Append bytes must be proportional to new tokens, and a generated token must not allocate or copy the complete past.

Q4 is the first fast-Qwen3 milestone. It exits when decode latency is measured over growing context, no hidden full-cache gather occurs, and token outputs remain identical to the tensor-cache reference.

Phase Q5: scheduling and fusion

Runtime PR01, PR02, and PR04 from the Runtime Execution Controls roadmap are complete. Registered Qwen kernels must reuse the delivered session executor and nesting guard; standalone entry points remain serial. Qwen PR07 tunes participants and fusions but adds no executor or private pool.

Tune distinct policies for:

• INT4 GEMV decode, usually partitioned over output channels;

• INT4 GEMM prefill, partitioned over row and output panels;

• RMSNorm and elementwise passes, kept serial below measured sizes;

• prefill attention, partitioned by head and query block;

• decode attention, partitioned by KV/query-head group only when outer work is insufficient.

Only then measure graph-level fusions:

• RMSNorm into packed projection input;

• Q/K projection layout, RoPE, and cache append;

• gate projection, SiLU, and gate/up multiplication;

• residual Add with the following RMSNorm;

• final RMSNorm, LM-head GEMV, and greedy ArgMax.

Each unfused standard graph remains a correctness fallback. A fusion lands only when it removes measured memory traffic and preserves graph-visible values.

Deferred work

The following work is explicitly not required for the first fast dense Qwen3 result:

• SVM, tree, convolution, and broad unary/binary parity roadmaps;

• Float8, float64, and generic integer MatMul completion;

• generic Attention masks and every MHA layout;

• paged/quantized KV cache, continuous batching, beam search, and speculative decoding;

• in-graph sampling and full generation control flow;

• Qwen3 MoE, VL, multimodal rotary, and later hybrid recurrent layers.

These items return to priority only after Q4 profiling shows that their corresponding functionality is needed.

com.microsoft operator inventory

No Microsoft-domain operator in this table is currently registered by onnx-light-cpu. The audited artifact makes MatMulNBits, GroupQueryAttention and SkipSimplifiedLayerNormalization immediate execution dependencies rather than optional comparator formats. Their compatibility registrations may remain in the dedicated Microsoft-domain repository, but they must lower to the shared packed MatMul, Attention, Add, and RMSNormalization plans owned here. The remaining rows stay conditional until a frozen graph contains them.

Operator	Priority	Qwen role	Standard/internal counterpart here
com.microsoft.MatMulNBits v1	C0	Primary weight-only 2/4/8-bit projection format used by ONNX Runtime.	QDQ INT4 matcher plus internal packed MatMul plan.
com.microsoft.MatMulBnb4 v1	C0 if present	Alternative bitsandbytes-style 4-bit matrix format.	Format adapter into the same internal weight-only plan.
com.microsoft.GroupQueryAttention v1	C1	Fused GQA, RoPE options, and tensor KV-cache path used by Qwen exports.	Standard attention-pattern matcher plus internal GQA/cache plan.
com.microsoft.RotaryEmbedding v1	C1	Legacy/contrib RoPE representation.	Standard RotaryEmbedding v23.
com.microsoft.SkipSimplifiedLayerNormalization v1	C2	Residual Add fused with RMS-style normalization.	Standard Add plus RMSNormalization matcher.
com.microsoft.Attention v1	C2	Older fused self-attention representation.	Standard attention matcher.
com.microsoft.MultiHeadAttention v1	C2	Fused dense MHA alternative.	Generic Attention roadmap.
com.microsoft.QAttention v1	C3	Legacy activation-quantized attention alternative; not the preferred Qwen3 INT4 path.	No first-milestone counterpart.
com.microsoft.Sampling v1	C3	In-graph generation and sampling control.	External sampler first; standard TopK/ArgMax later.
com.microsoft.PagedAttention v1	Conditional	Continuous batching and paged KV storage. Current ONNX Runtime sources register CUDA/WebGPU implementations, not a CPU compute kernel.	Later paged KV-cache phase.
com.microsoft.MoE and com.microsoft.QMoE v1	Conditional	Dense and quantized expert routing for Qwen3 MoE variants.	Separate MoE roadmap after dense Qwen3.
com.microsoft.MRotaryEmbedding v1	Conditional	Multimodal rotary positions used by VL variants.	Separate Qwen3-VL scope.
com.microsoft.SparseAttention v1	Conditional	Sparse/sliding attention variants.	Generic sparse Attention work.
com.microsoft.GatedRMSNorm and com.microsoft.CausalConvWithState v1	Conditional	Later hybrid Qwen-family recurrent/linear-attention blocks.	Out of the dense Qwen3 scope.

ONNX Runtime also carries experimental SimplifiedLayerNormalization in the default ONNX domain at version 1. It is not the standard ONNX RMSNormalization v23 schema and should be treated as another compatibility adapter that lowers to the shared normalization engine.

Benchmark gates

Gate	Exit criterion
Graph	Every executed node is reported with its backend; no silent reference fallback occurs in a timed priority region.
INT4 memory	Full BF16/float weights are never materialized. Persistent packed weights plus scales/metadata stay within 1.25x serialized UINT4 weights plus serialized scales/zero points, with tied storage counted once.
Correctness	Float and INT4 teacher-forced logits satisfy the frozen tolerances. Greedy ids match for stable top-1 margins; documented near ties satisfy the bounded candidate rule. Stable prompts match exact token sequences.
Decode structure	One token performs no full-weight conversion, complete score-matrix allocation, or full-KV copy.
First milestone	Qwen3-0.6B and Qwen3-4B complete prefill and 128-token greedy decode at batch 1 with bounded memory and stable repeated latency.
Performance	Q4 publishes both standard-contract and native-performance TTFT and tokens/second on the frozen CPU, threads, affinity, and prompt/context matrix. Against the native-performance baseline, initial acceptance requires no priority case below 0.9x and final tuning reaches at least 1.0x median.

Pull-request sequence

PR	Deliverable	Merge criterion	Depends on	Status
Qwen PR01	Frozen graphs and generation benchmark.	The audited native artifact and portable graphs have pinned model/exporter/tokenizer revisions, external-data digests, exhaustive kernel inventories, lazy backend cases, correctness rules, comparator contracts, TTFT, decode, memory, and per-node profiles.	None	Pending
Qwen PR02	Native MatMulNBits and standard QDQ INT4 plan plus decode GEMV.	MatMulNBits and constant QDQ lower to one prepared plan; every decode projection keeps weights compressed and passes exact packing/tail plus model-logit tests.	Qwen PR01; reuses completed Gemm PR09.6 primitives	Pending
Qwen PR03	INT4 small-M and prefill GEMM.	Qwen prompt shapes reuse packed weights and improve TTFT without regressing decode or exceeding the memory bound.	Qwen PR02	Pending
Qwen PR04	Qwen block operator slice.	The dedicated non-MatMulNBits operator plan is complete and runs one Qwen block; the portable RMSNorm, RoPE, Gather, SiLU, Softmax, and layout paths remain differential coverage.	Qwen PR01; completed ExpLog PR01-PR03	Partially implemented: RMSNormalization, Sigmoid, Mul, and Sub primitives are delivered; adapters, layout/input kernels, and the complete block remain
Qwen PR05	Frozen-graph integration with shared Attention.	The delivered native GroupQueryAttention adapter and standard Attention execute from the frozen graph through the shared engine. Batch-1 causal prefill/decode retain zero-copy query/KV-head grouping.	Qwen PR04; Attention PR14 / #391	Partially implemented: the GQA adapter and engine are delivered; frozen-graph integration remains
Qwen PR06a	Backend-neutral persistent cache in onnx-light.	Request-owned state survives invocation cleanup, reset and isolation are explicit, safe tensor-cache rewrites preserve observable outputs, and execution planning accounts for persistent lifetime and bytes.	Qwen PR05; KV C0	Pending
Qwen PR06b	Contiguous CPU cache and cache-aware decode in onnx-light-cpu.	Append work is proportional to new tokens; decode reads the cache directly through the shared Attention block iterator and produces no full present-cache copy or gather.	Qwen PR06a; KV C1-C2	Pending
Qwen PR07	Participant tuning and measured fusion.	The completed Runtime PR01/02/04 executor is reused without a private pool; only measured Qwen fusions land, with inspectable scheduling decisions.	Qwen PR06b	Pending
Qwen PR08	First dense-Qwen3 performance gate.	Both reference models pass correctness/memory gates, every priority case is at least 0.9x ONNX Runtime, and median performance reaches 1.0x. If the target is missed, bottleneck evidence is published and PR08 remains open.	Qwen PR01-PR07, including PR06a and PR06b	Pending

Qwen PR08 closes the first fast dense-Qwen3 milestone. Later work starts from its end-to-end profile rather than from an operator checklist.

Reference evidence

The ordering follows the current ONNX and ONNX Runtime contracts:

• ONNX opset 23 defines RMSNormalization and RotaryEmbedding;

• ONNX Runtime’s CPU MatMulNBits is the direct weight-only comparison;

• its CPU GroupQueryAttention confirms the GQA/KV-cache execution target;

• the pinned model contracts originate from the official Qwen3-0.6B configuration and Qwen3-4B configuration;

• the ONNX Runtime GenAI Qwen builder distinguishes full-attention layers and their KV-cache handling.