.. _l-next-steps-quantization: Quantization ============ :Date: 2026-08 **discussion** .. note:: The structures on this page are descriptive quantization profiles. They should be implemented with :ref:`l-next-steps-custom-types`, not as a parallel protobuf hierarchy. Each specialized structure below is followed by its ``StructTypeProto`` translation. The families remain useful as format names, validation profiles, and decoder specifications. Format coverage summary +++++++++++++++++++++++ .. list-table:: :header-rows: 1 :widths: 25 8 25 10 * - Format - bpw - Proto used - Possible * - 1.58-bit Ternary (BitNet) - 1.63 - ``Codebook`` - ✅ * - AQLM 2×8 - 3.0 - ``Codebook`` - ✅ * - Binary/XNOR (1-bit) - 1.0 - ``Codebook`` - ✅ * - EXL2 (variable bpw) - 3.5 - ``Tiling`` + ``Linear`` (multiple) - ✅ * - EETQ (INT8 weight-only) - 8.0 - ``Linear`` - ✅ * - EXL3 (improved EXL2) - 2–6 - ``Tiling`` + ``Linear`` or ``Codebook`` - ✅ * - FP6 LLM (TC-FPn) - 6.125 - ``Tiling`` + ``FloatingPoint`` - ✅ * - FP8 E4M3 - 8.0 - ``FloatingPoint`` - ✅ * - HQQ (mixed-precision per head) - 2–4 - ``Tiling`` + ``Linear`` - ✅ * - INT4 AWQ (per-group) - 4.5 - ``Tiling`` + ``Linear`` - ✅ * - INT4 GPTQ (per-group) - 4.5 - ``Tiling`` + ``Linear`` - ✅ * - INT4 Symmetric - 4.5 - ``Linear`` - ✅ * - INT8 per-channel - 8.0 - ``Linear`` - ✅ * - INT8 symmetric - 8.0 - ``Linear`` - ✅ * - IQ1_S - 1.56 - ``Tiling`` + ``Codebook`` - ✅ * - IQ4_NL - 4.5 - ``Codebook`` - ✅ * - Log quantization - 4.0 - ``Log`` - ✅ * - MatMulNBits INT4 (ORT) - 4.5 - ``Tiling`` + ``Linear`` - ✅ * - MXFP4 - 5.0 - ``Tiling`` + ``FloatingPoint`` - ✅ * - MXFP6 E3M2 - 6.125 - ``Tiling`` + ``FloatingPoint`` - ✅ * - NF4 (QLoRA) - 4.5 - ``Codebook`` - ✅ * - NVFP4 (E2M1 + FP8 scale) - 4.5 - ``Tiling`` + ``FloatingPoint`` - ✅ * - Q2_K - 2.625 - ``Tiling`` × 2 + ``Linear`` - ✅ * - Q3_K - 3.4 - ``Tiling`` × 2 + ``Linear`` - ✅ * - Q4_K - 4.5 - ``Tiling`` × 2 + ``Linear`` - ✅ * - Q5_K - 5.5 - ``Tiling`` × 2 + ``Linear`` - ✅ * - Q6_K - 6.6 - ``Tiling`` × 2 + ``Linear`` - ✅ * - QuaRot (rotational quantization) - 4.0 - ``Linear`` + ``RotationProto`` - ✅ * - QuIP# - 2.0 - ``Codebook`` + rotation - ✅ * - SpQR - 3.4 - ``Sparse`` + ``Tiling`` + ``Linear`` - ✅ * - SmoothQuant (W8A8) - 8.0 - ``Linear`` + ``RotationProto`` - ✅ * - STQ1_0 (Sherry) - 1.3125 - ``StructuredBlock`` - ✅ * - TQ1_0 - 1.6 - ``Codebook`` - ✅ * - TQ2_0 - 2.0 - ``Codebook`` - ✅ * - BitNet b1.58 - 1.58 - ``Codebook`` - ✅ * - ParetoQ (SEQ ternary) - 1.58–2.0 - ``Function`` or ``Codebook`` - ✅ * - Tequila (ICLR 2026) - 1.58 - ``Codebook`` - ✅ CPU prepacking coverage +++++++++++++++++++++++ The formats above describe values and portable physical layouts. They do not cover every representation produced by a CPU kernel prepacker. A prepacked tensor is tied to an operand role, a microkernel ABI, and an exact processor feature set; it must therefore be stored as a :ref:`l-next-steps-compiled-tensor` cache entry while the original initializer remains the portable fallback. Prepacking is useful for floating-point and integer tensors, not only INT4. The first CPU implementation should cover the following persistent matrix packing families. ``B`` denotes the usually constant right-hand operand. .. list-table:: :header-rows: 1 :widths: 18 22 25 35 * - Input family - CPU target - Packed representation - Main consumers * - FP32 / FP64 - SSE, AVX2, AVX-512F - Transposed or blocked ``B`` panels, padded and interleaved by the microkernel ``N`` and ``K`` tiles - ``Gemm``, ``MatMul``, batched ``MatMul``, linear projections * - BF16 - AVX-512 BF16 - K-paired BF16 words arranged for ``VDPBF16PS``, with an FP32 accumulation tile - ``Gemm``, ``MatMul``, attention projections * - FP16 - AVX-512 FP16 - FP16 ``B`` panels in the kernel's ``K × N`` lane order - ``Gemm``, ``MatMul``, attention projections * - INT8 / UINT8 - AVX2 - K-paired or K-quad interleaving for the selected emulated dot-product kernel, plus correction sums when required - ``QLinearMatMul``, ``MatMulInteger``, quantized ``Gemm`` * - INT8 / UINT8 - AVX-512 VNNI - Four K values per dot-product group in VNNI lane order, with per-column sums and zero-point correction data - ``QLinearMatMul``, ``MatMulInteger``, quantized ``Gemm`` * - INT4 / UINT4 - AVX2, AVX-512F/BW - Nibble-interleaved ``B`` panels with scales, zero points, and optional group metadata placed in the order consumed by vector unpacking - ``MatMulNBits``, GPTQ, AWQ, HQQ, weight-only ``Gemm`` * - INT4 / UINT4 - AVX-512 VNNI - INT4 panels expanded or streamed into K-quad VNNI groups, with per-group scales and corrections adjacent to their tiles - Weight-only ``Gemm`` and ``MatMul`` with INT8 activations * - INT2--INT7, codebook, binary, ternary - AVX2, AVX-512 BW/VBMI/VBMI2/VPOPCNTDQ - Bit planes, lane-interleaved codes, lookup tables, scales, and sparse outliers arranged for unpack, lookup, or population-count kernels - AQLM, NF4/IQ, BitNet, TQ, SpQR, and other low-bit ``MatMul`` kernels * - BF16 / INT8 - AMX BF16 / AMX INT8 - Tile-major ``B`` panels matching AMX tile geometry and K grouping - Large ``Gemm``, ``MatMul``, and fused projections Feature names in this table are compatibility requirements, not format names. For example, AVX-512F, AVX-512 BW, AVX-512 VNNI, AVX-512 BF16, AVX-512 FP16, VBMI, VBMI2, VPOPCNTDQ, and AMX must be distinguished. An ``avx512-int4`` label is insufficient because two kernels may require different subsets or use different ``MR × NR × KR`` tiles. Operator-specific prepacking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Matrix-panel packing is the base layer, but the plan must also register prepackers for every constant operand whose reuse amortizes preparation: .. list-table:: :header-rows: 1 :widths: 20 34 46 * - Operator family - Persistent inputs - Required prepacking * - ``Gemm`` / ``MatMul`` - Constant ``B`` and optional bias - Transpose while packing, create K/N panels, pad tails, interleave quantization parameters and correction sums, and optionally fuse bias and epilogue constants * - Batched ``MatMul`` - Shared or constant batched ``B`` - Pack each distinct batch matrix, retain batch/head strides, and share one packed object when broadcasting makes the logical ``B`` identical * - ``QLinearMatMul`` / ``MatMulInteger`` / ``MatMulNBits`` - Quantized ``B``, scales, zero points, bias, sparse outliers - Pack codes and all auxiliary data in consumption order; precompute column sums and zero-point compensation when the kernel ABI uses them * - ``Conv`` / ``QLinearConv`` - Weights, bias, scales, zero points - Reorder OIHW/HWIO weights into output-channel, input-channel, and spatial tiles for direct, 1x1-GEMM, im2col-GEMM, grouped, or depthwise kernels; each algorithm has a separate packed cache key * - Attention projections - Q, K, V, and output projection weights - Pack separate matrices or one fused QKV panel, preserving head and grouped-query-attention boundaries and the selected quantization groups * - Attention score/value products - Constant masks or reusable encoder K/V tensors - Store K in the transposed blocked layout used by ``Q × K^T`` and V in the blocked layout used by ``P × V``; include head, sequence, and head dimension tiles in the compatibility key * - Fused/flash attention - Projection weights and reusable K/V blocks - Pack by the fused kernel's query, key, value, head, and sequence tiles; precompute causal or constant-mask tiles only when they are model constants * - ``Gather`` / embedding - Quantized embedding table - Pack row-local codes, scales, and zero points so one row remains independently addressable without decoding neighboring rows * - Mixture-of-experts - Expert projection weights - Pack each expert with the common matrix format and add an expert index without merging independently loadable experts into one mandatory blob Mutable decoder K/V caches are not compiled initializers. Their blocked or quantized runtime layout belongs to :ref:`l-next-steps-prepared-execution` and must encode the same head, sequence-tile, feature, and kernel-ABI constraints. A cache populated during inference must never be published as a ``CompiledTensorProto`` for a model initializer. Packed-format contract ^^^^^^^^^^^^^^^^^^^^^^ Every registered prepacker must describe enough information to reject an incompatible cache entry without inspecting implementation-specific bytes: * source tensor digest, logical shape, element type, and quantization profile; * operator and operand role, including transpose flags and broadcast rules; * exact required CPU features and, when relevant, cache-line or alignment requirements; * kernel library, kernel ABI, packed-format name, and format version; * ``MR``, ``NR``, ``KR``, outer tile order, lane interleave, padding, and tail policy; * placement and type of scales, zero points, codebooks, bias, column sums, compensation values, and sparse outliers; * supported dynamic-shape bounds, batch/head sharing, and fused epilogue; * payload size and alignment, plus the source digest required by :ref:`l-next-steps-compiled-tensor`. Packed entries must use specific names such as ``matmul-b-int4-avx512-vnni-nr64-kr4-v2`` or ``attention-qkv-bf16-amx-tile16-v1`` rather than broad names such as ``INT4_AVX512``. Selection first matches the exact kernel ABI and CPU features, then applies the :ref:`l-next-steps-processor-aware-kernel-tuning` packing threshold. On a miss, the runtime prepacks from the portable initializer; it may also execute an unpacked kernel when packing would cost more than it saves. QuantizedTensorProto ++++++++++++++++++++ A quantized tensor cannot rely on ``shape × sizeof(data_type)`` to compute its storage size (sub-byte packing, block metadata, sparse outliers, etc.). It carries its own byte size explicitly. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message QuantizedTensorProto { repeated int64 dims = 1; // logical shape of the tensor bytes raw_data = 2; // quantized payload int64 n_bytes = 3; // byte size of raw_data int32 quantized_type = 4; // index into ModelProto.quantizations optional QuantizationProto quantization = 5; // inline if unique to this tensor string name = 6; string doc_string = 7; } .. tab-item:: Custom types .. code-block:: text StructProto { type: raw_data: ... name: ... doc_string: ... } ``QuantizedTensorProto`` becomes ``StructProto``. ``raw_data``, ``external_data``, ``name``, and ``doc_string`` are carried by that generic value. Its exact model-level or inline ``StructTypeProto`` replaces ``quantized_type`` and ``quantization``. Counts such as a block or tile number are concrete dimensions of that type. The size computed from the physical type must equal ``raw_data.size()`` or external-data ``length``. Logical dimensions and element type belong to the decoder output ``ValueInfoProto``. In both forms, ``name`` identifies the concrete value and ``doc_string`` documents it. TypeProto and containers ^^^^^^^^^^^^^^^^^^^^^^^^ The following specialized container additions are therefore not required by the recommended implementation. They document what would be necessary only if ``QuantizedTensorProto`` remained a distinct value category: .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message TypeProto { message QuantizedTensor { repeated int32 allowed_quantized_type = 1; optional int32 elem_type = 2; // decoded logical element type optional TensorShapeProto shape = 3; } oneof value { ... QuantizedTensor quantized_tensor_type = ; } } message SequenceProto { enum DataType { ... QUANTIZED_TENSOR = ; } repeated QuantizedTensorProto quantized_tensor_values = ; optional TypeProto value_type = ; } message OptionalProto { enum DataType { ... QUANTIZED_TENSOR = ; } optional QuantizedTensorProto quantized_tensor_value = ; optional TypeProto value_type = ; } .. tab-item:: Custom types .. code-block:: text message TypeProto { oneof value { ... StructTypeProto struct_type = ; } } message SequenceProto { enum DataType { ... STRUCT = ; } repeated StructProto struct_values = ; optional TypeProto value_type = ; } message OptionalProto { enum DataType { ... STRUCT = ; } optional StructProto struct_value = ; optional TypeProto value_type = ; } ``allowed_quantized_type`` is a set of indices into ``ModelProto.quantizations``. An empty set accepts any quantization declaration whose decoded element type and logical shape satisfy the remaining constraints. This allows a sequence or map to contain pages using different quantization formats without becoming untyped. The corresponding value containers require a new category. These specialized branches map to the generic integrations defined by :ref:`l-next-steps-custom-types`: * ``TypeProto.QuantizedTensor`` becomes ``TypeProto.struct_type``; * ``QUANTIZED_TENSOR`` becomes the ``STRUCT`` value category; * ``quantized_tensor_values`` and ``quantized_tensor_value`` become ``struct_values`` and ``struct_value``; * heterogeneous pages use the unconstrained static structured category, while each ``StructProto`` carries its exact physical type. ``MapProto`` inherits support because its values are represented by a ``SequenceProto``. Consequently, ``Sequence`` and ``Map`` can represent a paged KV-cache with a different quantization declaration for each page. ``value_type`` is required for these new categories and carries the complete ``TypeProto.QuantizedTensor`` constraint. It makes standalone sequence and map values self-describing and must agree with any enclosing ``ValueInfoProto``. The generalized design in :ref:`l-next-steps-custom-types` avoids duplicating this machinery: a quantized tensor is an structured value with an explicit physical type and a decoder with a typed output signature. QuantizationProto +++++++++++++++++ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Describes how a tensor is quantized. Nine variants cover the // common families without separate messages for scalar/vector // codebooks, one-/multi-dimensional tiling, identity casts, or // packed linear layouts. // TilingQuantizationProto can nest QuantizationProto to express // multi-level hierarchies (e.g. K-Quants, MXFP). // Optional pre/post rotations support QuIP#, SmoothQuant, etc. message QuantizationProto { oneof kind { LinearUniformProto linear = 1; CodebookProto codebook = 2; FloatingPointUniformProto floating_point = 4; SparseQuantizationProto sparse = 5; LogUniformProto log = 6; FunctionUniformProto function = 7; TilingQuantizationProto tiling = 13; CastUniformProto cast = 16; StructuredBlockUniformProto structured_block = 18; } int32 data_type = 15; // dequantized element type (same enum as TensorProto.data_type) string doc_string = 9; // human-readable description repeated StringStringEntryProto metadata_props = 10; // arbitrary key-value metadata optional RotationProto pre_rotation = 11; // rotation applied before quantization optional RotationProto post_rotation = 12; // rotation applied after dequantization } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { // One concrete array or structure and its decoder. }, ... ] } There is no direct ``QuantizationProto`` message in the recommended schema. Each entry becomes one concrete ``StructTypeProto`` in ``ModelProto.struct_types``: * the selected profile determines its ``array`` or ``structure``; * scalar parameters and lookup tables become fields with ``constant``; * ``data_type`` and the logical shape are declared by the decoder output ``ValueInfoProto``; * pre/post rotations become decoder or encoder nodes; * ``doc_string`` and ``metadata_props`` remain type documentation and metadata. The translation is lossless: every specialized field must map to serialized structure, a named constant field, decoder/encoder structure, or metadata. A decoder must not hide a profile parameter that was explicit in the specialized declaration. Code blocks containing names such as ``N``, ``K``, ``packed_bytes``, or ``tile_type`` are translation templates, not valid serialized declarations. Every such name is replaced by a concrete value or model type index in an ``StructTypeProto``. LinearUniformProto ^^^^^^^^^^^^^^^^^^ Classic affine/symmetric: ``value = (q - zero_point) * scale``. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message LinearUniformProto { int32 storage_type = 1; // quantized element type (same enum as TensorProto.data_type) int32 bits = 2; // number of bits (e.g. 4, 8) bool symmetric = 3; // true if zero_point is always 0 oneof scale { float scale_float = 4; // scale as float int32 scale_int = 5; // scale as shared exponent (value = q * 2^scale_int) } int64 zero_point = 6; // quantization zero point int32 axis = 7; // axis for per-channel, -1 if per-tensor } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "LINEAR" structure: Structure { field: { name: "values", type: array(INT4, dimension=N) } field: { name: "storage_type", constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], symmetric) } field: { name: "scale", constant: tensor(FLOAT, [], s) } field: { name: "zero_point", constant: tensor(INT64, [], z) } field: { name: "axis", constant: tensor(INT32, [], axis) } } decoder: FunctionProto { // Y = (Cast(values) - zero_point) * scale } } The physical codes are an ``Array``. When ``bits`` equals the canonical width of ``storage_type``, the element type is used directly. Otherwise the packed region is an ``Array(UINT8)`` and the decoder extracts exactly ``bits`` per logical code. Scale and zero point are constant fields when shared, or serialized array fields when they vary by channel or block. ``N`` is always replaced by the concrete physical count. The displayed ``scale`` field represents ``scale_float``. A declaration using ``scale_int`` gives that field type ``INT32`` instead, preserving which member of the specialized ``oneof`` was selected. .. code-block:: python # Dequantize values = unpack(data, q.bits) if q.scale_float: result = (values - q.zero_point) * q.scale_float else: result = (values - q.zero_point) * (2 ** q.scale_int) # Quantize if q.scale_float: values = round(tensor / q.scale_float) + q.zero_point else: values = round(tensor / (2 ** q.scale_int)) + q.zero_point values = clip(values, 0, (1 << q.bits) - 1) raw_data = pack(values, q.bits) CodebookProto ^^^^^^^^^^^^^ Scalar or additive vector lookup-table quantization. Scalar codebooks use ``num_codebooks = vector_size = 1``. For additive vector quantization, each vector is reconstructed as the sum of one entry from each codebook. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message CodebookProto { oneof scale { float scale_float = 1; // scale as float int32 scale_int = 2; // scale as shared exponent } repeated float codebook_data = 3; // concatenated codebooks int32 packed_count = 4; // number of values packed int32 packed_bytes = 5; // into this many bytes int32 num_codebooks = 6; // additive codebooks; 0 means 1 int32 codebook_size = 7; // entries per codebook; 0 means infer int32 vector_size = 8; // values per entry; 0 means 1 int32 index_bits = 9; // bits per index; 0 means base-N packing } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "CODEBOOK" structure: Structure { field: { name: "indices", type: array(UINT8, dimension=packed_bytes) } field: { name: "codebook" constant: tensor( FLOAT, [codebook_value_count], codebook_data ) } field: { name: "scale", constant: tensor(FLOAT, [], scale) } field: { name: "packed_count", constant: tensor(INT32, [], packed_count) } field: { name: "packed_bytes", constant: tensor(INT32, [], packed_bytes) } field: { name: "num_codebooks", constant: tensor(INT32, [], num_codebooks) } field: { name: "codebook_size", constant: tensor(INT32, [], codebook_size) } field: { name: "vector_size", constant: tensor(INT32, [], vector_size) } field: { name: "index_bits", constant: tensor(INT32, [], index_bits) } } decoder: FunctionProto { // Y = scale * additive_gather(codebook, unpack(indices)) } } Packed indices are serialized as an ``Array(UINT8)``. The codebooks and scale are constant fields, so they are stored once in the type and consume no payload bytes. The decoder unpacks each index, gathers the selected scalar or vector entry, sums additive codebooks, and applies the scale. As for linear quantization, the ``scale`` field is ``FLOAT`` for ``scale_float`` and ``INT32`` for ``scale_int``. .. code-block:: python # Dequantize num_codebooks = q.num_codebooks or 1 vector_size = q.vector_size or 1 codebook_size = q.codebook_size or ( len(q.codebook_data) // (num_codebooks * vector_size)) indices = unpack_indices(data, q.index_bits, codebook_size, q.packed_count, q.packed_bytes) result = zeros(n_vectors * vector_size) for k in range(num_codebooks): codebook_k = q.codebook_data[ k * codebook_size * vector_size: (k + 1) * codebook_size * vector_size] for i, idx in enumerate(indices[k]): begin = idx * vector_size result[i*vector_size:(i+1)*vector_size] += ( codebook_k[begin:begin + vector_size]) result *= q.scale # Quantize indices = nearest_additive_codewords(tensor / q.scale, q) raw_data = pack_indices(indices, q.index_bits, codebook_size, q.packed_count, q.packed_bytes) FloatingPointUniformProto ^^^^^^^^^^^^^^^^^^^^^^^^^ For finite normal values, micro-float quantization uses ``value = (-1)^sign * 2^(exp - bias) * (1 + mantissa)``. Covers FP6, FP4, MXFP and similar reduced-precision floating-point formats. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message FloatingPointUniformProto { int32 sign_bits = 1; // sign bits (usually 1) int32 exponent_bits = 2; // exponent bits (e.g. 3 for E3M2, 2 for E2M1) int32 mantissa_bits = 3; // mantissa bits (e.g. 2 for E3M2, 1 for E2M1) int32 exponent_bias = 4; // exponent bias (e.g. 3 for E3M2) bool has_inf = 5; // true if format supports infinity bool has_nan = 6; // true if format supports NaN bool split_storage = 7; // true if sign+exp and mantissa stored separately (TC-FPn) int32 packed_count = 8; // number of values packed int32 packed_bytes = 9; // into this many bytes } .. tab-item:: Custom types .. code-block:: text StructTypeProto { // translation template; not serialized as-is structure: Structure { // Serialized payload. field: { name: "packed" type: bit_packing( dimension=packed_count, components=[ sign:sign_bits, exponent:exponent_bits, mantissa:mantissa_bits ] ) } // Type-level parameters: these consume no payload bytes. field: { name: "exponent_bias" constant: tensor(INT32, [], exponent_bias) } field: { name: "has_inf" constant: tensor(BOOL, [], has_inf) } field: { name: "has_nan" constant: tensor(BOOL, [], has_nan) } } decoder: FunctionProto { // unpack packed_count values from packed_bytes bytes // interpret sign/exponent/mantissa using the declared constants // apply exceptional-value policy and reconstruct Y } } StructTypeProto { name: "FLOAT6_E3M2" structure: Structure { field: { name: "packed" type: bit_packing( dimension=4, components=[sign:1, exponent:3, mantissa:2] ) } field: { name: "exponent_bias", constant: tensor(INT32, [], 3) } field: { name: "has_inf", constant: tensor(BOOL, [], true) } field: { name: "has_nan", constant: tensor(BOOL, [], true) } } decoder: FunctionProto { output: "Y" value_info: { name: "Y" type: TypeProto { tensor_type: Tensor { elem_type: FLOAT shape: TensorShapeProto { dim: { dim_value: 4 } } } } } // unpack four 6-bit values and decode E3M2 with exponent bias 3 } } A standard ONNX low-precision type such as ``FLOAT4E2M1`` or a float8 type is an ordinary physical ``Array``. A non-standard width uses ``BitPacking`` so the component widths and packed element count remain explicit. Split storage is a ``Structure`` with one field per bit plane or component. Parameters not already expressed by the physical structure remain named constant fields. Thus ``sign_bits``, ``exponent_bits``, ``mantissa_bits``, ``packed_count``, ``packed_bytes``, and ``split_storage`` are checked from the physical layout rather than duplicated as constants. This is a generic profile template, not a serializable element type: ``sign_bits``, ``exponent_bits``, and the other symbolic values are replaced when the concrete physical type is declared. A generic ``IEEE_FLOAT`` cannot be a ``TensorProto.data_type`` or ``TypeProto.Tensor.elem_type`` because it does not determine the width or the decoder's standard ONNX output type. The following concrete ``FLOAT6_E3M2`` declaration packs four 6-bit values into three bytes. It uses an IEEE-like exceptional-value policy, fixes every profile parameter with ``Field.constant``, and decodes to a standard ONNX ``FLOAT`` tensor: When ``split_storage`` is true, the serialized ``packed`` field is replaced by explicitly named component arrays such as ``sign_exponent`` and ``mantissa``. The constant remains useful to profiles and validation, while the physical structure itself is sufficient to locate the components. .. code-block:: python # Dequantize bits_per_elem = q.sign_bits + q.exponent_bits + q.mantissa_bits raw_values = unpack(data, bits_per_elem, q.split_storage) result = [fp_decode(v, q.sign_bits, q.exponent_bits, q.mantissa_bits, q.exponent_bias) for v in raw_values] SparseQuantizationProto ^^^^^^^^^^^^^^^^^^^^^^^ Sparse + dense decomposition (SpQR, SqueezeLLM). Outlier values above a threshold are stored separately in higher precision; the rest uses a base quantization scheme. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message SparseQuantizationProto { QuantizationProto base_quant = 1; // quantization for non-outlier values int32 outlier_data_type = 2; // data type for outlier values (e.g. FLOAT16) float outlier_threshold = 3; // absolute value threshold for outlier detection float outlier_ratio = 4; // fraction of values stored as outliers (e.g. 0.01) } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "SPARSE_QUANTIZED" structure: Structure { field: { name: "base", type: struct(type_index=base_type) } field: { name: "indices", type: array(INT64, dimension=K) } field: { name: "values", type: array(FLOAT16, dimension=K) } field: { name: "outlier_threshold" constant: tensor(FLOAT, [], outlier_threshold) } field: { name: "outlier_ratio" constant: tensor(FLOAT, [], outlier_ratio) } } decoder: FunctionProto { // Y = scatter(decode(base), indices, values) } } The dense/base region is a referenced structured physical type. Outlier indices and values are concrete arrays in the same root ``Structure``. The concrete element type of ``values`` preserves ``outlier_data_type``. Threshold and ratio guide the encoder but are not required to decode an already serialized value; they remain explicit constants. The root decoder incorporates the base decoder nodes explicitly; nested decoders are not invoked implicitly. ``K`` is a concrete outlier count in each physical declaration. .. code-block:: python # Dequantize base_values = dequantize(base_data, q.base_quant) outlier_indices, outlier_values = read_sparse(outlier_data, q.outlier_data_type) result = base_values result[outlier_indices] = outlier_values # Quantize mask = abs(tensor) > q.outlier_threshold outlier_indices, outlier_values = where(mask), tensor[mask] base_tensor = tensor.copy(); base_tensor[mask] = 0 base_data = quantize(base_tensor, q.base_quant) outlier_data = write_sparse(outlier_indices, outlier_values, q.outlier_data_type) LogUniformProto ^^^^^^^^^^^^^^^ Logarithmic quantization: ``value = sign * base^(q + offset)``. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message LogUniformProto { int32 bits = 1; // number of bits (e.g. 4, 8) float base = 2; // logarithm base (e.g. 2.0) float offset = 3; // exponent offset bool has_sign = 4; // true if sign bit is stored separately } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "LOG_UNIFORM" structure: Structure { field: { name: "codes", type: array(UINT4, dimension=N) } field: { name: "signs", type: array(UINT8, dimension=sign_bytes) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "base", constant: tensor(FLOAT, [], base) } field: { name: "offset", constant: tensor(FLOAT, [], offset) } field: { name: "has_sign", constant: tensor(BOOL, [], has_sign) } } decoder: FunctionProto { // Y = sign * Pow(base, Cast(codes) + offset) } } Codes use a native fixed-width ``Array`` when possible and packed ``Array(UINT8)`` otherwise. A separate sign plane is another array field. ``base`` and ``offset`` are constants consumed by the decoder. The ``signs`` physical field is omitted when ``has_sign`` is false. .. code-block:: python # Dequantize values = unpack(data, q.bits) if q.has_sign: signs = extract_sign_bits(values) exponents = extract_magnitude(values) result = signs * (q.base ** (exponents + q.offset)) else: result = q.base ** (values + q.offset) # Quantize if q.has_sign: signs = sign(tensor) exponents = round(log(abs(tensor)) / log(q.base) - q.offset) values = pack_sign_magnitude(signs, exponents) else: values = round(log(tensor) / log(q.base) - q.offset) values = clip(values, 0, (1 << q.bits) - 1) raw_data = pack(values, q.bits) FunctionUniformProto ^^^^^^^^^^^^^^^^^^^^ Custom quantization/dequantization defined by op names. ``storage_type`` and ``bits`` describe the storage format of the quantized data so the runtime can read the raw bytes correctly. The custom ops referenced by ``quantize_op`` and ``dequantize_op`` handle the actual conversion logic. The dequantize op takes a tensor of ``storage_type`` and returns the type specified by ``QuantizationProto.data_type``; the quantize op does the reverse. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message FunctionUniformProto { int32 storage_type = 1; // storage element type (how raw bytes are interpreted) int32 bits = 2; // bits per element (for sub-byte packing) string quantize_op = 3; // op: data_type -> storage_type (e.g. "custom::Quantize") string dequantize_op = 4; // op: storage_type -> data_type (e.g. "custom::Dequantize") optional BlockLayoutProto block_layout = 5; // physical block structure (for introspection) } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "FUNCTION_QUANTIZED" structure: Structure { field: { name: "payload" type: array(UINT8, dimension=packed_bytes) } field: { name: "storage_type" constant: tensor(INT32, [], storage_type) } field: { name: "bits", constant: tensor(INT32, [], bits) } } decoder: FunctionProto { ... } // standard-domain form, when available encoder: FunctionProto { ... } metadata_props: { key: "quantize_op", value: quantize_op } metadata_props: { key: "dequantize_op", value: dequantize_op } } ``storage_type``, ``bits``, and ``block_layout`` become the concrete ``Array``/``Structure`` schema. ``dequantize_op`` becomes ``StructTypeProto.decoder`` and ``quantize_op`` becomes ``encoder`` when both can be expressed as deterministic standard-domain ONNX functions. If either operation requires a custom-domain operator, the physical type remains valid but omits that portable function. The consuming custom operator is then responsible for interpretation. The original operator identifiers are preserved as ``quantize_op`` and ``dequantize_op`` metadata so the translation remains lossless. When ``block_layout`` is present, its concrete fields replace the raw ``payload`` array and determine the same exact byte size. .. code-block:: python # Dequantize if q.block_layout: blocks = split_blocks(data, q.block_layout.bytes_per_block) result = concat([call_op(q.dequantize_op, block) for block in blocks]) else: raw = interpret(data, q.storage_type, q.bits) result = call_op(q.dequantize_op, raw) # Quantize if q.block_layout: blocks = tile(tensor, q.block_layout.block_size) raw_data = concat([call_op(q.quantize_op, block) for block in blocks]) else: raw_data = call_op(q.quantize_op, tensor) TilingQuantizationProto ^^^^^^^^^^^^^^^^^^^^^^^ Recursive one- or multi-dimensional block quantization. Weights are partitioned into tiles along ``axes``. An empty ``axes`` denotes the flattened tensor, making classic one-dimensional block quantization a special case. If there are more tiles than entries in ``elem_quant``, the list is cycled from the beginning (``elem_quant[i % len(elem_quant)]``). Nesting supports multi-level hierarchies such as K-Quants and MXFP. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message TilingQuantizationProto { repeated int64 tile_shape = 1; // tile size; one value if axes is empty repeated int32 axes = 2; // empty = flattened tensor repeated QuantizationProto elem_quant = 3; // cycled over tiles repeated int32 perm = 4; // permutation of axes in memory layout } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "TILED" structure: Structure { field: { name: "tiles" type: array( struct(type_index=tile_type), dimension=concrete_tile_count ) } field: { name: "tile_shape" constant: tensor(INT64, [tile_rank], concrete_tile_shape) } field: { name: "axes" constant: tensor(INT32, [axis_count], concrete_axes) } field: { name: "perm" constant: tensor(INT32, [perm_rank], concrete_perm) } } decoder: FunctionProto { // decode each tile, place it at concrete coordinates, inverse perm } } A homogeneous tiling stores a nested ``Array`` whose element references the concrete tile type. The root is a ``Structure`` so concrete ``tile_shape``, ``axes``, and ``perm`` values remain explicit constant fields. If tiles cycle through different quantizations, the root has one physical field per concrete tile or homogeneous tile region. The decoder concatenates or scatters tile outputs and incorporates the tile decoder nodes explicitly. No symbolic tile count, modulo rule, or implicit nested-decoder invocation remains in the physical type. .. code-block:: python # Dequantize shape = resolve_tile_shape(q.tile_shape, q.axes, tensor_shape) tiles = split_tiles(data, shape, q.axes, q.perm) result = empty(tensor_shape) for i, (coords, tile_data) in enumerate(tiles): eq = q.elem_quant[i % len(q.elem_quant)] result[coords] = dequantize(tile_data, eq) if q.perm: result = inverse_permute(result, q.perm) CastUniformProto ^^^^^^^^^^^^^^^^ Type conversion without quantization. Stores values cast from the original type to a different type (e.g. float32 → bfloat16). The source type is ``QuantizationProto.data_type``, the target type is ``storage_type``. Setting ``storage_type`` equal to ``data_type`` is the identity operation, useful for pure retiling. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text message CastUniformProto { int32 storage_type = 1; // target element type (same enum as TensorProto.data_type) } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "CAST" array: Array { element_type: BFLOAT16 dimension: N } decoder: FunctionProto { // Y = Cast(values, original_type) } } The payload is a concrete ``Array`` of ``storage_type``. The decoder output ``ValueInfoProto`` declares the original type and shape; the decoder contains the standard ONNX ``Cast`` and any required reshape. .. code-block:: python # Dequantize result = cast(data, from_type=q.storage_type, to_type=quant_proto.data_type) # Quantize raw_data = cast(tensor, from_type=quant_proto.data_type, to_type=q.storage_type) StructuredBlockUniformProto ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Generic block quantization with explicit physical layout. Each block contains named fields at known bit offsets. ``VALUES`` can be decoded directly for packed linear formats, or an index formula can combine fields into codebook indices. Optional scale, zero point, bias, and scatter fields cover prepacked linear formats (CompInt8, QGEMM), exotic formats (STQ1_0), and future layouts while keeping dequantization fully deducible from the proto structure. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text enum BlockFieldRole { VALUES = 0; // quantized values or codebook indices SIGN = 1; // sign bits SCALE = 2; // per-block scale ZERO_POINT = 3; // per-block zero point BIAS = 4; // per-block bias correction EXPONENT = 5; // shared exponent CODE = 6; // codebook index (partial) MASK = 7; // sparsity mask } message BlockFieldProto { BlockFieldRole role = 1; // semantic role of this field int32 bit_offset = 2; // offset in block or field record int32 bit_width = 3; // bits per element int32 count = 4; // number of elements int32 data_type = 5; // element type (0 = raw unsigned bits) } message BlockLayoutProto { int32 block_size = 1; // logical values per block int32 bytes_per_block = 2; // physical bytes per block repeated BlockFieldProto fields = 3; // physical fields in the block } enum BlockStorageOrder { INTERLEAVED = 0; // all fields for block 0, then all fields for block 1 SEQUENTIAL = 1; // one region per field, each containing all blocks } message FieldWeightProto { BlockFieldRole field = 1; // which field to use int32 multiplier = 2; // coefficient in index formula } message ScatterProto { int32 group_size = 1; // logical group size (e.g. 64) int32 vector_size = 2; // lanes per codebook entry (e.g. 4) int32 stride = 3; // stride within group (e.g. 16) } message StructuredBlockUniformProto { BlockLayoutProto block_layout = 1; // physical block structure repeated float codebook_data = 2; // empty = use VALUES directly int32 codebook_vector_size = 3; // values per codebook entry repeated FieldWeightProto index_formula = 4; // index = sum(field * multiplier) optional ScatterProto scatter = 5; // output element placement (absent = contiguous) BlockStorageOrder storage_order = 6; // physical organization of block fields optional int32 axis = 7; // absent = flattened tensor } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "STRUCTURED_BLOCK" structure: Structure { field: { name: "blocks" type: array( struct(type_index=block_type), dimension=concrete_block_count ) } field: { name: "codebook" constant: tensor( FLOAT, [codebook_value_count], codebook_data ) } field: { name: "codebook_vector_size" constant: tensor(INT32, [], codebook_vector_size) } field: { name: "index_fields" constant: tensor( INT32, [index_term_count], concrete_index_fields ) } field: { name: "index_multipliers" constant: tensor( INT32, [index_term_count], concrete_index_multipliers ) } field: { name: "storage_order" constant: tensor(INT32, [], concrete_storage_order) } field: { name: "block_size" constant: tensor(INT32, [], concrete_block_size) } field: { name: "axis" constant: tensor(INT32, [], concrete_axis) } } decoder: FunctionProto { // index = sum(named_field * multiplier) // Y = scatter(codebook[index] * scale + bias) } } This profile maps directly to the generic mechanism: * ``BlockFieldProto`` becomes a named ``Structure.Field``; * ``count`` becomes ``Array.dimension``; * ``data_type`` becomes the scalar array element type; * field roles become ordinary field names used by the decoder; * explicit padding replaces ``bit_offset`` and ``bytes_per_block``; * ``codebook_data`` becomes a constant field; * ``index_formula`` and ``ScatterProto`` become constant parameters consumed by decoder nodes; * ``INTERLEAVED`` is an array of block structures; * ``SEQUENTIAL`` is a structure containing one array per field. The ``codebook``, index-formula, and ``axis`` fields are omitted when their specialized counterparts are absent. When ``scatter`` is present, its three integer members are additional constant fields named ``scatter_group_size``, ``scatter_vector_size``, and ``scatter_stride``. The concrete block type itself preserves ``block_size``, field widths and counts, explicit padding, and therefore ``bytes_per_block``. No separate ``BlockFieldProto``, ``BlockLayoutProto``, ``FieldWeightProto``, or ``ScatterProto`` is implemented. .. code-block:: python # Dequantize blocks = parse_structured_blocks( data, q.block_layout, q.storage_order, q.axis, tensor_shape) result = [] for block in blocks: fields = parse_fields(block, q.block_layout.fields) scale = fields[SCALE][0] if SCALE in fields else 1.0 zp = fields[ZERO_POINT][0] if ZERO_POINT in fields else 0.0 if q.codebook_data: assert q.index_formula, "index_formula is required when codebook_data is set" assert q.codebook_vector_size > 0, "codebook_vector_size must be > 0" n_entries = len(q.codebook_data) // q.codebook_vector_size values = [] n_codes = None for fw in q.index_formula: assert fw.field in fields, f"missing field {fw.field} for index_formula" if n_codes is None: n_codes = len(fields[fw.field]) else: assert len(fields[fw.field]) == n_codes, f"field {fw.field} has inconsistent length (expected {n_codes})" assert n_codes is not None, "index_formula must not be empty" for i in range(n_codes): raw_idx = sum(fields[fw.field][i] * fw.multiplier for fw in q.index_formula) idx = int(raw_idx) assert raw_idx == idx, f"codebook index {raw_idx} is not an integer" assert 0 <= idx < n_entries, f"codebook index {idx} out of range" begin = idx * q.codebook_vector_size values.extend(q.codebook_data[begin:begin + q.codebook_vector_size]) else: values = fields[VALUES] if q.scatter: values = inverse_scatter(values, q.scatter) bias = fields[BIAS][0] if BIAS in fields else 0.0 result.append((values - zp) * scale + bias) result = concat(result) RotationProto ^^^^^^^^^^^^^ Pre/post rotation applied to the tensor (QuIP#, SmoothQuant). Dequantization with rotation: ``values = post_rotation @ dequant(data) @ pre_rotation``. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text enum RotationType { HADAMARD = 0; PLAIN = 1; } message RotationProto { RotationType matrix_type = 1; // type of rotation repeated int32 dims = 2; // shape of the rotation matrix optional int32 matrix_index = 3; // index into ModelProto.rotation_matrices (for PLAIN) } .. tab-item:: Custom types .. code-block:: text field: { name: "rotation" constant: tensor(FLOAT, concrete_rotation_dims, concrete_matrix) } field: { name: "rotation_type" constant: tensor(INT32, [], PLAIN) } decoder: FunctionProto { // Y = MatMul(rotation, decode_physical_fields(...)) } Rotation is not a physical root type. A plain matrix is a field with ``constant`` in the quantized type and matrix multiplication is part of the decoder or encoder. A Hadamard rotation needs no stored matrix and is expressed directly by standard ONNX nodes. ``ModelProto.rotation_matrices`` and ``matrix_index`` are therefore resolved rather than copied into the generic type. Known quantization schemes +++++++++++++++++++++++++++ The examples below retain the specialized profile notation because it is compact and makes comparison with existing formats easy. They are not additional wire messages. Each example is implemented by applying the corresponding custom-type translation above, with all dimensions and payload sizes made concrete. INT8 symmetric (per-tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: text QuantizationProto { linear: LinearUniformProto { storage_type: INT8, bits: 8, symmetric: true, scale_float: 0.02, zero_point: 0, axis: -1 } } INT4 symmetric (per-tensor) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ For a tensor of 1024 elements: .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { linear: LinearUniformProto { storage_type: INT4 bits: 4 symmetric: true scale_float: 0.02 zero_point: 0 axis: -1 } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "INT4_SYMMETRIC_1024" structure: Structure { field: { name: "values" type: array(INT4, dimension=1024) } field: { name: "storage_type" constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 0.02) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], -1) } } } INT4 asymmetric (GPTQ, per-group of 128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For a tensor of 1024 elements (8 groups): .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [128] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.015, zero_point: 8, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.012, zero_point: 7, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.014, zero_point: 8, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.013, zero_point: 8, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.016, zero_point: 7, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.011, zero_point: 8, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.015, zero_point: 9, axis: -1 }}, QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.012, zero_point: 8, axis: -1 }} ] } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "GPTQ_INT4_1024_GROUP128" structure: Structure { field: { name: "values" type: array(INT4, dimensions=[8, 128]) } field: { name: "scales" constant: tensor( FLOAT, [8], [0.015, 0.012, 0.014, 0.013, 0.016, 0.011, 0.015, 0.012] ) } field: { name: "zero_points" constant: tensor(INT64, [8], [8, 7, 8, 8, 7, 8, 9, 8]) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } field: { name: "storage_type" constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "axis", constant: tensor(INT32, [], -1) } } } NF4 (QLoRA / bitsandbytes) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { codebook: CodebookProto { scale_float: 1.0, codebook_data: [-1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911, 0.0, 0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1.0], packed_count: 2, packed_bytes: 1 } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "NF4" structure: Structure { field: { name: "indices" type: array(UINT8, dimension=packed_byte_count) } field: { name: "codebook" constant: tensor( FLOAT, [16], [-1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911, 0.0, 0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1.0] ) } field: { name: "scale", constant: tensor(FLOAT, [], 1.0) } field: { name: "packed_count", constant: tensor(INT32, [], 2) } field: { name: "packed_bytes", constant: tensor(INT32, [], 1) } field: { name: "num_codebooks", constant: tensor(INT32, [], 1) } field: { name: "codebook_size", constant: tensor(INT32, [], 16) } field: { name: "vector_size", constant: tensor(INT32, [], 1) } field: { name: "index_bits", constant: tensor(INT32, [], 0) } } } AWQ INT4 (per-group of 128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [128] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.01, zero_point: 8, axis: -1 }}, // ... one per block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "AWQ_INT4_BLOCK_128" structure: Structure { field: { name: "values", type: array(INT4, dimension=128) } field: { name: "storage_type" constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.01) } field: { name: "zero_point", constant: tensor(INT64, [], 8) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "AWQ_INT4_TILES" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } } } ] } Q2_K (llama.cpp, nested 256/16) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [256] elem_quant: [ QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [16] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: UINT2, bits: 2, symmetric: false, scale_float: 0.005, zero_point: 1, axis: -1 }}, // ... 16 sub-blocks per super-block ] } }, // ... one per super-block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "Q2_K_SUBBLOCK_16" structure: Structure { field: { name: "values", type: array(UINT8, dimension=4) } field: { name: "storage_type" constant: tensor(INT32, [], UINT2) } field: { name: "bits", constant: tensor(INT32, [], 2) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.005) } field: { name: "zero_point", constant: tensor(INT64, [], 1) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "Q2_K_SUPERBLOCK_256" structure: Structure { field: { name: "subblocks" type: array(struct(type_index=0), dimension=16) } field: { name: "tile_shape" constant: tensor(INT64, [1], [16]) } } }, StructTypeProto { name: "Q2_K_TENSOR" structure: Structure { field: { name: "superblocks" type: array( struct(type_index=1), dimension=superblock_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } } } ] } Q3_K (llama.cpp, nested 256/32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [256] elem_quant: [ QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: UINT4, bits: 3, symmetric: false, scale_float: 0.004, zero_point: 3, axis: -1 }}, // ... 8 sub-blocks per super-block ] } }, // ... one per super-block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "Q3_K_SUBBLOCK_32" structure: Structure { field: { name: "values", type: array(UINT8, dimension=12) } field: { name: "storage_type" constant: tensor(INT32, [], UINT4) } field: { name: "bits", constant: tensor(INT32, [], 3) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.004) } field: { name: "zero_point", constant: tensor(INT64, [], 3) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "Q3_K_SUPERBLOCK_256" structure: Structure { field: { name: "subblocks" type: array(struct(type_index=0), dimension=8) } field: { name: "tile_shape" constant: tensor(INT64, [1], [32]) } } }, StructTypeProto { name: "Q3_K_TENSOR" structure: Structure { field: { name: "superblocks" type: array( struct(type_index=1), dimension=superblock_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } } } ] } Q4_K (llama.cpp, nested 256/32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [256] elem_quant: [ QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: UINT4, bits: 4, symmetric: false, scale_float: 0.003, zero_point: 2, axis: -1 }}, // ... 8 sub-blocks per super-block ] } }, // ... one per super-block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "Q4_K_SUBBLOCK_32" structure: Structure { field: { name: "values", type: array(UINT4, dimension=32) } field: { name: "storage_type" constant: tensor(INT32, [], UINT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.003) } field: { name: "zero_point", constant: tensor(INT64, [], 2) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "Q4_K_SUPERBLOCK_256" structure: Structure { field: { name: "subblocks" type: array(struct(type_index=0), dimension=8) } field: { name: "tile_shape" constant: tensor(INT64, [1], [32]) } } }, StructTypeProto { name: "Q4_K_TENSOR" structure: Structure { field: { name: "superblocks" type: array( struct(type_index=1), dimension=superblock_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } } } ] } Q5_K (llama.cpp, nested 256/32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [256] elem_quant: [ QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: UINT8, bits: 5, symmetric: false, scale_float: 0.002, zero_point: 4, axis: -1 }}, // ... 8 sub-blocks per super-block ] } }, // ... one per super-block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "Q5_K_SUBBLOCK_32" structure: Structure { field: { name: "values", type: array(UINT8, dimension=20) } field: { name: "storage_type" constant: tensor(INT32, [], UINT8) } field: { name: "bits", constant: tensor(INT32, [], 5) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.002) } field: { name: "zero_point", constant: tensor(INT64, [], 4) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "Q5_K_SUPERBLOCK_256" structure: Structure { field: { name: "subblocks" type: array(struct(type_index=0), dimension=8) } field: { name: "tile_shape" constant: tensor(INT64, [1], [32]) } } }, StructTypeProto { name: "Q5_K_TENSOR" structure: Structure { field: { name: "superblocks" type: array( struct(type_index=1), dimension=superblock_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } } } ] } Q6_K (llama.cpp, nested 256/16) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [256] elem_quant: [ QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [16] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: UINT8, bits: 6, symmetric: false, scale_float: 0.001, zero_point: 5, axis: -1 }}, // ... 16 sub-blocks per super-block ] } }, // ... one per super-block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "Q6_K_SUBBLOCK_16" structure: Structure { field: { name: "values", type: array(UINT8, dimension=12) } field: { name: "storage_type" constant: tensor(INT32, [], UINT8) } field: { name: "bits", constant: tensor(INT32, [], 6) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.001) } field: { name: "zero_point", constant: tensor(INT64, [], 5) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "Q6_K_SUPERBLOCK_256" structure: Structure { field: { name: "subblocks" type: array(struct(type_index=0), dimension=16) } field: { name: "tile_shape" constant: tensor(INT64, [1], [16]) } } }, StructTypeProto { name: "Q6_K_TENSOR" structure: Structure { field: { name: "superblocks" type: array( struct(type_index=1), dimension=superblock_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } } } ] } MXFP4 (OCP Microscaling, shared exponent per group of 32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32] elem_quant: [ QuantizationProto { floating_point: FloatingPointUniformProto { sign_bits: 1, exponent_bits: 2, mantissa_bits: 1, exponent_bias: 1, has_inf: false, has_nan: false, split_storage: false, packed_count: 2, packed_bytes: 1 }}, // ... one per block, shared exponent at super-block level ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "MXFP4_E2M1_BLOCK_32" structure: Structure { field: { name: "values" type: array(FLOAT4E2M1, dimension=32) } field: { name: "scale" type: array(FLOAT8E8M0, dimension=1) } field: { name: "sign_bits", constant: tensor(INT32, [], 1) } field: { name: "exponent_bits", constant: tensor(INT32, [], 2) } field: { name: "mantissa_bits", constant: tensor(INT32, [], 1) } field: { name: "exponent_bias", constant: tensor(INT32, [], 1) } field: { name: "has_inf", constant: tensor(BOOL, [], false) } field: { name: "has_nan", constant: tensor(BOOL, [], false) } field: { name: "split_storage", constant: tensor(BOOL, [], false) } field: { name: "packed_count", constant: tensor(INT32, [], 2) } field: { name: "packed_bytes", constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "MXFP4_TILES" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [32]) } } } ] } MXFP6 E3M2 (OCP Microscaling, 6-bit float per group of 32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32] elem_quant: [ QuantizationProto { floating_point: FloatingPointUniformProto { sign_bits: 1, exponent_bits: 3, mantissa_bits: 2, exponent_bias: 3, has_inf: false, has_nan: false, split_storage: false, packed_count: 4, packed_bytes: 3 }}, // ... one per block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "MXFP6_E3M2_BLOCK_32" structure: Structure { field: { name: "values", type: array(UINT8, dimension=24) } field: { name: "scale" type: array(FLOAT8E8M0, dimension=1) } field: { name: "sign_bits", constant: tensor(INT32, [], 1) } field: { name: "exponent_bits", constant: tensor(INT32, [], 3) } field: { name: "mantissa_bits", constant: tensor(INT32, [], 2) } field: { name: "exponent_bias", constant: tensor(INT32, [], 3) } field: { name: "has_inf", constant: tensor(BOOL, [], false) } field: { name: "has_nan", constant: tensor(BOOL, [], false) } field: { name: "split_storage", constant: tensor(BOOL, [], false) } field: { name: "packed_count", constant: tensor(INT32, [], 4) } field: { name: "packed_bytes", constant: tensor(INT32, [], 3) } } }, StructTypeProto { name: "MXFP6_E3M2_TILES" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [32]) } } } ] } FP6 LLM (DeepSpeed TC-FPn, per-group of 128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Same FP6 E3M2 quantization as MXFP6 but with split storage (4-bit sign+exponent and 2-bit mantissa stored separately) for Tensor Core alignment. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [128] elem_quant: [ QuantizationProto { floating_point: FloatingPointUniformProto { sign_bits: 1 exponent_bits: 3 mantissa_bits: 2 exponent_bias: 3 has_inf: false has_nan: false split_storage: true packed_count: 4 packed_bytes: 3 } } ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { // index 0: four FLOAT6_E3M2 values name: "FLOAT6_E3M2" structure: Structure { field: { name: "sign_exponent" type: bit_packing( dimension=4, components=[sign:1, exponent:3] ) } field: { name: "mantissa" type: array(UINT2, dimension=4) } field: { name: "exponent_bias", constant: tensor(INT32, [], 3) } field: { name: "has_inf", constant: tensor(BOOL, [], false) } field: { name: "has_nan", constant: tensor(BOOL, [], false) } } }, StructTypeProto { // index 1: one tile name: "FP6_LLM_TILE_128" structure: Structure { field: { name: "values" type: array( struct(type_index=0), dimension=32 ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } } } ] } INT8 per-channel (classic CNN) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { linear: LinearUniformProto { storage_type: INT8, bits: 8, symmetric: true, scale_float: 0.03, zero_point: 0, axis: 0 } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "INT8_PER_CHANNEL" structure: Structure { field: { name: "values", type: array(INT8, dimension=N) } field: { name: "storage_type" constant: tensor(INT32, [], INT8) } field: { name: "bits", constant: tensor(INT32, [], 8) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 0.03) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], 0) } } } 1.58-bit ternary (BitNet b1.58) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { codebook: CodebookProto { scale_float: 0.5, codebook_data: [-1.0, 0.0, 1.0], packed_count: 5, packed_bytes: 1 } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "TERNARY_1_58" structure: Structure { field: { name: "indices" type: array(UINT8, dimension=packed_byte_count) } field: { name: "codebook" constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0]) } field: { name: "scale", constant: tensor(FLOAT, [], 0.5) } field: { name: "packed_count", constant: tensor(INT32, [], 5) } field: { name: "packed_bytes", constant: tensor(INT32, [], 1) } field: { name: "num_codebooks", constant: tensor(INT32, [], 1) } field: { name: "codebook_size", constant: tensor(INT32, [], 3) } field: { name: "vector_size", constant: tensor(INT32, [], 1) } field: { name: "index_bits", constant: tensor(INT32, [], 0) } } } FP8 E4M3 (per-tensor) ^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { linear: LinearUniformProto { storage_type: FLOAT8E4M3FN, bits: 8, symmetric: true, scale_float: 1.0, zero_point: 0, axis: -1 } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "FP8_E4M3_PER_TENSOR" structure: Structure { field: { name: "values" type: array(FLOAT8E4M3FN, dimension=N) } field: { name: "storage_type" constant: tensor(INT32, [], FLOAT8E4M3FN) } field: { name: "bits", constant: tensor(INT32, [], 8) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 1.0) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], -1) } } } AQLM 2×8 (Additive Quantization, 2 bits/weight) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { codebook: CodebookProto { num_codebooks: 2, codebook_size: 256, vector_size: 8, index_bits: 8, codebook_data: [...] // 2 * 256 * 8 = 4096 floats } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "AQLM_2X8" structure: Structure { field: { name: "indices" type: array(UINT8, dimensions=[vector_count, 2]) } field: { name: "codebooks" constant: tensor( FLOAT, [2, 256, 8], aqlm_codebook_data ) } field: { name: "num_codebooks", constant: tensor(INT32, [], 2) } field: { name: "codebook_size", constant: tensor(INT32, [], 256) } field: { name: "vector_size", constant: tensor(INT32, [], 8) } field: { name: "index_bits", constant: tensor(INT32, [], 8) } } } IQ1_S (llama.cpp, 1.56 bits/weight, E8 lattice) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Uses a 2048-entry vector codebook (E8 lattice grid, 8 values per entry). Each sub-block of 8 weights is looked up with an 11-bit index. Wrapped in a block of 256 for the shared FP16 scale. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [256] elem_quant: [ QuantizationProto { codebook: CodebookProto { num_codebooks: 1, codebook_size: 2048, vector_size: 8, index_bits: 11, codebook_data: [...] // 2048 * 8 = 16384 values in {-1, 0, 1} }}, // ... one per super-block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "IQ1_S_BLOCK_256" structure: Structure { field: { name: "indices" type: array(UINT8, dimension=44) } field: { name: "scale" type: array(FLOAT16, dimension=1) } field: { name: "codebook" constant: tensor( FLOAT, [2048, 8], iq1_s_e8_codebook ) } field: { name: "num_codebooks" constant: tensor(INT32, [], 1) } field: { name: "codebook_size" constant: tensor(INT32, [], 2048) } field: { name: "vector_size" constant: tensor(INT32, [], 8) } field: { name: "index_bits" constant: tensor(INT32, [], 11) } } }, StructTypeProto { name: "IQ1_S_TILES" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } } } ] } SpQR (Sparse Quantization, ~3.4 bits/weight) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Outliers (>1% of values) stored in FP16, rest in INT3 per-group. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { sparse: SparseQuantizationProto { base_quant: QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [16] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 3, symmetric: false, scale_float: 0.01, zero_point: 4, axis: -1 }}, // ... one per block ] } }, outlier_data_type: FLOAT16, outlier_threshold: 6.0, outlier_ratio: 0.01 } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "SPQR_INT3_BLOCK_16" structure: Structure { field: { name: "values", type: array(UINT8, dimension=6) } field: { name: "storage_type" constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 3) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.01) } field: { name: "zero_point", constant: tensor(INT64, [], 4) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "SPQR_DENSE_TILES" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=dense_block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [16]) } } }, StructTypeProto { name: "SPQR" structure: Structure { field: { name: "base" type: struct(type_index=1) } field: { name: "outlier_indices" type: array(INT64, dimension=outlier_count) } field: { name: "outlier_values" type: array(FLOAT16, dimension=outlier_count) } field: { name: "outlier_data_type" constant: tensor(INT32, [], FLOAT16) } field: { name: "outlier_threshold" constant: tensor(FLOAT, [], 6.0) } field: { name: "outlier_ratio" constant: tensor(FLOAT, [], 0.01) } } } ] } QuIP# (Vector quantization with Hadamard rotation) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { codebook: CodebookProto { num_codebooks: 1, codebook_size: 256, vector_size: 8, index_bits: 8, codebook_data: [...] // learned codebook, 256 * 8 = 2048 floats }, pre_rotation: RotationProto { matrix_type: HADAMARD, dims: [4096, 4096] }, post_rotation: RotationProto { matrix_type: HADAMARD, dims: [4096, 4096] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "QUIP_SHARP" structure: Structure { field: { name: "indices" type: array(UINT8, dimension=vector_count) } field: { name: "codebook" constant: tensor( FLOAT, [256, 8], quip_sharp_codebook ) } field: { name: "num_codebooks" constant: tensor(INT32, [], 1) } field: { name: "codebook_size" constant: tensor(INT32, [], 256) } field: { name: "vector_size" constant: tensor(INT32, [], 8) } field: { name: "index_bits" constant: tensor(INT32, [], 8) } field: { name: "pre_rotation_type" constant: tensor(INT32, [], 0) // HADAMARD } field: { name: "pre_rotation_dims" constant: tensor(INT32, [2], [4096, 4096]) } field: { name: "post_rotation_type" constant: tensor(INT32, [], 0) // HADAMARD } field: { name: "post_rotation_dims" constant: tensor(INT32, [2], [4096, 4096]) } } } ] } EXL2 (variable bits per layer, ~3.5 bpw average) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ EXL2 assigns different bit-widths per layer to hit a target average bpw. Each layer gets its own ``QuantizationProto`` (via ``quantized_type`` index), so a model may use several quantization entries — e.g. 2-bit for less important layers and 4-bit for critical ones. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Layer A (less important): 2-bit per group of 128 QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [128] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 2, symmetric: false, scale_float: 0.005, zero_point: 2, axis: -1 }}, // ... one per block ] } } // Layer B (critical): 4-bit per group of 128 QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [128] elem_quant: [ QuantizationProto { linear: LinearUniformProto { storage_type: INT4, bits: 4, symmetric: false, scale_float: 0.01, zero_point: 8, axis: -1 }}, // ... one per block ] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "EXL2_INT2_BLOCK_128" structure: Structure { field: { name: "values", type: array(UINT8, dimension=32) } field: { name: "storage_type" constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 2) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.005) } field: { name: "zero_point", constant: tensor(INT64, [], 2) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "EXL2_INT4_BLOCK_128" structure: Structure { field: { name: "values", type: array(INT4, dimension=128) } field: { name: "storage_type" constant: tensor(INT32, [], INT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.01) } field: { name: "zero_point", constant: tensor(INT64, [], 8) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "EXL2_LAYER_A" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=layer_a_block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } } }, StructTypeProto { name: "EXL2_LAYER_B" structure: Structure { field: { name: "blocks" type: array( struct(type_index=1), dimension=layer_b_block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } } } ] } Log quantization (4-bit) ^^^^^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { log: LogUniformProto { bits: 4, base: 2.0, offset: -7.0, has_sign: true } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "LOG4" structure: Structure { field: { name: "codes", type: array(UINT4, dimension=N) } field: { name: "signs" type: array(UINT8, dimension=sign_byte_count) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "base", constant: tensor(FLOAT, [], 2.0) } field: { name: "offset", constant: tensor(FLOAT, [], -7.0) } field: { name: "has_sign", constant: tensor(BOOL, [], true) } } } Custom (plugin-based) ^^^^^^^^^^^^^^^^^^^^^ .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { function: FunctionUniformProto { storage_type: UINT4, bits: 4, quantize_op: "vendor::QuantizeV2", dequantize_op: "vendor::DequantizeV2" } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "VENDOR_QUANTIZE_V2" structure: Structure { field: { name: "payload", type: array(UINT4, dimension=N) } field: { name: "storage_type" constant: tensor(INT32, [], UINT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } } metadata_props: { key: "quantize_op" value: "vendor::QuantizeV2" } metadata_props: { key: "dequantize_op" value: "vendor::DequantizeV2" } } MatMulNBits INT4 (onnxruntime, per-group of 32, tiled) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Weights of shape ``[K, N]`` are tiled into blocks of 32 along axis K and 128 along axis N (SIMD width). All tiles share the same INT4 quantization scheme. ``perm: [1, 0]`` indicates N-major tile ordering in memory for cache locality. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Weight shape: [4096, 4096], group_size=32, N_tile=128 QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32, 128], axes: [0, 1], elem_quant: [QuantizationProto { linear: LinearUniformProto { storage_type: UINT4, bits: 4, symmetric: true, scale_float: 0.0, zero_point: 0, axis: -1 }}], perm: [1, 0] // N-major ordering in memory } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "MATMUL_NBITS_INT4_TILE_32X128" structure: Structure { field: { name: "values" type: array(UINT4, dimensions=[32, 128]) } field: { name: "storage_type" constant: tensor(INT32, [], UINT4) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 0.0) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "MATMUL_NBITS_INT4_4096X4096" structure: Structure { field: { name: "tiles" type: array( struct(type_index=0), dimensions=[32, 128] ) } field: { name: "tile_shape" constant: tensor(INT64, [2], [32, 128]) } field: { name: "axes" constant: tensor(INT32, [2], [0, 1]) } field: { name: "perm" constant: tensor(INT32, [2], [1, 0]) } } } ] } Block-tiled floats (no quantization) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Reorganizes a row-major float matrix into tiles of ``[32, 128]`` without any value transformation. Useful for prepack layouts that expect block-tiled storage. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Weight shape: [4096, 4096], tiled into [32, 128] blocks QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [32, 128], axes: [0, 1], elem_quant: [QuantizationProto { data_type: FLOAT, cast: CastUniformProto { storage_type: FLOAT } }] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "FLOAT_TILE_32X128" array: Array { element_type: FLOAT dimension: 32 dimension: 128 } }, StructTypeProto { name: "BLOCK_TILED_FLOAT_4096X4096" structure: Structure { field: { name: "tiles" type: array( struct(type_index=0), dimensions=[128, 32] ) } field: { name: "tile_shape" constant: tensor(INT64, [2], [32, 128]) } field: { name: "axes" constant: tensor(INT32, [2], [0, 1]) } } } ] } STQ1_0 (Sherry, 1.25 bits/weight, ternary codebook) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 256 values per 42-byte block. Each 5-bit combined index (4-bit code + 1-bit sign) selects from a 32-entry codebook of 4 ternary values. Output positions use a stride-16 scatter within each 64-value chunk. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { data_type: FLOAT, structured_block: StructuredBlockUniformProto { block_layout: BlockLayoutProto { block_size: 256, bytes_per_block: 42, fields: [ BlockFieldProto { role: CODE, bit_offset: 0, bit_width: 4, count: 64, data_type: 0 }, BlockFieldProto { role: SIGN, bit_offset: 256, bit_width: 1, count: 64, data_type: 0 }, BlockFieldProto { role: SCALE, bit_offset: 320, bit_width: 16, count: 1, data_type: FLOAT16 }, ] }, codebook_data: [...] // 32 * 4 = 128 ternary values in {-1, 0, 1} codebook_vector_size: 4, index_formula: [ FieldWeightProto { field: CODE, multiplier: 1 }, FieldWeightProto { field: SIGN, multiplier: 16 }, ], scatter: ScatterProto { group_size: 64, vector_size: 4, stride: 16 } } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "STQ1_0_BLOCK_256" structure: Structure { field: { name: "code", type: array(UINT4, dimension=64) } field: { name: "sign", type: array(UINT8, dimension=8) } field: { name: "scale", type: array(FLOAT16, dimension=1) } field: { name: "field_roles" constant: tensor(INT32, [3], [6, 1, 2]) } field: { name: "sign_bit_width" constant: tensor(INT32, [], 1) } field: { name: "sign_count" constant: tensor(INT32, [], 64) } field: { name: "block_size" constant: tensor(INT32, [], 256) } } }, StructTypeProto { name: "STQ1_0" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "codebook" constant: tensor( FLOAT, [32, 4], stq1_0_codebook ) } field: { name: "codebook_vector_size" constant: tensor(INT32, [], 4) } field: { name: "index_fields" constant: tensor(INT32, [2], [6, 1]) } field: { name: "index_multipliers" constant: tensor(INT32, [2], [1, 16]) } field: { name: "scatter_group_size" constant: tensor(INT32, [], 64) } field: { name: "scatter_vector_size" constant: tensor(INT32, [], 4) } field: { name: "scatter_stride" constant: tensor(INT32, [], 16) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT) } } } ] } Dequantization of one block (derived from proto fields): .. code-block:: python sb = proto.structured_block bl = sb.block_layout code_field = bl.fields[role == CODE] # bit_offset=0, bit_width=4, count=64 sign_field = bl.fields[role == SIGN] # bit_offset=256, bit_width=1, count=64 scale_field = bl.fields[role == SCALE] # bit_offset=320, bit_width=16, count=1 scale = extract(block, scale_field.bit_offset, scale_field.bit_width, scale_field.data_type) for g in range(code_field.count): # Extract fields code = extract(block, code_field.bit_offset + code_field.bit_width * g, code_field.bit_width) sign = extract(block, sign_field.bit_offset + sign_field.bit_width * g, sign_field.bit_width) # Combine fields into codebook index using index_formula index = sum(values[fw.field] * fw.multiplier for fw in sb.index_formula) # = code * 1 + sign * 16 # Codebook lookup vector = sb.codebook_data[index * sb.codebook_vector_size : (index+1) * sb.codebook_vector_size] # Scatter to output positions s = sb.scatter groups_per_chunk = s.group_size // s.vector_size # 64 // 4 = 16 chunk = g // groups_per_chunk lane = g % groups_per_chunk for p in range(s.vector_size): output[s.group_size * chunk + lane + s.stride * p] = vector[p] * scale Column-major layout ^^^^^^^^^^^^^^^^^^^ Expresses a column-major (Fortran-order) storage of a 2D matrix. A ``tile_shape`` value of 0 means "same as the tensor dimension on that axis", so ``[0, 0]`` is a single tile covering the whole tensor. ``perm: [1, 0]`` reverses the axis order in memory. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Weight shape: [M, N], stored column-major QuantizationProto { tiling: TilingQuantizationProto { tile_shape: [0, 0], axes: [0, 1], elem_quant: [QuantizationProto { data_type: FLOAT, cast: CastUniformProto { storage_type: FLOAT } }], perm: [1, 0] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "FLOAT_MATRIX_MXN" array: Array { element_type: FLOAT dimension: M dimension: N } }, StructTypeProto { name: "COLUMN_MAJOR_FLOAT_MXN" structure: Structure { field: { name: "tiles" type: array( struct(type_index=0), dimension=1 ) } field: { name: "tile_shape" constant: tensor(INT64, [2], [0, 0]) } field: { name: "axes" constant: tensor(INT32, [2], [0, 1]) } field: { name: "perm" constant: tensor(INT32, [2], [1, 0]) } } } ] } TQ1_0 (llama.cpp, 1.6 bits/weight, ternary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Pure ternary: each weight is {-1, 0, +1}. Five ternary values are packed into one byte (3⁵ = 243 ≤ 255). One FP16 scale per block of 256 weights. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Codebook: [-1, 0, +1], packed 5 per byte QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [256], elem_quant: [QuantizationProto { codebook: CodebookProto { scale_float: 1.0, // scale is per-block, stored separately codebook_data: [-1.0, 0.0, 1.0], packed_count: 5, // 5 ternary values per byte packed_bytes: 1 } }] } } // Dequantize: for each block of 256 weights // indices = unpack_base3(data, 5 values/byte) // result = [codebook[i] * scale for i in indices] // = [{-1,0,+1}[i] * d for i in indices] .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "TQ1_0_BLOCK_256" structure: Structure { field: { name: "indices", type: array(UINT8, dimension=52) } field: { name: "scale", type: array(FLOAT16, dimension=1) } field: { name: "codebook" constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0]) } field: { name: "packed_count" constant: tensor(INT32, [], 5) } field: { name: "packed_bytes" constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "TQ1_0" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } TQ2_0 (llama.cpp, 2 bits/weight, ternary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ternary with 2 bits per weight (simpler packing, 1 unused state). Mapping: 0 → -1, 1 → 0, 2 → +1. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Codebook: [-1, 0, +1], 2 bits per value QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [256], elem_quant: [QuantizationProto { codebook: CodebookProto { scale_float: 1.0, codebook_data: [-1.0, 0.0, 1.0], packed_count: 4, // 4 values per byte (2 bits each) packed_bytes: 1 } }] } } // Dequantize: same as TQ1_0 but simpler unpack (2 bits, not base-3) .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "TQ2_0_BLOCK_256" structure: Structure { field: { name: "indices", type: array(UINT8, dimension=64) } field: { name: "scale", type: array(FLOAT16, dimension=1) } field: { name: "codebook" constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0]) } field: { name: "packed_count" constant: tensor(INT32, [], 4) } field: { name: "packed_bytes" constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "TQ2_0" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } BitNet b1.58 (Microsoft, 1.58 bits/weight, native ternary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model trained natively with ternary weights {-1, 0, +1}. Storage is identical to TQ1_0 (5 ternary values per byte). The difference is in training (QAT), not in the storage format. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Same storage as TQ1_0 QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [256], elem_quant: [QuantizationProto { codebook: CodebookProto { scale_float: 1.0, codebook_data: [-1.0, 0.0, 1.0], packed_count: 5, packed_bytes: 1 } }] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "BITNET_B1_58_BLOCK_256" structure: Structure { field: { name: "indices", type: array(UINT8, dimension=52) } field: { name: "scale", type: array(FLOAT16, dimension=1) } field: { name: "codebook" constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0]) } field: { name: "packed_count" constant: tensor(INT32, [], 5) } field: { name: "packed_bytes" constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "BITNET_B1_58" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } ParetoQ (Meta, 1.58–2 bits/weight, SEQ ternary/2-bit) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Quantization-aware training with Stretched Elastic Quantization (SEQ). The storage format is standard ternary or 2-bit; the custom SEQ function is used only during training. At inference, the model uses a standard codebook. If the SEQ rounding function is needed at inference, use ``FunctionUniformProto``. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Ternary variant (same storage as TQ1_0) QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [256], elem_quant: [QuantizationProto { codebook: CodebookProto { scale_float: 1.0, codebook_data: [-1.0, 0.0, 1.0], packed_count: 5, packed_bytes: 1 } }] } } // 2-bit variant (4 levels) QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [128], elem_quant: [QuantizationProto { linear: LinearUniformProto { storage_type: UINT8, bits: 2, symmetric: false, scale_float: 0.023, zero_point: 1, axis: -1 } }] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "PARETOQ_TERNARY_BLOCK_256" structure: Structure { field: { name: "indices", type: array(UINT8, dimension=52) } field: { name: "scale", type: array(FLOAT16, dimension=1) } field: { name: "codebook" constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0]) } field: { name: "packed_count" constant: tensor(INT32, [], 5) } field: { name: "packed_bytes" constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "PARETOQ_UINT2_BLOCK_128" structure: Structure { field: { name: "values", type: array(UINT8, dimension=32) } field: { name: "storage_type" constant: tensor(INT32, [], UINT8) } field: { name: "bits", constant: tensor(INT32, [], 2) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.023) } field: { name: "zero_point", constant: tensor(INT64, [], 1) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "PARETOQ_TERNARY" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=ternary_block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } }, StructTypeProto { name: "PARETOQ_UINT2" structure: Structure { field: { name: "blocks" type: array( struct(type_index=1), dimension=uint2_block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } Tequila (ICLR 2026, 1.58 bits/weight, deadzone-free ternary) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Improved ternary QAT that removes the dead-zone around zero. Storage is identical to TQ1_0/BitNet (ternary codebook). The deadzone-free quantization function is only used during training. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Same storage as TQ1_0 / BitNet b1.58 QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [256], elem_quant: [QuantizationProto { codebook: CodebookProto { scale_float: 1.0, codebook_data: [-1.0, 0.0, 1.0], packed_count: 5, packed_bytes: 1 } }] } } // Note: the difference with BitNet is in training only. // At inference, both use the same ternary codebook. .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "TEQUILA_BLOCK_256" structure: Structure { field: { name: "indices", type: array(UINT8, dimension=52) } field: { name: "scale", type: array(FLOAT16, dimension=1) } field: { name: "codebook" constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0]) } field: { name: "packed_count" constant: tensor(INT32, [], 5) } field: { name: "packed_bytes" constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "TEQUILA" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [256]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } EETQ (INT8 weight-only per-channel) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Simple INT8 per-channel weight-only quantization (NetEase FuXi). No calibration, no QAT. Uses optimized W8A16 GEMM kernels from FasterTransformer / TensorRT-LLM. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { data_type: FLOAT16, linear: LinearUniformProto { storage_type: INT8, bits: 8, symmetric: true, scale_float: 0.0042, // per-channel scale zero_point: 0, axis: 0 // per output channel (row) } } .. tab-item:: Custom types .. code-block:: text StructTypeProto { name: "EETQ_INT8_PER_CHANNEL" structure: Structure { field: { name: "values", type: array(INT8, dimension=N) } field: { name: "storage_type" constant: tensor(INT32, [], INT8) } field: { name: "bits", constant: tensor(INT32, [], 8) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 0.0042) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], 0) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } EXL3 (improved EXL2, 2–6 bpw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Successor to EXL2 with improved codebook and mixed-precision per layer. Storage is structurally identical to EXL2 (variable bpw per layer using nested ``TilingQuantizationProto``). .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Same structure as EXL2, different calibration QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [128], elem_quant: [QuantizationProto { linear: LinearUniformProto { storage_type: UINT8, bits: 3, // varies per layer (2–6) symmetric: false, scale_float: 0.015, zero_point: 4, axis: -1 } }] } } .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "EXL3_UINT3_BLOCK_128" structure: Structure { field: { name: "values", type: array(UINT8, dimension=48) } field: { name: "storage_type" constant: tensor(INT32, [], UINT8) } field: { name: "bits", constant: tensor(INT32, [], 3) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.015) } field: { name: "zero_point", constant: tensor(INT64, [], 4) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "EXL3_LAYER" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } HQQ (Hybrid Quantization, 2–4 bits, mixed-precision per head) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Different bit-widths for Q, K, V projections in attention layers. Each tensor gets its own ``QuantizationProto``; the example shows a 2-bit weight with per-group scales. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text // Example: 2-bit weight with group size 64 QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [64], elem_quant: [QuantizationProto { linear: LinearUniformProto { storage_type: UINT8, bits: 2, symmetric: false, scale_float: 0.032, zero_point: 2, axis: -1 } }] } } // In practice, Q proj may use 4-bit while V proj uses 2-bit. // Each tensor simply references a different QuantizationProto index. .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "HQQ_UINT2_BLOCK_64" structure: Structure { field: { name: "values", type: array(UINT8, dimension=16) } field: { name: "storage_type" constant: tensor(INT32, [], UINT8) } field: { name: "bits", constant: tensor(INT32, [], 2) } field: { name: "symmetric", constant: tensor(BOOL, [], false) } field: { name: "scale", constant: tensor(FLOAT, [], 0.032) } field: { name: "zero_point", constant: tensor(INT64, [], 2) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "HQQ_HEAD" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [64]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } NVFP4 (NVIDIA FP4 E2M1 with FP8 scale, 4.5 bpw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ NVIDIA's block floating-point FP4 format. Each element is E2M1 (2 exponent bits, 1 mantissa bit), with one FP8 E4M3 scale per block of 16 elements. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { data_type: FLOAT16, tiling: TilingQuantizationProto { tile_shape: [16], elem_quant: [QuantizationProto { floating_point: FloatingPointUniformProto { sign_bits: 1, exponent_bits: 2, mantissa_bits: 1, exponent_bias: 1, has_inf: false, has_nan: false, packed_count: 2, // 2 FP4 values per byte packed_bytes: 1 } }] } } // Dequantize: for each block of 16 elements // values = [fp_decode(v, 1, 2, 1, bias=1) for v in unpack_fp4(data)] // result = values * scale_fp8 .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "NVFP4_E2M1_BLOCK_16" structure: Structure { field: { name: "values" type: array(FLOAT4E2M1, dimension=16) } field: { name: "scale" type: array(FLOAT8E4M3FN, dimension=1) } field: { name: "sign_bits", constant: tensor(INT32, [], 1) } field: { name: "exponent_bits", constant: tensor(INT32, [], 2) } field: { name: "mantissa_bits", constant: tensor(INT32, [], 1) } field: { name: "exponent_bias", constant: tensor(INT32, [], 1) } field: { name: "has_inf", constant: tensor(BOOL, [], false) } field: { name: "has_nan", constant: tensor(BOOL, [], false) } field: { name: "split_storage", constant: tensor(BOOL, [], false) } field: { name: "packed_count", constant: tensor(INT32, [], 2) } field: { name: "packed_bytes", constant: tensor(INT32, [], 1) } } }, StructTypeProto { name: "NVFP4" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [16]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } QuaRot (rotational quantization, 4 bits) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Applies a Hadamard rotation before INT4 quantization to spread outlier magnitudes evenly across channels. The rotation is stored in ``RotationProto``. .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { data_type: FLOAT16, pre_rotation: RotationProto { type: HADAMARD, dims: [4096] }, tiling: TilingQuantizationProto { tile_shape: [128], elem_quant: [QuantizationProto { linear: LinearUniformProto { storage_type: UINT8, bits: 4, symmetric: true, scale_float: 0.018, zero_point: 0, axis: -1 } }] } } // Dequantize: // 1. values = unpack(data, 4 bits) // 2. result = values * scale // 3. result = hadamard_inverse(result) ← RotationProto .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "QUAROT_UINT4_BLOCK_128" structure: Structure { field: { name: "values", type: array(UINT4, dimension=128) } field: { name: "storage_type" constant: tensor(INT32, [], UINT8) } field: { name: "bits", constant: tensor(INT32, [], 4) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 0.018) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], -1) } } }, StructTypeProto { name: "QUAROT" structure: Structure { field: { name: "blocks" type: array( struct(type_index=0), dimension=block_count ) } field: { name: "tile_shape" constant: tensor(INT64, [1], [128]) } field: { name: "pre_rotation_type" constant: tensor(INT32, [], 0) // HADAMARD } field: { name: "pre_rotation_dims" constant: tensor(INT32, [1], [4096]) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT16) } } } ] } SmoothQuant (W8A8, smoothed INT8) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Shifts quantization difficulty from activations to weights via a per-channel scaling transform. The smoothing factor is a diagonal matrix stored as a rotation matrix (``PLAIN`` type). .. tab-set:: .. tab-item:: Specialized proto .. code-block:: text QuantizationProto { data_type: FLOAT, pre_rotation: RotationProto { type: PLAIN, dims: [4096], matrix_index: 0 // index into ModelProto.rotation_matrices }, linear: LinearUniformProto { storage_type: INT8, bits: 8, symmetric: true, scale_float: 0.0042, zero_point: 0, axis: 1 // per-channel } } // Dequantize: // 1. result = int8_values * scale (per-channel) // 2. result = result @ diag(smooth_factors)^(-1) ← RotationProto .. tab-item:: Custom types .. code-block:: text ModelProto { struct_types: [ StructTypeProto { name: "SMOOTHQUANT_INT8" structure: Structure { field: { name: "values" type: array(INT8, dimension=N) } field: { name: "storage_type" constant: tensor(INT32, [], INT8) } field: { name: "bits", constant: tensor(INT32, [], 8) } field: { name: "symmetric", constant: tensor(BOOL, [], true) } field: { name: "scale", constant: tensor(FLOAT, [], 0.0042) } field: { name: "zero_point", constant: tensor(INT64, [], 0) } field: { name: "axis", constant: tensor(INT32, [], 1) } field: { name: "pre_rotation_type" constant: tensor(INT32, [], 1) // PLAIN } field: { name: "pre_rotation_dims" constant: tensor(INT32, [1], [4096]) } field: { name: "pre_rotation" constant: tensor(FLOAT, [4096], smooth_factors) } field: { name: "data_type" constant: tensor(INT32, [], FLOAT) } } } ] } Pseudo-code +++++++++++ The dequantize and quantize pseudo-code for each format is shown inline in the corresponding proto section above. The top-level dispatcher is: .. code-block:: python def dequantize(qtensor: QuantizedTensorProto, model: ModelProto) -> float[]: quant = model.quantizations[qtensor.quantized_type] data = qtensor.raw_data match quant.kind: case LinearUniformProto as q: ... # see LinearUniformProto section case CodebookProto as q: ... # see CodebookProto section case FloatingPointUniformProto: ... case SparseQuantizationProto: ... case LogUniformProto: ... case FunctionUniformProto: ... case TilingQuantizationProto: ... case CastUniformProto: ... case StructuredBlockUniformProto: ... # Apply post-rotation if present if quant.post_rotation: result = apply_rotation(result, quant.post_rotation, model) assert len(result) == product(qtensor.dims) return result