Quantization#

Date:

2026-08

discussion

Note

The structures on this page are descriptive quantization profiles. They should be implemented with Structured views over byte buffers, not as a parallel protobuf hierarchy. Each specialized structure below is followed by its StructTypeProto translation. The families remain useful as format names, validation profiles, and decoder specifications.

Format coverage summary#

Format

bpw

Proto used

Possible

1.58-bit Ternary (BitNet)

1.63

Codebook

✅

AQLM 2×8

3.0

Codebook

✅

Binary/XNOR (1-bit)

1.0

Codebook

✅

EXL2 (variable bpw)

3.5

Tiling + Linear (multiple)

✅

EETQ (INT8 weight-only)

8.0

Linear

✅

EXL3 (improved EXL2)

2–6

Tiling + Linear or Codebook

✅

FP6 LLM (TC-FPn)

6.125

Tiling + FloatingPoint

✅

FP8 E4M3

8.0

FloatingPoint

✅

HQQ (mixed-precision per head)

2–4

Tiling + Linear

✅

INT4 AWQ (per-group)

4.5

Tiling + Linear

✅

INT4 GPTQ (per-group)

4.5

Tiling + Linear

✅

INT4 Symmetric

4.5

Linear

✅

INT8 per-channel

8.0

Linear

✅

INT8 symmetric

8.0

Linear

✅

IQ1_S

1.56

Tiling + Codebook

✅

IQ4_NL

4.5

Codebook

✅

Log quantization

4.0

Log

✅

MatMulNBits INT4 (ORT)

4.5

Tiling + Linear

✅

MXFP4

5.0

Tiling + FloatingPoint

✅

MXFP6 E3M2

6.125

Tiling + FloatingPoint

✅

NF4 (QLoRA)

4.5

Codebook

✅

NVFP4 (E2M1 + FP8 scale)

4.5

Tiling + FloatingPoint

✅

Q2_K

2.625

Tiling × 2 + Linear

✅

Q3_K

3.4

Tiling × 2 + Linear

✅

Q4_K

4.5

Tiling × 2 + Linear

✅

Q5_K

5.5

Tiling × 2 + Linear

✅

Q6_K

6.6

Tiling × 2 + Linear

✅

QuaRot (rotational quantization)

4.0

Linear + RotationProto

✅

QuIP#

2.0

Codebook + rotation

✅

SpQR

3.4

Sparse + Tiling + Linear

✅

SmoothQuant (W8A8)

8.0

Linear + RotationProto

✅

STQ1_0 (Sherry)

1.3125

StructuredBlock

✅

TQ1_0

1.6

Codebook

✅

TQ2_0

2.0

Codebook

✅

BitNet b1.58

1.58

Codebook

✅

ParetoQ (SEQ ternary)

1.58–2.0

Function or Codebook

✅

Tequila (ICLR 2026)

1.58

Codebook

✅

CPU prepacking coverage#

The formats above describe values and portable physical layouts. They do not cover every representation produced by a CPU kernel prepacker. A prepacked tensor is tied to an operand role, a microkernel ABI, and an exact processor feature set; it must therefore be stored as a CompiledTensorProto cache entry while the original initializer remains the portable fallback. Prepacking is useful for floating-point and integer tensors, not only INT4.

The first CPU implementation should cover the following persistent matrix packing families. B denotes the usually constant right-hand operand.

Input family

CPU target

Packed representation

Main consumers

FP32 / FP64

SSE, AVX2, AVX-512F

Transposed or blocked B panels, padded and interleaved by the microkernel N and K tiles

Gemm, MatMul, batched MatMul, linear projections

BF16

AVX-512 BF16

K-paired BF16 words arranged for VDPBF16PS, with an FP32 accumulation tile

Gemm, MatMul, attention projections

FP16

AVX-512 FP16

FP16 B panels in the kernel’s K × N lane order

Gemm, MatMul, attention projections

INT8 / UINT8

AVX2

K-paired or K-quad interleaving for the selected emulated dot-product kernel, plus correction sums when required

QLinearMatMul, MatMulInteger, quantized Gemm

INT8 / UINT8

AVX-512 VNNI

Four K values per dot-product group in VNNI lane order, with per-column sums and zero-point correction data

QLinearMatMul, MatMulInteger, quantized Gemm

INT4 / UINT4

AVX2, AVX-512F/BW

Nibble-interleaved B panels with scales, zero points, and optional group metadata placed in the order consumed by vector unpacking

MatMulNBits, GPTQ, AWQ, HQQ, weight-only Gemm

INT4 / UINT4

AVX-512 VNNI

INT4 panels expanded or streamed into K-quad VNNI groups, with per-group scales and corrections adjacent to their tiles

Weight-only Gemm and MatMul with INT8 activations

INT2–INT7, codebook, binary, ternary

AVX2, AVX-512 BW/VBMI/VBMI2/VPOPCNTDQ

Bit planes, lane-interleaved codes, lookup tables, scales, and sparse outliers arranged for unpack, lookup, or population-count kernels

AQLM, NF4/IQ, BitNet, TQ, SpQR, and other low-bit MatMul kernels

BF16 / INT8

AMX BF16 / AMX INT8

Tile-major B panels matching AMX tile geometry and K grouping

Large Gemm, MatMul, and fused projections

Feature names in this table are compatibility requirements, not format names. For example, AVX-512F, AVX-512 BW, AVX-512 VNNI, AVX-512 BF16, AVX-512 FP16, VBMI, VBMI2, VPOPCNTDQ, and AMX must be distinguished. An avx512-int4 label is insufficient because two kernels may require different subsets or use different MR × NR × KR tiles.

Operator-specific prepacking#

Matrix-panel packing is the base layer, but the plan must also register prepackers for every constant operand whose reuse amortizes preparation:

Operator family

Persistent inputs

Required prepacking

Gemm / MatMul

Constant B and optional bias

Transpose while packing, create K/N panels, pad tails, interleave quantization parameters and correction sums, and optionally fuse bias and epilogue constants

Batched MatMul

Shared or constant batched B

Pack each distinct batch matrix, retain batch/head strides, and share one packed object when broadcasting makes the logical B identical

QLinearMatMul / MatMulInteger / MatMulNBits

Quantized B, scales, zero points, bias, sparse outliers

Pack codes and all auxiliary data in consumption order; precompute column sums and zero-point compensation when the kernel ABI uses them

Conv / QLinearConv

Weights, bias, scales, zero points

Reorder OIHW/HWIO weights into output-channel, input-channel, and spatial tiles for direct, 1x1-GEMM, im2col-GEMM, grouped, or depthwise kernels; each algorithm has a separate packed cache key

Attention projections

Q, K, V, and output projection weights

Pack separate matrices or one fused QKV panel, preserving head and grouped-query-attention boundaries and the selected quantization groups

Attention score/value products

Constant masks or reusable encoder K/V tensors

Store K in the transposed blocked layout used by Q × K^T and V in the blocked layout used by P × V; include head, sequence, and head dimension tiles in the compatibility key

Fused/flash attention

Projection weights and reusable K/V blocks

Pack by the fused kernel’s query, key, value, head, and sequence tiles; precompute causal or constant-mask tiles only when they are model constants

Gather / embedding

Quantized embedding table

Pack row-local codes, scales, and zero points so one row remains independently addressable without decoding neighboring rows

Mixture-of-experts

Expert projection weights

Pack each expert with the common matrix format and add an expert index without merging independently loadable experts into one mandatory blob

Mutable decoder K/V caches are not compiled initializers. Their blocked or quantized runtime layout belongs to Prepared and asynchronous execution and must encode the same head, sequence-tile, feature, and kernel-ABI constraints. A cache populated during inference must never be published as a CompiledTensorProto for a model initializer.

Packed-format contract#

Every registered prepacker must describe enough information to reject an incompatible cache entry without inspecting implementation-specific bytes:

  • source tensor digest, logical shape, element type, and quantization profile;

  • operator and operand role, including transpose flags and broadcast rules;

  • exact required CPU features and, when relevant, cache-line or alignment requirements;

  • kernel library, kernel ABI, packed-format name, and format version;

  • MR, NR, KR, outer tile order, lane interleave, padding, and tail policy;

  • placement and type of scales, zero points, codebooks, bias, column sums, compensation values, and sparse outliers;

  • supported dynamic-shape bounds, batch/head sharing, and fused epilogue;

  • payload size and alignment, plus the source digest required by CompiledTensorProto.

Packed entries must use specific names such as matmul-b-int4-avx512-vnni-nr64-kr4-v2 or attention-qkv-bf16-amx-tile16-v1 rather than broad names such as INT4_AVX512. Selection first matches the exact kernel ABI and CPU features, then applies the Processor-aware kernel tuning packing threshold. On a miss, the runtime prepacks from the portable initializer; it may also execute an unpacked kernel when packing would cost more than it saves.

QuantizedTensorProto#

A quantized tensor cannot rely on shape × sizeof(data_type) to compute its storage size (sub-byte packing, block metadata, sparse outliers, etc.). It carries its own byte size explicitly.

message QuantizedTensorProto {
    repeated int64 dims = 1;       // logical shape of the tensor
    bytes raw_data = 2;            // quantized payload
    int64 n_bytes = 3;            // byte size of raw_data
    int32 quantized_type = 4;      // index into ModelProto.quantizations
    optional QuantizationProto quantization = 5;  // inline if unique to this tensor
    string name = 6;
    string doc_string = 7;
}

StructProto {
    type: <quantization-profile type index>
    raw_data: ...
    name: ...
    doc_string: ...
}

QuantizedTensorProto becomes StructProto. raw_data, external_data, name, and doc_string are carried by that generic value. Its exact model-level or inline StructTypeProto replaces quantized_type and quantization. Counts such as a block or tile number are concrete dimensions of that type. The size computed from the physical type must equal raw_data.size() or external-data length. Logical dimensions and element type belong to the decoder output ValueInfoProto.

In both forms, name identifies the concrete value and doc_string documents it.

TypeProto and containers#

The following specialized container additions are therefore not required by the recommended implementation. They document what would be necessary only if QuantizedTensorProto remained a distinct value category:

message TypeProto {
    message QuantizedTensor {
        repeated int32 allowed_quantized_type = 1;
        optional int32 elem_type = 2;       // decoded logical element type
        optional TensorShapeProto shape = 3;
    }

    oneof value {
        ...
        QuantizedTensor quantized_tensor_type = <N>;
    }
}

message SequenceProto {
    enum DataType {
        ...
        QUANTIZED_TENSOR = <N>;
    }
    repeated QuantizedTensorProto quantized_tensor_values = <N>;
    optional TypeProto value_type = <N+1>;
}

message OptionalProto {
    enum DataType {
        ...
        QUANTIZED_TENSOR = <N>;
    }
    optional QuantizedTensorProto quantized_tensor_value = <N>;
    optional TypeProto value_type = <N+1>;
}

message TypeProto {
    oneof value {
        ...
        StructTypeProto struct_type = <N>;
    }
}

message SequenceProto {
    enum DataType {
        ...
        STRUCT = <N>;
    }
    repeated StructProto struct_values = <N>;
    optional TypeProto value_type = <N+1>;
}

message OptionalProto {
    enum DataType {
        ...
        STRUCT = <N>;
    }
    optional StructProto struct_value = <N>;
    optional TypeProto value_type = <N+1>;
}

allowed_quantized_type is a set of indices into ModelProto.quantizations. An empty set accepts any quantization declaration whose decoded element type and logical shape satisfy the remaining constraints. This allows a sequence or map to contain pages using different quantization formats without becoming untyped.

The corresponding value containers require a new category.

These specialized branches map to the generic integrations defined by Structured views over byte buffers:

  • TypeProto.QuantizedTensor becomes TypeProto.struct_type;

  • QUANTIZED_TENSOR becomes the STRUCT value category;

  • quantized_tensor_values and quantized_tensor_value become struct_values and struct_value;

  • heterogeneous pages use the unconstrained static structured category, while each StructProto carries its exact physical type.

MapProto inherits support because its values are represented by a SequenceProto. Consequently, Sequence<QuantizedTensor> and Map<int64, QuantizedTensor> can represent a paged KV-cache with a different quantization declaration for each page.

value_type is required for these new categories and carries the complete TypeProto.QuantizedTensor constraint. It makes standalone sequence and map values self-describing and must agree with any enclosing ValueInfoProto.

The generalized design in Structured views over byte buffers avoids duplicating this machinery: a quantized tensor is an structured value with an explicit physical type and a decoder with a typed output signature.

QuantizationProto#

// Describes how a tensor is quantized. Nine variants cover the
// common families without separate messages for scalar/vector
// codebooks, one-/multi-dimensional tiling, identity casts, or
// packed linear layouts.
// TilingQuantizationProto can nest QuantizationProto to express
// multi-level hierarchies (e.g. K-Quants, MXFP).
// Optional pre/post rotations support QuIP#, SmoothQuant, etc.
message QuantizationProto {
    oneof kind {
        LinearUniformProto linear = 1;
        CodebookProto codebook = 2;
        FloatingPointUniformProto floating_point = 4;
        SparseQuantizationProto sparse = 5;
        LogUniformProto log = 6;
        FunctionUniformProto function = 7;
        TilingQuantizationProto tiling = 13;
        CastUniformProto cast = 16;
        StructuredBlockUniformProto structured_block = 18;
    }
    int32 data_type = 15;              // dequantized element type (same enum as TensorProto.data_type)
    string doc_string = 9;             // human-readable description
    repeated StringStringEntryProto metadata_props = 10;  // arbitrary key-value metadata
    optional RotationProto pre_rotation = 11;   // rotation applied before quantization
    optional RotationProto post_rotation = 12;  // rotation applied after dequantization
}

ModelProto {
    struct_types: [
        StructTypeProto {
            // One concrete array or structure and its decoder.
        },
        ...
    ]
}

There is no direct QuantizationProto message in the recommended schema. Each entry becomes one concrete StructTypeProto in ModelProto.struct_types:

  • the selected profile determines its array or structure;

  • scalar parameters and lookup tables become fields with constant;

  • data_type and the logical shape are declared by the decoder output ValueInfoProto;

  • pre/post rotations become decoder or encoder nodes;

  • doc_string and metadata_props remain type documentation and metadata.

The translation is lossless: every specialized field must map to serialized structure, a named constant field, decoder/encoder structure, or metadata. A decoder must not hide a profile parameter that was explicit in the specialized declaration.

Code blocks containing names such as N, K, packed_bytes, or tile_type are translation templates, not valid serialized declarations. Every such name is replaced by a concrete value or model type index in an StructTypeProto.

LinearUniformProto#

Classic affine/symmetric: value = (q - zero_point) * scale.

message LinearUniformProto {
    int32 storage_type = 1;       // quantized element type (same enum as TensorProto.data_type)
    int32 bits = 2;               // number of bits (e.g. 4, 8)
    bool symmetric = 3;           // true if zero_point is always 0
    oneof scale {
        float scale_float = 4;    // scale as float
        int32 scale_int = 5;      // scale as shared exponent (value = q * 2^scale_int)
    }
    int64 zero_point = 6;         // quantization zero point
    int32 axis = 7;               // axis for per-channel, -1 if per-tensor
}

StructTypeProto {
    name: "LINEAR"
    structure: Structure {
        field: { name: "values",       type: array(INT4, dimension=N) }
        field: { name: "storage_type", constant: tensor(INT32, [], INT4) }
        field: { name: "bits",         constant: tensor(INT32, [], 4) }
        field: { name: "symmetric",    constant: tensor(BOOL, [], symmetric) }
        field: { name: "scale",        constant: tensor(FLOAT, [], s) }
        field: { name: "zero_point",   constant: tensor(INT64, [], z) }
        field: { name: "axis",         constant: tensor(INT32, [], axis) }
    }
    decoder: FunctionProto {
        // Y = (Cast(values) - zero_point) * scale
    }
}

The physical codes are an Array. When bits equals the canonical width of storage_type, the element type is used directly. Otherwise the packed region is an Array(UINT8) and the decoder extracts exactly bits per logical code. Scale and zero point are constant fields when shared, or serialized array fields when they vary by channel or block.

N is always replaced by the concrete physical count. The displayed scale field represents scale_float. A declaration using scale_int gives that field type INT32 instead, preserving which member of the specialized oneof was selected.

# Dequantize
values = unpack(data, q.bits)
if q.scale_float:
    result = (values - q.zero_point) * q.scale_float
else:
    result = (values - q.zero_point) * (2 ** q.scale_int)

# Quantize
if q.scale_float:
    values = round(tensor / q.scale_float) + q.zero_point
else:
    values = round(tensor / (2 ** q.scale_int)) + q.zero_point
values = clip(values, 0, (1 << q.bits) - 1)
raw_data = pack(values, q.bits)

CodebookProto#

Scalar or additive vector lookup-table quantization. Scalar codebooks use num_codebooks = vector_size = 1. For additive vector quantization, each vector is reconstructed as the sum of one entry from each codebook.

message CodebookProto {
    oneof scale {
        float scale_float = 1;    // scale as float
        int32 scale_int = 2;      // scale as shared exponent
    }
    repeated float codebook_data = 3;  // concatenated codebooks
    int32 packed_count = 4;       // number of values packed
    int32 packed_bytes = 5;       // into this many bytes
    int32 num_codebooks = 6;      // additive codebooks; 0 means 1
    int32 codebook_size = 7;      // entries per codebook; 0 means infer
    int32 vector_size = 8;        // values per entry; 0 means 1
    int32 index_bits = 9;         // bits per index; 0 means base-N packing
}

StructTypeProto {
    name: "CODEBOOK"
    structure: Structure {
        field: { name: "indices",  type: array(UINT8, dimension=packed_bytes) }
        field: {
            name: "codebook"
            constant: tensor(
                FLOAT,
                [codebook_value_count],
                codebook_data
            )
        }
        field: { name: "scale", constant: tensor(FLOAT, [], scale) }
        field: { name: "packed_count", constant: tensor(INT32, [], packed_count) }
        field: { name: "packed_bytes", constant: tensor(INT32, [], packed_bytes) }
        field: { name: "num_codebooks", constant: tensor(INT32, [], num_codebooks) }
        field: { name: "codebook_size", constant: tensor(INT32, [], codebook_size) }
        field: { name: "vector_size", constant: tensor(INT32, [], vector_size) }
        field: { name: "index_bits", constant: tensor(INT32, [], index_bits) }
    }
    decoder: FunctionProto {
        // Y = scale * additive_gather(codebook, unpack(indices))
    }
}

Packed indices are serialized as an Array(UINT8). The codebooks and scale are constant fields, so they are stored once in the type and consume no payload bytes. The decoder unpacks each index, gathers the selected scalar or vector entry, sums additive codebooks, and applies the scale.

As for linear quantization, the scale field is FLOAT for scale_float and INT32 for scale_int.

# Dequantize
num_codebooks = q.num_codebooks or 1
vector_size = q.vector_size or 1
codebook_size = q.codebook_size or (
    len(q.codebook_data) // (num_codebooks * vector_size))
indices = unpack_indices(data, q.index_bits, codebook_size,
                         q.packed_count, q.packed_bytes)
result = zeros(n_vectors * vector_size)
for k in range(num_codebooks):
    codebook_k = q.codebook_data[
        k * codebook_size * vector_size:
        (k + 1) * codebook_size * vector_size]
    for i, idx in enumerate(indices[k]):
        begin = idx * vector_size
        result[i*vector_size:(i+1)*vector_size] += (
            codebook_k[begin:begin + vector_size])
result *= q.scale

# Quantize
indices = nearest_additive_codewords(tensor / q.scale, q)
raw_data = pack_indices(indices, q.index_bits, codebook_size,
                        q.packed_count, q.packed_bytes)

FloatingPointUniformProto#

For finite normal values, micro-float quantization uses value = (-1)^sign * 2^(exp - bias) * (1 + mantissa). Covers FP6, FP4, MXFP and similar reduced-precision floating-point formats.

message FloatingPointUniformProto {
    int32 sign_bits = 1;          // sign bits (usually 1)
    int32 exponent_bits = 2;      // exponent bits (e.g. 3 for E3M2, 2 for E2M1)
    int32 mantissa_bits = 3;      // mantissa bits (e.g. 2 for E3M2, 1 for E2M1)
    int32 exponent_bias = 4;      // exponent bias (e.g. 3 for E3M2)
    bool has_inf = 5;             // true if format supports infinity
    bool has_nan = 6;             // true if format supports NaN
    bool split_storage = 7;       // true if sign+exp and mantissa stored separately (TC-FPn)
    int32 packed_count = 8;       // number of values packed
    int32 packed_bytes = 9;       // into this many bytes
}

StructTypeProto {  // translation template; not serialized as-is
    structure: Structure {
        // Serialized payload.
        field: {
            name: "packed"
            type: bit_packing(
                dimension=packed_count,
                components=[
                    sign:sign_bits,
                    exponent:exponent_bits,
                    mantissa:mantissa_bits
                ]
            )
        }

        // Type-level parameters: these consume no payload bytes.
        field: {
            name: "exponent_bias"
            constant: tensor(INT32, [], exponent_bias)
        }
        field: {
            name: "has_inf"
            constant: tensor(BOOL, [], has_inf)
        }
        field: {
            name: "has_nan"
            constant: tensor(BOOL, [], has_nan)
        }
    }
    decoder: FunctionProto {
        // unpack packed_count values from packed_bytes bytes
        // interpret sign/exponent/mantissa using the declared constants
        // apply exceptional-value policy and reconstruct Y
    }
}

StructTypeProto {
    name: "FLOAT6_E3M2"
    structure: Structure {
        field: {
            name: "packed"
            type: bit_packing(
                dimension=4,
                components=[sign:1, exponent:3, mantissa:2]
            )
        }
        field: { name: "exponent_bias", constant: tensor(INT32, [], 3) }
        field: { name: "has_inf", constant: tensor(BOOL, [], true) }
        field: { name: "has_nan", constant: tensor(BOOL, [], true) }
    }
    decoder: FunctionProto {
        output: "Y"
        value_info: {
            name: "Y"
            type: TypeProto {
                tensor_type: Tensor {
                    elem_type: FLOAT
                    shape: TensorShapeProto {
                        dim: { dim_value: 4 }
                    }
                }
            }
        }
        // unpack four 6-bit values and decode E3M2 with exponent bias 3
    }
}

A standard ONNX low-precision type such as FLOAT4E2M1 or a float8 type is an ordinary physical Array. A non-standard width uses BitPacking so the component widths and packed element count remain explicit. Split storage is a Structure with one field per bit plane or component. Parameters not already expressed by the physical structure remain named constant fields. Thus sign_bits, exponent_bits, mantissa_bits, packed_count, packed_bytes, and split_storage are checked from the physical layout rather than duplicated as constants.

This is a generic profile template, not a serializable element type: sign_bits, exponent_bits, and the other symbolic values are replaced when the concrete physical type is declared. A generic IEEE_FLOAT cannot be a TensorProto.data_type or TypeProto.Tensor.elem_type because it does not determine the width or the decoder’s standard ONNX output type.

The following concrete FLOAT6_E3M2 declaration packs four 6-bit values into three bytes. It uses an IEEE-like exceptional-value policy, fixes every profile parameter with Field.constant, and decodes to a standard ONNX FLOAT tensor:

When split_storage is true, the serialized packed field is replaced by explicitly named component arrays such as sign_exponent and mantissa. The constant remains useful to profiles and validation, while the physical structure itself is sufficient to locate the components.

# Dequantize
bits_per_elem = q.sign_bits + q.exponent_bits + q.mantissa_bits
raw_values = unpack(data, bits_per_elem, q.split_storage)
result = [fp_decode(v, q.sign_bits, q.exponent_bits,
                    q.mantissa_bits, q.exponent_bias)
          for v in raw_values]

SparseQuantizationProto#

Sparse + dense decomposition (SpQR, SqueezeLLM). Outlier values above a threshold are stored separately in higher precision; the rest uses a base quantization scheme.

message SparseQuantizationProto {
    QuantizationProto base_quant = 1;  // quantization for non-outlier values
    int32 outlier_data_type = 2;       // data type for outlier values (e.g. FLOAT16)
    float outlier_threshold = 3;       // absolute value threshold for outlier detection
    float outlier_ratio = 4;           // fraction of values stored as outliers (e.g. 0.01)
}

StructTypeProto {
    name: "SPARSE_QUANTIZED"
    structure: Structure {
        field: { name: "base",    type: struct(type_index=base_type) }
        field: { name: "indices", type: array(INT64, dimension=K) }
        field: { name: "values",  type: array(FLOAT16, dimension=K) }
        field: {
            name: "outlier_threshold"
            constant: tensor(FLOAT, [], outlier_threshold)
        }
        field: {
            name: "outlier_ratio"
            constant: tensor(FLOAT, [], outlier_ratio)
        }
    }
    decoder: FunctionProto {
        // Y = scatter(decode(base), indices, values)
    }
}

The dense/base region is a referenced structured physical type. Outlier indices and values are concrete arrays in the same root Structure. The concrete element type of values preserves outlier_data_type. Threshold and ratio guide the encoder but are not required to decode an already serialized value; they remain explicit constants. The root decoder incorporates the base decoder nodes explicitly; nested decoders are not invoked implicitly.

K is a concrete outlier count in each physical declaration.

# Dequantize
base_values = dequantize(base_data, q.base_quant)
outlier_indices, outlier_values = read_sparse(outlier_data, q.outlier_data_type)
result = base_values
result[outlier_indices] = outlier_values

# Quantize
mask = abs(tensor) > q.outlier_threshold
outlier_indices, outlier_values = where(mask), tensor[mask]
base_tensor = tensor.copy(); base_tensor[mask] = 0
base_data = quantize(base_tensor, q.base_quant)
outlier_data = write_sparse(outlier_indices, outlier_values, q.outlier_data_type)

LogUniformProto#

Logarithmic quantization: value = sign * base^(q + offset).

message LogUniformProto {
    int32 bits = 1;               // number of bits (e.g. 4, 8)
    float base = 2;              // logarithm base (e.g. 2.0)
    float offset = 3;            // exponent offset
    bool has_sign = 4;           // true if sign bit is stored separately
}

StructTypeProto {
    name: "LOG_UNIFORM"
    structure: Structure {
        field: { name: "codes",  type: array(UINT4, dimension=N) }
        field: { name: "signs",  type: array(UINT8, dimension=sign_bytes) }
        field: { name: "bits", constant: tensor(INT32, [], 4) }
        field: { name: "base", constant: tensor(FLOAT, [], base) }
        field: { name: "offset", constant: tensor(FLOAT, [], offset) }
        field: { name: "has_sign", constant: tensor(BOOL, [], has_sign) }
    }
    decoder: FunctionProto {
        // Y = sign * Pow(base, Cast(codes) + offset)
    }
}

Codes use a native fixed-width Array when possible and packed Array(UINT8) otherwise. A separate sign plane is another array field. base and offset are constants consumed by the decoder.

The signs physical field is omitted when has_sign is false.

# Dequantize
values = unpack(data, q.bits)
if q.has_sign:
    signs = extract_sign_bits(values)
    exponents = extract_magnitude(values)
    result = signs * (q.base ** (exponents + q.offset))
else:
    result = q.base ** (values + q.offset)

# Quantize
if q.has_sign:
    signs = sign(tensor)
    exponents = round(log(abs(tensor)) / log(q.base) - q.offset)
    values = pack_sign_magnitude(signs, exponents)
else:
    values = round(log(tensor) / log(q.base) - q.offset)
values = clip(values, 0, (1 << q.bits) - 1)
raw_data = pack(values, q.bits)

FunctionUniformProto#

Custom quantization/dequantization defined by op names. storage_type and bits describe the storage format of the quantized data so the runtime can read the raw bytes correctly. The custom ops referenced by quantize_op and dequantize_op handle the actual conversion logic. The dequantize op takes a tensor of storage_type and returns the type specified by QuantizationProto.data_type; the quantize op does the reverse.

message FunctionUniformProto {
    int32 storage_type = 1;       // storage element type (how raw bytes are interpreted)
    int32 bits = 2;               // bits per element (for sub-byte packing)
    string quantize_op = 3;       // op: data_type -> storage_type (e.g. "custom::Quantize")
    string dequantize_op = 4;     // op: storage_type -> data_type (e.g. "custom::Dequantize")
    optional BlockLayoutProto block_layout = 5;  // physical block structure (for introspection)
}

StructTypeProto {
    name: "FUNCTION_QUANTIZED"
    structure: Structure {
        field: {
            name: "payload"
            type: array(UINT8, dimension=packed_bytes)
        }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], storage_type)
        }
        field: { name: "bits", constant: tensor(INT32, [], bits) }
    }
    decoder: FunctionProto { ... }  // standard-domain form, when available
    encoder: FunctionProto { ... }
    metadata_props: { key: "quantize_op", value: quantize_op }
    metadata_props: { key: "dequantize_op", value: dequantize_op }
}

storage_type, bits, and block_layout become the concrete Array/Structure schema. dequantize_op becomes StructTypeProto.decoder and quantize_op becomes encoder when both can be expressed as deterministic standard-domain ONNX functions.

If either operation requires a custom-domain operator, the physical type remains valid but omits that portable function. The consuming custom operator is then responsible for interpretation. The original operator identifiers are preserved as quantize_op and dequantize_op metadata so the translation remains lossless.

When block_layout is present, its concrete fields replace the raw payload array and determine the same exact byte size.

# Dequantize
if q.block_layout:
    blocks = split_blocks(data, q.block_layout.bytes_per_block)
    result = concat([call_op(q.dequantize_op, block) for block in blocks])
else:
    raw = interpret(data, q.storage_type, q.bits)
    result = call_op(q.dequantize_op, raw)

# Quantize
if q.block_layout:
    blocks = tile(tensor, q.block_layout.block_size)
    raw_data = concat([call_op(q.quantize_op, block) for block in blocks])
else:
    raw_data = call_op(q.quantize_op, tensor)

TilingQuantizationProto#

Recursive one- or multi-dimensional block quantization. Weights are partitioned into tiles along axes. An empty axes denotes the flattened tensor, making classic one-dimensional block quantization a special case. If there are more tiles than entries in elem_quant, the list is cycled from the beginning (elem_quant[i % len(elem_quant)]). Nesting supports multi-level hierarchies such as K-Quants and MXFP.

message TilingQuantizationProto {
    repeated int64 tile_shape = 1;                 // tile size; one value if axes is empty
    repeated int32 axes = 2;                       // empty = flattened tensor
    repeated QuantizationProto elem_quant = 3;     // cycled over tiles
    repeated int32 perm = 4;                       // permutation of axes in memory layout
}

StructTypeProto {
    name: "TILED"
    structure: Structure {
        field: {
            name: "tiles"
            type: array(
                struct(type_index=tile_type),
                dimension=concrete_tile_count
            )
        }
        field: {
            name: "tile_shape"
            constant: tensor(INT64, [tile_rank], concrete_tile_shape)
        }
        field: {
            name: "axes"
            constant: tensor(INT32, [axis_count], concrete_axes)
        }
        field: {
            name: "perm"
            constant: tensor(INT32, [perm_rank], concrete_perm)
        }
    }
    decoder: FunctionProto {
        // decode each tile, place it at concrete coordinates, inverse perm
    }
}

A homogeneous tiling stores a nested Array whose element references the concrete tile type. The root is a Structure so concrete tile_shape, axes, and perm values remain explicit constant fields. If tiles cycle through different quantizations, the root has one physical field per concrete tile or homogeneous tile region. The decoder concatenates or scatters tile outputs and incorporates the tile decoder nodes explicitly. No symbolic tile count, modulo rule, or implicit nested-decoder invocation remains in the physical type.

# Dequantize
shape = resolve_tile_shape(q.tile_shape, q.axes, tensor_shape)
tiles = split_tiles(data, shape, q.axes, q.perm)
result = empty(tensor_shape)
for i, (coords, tile_data) in enumerate(tiles):
    eq = q.elem_quant[i % len(q.elem_quant)]
    result[coords] = dequantize(tile_data, eq)
if q.perm:
    result = inverse_permute(result, q.perm)

CastUniformProto#

Type conversion without quantization. Stores values cast from the original type to a different type (e.g. float32 → bfloat16). The source type is QuantizationProto.data_type, the target type is storage_type. Setting storage_type equal to data_type is the identity operation, useful for pure retiling.

message CastUniformProto {
    int32 storage_type = 1;        // target element type (same enum as TensorProto.data_type)
}

StructTypeProto {
    name: "CAST"
    array: Array {
        element_type: BFLOAT16
        dimension: N
    }
    decoder: FunctionProto {
        // Y = Cast(values, original_type)
    }
}

The payload is a concrete Array of storage_type. The decoder output ValueInfoProto declares the original type and shape; the decoder contains the standard ONNX Cast and any required reshape.

# Dequantize
result = cast(data, from_type=q.storage_type, to_type=quant_proto.data_type)

# Quantize
raw_data = cast(tensor, from_type=quant_proto.data_type, to_type=q.storage_type)

StructuredBlockUniformProto#

Generic block quantization with explicit physical layout. Each block contains named fields at known bit offsets. VALUES can be decoded directly for packed linear formats, or an index formula can combine fields into codebook indices. Optional scale, zero point, bias, and scatter fields cover prepacked linear formats (CompInt8, QGEMM), exotic formats (STQ1_0), and future layouts while keeping dequantization fully deducible from the proto structure.

enum BlockFieldRole {
    VALUES = 0;           // quantized values or codebook indices
    SIGN = 1;             // sign bits
    SCALE = 2;            // per-block scale
    ZERO_POINT = 3;       // per-block zero point
    BIAS = 4;             // per-block bias correction
    EXPONENT = 5;         // shared exponent
    CODE = 6;             // codebook index (partial)
    MASK = 7;             // sparsity mask
}

message BlockFieldProto {
    BlockFieldRole role = 1;       // semantic role of this field
    int32 bit_offset = 2;          // offset in block or field record
    int32 bit_width = 3;           // bits per element
    int32 count = 4;               // number of elements
    int32 data_type = 5;           // element type (0 = raw unsigned bits)
}

message BlockLayoutProto {
    int32 block_size = 1;                  // logical values per block
    int32 bytes_per_block = 2;             // physical bytes per block
    repeated BlockFieldProto fields = 3;   // physical fields in the block
}

enum BlockStorageOrder {
    INTERLEAVED = 0;       // all fields for block 0, then all fields for block 1
    SEQUENTIAL = 1;        // one region per field, each containing all blocks
}

message FieldWeightProto {
    BlockFieldRole field = 1;      // which field to use
    int32 multiplier = 2;          // coefficient in index formula
}

message ScatterProto {
    int32 group_size = 1;          // logical group size (e.g. 64)
    int32 vector_size = 2;         // lanes per codebook entry (e.g. 4)
    int32 stride = 3;             // stride within group (e.g. 16)
}

message StructuredBlockUniformProto {
    BlockLayoutProto block_layout = 1;             // physical block structure
    repeated float codebook_data = 2;              // empty = use VALUES directly
    int32 codebook_vector_size = 3;                // values per codebook entry
    repeated FieldWeightProto index_formula = 4;   // index = sum(field * multiplier)
    optional ScatterProto scatter = 5;             // output element placement (absent = contiguous)
    BlockStorageOrder storage_order = 6;           // physical organization of block fields
    optional int32 axis = 7;                       // absent = flattened tensor
}

StructTypeProto {
    name: "STRUCTURED_BLOCK"
    structure: Structure {
        field: {
            name: "blocks"
            type: array(
                struct(type_index=block_type),
                dimension=concrete_block_count
            )
        }
        field: {
            name: "codebook"
            constant: tensor(
                FLOAT,
                [codebook_value_count],
                codebook_data
            )
        }
        field: {
            name: "codebook_vector_size"
            constant: tensor(INT32, [], codebook_vector_size)
        }
        field: {
            name: "index_fields"
            constant: tensor(
                INT32,
                [index_term_count],
                concrete_index_fields
            )
        }
        field: {
            name: "index_multipliers"
            constant: tensor(
                INT32,
                [index_term_count],
                concrete_index_multipliers
            )
        }
        field: {
            name: "storage_order"
            constant: tensor(INT32, [], concrete_storage_order)
        }
        field: {
            name: "block_size"
            constant: tensor(INT32, [], concrete_block_size)
        }
        field: {
            name: "axis"
            constant: tensor(INT32, [], concrete_axis)
        }
    }
    decoder: FunctionProto {
        // index = sum(named_field * multiplier)
        // Y = scatter(codebook[index] * scale + bias)
    }
}

This profile maps directly to the generic mechanism:

  • BlockFieldProto becomes a named Structure.Field;

  • count becomes Array.dimension;

  • data_type becomes the scalar array element type;

  • field roles become ordinary field names used by the decoder;

  • explicit padding replaces bit_offset and bytes_per_block;

  • codebook_data becomes a constant field;

  • index_formula and ScatterProto become constant parameters consumed by decoder nodes;

  • INTERLEAVED is an array of block structures;

  • SEQUENTIAL is a structure containing one array per field.

The codebook, index-formula, and axis fields are omitted when their specialized counterparts are absent. When scatter is present, its three integer members are additional constant fields named scatter_group_size, scatter_vector_size, and scatter_stride. The concrete block type itself preserves block_size, field widths and counts, explicit padding, and therefore bytes_per_block.

No separate BlockFieldProto, BlockLayoutProto, FieldWeightProto, or ScatterProto is implemented.

# Dequantize
blocks = parse_structured_blocks(
    data, q.block_layout, q.storage_order, q.axis, tensor_shape)
result = []
for block in blocks:
    fields = parse_fields(block, q.block_layout.fields)
    scale = fields[SCALE][0] if SCALE in fields else 1.0
    zp = fields[ZERO_POINT][0] if ZERO_POINT in fields else 0.0
    if q.codebook_data:
        assert q.index_formula, "index_formula is required when codebook_data is set"
        assert q.codebook_vector_size > 0, "codebook_vector_size must be > 0"
        n_entries = len(q.codebook_data) // q.codebook_vector_size
        values = []
        n_codes = None
        for fw in q.index_formula:
            assert fw.field in fields, f"missing field {fw.field} for index_formula"
            if n_codes is None:
                n_codes = len(fields[fw.field])
            else:
                assert len(fields[fw.field]) == n_codes, f"field {fw.field} has inconsistent length (expected {n_codes})"
        assert n_codes is not None, "index_formula must not be empty"
        for i in range(n_codes):
            raw_idx = sum(fields[fw.field][i] * fw.multiplier for fw in q.index_formula)
            idx = int(raw_idx)
            assert raw_idx == idx, f"codebook index {raw_idx} is not an integer"
            assert 0 <= idx < n_entries, f"codebook index {idx} out of range"
            begin = idx * q.codebook_vector_size
            values.extend(q.codebook_data[begin:begin + q.codebook_vector_size])
    else:
        values = fields[VALUES]
    if q.scatter:
        values = inverse_scatter(values, q.scatter)
    bias = fields[BIAS][0] if BIAS in fields else 0.0
    result.append((values - zp) * scale + bias)
result = concat(result)

RotationProto#

Pre/post rotation applied to the tensor (QuIP#, SmoothQuant). Dequantization with rotation: values = post_rotation @ dequant(data) @ pre_rotation.

enum RotationType {
    HADAMARD = 0;
    PLAIN = 1;
}

message RotationProto {
    RotationType matrix_type = 1;         // type of rotation
    repeated int32 dims = 2;              // shape of the rotation matrix
    optional int32 matrix_index = 3;      // index into ModelProto.rotation_matrices (for PLAIN)
}

field: {
    name: "rotation"
    constant: tensor(FLOAT, concrete_rotation_dims, concrete_matrix)
}
field: {
    name: "rotation_type"
    constant: tensor(INT32, [], PLAIN)
}

decoder: FunctionProto {
    // Y = MatMul(rotation, decode_physical_fields(...))
}

Rotation is not a physical root type. A plain matrix is a field with constant in the quantized type and matrix multiplication is part of the decoder or encoder. A Hadamard rotation needs no stored matrix and is expressed directly by standard ONNX nodes. ModelProto.rotation_matrices and matrix_index are therefore resolved rather than copied into the generic type.

Known quantization schemes#

The examples below retain the specialized profile notation because it is compact and makes comparison with existing formats easy. They are not additional wire messages. Each example is implemented by applying the corresponding custom-type translation above, with all dimensions and payload sizes made concrete.

INT8 symmetric (per-tensor)#

QuantizationProto {
    linear: LinearUniformProto {
        storage_type: INT8, bits: 8, symmetric: true,
        scale_float: 0.02, zero_point: 0, axis: -1
    }
}

INT4 symmetric (per-tensor)#

For a tensor of 1024 elements:

QuantizationProto {
    linear: LinearUniformProto {
        storage_type: INT4
        bits: 4
        symmetric: true
        scale_float: 0.02
        zero_point: 0
        axis: -1
    }
}

StructTypeProto {
    name: "INT4_SYMMETRIC_1024"
    structure: Structure {
        field: {
            name: "values"
            type: array(INT4, dimension=1024)
        }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], INT4)
        }
        field: { name: "bits", constant: tensor(INT32, [], 4) }
        field: { name: "symmetric", constant: tensor(BOOL, [], true) }
        field: { name: "scale", constant: tensor(FLOAT, [], 0.02) }
        field: { name: "zero_point", constant: tensor(INT64, [], 0) }
        field: { name: "axis", constant: tensor(INT32, [], -1) }
    }
}

INT4 asymmetric (GPTQ, per-group of 128)#

For a tensor of 1024 elements (8 groups):

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [128]
        elem_quant: [
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.015, zero_point: 8, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.012, zero_point: 7, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.014, zero_point: 8, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.013, zero_point: 8, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.016, zero_point: 7, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.011, zero_point: 8, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.015, zero_point: 9, axis: -1
            }},
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4, symmetric: false,
                scale_float: 0.012, zero_point: 8, axis: -1
            }}
        ]
    }
}

StructTypeProto {
    name: "GPTQ_INT4_1024_GROUP128"
    structure: Structure {
        field: {
            name: "values"
            type: array(INT4, dimensions=[8, 128])
        }
        field: {
            name: "scales"
            constant: tensor(
                FLOAT,
                [8],
                [0.015, 0.012, 0.014, 0.013,
                 0.016, 0.011, 0.015, 0.012]
            )
        }
        field: {
            name: "zero_points"
            constant: tensor(INT64, [8], [8, 7, 8, 8, 7, 8, 9, 8])
        }
        field: {
            name: "tile_shape"
            constant: tensor(INT64, [1], [128])
        }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], INT4)
        }
        field: { name: "bits", constant: tensor(INT32, [], 4) }
        field: { name: "symmetric", constant: tensor(BOOL, [], false) }
        field: { name: "axis", constant: tensor(INT32, [], -1) }
    }
}

NF4 (QLoRA / bitsandbytes)#

QuantizationProto {
    codebook: CodebookProto {
        scale_float: 1.0,
        codebook_data: [-1.0, -0.6962, -0.5251, -0.3949, -0.2844,
                   -0.1848, -0.0911, 0.0, 0.0796, 0.1609,
                   0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1.0],
        packed_count: 2, packed_bytes: 1
    }
}

StructTypeProto {
    name: "NF4"
    structure: Structure {
        field: {
            name: "indices"
            type: array(UINT8, dimension=packed_byte_count)
        }
        field: {
            name: "codebook"
            constant: tensor(
                FLOAT,
                [16],
                [-1.0, -0.6962, -0.5251, -0.3949,
                 -0.2844, -0.1848, -0.0911, 0.0,
                 0.0796, 0.1609, 0.2461, 0.3379,
                 0.4407, 0.5626, 0.7230, 1.0]
            )
        }
        field: { name: "scale", constant: tensor(FLOAT, [], 1.0) }
        field: { name: "packed_count", constant: tensor(INT32, [], 2) }
        field: { name: "packed_bytes", constant: tensor(INT32, [], 1) }
        field: { name: "num_codebooks", constant: tensor(INT32, [], 1) }
        field: { name: "codebook_size", constant: tensor(INT32, [], 16) }
        field: { name: "vector_size", constant: tensor(INT32, [], 1) }
        field: { name: "index_bits", constant: tensor(INT32, [], 0) }
    }
}

AWQ INT4 (per-group of 128)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [128]
        elem_quant: [
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4,
                symmetric: false, scale_float: 0.01, zero_point: 8, axis: -1
            }},
            // ... one per block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "AWQ_INT4_BLOCK_128"
            structure: Structure {
                field: { name: "values", type: array(INT4, dimension=128) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], INT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 4) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.01) }
                field: { name: "zero_point", constant: tensor(INT64, [], 8) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "AWQ_INT4_TILES"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
            }
        }
    ]
}

Q2_K (llama.cpp, nested 256/16)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [256]
        elem_quant: [
            QuantizationProto {
                tiling: TilingQuantizationProto {
                    tile_shape: [16]
                    elem_quant: [
                        QuantizationProto { linear: LinearUniformProto {
                            storage_type: UINT2, bits: 2,
                            symmetric: false, scale_float: 0.005, zero_point: 1, axis: -1
                        }},
                        // ... 16 sub-blocks per super-block
                    ]
                }
            },
            // ... one per super-block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "Q2_K_SUBBLOCK_16"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=4) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT2)
                }
                field: { name: "bits", constant: tensor(INT32, [], 2) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.005) }
                field: { name: "zero_point", constant: tensor(INT64, [], 1) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "Q2_K_SUPERBLOCK_256"
            structure: Structure {
                field: {
                    name: "subblocks"
                    type: array(struct(type_index=0), dimension=16)
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [16])
                }
            }
        },
        StructTypeProto {
            name: "Q2_K_TENSOR"
            structure: Structure {
                field: {
                    name: "superblocks"
                    type: array(
                        struct(type_index=1),
                        dimension=superblock_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
            }
        }
    ]
}

Q3_K (llama.cpp, nested 256/32)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [256]
        elem_quant: [
            QuantizationProto {
                tiling: TilingQuantizationProto {
                    tile_shape: [32]
                    elem_quant: [
                        QuantizationProto { linear: LinearUniformProto {
                            storage_type: UINT4, bits: 3,
                            symmetric: false, scale_float: 0.004, zero_point: 3, axis: -1
                        }},
                        // ... 8 sub-blocks per super-block
                    ]
                }
            },
            // ... one per super-block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "Q3_K_SUBBLOCK_32"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=12) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 3) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.004) }
                field: { name: "zero_point", constant: tensor(INT64, [], 3) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "Q3_K_SUPERBLOCK_256"
            structure: Structure {
                field: {
                    name: "subblocks"
                    type: array(struct(type_index=0), dimension=8)
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [32])
                }
            }
        },
        StructTypeProto {
            name: "Q3_K_TENSOR"
            structure: Structure {
                field: {
                    name: "superblocks"
                    type: array(
                        struct(type_index=1),
                        dimension=superblock_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
            }
        }
    ]
}

Q4_K (llama.cpp, nested 256/32)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [256]
        elem_quant: [
            QuantizationProto {
                tiling: TilingQuantizationProto {
                    tile_shape: [32]
                    elem_quant: [
                        QuantizationProto { linear: LinearUniformProto {
                            storage_type: UINT4, bits: 4,
                            symmetric: false, scale_float: 0.003, zero_point: 2, axis: -1
                        }},
                        // ... 8 sub-blocks per super-block
                    ]
                }
            },
            // ... one per super-block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "Q4_K_SUBBLOCK_32"
            structure: Structure {
                field: { name: "values", type: array(UINT4, dimension=32) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 4) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.003) }
                field: { name: "zero_point", constant: tensor(INT64, [], 2) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "Q4_K_SUPERBLOCK_256"
            structure: Structure {
                field: {
                    name: "subblocks"
                    type: array(struct(type_index=0), dimension=8)
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [32])
                }
            }
        },
        StructTypeProto {
            name: "Q4_K_TENSOR"
            structure: Structure {
                field: {
                    name: "superblocks"
                    type: array(
                        struct(type_index=1),
                        dimension=superblock_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
            }
        }
    ]
}

Q5_K (llama.cpp, nested 256/32)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [256]
        elem_quant: [
            QuantizationProto {
                tiling: TilingQuantizationProto {
                    tile_shape: [32]
                    elem_quant: [
                        QuantizationProto { linear: LinearUniformProto {
                            storage_type: UINT8, bits: 5,
                            symmetric: false, scale_float: 0.002, zero_point: 4, axis: -1
                        }},
                        // ... 8 sub-blocks per super-block
                    ]
                }
            },
            // ... one per super-block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "Q5_K_SUBBLOCK_32"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=20) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 5) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.002) }
                field: { name: "zero_point", constant: tensor(INT64, [], 4) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "Q5_K_SUPERBLOCK_256"
            structure: Structure {
                field: {
                    name: "subblocks"
                    type: array(struct(type_index=0), dimension=8)
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [32])
                }
            }
        },
        StructTypeProto {
            name: "Q5_K_TENSOR"
            structure: Structure {
                field: {
                    name: "superblocks"
                    type: array(
                        struct(type_index=1),
                        dimension=superblock_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
            }
        }
    ]
}

Q6_K (llama.cpp, nested 256/16)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [256]
        elem_quant: [
            QuantizationProto {
                tiling: TilingQuantizationProto {
                    tile_shape: [16]
                    elem_quant: [
                        QuantizationProto { linear: LinearUniformProto {
                            storage_type: UINT8, bits: 6,
                            symmetric: false, scale_float: 0.001, zero_point: 5, axis: -1
                        }},
                        // ... 16 sub-blocks per super-block
                    ]
                }
            },
            // ... one per super-block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "Q6_K_SUBBLOCK_16"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=12) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 6) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.001) }
                field: { name: "zero_point", constant: tensor(INT64, [], 5) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "Q6_K_SUPERBLOCK_256"
            structure: Structure {
                field: {
                    name: "subblocks"
                    type: array(struct(type_index=0), dimension=16)
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [16])
                }
            }
        },
        StructTypeProto {
            name: "Q6_K_TENSOR"
            structure: Structure {
                field: {
                    name: "superblocks"
                    type: array(
                        struct(type_index=1),
                        dimension=superblock_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
            }
        }
    ]
}

MXFP4 (OCP Microscaling, shared exponent per group of 32)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [32]
        elem_quant: [
            QuantizationProto { floating_point: FloatingPointUniformProto {
                sign_bits: 1, exponent_bits: 2, mantissa_bits: 1,
                exponent_bias: 1, has_inf: false, has_nan: false,
                split_storage: false, packed_count: 2, packed_bytes: 1
            }},
            // ... one per block, shared exponent at super-block level
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "MXFP4_E2M1_BLOCK_32"
            structure: Structure {
                field: {
                    name: "values"
                    type: array(FLOAT4E2M1, dimension=32)
                }
                field: {
                    name: "scale"
                    type: array(FLOAT8E8M0, dimension=1)
                }
                field: { name: "sign_bits", constant: tensor(INT32, [], 1) }
                field: { name: "exponent_bits", constant: tensor(INT32, [], 2) }
                field: { name: "mantissa_bits", constant: tensor(INT32, [], 1) }
                field: { name: "exponent_bias", constant: tensor(INT32, [], 1) }
                field: { name: "has_inf", constant: tensor(BOOL, [], false) }
                field: { name: "has_nan", constant: tensor(BOOL, [], false) }
                field: { name: "split_storage", constant: tensor(BOOL, [], false) }
                field: { name: "packed_count", constant: tensor(INT32, [], 2) }
                field: { name: "packed_bytes", constant: tensor(INT32, [], 1) }
            }
        },
        StructTypeProto {
            name: "MXFP4_TILES"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [32])
                }
            }
        }
    ]
}

MXFP6 E3M2 (OCP Microscaling, 6-bit float per group of 32)#

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [32]
        elem_quant: [
            QuantizationProto { floating_point: FloatingPointUniformProto {
                sign_bits: 1, exponent_bits: 3, mantissa_bits: 2,
                exponent_bias: 3, has_inf: false, has_nan: false,
                split_storage: false, packed_count: 4, packed_bytes: 3
            }},
            // ... one per block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "MXFP6_E3M2_BLOCK_32"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=24) }
                field: {
                    name: "scale"
                    type: array(FLOAT8E8M0, dimension=1)
                }
                field: { name: "sign_bits", constant: tensor(INT32, [], 1) }
                field: { name: "exponent_bits", constant: tensor(INT32, [], 3) }
                field: { name: "mantissa_bits", constant: tensor(INT32, [], 2) }
                field: { name: "exponent_bias", constant: tensor(INT32, [], 3) }
                field: { name: "has_inf", constant: tensor(BOOL, [], false) }
                field: { name: "has_nan", constant: tensor(BOOL, [], false) }
                field: { name: "split_storage", constant: tensor(BOOL, [], false) }
                field: { name: "packed_count", constant: tensor(INT32, [], 4) }
                field: { name: "packed_bytes", constant: tensor(INT32, [], 3) }
            }
        },
        StructTypeProto {
            name: "MXFP6_E3M2_TILES"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [32])
                }
            }
        }
    ]
}

FP6 LLM (DeepSpeed TC-FPn, per-group of 128)#

Same FP6 E3M2 quantization as MXFP6 but with split storage (4-bit sign+exponent and 2-bit mantissa stored separately) for Tensor Core alignment.

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [128]
        elem_quant: [
            QuantizationProto {
                floating_point: FloatingPointUniformProto {
                    sign_bits: 1
                    exponent_bits: 3
                    mantissa_bits: 2
                    exponent_bias: 3
                    has_inf: false
                    has_nan: false
                    split_storage: true
                    packed_count: 4
                    packed_bytes: 3
                }
            }
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {  // index 0: four FLOAT6_E3M2 values
            name: "FLOAT6_E3M2"
            structure: Structure {
                field: {
                    name: "sign_exponent"
                    type: bit_packing(
                        dimension=4,
                        components=[sign:1, exponent:3]
                    )
                }
                field: {
                    name: "mantissa"
                    type: array(UINT2, dimension=4)
                }
                field: { name: "exponent_bias", constant: tensor(INT32, [], 3) }
                field: { name: "has_inf", constant: tensor(BOOL, [], false) }
                field: { name: "has_nan", constant: tensor(BOOL, [], false) }
            }
        },
        StructTypeProto {  // index 1: one tile
            name: "FP6_LLM_TILE_128"
            structure: Structure {
                field: {
                    name: "values"
                    type: array(
                        struct(type_index=0),
                        dimension=32
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
            }
        }
    ]
}

INT8 per-channel (classic CNN)#

QuantizationProto {
    linear: LinearUniformProto {
        storage_type: INT8, bits: 8, symmetric: true,
        scale_float: 0.03, zero_point: 0, axis: 0
    }
}

StructTypeProto {
    name: "INT8_PER_CHANNEL"
    structure: Structure {
        field: { name: "values", type: array(INT8, dimension=N) }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], INT8)
        }
        field: { name: "bits", constant: tensor(INT32, [], 8) }
        field: { name: "symmetric", constant: tensor(BOOL, [], true) }
        field: { name: "scale", constant: tensor(FLOAT, [], 0.03) }
        field: { name: "zero_point", constant: tensor(INT64, [], 0) }
        field: { name: "axis", constant: tensor(INT32, [], 0) }
    }
}

1.58-bit ternary (BitNet b1.58)#

QuantizationProto {
    codebook: CodebookProto {
        scale_float: 0.5,
        codebook_data: [-1.0, 0.0, 1.0],
        packed_count: 5, packed_bytes: 1
    }
}

StructTypeProto {
    name: "TERNARY_1_58"
    structure: Structure {
        field: {
            name: "indices"
            type: array(UINT8, dimension=packed_byte_count)
        }
        field: {
            name: "codebook"
            constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0])
        }
        field: { name: "scale", constant: tensor(FLOAT, [], 0.5) }
        field: { name: "packed_count", constant: tensor(INT32, [], 5) }
        field: { name: "packed_bytes", constant: tensor(INT32, [], 1) }
        field: { name: "num_codebooks", constant: tensor(INT32, [], 1) }
        field: { name: "codebook_size", constant: tensor(INT32, [], 3) }
        field: { name: "vector_size", constant: tensor(INT32, [], 1) }
        field: { name: "index_bits", constant: tensor(INT32, [], 0) }
    }
}

FP8 E4M3 (per-tensor)#

QuantizationProto {
    linear: LinearUniformProto {
        storage_type: FLOAT8E4M3FN, bits: 8, symmetric: true,
        scale_float: 1.0, zero_point: 0, axis: -1
    }
}

StructTypeProto {
    name: "FP8_E4M3_PER_TENSOR"
    structure: Structure {
        field: {
            name: "values"
            type: array(FLOAT8E4M3FN, dimension=N)
        }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], FLOAT8E4M3FN)
        }
        field: { name: "bits", constant: tensor(INT32, [], 8) }
        field: { name: "symmetric", constant: tensor(BOOL, [], true) }
        field: { name: "scale", constant: tensor(FLOAT, [], 1.0) }
        field: { name: "zero_point", constant: tensor(INT64, [], 0) }
        field: { name: "axis", constant: tensor(INT32, [], -1) }
    }
}

AQLM 2×8 (Additive Quantization, 2 bits/weight)#

QuantizationProto {
    codebook: CodebookProto {
        num_codebooks: 2,
        codebook_size: 256,
        vector_size: 8,
        index_bits: 8,
        codebook_data: [...]  // 2 * 256 * 8 = 4096 floats
    }
}

StructTypeProto {
    name: "AQLM_2X8"
    structure: Structure {
        field: {
            name: "indices"
            type: array(UINT8, dimensions=[vector_count, 2])
        }
        field: {
            name: "codebooks"
            constant: tensor(
                FLOAT,
                [2, 256, 8],
                aqlm_codebook_data
            )
        }
        field: { name: "num_codebooks", constant: tensor(INT32, [], 2) }
        field: { name: "codebook_size", constant: tensor(INT32, [], 256) }
        field: { name: "vector_size", constant: tensor(INT32, [], 8) }
        field: { name: "index_bits", constant: tensor(INT32, [], 8) }
    }
}

IQ1_S (llama.cpp, 1.56 bits/weight, E8 lattice)#

Uses a 2048-entry vector codebook (E8 lattice grid, 8 values per entry). Each sub-block of 8 weights is looked up with an 11-bit index. Wrapped in a block of 256 for the shared FP16 scale.

QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [256]
        elem_quant: [
            QuantizationProto { codebook: CodebookProto {
                num_codebooks: 1,
                codebook_size: 2048,
                vector_size: 8,
                index_bits: 11,
                codebook_data: [...]  // 2048 * 8 = 16384 values in {-1, 0, 1}
            }},
            // ... one per super-block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "IQ1_S_BLOCK_256"
            structure: Structure {
                field: {
                    name: "indices"
                    type: array(UINT8, dimension=44)
                }
                field: {
                    name: "scale"
                    type: array(FLOAT16, dimension=1)
                }
                field: {
                    name: "codebook"
                    constant: tensor(
                        FLOAT,
                        [2048, 8],
                        iq1_s_e8_codebook
                    )
                }
                field: {
                    name: "num_codebooks"
                    constant: tensor(INT32, [], 1)
                }
                field: {
                    name: "codebook_size"
                    constant: tensor(INT32, [], 2048)
                }
                field: {
                    name: "vector_size"
                    constant: tensor(INT32, [], 8)
                }
                field: {
                    name: "index_bits"
                    constant: tensor(INT32, [], 11)
                }
            }
        },
        StructTypeProto {
            name: "IQ1_S_TILES"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
            }
        }
    ]
}

SpQR (Sparse Quantization, ~3.4 bits/weight)#

Outliers (>1% of values) stored in FP16, rest in INT3 per-group.

QuantizationProto {
    sparse: SparseQuantizationProto {
        base_quant: QuantizationProto {
            tiling: TilingQuantizationProto {
                tile_shape: [16]
                elem_quant: [
                    QuantizationProto { linear: LinearUniformProto {
                        storage_type: INT4, bits: 3,
                        symmetric: false, scale_float: 0.01, zero_point: 4, axis: -1
                    }},
                    // ... one per block
                ]
            }
        },
        outlier_data_type: FLOAT16,
        outlier_threshold: 6.0,
        outlier_ratio: 0.01
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "SPQR_INT3_BLOCK_16"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=6) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], INT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 3) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.01) }
                field: { name: "zero_point", constant: tensor(INT64, [], 4) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "SPQR_DENSE_TILES"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=dense_block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [16])
                }
            }
        },
        StructTypeProto {
            name: "SPQR"
            structure: Structure {
                field: {
                    name: "base"
                    type: struct(type_index=1)
                }
                field: {
                    name: "outlier_indices"
                    type: array(INT64, dimension=outlier_count)
                }
                field: {
                    name: "outlier_values"
                    type: array(FLOAT16, dimension=outlier_count)
                }
                field: {
                    name: "outlier_data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
                field: {
                    name: "outlier_threshold"
                    constant: tensor(FLOAT, [], 6.0)
                }
                field: {
                    name: "outlier_ratio"
                    constant: tensor(FLOAT, [], 0.01)
                }
            }
        }
    ]
}

QuIP# (Vector quantization with Hadamard rotation)#

QuantizationProto {
    codebook: CodebookProto {
        num_codebooks: 1,
        codebook_size: 256,
        vector_size: 8,
        index_bits: 8,
        codebook_data: [...]  // learned codebook, 256 * 8 = 2048 floats
    },
    pre_rotation: RotationProto { matrix_type: HADAMARD, dims: [4096, 4096] },
    post_rotation: RotationProto { matrix_type: HADAMARD, dims: [4096, 4096] }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "QUIP_SHARP"
            structure: Structure {
                field: {
                    name: "indices"
                    type: array(UINT8, dimension=vector_count)
                }
                field: {
                    name: "codebook"
                    constant: tensor(
                        FLOAT,
                        [256, 8],
                        quip_sharp_codebook
                    )
                }
                field: {
                    name: "num_codebooks"
                    constant: tensor(INT32, [], 1)
                }
                field: {
                    name: "codebook_size"
                    constant: tensor(INT32, [], 256)
                }
                field: {
                    name: "vector_size"
                    constant: tensor(INT32, [], 8)
                }
                field: {
                    name: "index_bits"
                    constant: tensor(INT32, [], 8)
                }
                field: {
                    name: "pre_rotation_type"
                    constant: tensor(INT32, [], 0)  // HADAMARD
                }
                field: {
                    name: "pre_rotation_dims"
                    constant: tensor(INT32, [2], [4096, 4096])
                }
                field: {
                    name: "post_rotation_type"
                    constant: tensor(INT32, [], 0)  // HADAMARD
                }
                field: {
                    name: "post_rotation_dims"
                    constant: tensor(INT32, [2], [4096, 4096])
                }
            }
        }
    ]
}

EXL2 (variable bits per layer, ~3.5 bpw average)#

EXL2 assigns different bit-widths per layer to hit a target average bpw. Each layer gets its own QuantizationProto (via quantized_type index), so a model may use several quantization entries — e.g. 2-bit for less important layers and 4-bit for critical ones.

// Layer A (less important): 2-bit per group of 128
QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [128]
        elem_quant: [
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 2,
                symmetric: false, scale_float: 0.005, zero_point: 2, axis: -1
            }},
            // ... one per block
        ]
    }
}

// Layer B (critical): 4-bit per group of 128
QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [128]
        elem_quant: [
            QuantizationProto { linear: LinearUniformProto {
                storage_type: INT4, bits: 4,
                symmetric: false, scale_float: 0.01, zero_point: 8, axis: -1
            }},
            // ... one per block
        ]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "EXL2_INT2_BLOCK_128"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=32) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], INT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 2) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.005) }
                field: { name: "zero_point", constant: tensor(INT64, [], 2) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "EXL2_INT4_BLOCK_128"
            structure: Structure {
                field: { name: "values", type: array(INT4, dimension=128) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], INT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 4) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.01) }
                field: { name: "zero_point", constant: tensor(INT64, [], 8) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "EXL2_LAYER_A"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=layer_a_block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
            }
        },
        StructTypeProto {
            name: "EXL2_LAYER_B"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=1),
                        dimension=layer_b_block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
            }
        }
    ]
}

Log quantization (4-bit)#

QuantizationProto {
    log: LogUniformProto {
        bits: 4,
        base: 2.0,
        offset: -7.0,
        has_sign: true
    }
}

StructTypeProto {
    name: "LOG4"
    structure: Structure {
        field: { name: "codes", type: array(UINT4, dimension=N) }
        field: {
            name: "signs"
            type: array(UINT8, dimension=sign_byte_count)
        }
        field: { name: "bits", constant: tensor(INT32, [], 4) }
        field: { name: "base", constant: tensor(FLOAT, [], 2.0) }
        field: { name: "offset", constant: tensor(FLOAT, [], -7.0) }
        field: { name: "has_sign", constant: tensor(BOOL, [], true) }
    }
}

Custom (plugin-based)#

QuantizationProto {
    function: FunctionUniformProto {
        storage_type: UINT4, bits: 4,
        quantize_op: "vendor::QuantizeV2",
        dequantize_op: "vendor::DequantizeV2"
    }
}

StructTypeProto {
    name: "VENDOR_QUANTIZE_V2"
    structure: Structure {
        field: { name: "payload", type: array(UINT4, dimension=N) }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], UINT4)
        }
        field: { name: "bits", constant: tensor(INT32, [], 4) }
    }
    metadata_props: {
        key: "quantize_op"
        value: "vendor::QuantizeV2"
    }
    metadata_props: {
        key: "dequantize_op"
        value: "vendor::DequantizeV2"
    }
}

MatMulNBits INT4 (onnxruntime, per-group of 32, tiled)#

Weights of shape [K, N] are tiled into blocks of 32 along axis K and 128 along axis N (SIMD width). All tiles share the same INT4 quantization scheme. perm: [1, 0] indicates N-major tile ordering in memory for cache locality.

// Weight shape: [4096, 4096], group_size=32, N_tile=128
QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [32, 128],
        axes: [0, 1],
        elem_quant: [QuantizationProto { linear: LinearUniformProto {
            storage_type: UINT4, bits: 4,
            symmetric: true, scale_float: 0.0, zero_point: 0, axis: -1
        }}],
        perm: [1, 0]   // N-major ordering in memory
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "MATMUL_NBITS_INT4_TILE_32X128"
            structure: Structure {
                field: {
                    name: "values"
                    type: array(UINT4, dimensions=[32, 128])
                }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT4)
                }
                field: { name: "bits", constant: tensor(INT32, [], 4) }
                field: { name: "symmetric", constant: tensor(BOOL, [], true) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.0) }
                field: { name: "zero_point", constant: tensor(INT64, [], 0) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "MATMUL_NBITS_INT4_4096X4096"
            structure: Structure {
                field: {
                    name: "tiles"
                    type: array(
                        struct(type_index=0),
                        dimensions=[32, 128]
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [2], [32, 128])
                }
                field: {
                    name: "axes"
                    constant: tensor(INT32, [2], [0, 1])
                }
                field: {
                    name: "perm"
                    constant: tensor(INT32, [2], [1, 0])
                }
            }
        }
    ]
}

Block-tiled floats (no quantization)#

Reorganizes a row-major float matrix into tiles of [32, 128] without any value transformation. Useful for prepack layouts that expect block-tiled storage.

// Weight shape: [4096, 4096], tiled into [32, 128] blocks
QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [32, 128],
        axes: [0, 1],
        elem_quant: [QuantizationProto {
            data_type: FLOAT,
            cast: CastUniformProto { storage_type: FLOAT }
        }]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "FLOAT_TILE_32X128"
            array: Array {
                element_type: FLOAT
                dimension: 32
                dimension: 128
            }
        },
        StructTypeProto {
            name: "BLOCK_TILED_FLOAT_4096X4096"
            structure: Structure {
                field: {
                    name: "tiles"
                    type: array(
                        struct(type_index=0),
                        dimensions=[128, 32]
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [2], [32, 128])
                }
                field: {
                    name: "axes"
                    constant: tensor(INT32, [2], [0, 1])
                }
            }
        }
    ]
}

STQ1_0 (Sherry, 1.25 bits/weight, ternary codebook)#

256 values per 42-byte block. Each 5-bit combined index (4-bit code + 1-bit sign) selects from a 32-entry codebook of 4 ternary values. Output positions use a stride-16 scatter within each 64-value chunk.

QuantizationProto {
    data_type: FLOAT,
    structured_block: StructuredBlockUniformProto {
        block_layout: BlockLayoutProto {
            block_size: 256, bytes_per_block: 42,
            fields: [
                BlockFieldProto { role: CODE,  bit_offset: 0,   bit_width: 4,  count: 64, data_type: 0 },
                BlockFieldProto { role: SIGN,  bit_offset: 256, bit_width: 1,  count: 64, data_type: 0 },
                BlockFieldProto { role: SCALE, bit_offset: 320, bit_width: 16, count: 1,  data_type: FLOAT16 },
            ]
        },
        codebook_data: [...]  // 32 * 4 = 128 ternary values in {-1, 0, 1}
        codebook_vector_size: 4,
        index_formula: [
            FieldWeightProto { field: CODE, multiplier: 1 },
            FieldWeightProto { field: SIGN, multiplier: 16 },
        ],
        scatter: ScatterProto { group_size: 64, vector_size: 4, stride: 16 }
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "STQ1_0_BLOCK_256"
            structure: Structure {
                field: { name: "code", type: array(UINT4, dimension=64) }
                field: { name: "sign", type: array(UINT8, dimension=8) }
                field: { name: "scale", type: array(FLOAT16, dimension=1) }
                field: {
                    name: "field_roles"
                    constant: tensor(INT32, [3], [6, 1, 2])
                }
                field: {
                    name: "sign_bit_width"
                    constant: tensor(INT32, [], 1)
                }
                field: {
                    name: "sign_count"
                    constant: tensor(INT32, [], 64)
                }
                field: {
                    name: "block_size"
                    constant: tensor(INT32, [], 256)
                }
            }
        },
        StructTypeProto {
            name: "STQ1_0"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "codebook"
                    constant: tensor(
                        FLOAT,
                        [32, 4],
                        stq1_0_codebook
                    )
                }
                field: {
                    name: "codebook_vector_size"
                    constant: tensor(INT32, [], 4)
                }
                field: {
                    name: "index_fields"
                    constant: tensor(INT32, [2], [6, 1])
                }
                field: {
                    name: "index_multipliers"
                    constant: tensor(INT32, [2], [1, 16])
                }
                field: {
                    name: "scatter_group_size"
                    constant: tensor(INT32, [], 64)
                }
                field: {
                    name: "scatter_vector_size"
                    constant: tensor(INT32, [], 4)
                }
                field: {
                    name: "scatter_stride"
                    constant: tensor(INT32, [], 16)
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT)
                }
            }
        }
    ]
}

Dequantization of one block (derived from proto fields):

sb = proto.structured_block
bl = sb.block_layout
code_field = bl.fields[role == CODE]    # bit_offset=0, bit_width=4, count=64
sign_field = bl.fields[role == SIGN]    # bit_offset=256, bit_width=1, count=64
scale_field = bl.fields[role == SCALE]  # bit_offset=320, bit_width=16, count=1

scale = extract(block, scale_field.bit_offset, scale_field.bit_width, scale_field.data_type)

for g in range(code_field.count):
    # Extract fields
    code = extract(block, code_field.bit_offset + code_field.bit_width * g, code_field.bit_width)
    sign = extract(block, sign_field.bit_offset + sign_field.bit_width * g, sign_field.bit_width)

    # Combine fields into codebook index using index_formula
    index = sum(values[fw.field] * fw.multiplier for fw in sb.index_formula)
    # = code * 1 + sign * 16

    # Codebook lookup
    vector = sb.codebook_data[index * sb.codebook_vector_size : (index+1) * sb.codebook_vector_size]

    # Scatter to output positions
    s = sb.scatter
    groups_per_chunk = s.group_size // s.vector_size  # 64 // 4 = 16
    chunk = g // groups_per_chunk
    lane = g % groups_per_chunk
    for p in range(s.vector_size):
        output[s.group_size * chunk + lane + s.stride * p] = vector[p] * scale

Column-major layout#

Expresses a column-major (Fortran-order) storage of a 2D matrix. A tile_shape value of 0 means “same as the tensor dimension on that axis”, so [0, 0] is a single tile covering the whole tensor. perm: [1, 0] reverses the axis order in memory.

// Weight shape: [M, N], stored column-major
QuantizationProto {
    tiling: TilingQuantizationProto {
        tile_shape: [0, 0],
        axes: [0, 1],
        elem_quant: [QuantizationProto {
            data_type: FLOAT,
            cast: CastUniformProto { storage_type: FLOAT }
        }],
        perm: [1, 0]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "FLOAT_MATRIX_MXN"
            array: Array {
                element_type: FLOAT
                dimension: M
                dimension: N
            }
        },
        StructTypeProto {
            name: "COLUMN_MAJOR_FLOAT_MXN"
            structure: Structure {
                field: {
                    name: "tiles"
                    type: array(
                        struct(type_index=0),
                        dimension=1
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [2], [0, 0])
                }
                field: {
                    name: "axes"
                    constant: tensor(INT32, [2], [0, 1])
                }
                field: {
                    name: "perm"
                    constant: tensor(INT32, [2], [1, 0])
                }
            }
        }
    ]
}

TQ1_0 (llama.cpp, 1.6 bits/weight, ternary)#

Pure ternary: each weight is {-1, 0, +1}. Five ternary values are packed into one byte (3⁵ = 243 ≤ 255). One FP16 scale per block of 256 weights.

// Codebook: [-1, 0, +1], packed 5 per byte
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [256],
        elem_quant: [QuantizationProto {
            codebook: CodebookProto {
                scale_float: 1.0,       // scale is per-block, stored separately
                codebook_data: [-1.0, 0.0, 1.0],
                packed_count: 5,        // 5 ternary values per byte
                packed_bytes: 1
            }
        }]
    }
}

// Dequantize: for each block of 256 weights
// indices = unpack_base3(data, 5 values/byte)
// result = [codebook[i] * scale for i in indices]
//        = [{-1,0,+1}[i] * d for i in indices]

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "TQ1_0_BLOCK_256"
            structure: Structure {
                field: { name: "indices", type: array(UINT8, dimension=52) }
                field: { name: "scale", type: array(FLOAT16, dimension=1) }
                field: {
                    name: "codebook"
                    constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0])
                }
                field: {
                    name: "packed_count"
                    constant: tensor(INT32, [], 5)
                }
                field: {
                    name: "packed_bytes"
                    constant: tensor(INT32, [], 1)
                }
            }
        },
        StructTypeProto {
            name: "TQ1_0"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

TQ2_0 (llama.cpp, 2 bits/weight, ternary)#

Ternary with 2 bits per weight (simpler packing, 1 unused state). Mapping: 0 → -1, 1 → 0, 2 → +1.

// Codebook: [-1, 0, +1], 2 bits per value
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [256],
        elem_quant: [QuantizationProto {
            codebook: CodebookProto {
                scale_float: 1.0,
                codebook_data: [-1.0, 0.0, 1.0],
                packed_count: 4,        // 4 values per byte (2 bits each)
                packed_bytes: 1
            }
        }]
    }
}

// Dequantize: same as TQ1_0 but simpler unpack (2 bits, not base-3)

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "TQ2_0_BLOCK_256"
            structure: Structure {
                field: { name: "indices", type: array(UINT8, dimension=64) }
                field: { name: "scale", type: array(FLOAT16, dimension=1) }
                field: {
                    name: "codebook"
                    constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0])
                }
                field: {
                    name: "packed_count"
                    constant: tensor(INT32, [], 4)
                }
                field: {
                    name: "packed_bytes"
                    constant: tensor(INT32, [], 1)
                }
            }
        },
        StructTypeProto {
            name: "TQ2_0"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

BitNet b1.58 (Microsoft, 1.58 bits/weight, native ternary)#

Model trained natively with ternary weights {-1, 0, +1}. Storage is identical to TQ1_0 (5 ternary values per byte). The difference is in training (QAT), not in the storage format.

// Same storage as TQ1_0
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [256],
        elem_quant: [QuantizationProto {
            codebook: CodebookProto {
                scale_float: 1.0,
                codebook_data: [-1.0, 0.0, 1.0],
                packed_count: 5,
                packed_bytes: 1
            }
        }]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "BITNET_B1_58_BLOCK_256"
            structure: Structure {
                field: { name: "indices", type: array(UINT8, dimension=52) }
                field: { name: "scale", type: array(FLOAT16, dimension=1) }
                field: {
                    name: "codebook"
                    constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0])
                }
                field: {
                    name: "packed_count"
                    constant: tensor(INT32, [], 5)
                }
                field: {
                    name: "packed_bytes"
                    constant: tensor(INT32, [], 1)
                }
            }
        },
        StructTypeProto {
            name: "BITNET_B1_58"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

ParetoQ (Meta, 1.58–2 bits/weight, SEQ ternary/2-bit)#

Quantization-aware training with Stretched Elastic Quantization (SEQ). The storage format is standard ternary or 2-bit; the custom SEQ function is used only during training. At inference, the model uses a standard codebook. If the SEQ rounding function is needed at inference, use FunctionUniformProto.

// Ternary variant (same storage as TQ1_0)
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [256],
        elem_quant: [QuantizationProto {
            codebook: CodebookProto {
                scale_float: 1.0,
                codebook_data: [-1.0, 0.0, 1.0],
                packed_count: 5,
                packed_bytes: 1
            }
        }]
    }
}

// 2-bit variant (4 levels)
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [128],
        elem_quant: [QuantizationProto {
            linear: LinearUniformProto {
                storage_type: UINT8,
                bits: 2,
                symmetric: false,
                scale_float: 0.023,
                zero_point: 1,
                axis: -1
            }
        }]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "PARETOQ_TERNARY_BLOCK_256"
            structure: Structure {
                field: { name: "indices", type: array(UINT8, dimension=52) }
                field: { name: "scale", type: array(FLOAT16, dimension=1) }
                field: {
                    name: "codebook"
                    constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0])
                }
                field: {
                    name: "packed_count"
                    constant: tensor(INT32, [], 5)
                }
                field: {
                    name: "packed_bytes"
                    constant: tensor(INT32, [], 1)
                }
            }
        },
        StructTypeProto {
            name: "PARETOQ_UINT2_BLOCK_128"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=32) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 2) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.023) }
                field: { name: "zero_point", constant: tensor(INT64, [], 1) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "PARETOQ_TERNARY"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=ternary_block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        },
        StructTypeProto {
            name: "PARETOQ_UINT2"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=1),
                        dimension=uint2_block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

Tequila (ICLR 2026, 1.58 bits/weight, deadzone-free ternary)#

Improved ternary QAT that removes the dead-zone around zero. Storage is identical to TQ1_0/BitNet (ternary codebook). The deadzone-free quantization function is only used during training.

// Same storage as TQ1_0 / BitNet b1.58
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [256],
        elem_quant: [QuantizationProto {
            codebook: CodebookProto {
                scale_float: 1.0,
                codebook_data: [-1.0, 0.0, 1.0],
                packed_count: 5,
                packed_bytes: 1
            }
        }]
    }
}

// Note: the difference with BitNet is in training only.
// At inference, both use the same ternary codebook.

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "TEQUILA_BLOCK_256"
            structure: Structure {
                field: { name: "indices", type: array(UINT8, dimension=52) }
                field: { name: "scale", type: array(FLOAT16, dimension=1) }
                field: {
                    name: "codebook"
                    constant: tensor(FLOAT, [3], [-1.0, 0.0, 1.0])
                }
                field: {
                    name: "packed_count"
                    constant: tensor(INT32, [], 5)
                }
                field: {
                    name: "packed_bytes"
                    constant: tensor(INT32, [], 1)
                }
            }
        },
        StructTypeProto {
            name: "TEQUILA"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [256])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

EETQ (INT8 weight-only per-channel)#

Simple INT8 per-channel weight-only quantization (NetEase FuXi). No calibration, no QAT. Uses optimized W8A16 GEMM kernels from FasterTransformer / TensorRT-LLM.

QuantizationProto {
    data_type: FLOAT16,
    linear: LinearUniformProto {
        storage_type: INT8,
        bits: 8,
        symmetric: true,
        scale_float: 0.0042,    // per-channel scale
        zero_point: 0,
        axis: 0                 // per output channel (row)
    }
}

StructTypeProto {
    name: "EETQ_INT8_PER_CHANNEL"
    structure: Structure {
        field: { name: "values", type: array(INT8, dimension=N) }
        field: {
            name: "storage_type"
            constant: tensor(INT32, [], INT8)
        }
        field: { name: "bits", constant: tensor(INT32, [], 8) }
        field: { name: "symmetric", constant: tensor(BOOL, [], true) }
        field: { name: "scale", constant: tensor(FLOAT, [], 0.0042) }
        field: { name: "zero_point", constant: tensor(INT64, [], 0) }
        field: { name: "axis", constant: tensor(INT32, [], 0) }
        field: {
            name: "data_type"
            constant: tensor(INT32, [], FLOAT16)
        }
    }
}

EXL3 (improved EXL2, 2–6 bpw)#

Successor to EXL2 with improved codebook and mixed-precision per layer. Storage is structurally identical to EXL2 (variable bpw per layer using nested TilingQuantizationProto).

// Same structure as EXL2, different calibration
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [128],
        elem_quant: [QuantizationProto {
            linear: LinearUniformProto {
                storage_type: UINT8,
                bits: 3,               // varies per layer (2–6)
                symmetric: false,
                scale_float: 0.015,
                zero_point: 4,
                axis: -1
            }
        }]
    }
}

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "EXL3_UINT3_BLOCK_128"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=48) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 3) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.015) }
                field: { name: "zero_point", constant: tensor(INT64, [], 4) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "EXL3_LAYER"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

HQQ (Hybrid Quantization, 2–4 bits, mixed-precision per head)#

Different bit-widths for Q, K, V projections in attention layers. Each tensor gets its own QuantizationProto; the example shows a 2-bit weight with per-group scales.

// Example: 2-bit weight with group size 64
QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [64],
        elem_quant: [QuantizationProto {
            linear: LinearUniformProto {
                storage_type: UINT8,
                bits: 2,
                symmetric: false,
                scale_float: 0.032,
                zero_point: 2,
                axis: -1
            }
        }]
    }
}

// In practice, Q proj may use 4-bit while V proj uses 2-bit.
// Each tensor simply references a different QuantizationProto index.

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "HQQ_UINT2_BLOCK_64"
            structure: Structure {
                field: { name: "values", type: array(UINT8, dimension=16) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 2) }
                field: { name: "symmetric", constant: tensor(BOOL, [], false) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.032) }
                field: { name: "zero_point", constant: tensor(INT64, [], 2) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "HQQ_HEAD"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [64])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

NVFP4 (NVIDIA FP4 E2M1 with FP8 scale, 4.5 bpw)#

NVIDIA’s block floating-point FP4 format. Each element is E2M1 (2 exponent bits, 1 mantissa bit), with one FP8 E4M3 scale per block of 16 elements.

QuantizationProto {
    data_type: FLOAT16,
    tiling: TilingQuantizationProto {
        tile_shape: [16],
        elem_quant: [QuantizationProto {
            floating_point: FloatingPointUniformProto {
                sign_bits: 1,
                exponent_bits: 2,
                mantissa_bits: 1,
                exponent_bias: 1,
                has_inf: false,
                has_nan: false,
                packed_count: 2,      // 2 FP4 values per byte
                packed_bytes: 1
            }
        }]
    }
}

// Dequantize: for each block of 16 elements
// values = [fp_decode(v, 1, 2, 1, bias=1) for v in unpack_fp4(data)]
// result = values * scale_fp8

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "NVFP4_E2M1_BLOCK_16"
            structure: Structure {
                field: {
                    name: "values"
                    type: array(FLOAT4E2M1, dimension=16)
                }
                field: {
                    name: "scale"
                    type: array(FLOAT8E4M3FN, dimension=1)
                }
                field: { name: "sign_bits", constant: tensor(INT32, [], 1) }
                field: { name: "exponent_bits", constant: tensor(INT32, [], 2) }
                field: { name: "mantissa_bits", constant: tensor(INT32, [], 1) }
                field: { name: "exponent_bias", constant: tensor(INT32, [], 1) }
                field: { name: "has_inf", constant: tensor(BOOL, [], false) }
                field: { name: "has_nan", constant: tensor(BOOL, [], false) }
                field: { name: "split_storage", constant: tensor(BOOL, [], false) }
                field: { name: "packed_count", constant: tensor(INT32, [], 2) }
                field: { name: "packed_bytes", constant: tensor(INT32, [], 1) }
            }
        },
        StructTypeProto {
            name: "NVFP4"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [16])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

QuaRot (rotational quantization, 4 bits)#

Applies a Hadamard rotation before INT4 quantization to spread outlier magnitudes evenly across channels. The rotation is stored in RotationProto.

QuantizationProto {
    data_type: FLOAT16,
    pre_rotation: RotationProto {
        type: HADAMARD,
        dims: [4096]
    },
    tiling: TilingQuantizationProto {
        tile_shape: [128],
        elem_quant: [QuantizationProto {
            linear: LinearUniformProto {
                storage_type: UINT8,
                bits: 4,
                symmetric: true,
                scale_float: 0.018,
                zero_point: 0,
                axis: -1
            }
        }]
    }
}

// Dequantize:
// 1. values = unpack(data, 4 bits)
// 2. result = values * scale
// 3. result = hadamard_inverse(result)  ← RotationProto

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "QUAROT_UINT4_BLOCK_128"
            structure: Structure {
                field: { name: "values", type: array(UINT4, dimension=128) }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], UINT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 4) }
                field: { name: "symmetric", constant: tensor(BOOL, [], true) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.018) }
                field: { name: "zero_point", constant: tensor(INT64, [], 0) }
                field: { name: "axis", constant: tensor(INT32, [], -1) }
            }
        },
        StructTypeProto {
            name: "QUAROT"
            structure: Structure {
                field: {
                    name: "blocks"
                    type: array(
                        struct(type_index=0),
                        dimension=block_count
                    )
                }
                field: {
                    name: "tile_shape"
                    constant: tensor(INT64, [1], [128])
                }
                field: {
                    name: "pre_rotation_type"
                    constant: tensor(INT32, [], 0)  // HADAMARD
                }
                field: {
                    name: "pre_rotation_dims"
                    constant: tensor(INT32, [1], [4096])
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT16)
                }
            }
        }
    ]
}

SmoothQuant (W8A8, smoothed INT8)#

Shifts quantization difficulty from activations to weights via a per-channel scaling transform. The smoothing factor is a diagonal matrix stored as a rotation matrix (PLAIN type).

QuantizationProto {
    data_type: FLOAT,
    pre_rotation: RotationProto {
        type: PLAIN,
        dims: [4096],
        matrix_index: 0          // index into ModelProto.rotation_matrices
    },
    linear: LinearUniformProto {
        storage_type: INT8,
        bits: 8,
        symmetric: true,
        scale_float: 0.0042,
        zero_point: 0,
        axis: 1                  // per-channel
    }
}

// Dequantize:
// 1. result = int8_values * scale (per-channel)
// 2. result = result @ diag(smooth_factors)^(-1)  ← RotationProto

ModelProto {
    struct_types: [
        StructTypeProto {
            name: "SMOOTHQUANT_INT8"
            structure: Structure {
                field: {
                    name: "values"
                    type: array(INT8, dimension=N)
                }
                field: {
                    name: "storage_type"
                    constant: tensor(INT32, [], INT8)
                }
                field: { name: "bits", constant: tensor(INT32, [], 8) }
                field: { name: "symmetric", constant: tensor(BOOL, [], true) }
                field: { name: "scale", constant: tensor(FLOAT, [], 0.0042) }
                field: { name: "zero_point", constant: tensor(INT64, [], 0) }
                field: { name: "axis", constant: tensor(INT32, [], 1) }
                field: {
                    name: "pre_rotation_type"
                    constant: tensor(INT32, [], 1)  // PLAIN
                }
                field: {
                    name: "pre_rotation_dims"
                    constant: tensor(INT32, [1], [4096])
                }
                field: {
                    name: "pre_rotation"
                    constant: tensor(FLOAT, [4096], smooth_factors)
                }
                field: {
                    name: "data_type"
                    constant: tensor(INT32, [], FLOAT)
                }
            }
        }
    ]
}

Pseudo-code#

The dequantize and quantize pseudo-code for each format is shown inline in the corresponding proto section above. The top-level dispatcher is:

def dequantize(qtensor: QuantizedTensorProto, model: ModelProto) -> float[]:
    quant = model.quantizations[qtensor.quantized_type]
    data = qtensor.raw_data

    match quant.kind:
        case LinearUniformProto as q:     ...  # see LinearUniformProto section
        case CodebookProto as q:          ...  # see CodebookProto section
        case FloatingPointUniformProto:   ...
        case SparseQuantizationProto:     ...
        case LogUniformProto:             ...
        case FunctionUniformProto:        ...
        case TilingQuantizationProto:     ...
        case CastUniformProto:            ...
        case StructuredBlockUniformProto: ...

    # Apply post-rotation if present
    if quant.post_rotation:
        result = apply_rotation(result, quant.post_rotation, model)

    assert len(result) == product(qtensor.dims)
    return result
On this page
Show Source