Quantize

• Domain: ai.rt

• Since version: 1

Encodes a finite floating tensor in the storage layout specified by type (a TYPE_PROTO attribute containing StructTypeProto). Optional inputs are scales, zero_points, offsets, codebooks, permutation, forward, inverse and outliers. Omitted scales are calibrated per block from the source range; cast blocks use one. Scalar numerical parameters broadcast, otherwise one value is required per block. Fixed scalar codebooks have profile defaults. Learned codebooks, vector/additive codebook scales and nonempty transforms/permutations/outlier indices must be supplied. This does not train GPTQ/AWQ/codebooks. Output preserves the source logical shape and dtype. With parameter_ref, a model-local fixed parameter set replaces all optional inputs and calibration; the output carries only local codes/outlier values in a compact storage schema plus the reference. Otherwise physical storage uses the requested type.

Inputs

• X (T): Floating tensor to encode.

• scales (P1): Optional scalar or per-block scales.

• zero_points (P2): Optional scalar or per-block zero points.

• offsets (P3): Optional scalar or per-block offsets.

• codebooks (P4): Optional concatenated per-block codebooks.

• permutation (I): Optional gather permutation.

• forward (P5): Optional forward transform matrix.

• inverse (P6): Optional inverse transform matrix.

• outliers (I): Optional original-source outlier indices.

Outputs

• Y (E): EncodedValueProto carrying its local storage type and parameter ownership.

Attributes

• parameter_ref (string): Model-local fixed numerical parameter set; excludes optional inputs.

• type (type_proto): Required quantization StructTypeProto wrapped in TypeProto.

Type Constraints

• T Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• P1 Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• P2 Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• P3 Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• P4 Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• P5 Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• P6 Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• I Allowed types: tensor(int64).

• E: Structured encoded value. Allowed types: struct.

Examples

test_cc_quantize_int4

Node:
  ai.rt.Quantize(X) -> (Q)
  Attributes:
    type = <attribute type 13>

Inputs:
  X: shape=(3,), dtype=float32
    [-1.,  0.,  1.]

Outputs:
  Q: shape=(3,), dtype=float64
    [-1.,  0.,  1.]

test_cc_quantize_nf4

Node:
  ai.rt.Quantize(X) -> (Q)
  Attributes:
    type = <attribute type 13>

Inputs:
  X: shape=(3,), dtype=float32
    [-1.,  0.,  1.]

Outputs:
  Q: shape=(3,), dtype=float64
    [-1.,  0.,  1.]

test_cc_quantize_ternary

Node:
  ai.rt.Quantize(X) -> (Q)
  Attributes:
    type = <attribute type 13>

Inputs:
  X: shape=(3,), dtype=float32
    [-1.,  0.,  1.]

Outputs:
  Q: shape=(3,), dtype=float64
    [-1.,  0.,  1.]

test_cc_quantize_tiled_float

Node:
  ai.rt.Quantize(X) -> (Q)
  Attributes:
    type = <attribute type 13>

Inputs:
  X: shape=(3,), dtype=float32
    [-1.,  0.,  1.]

Outputs:
  Q: shape=(3,), dtype=float64
    [-1.,  0.,  1.]