QuantizePagedCache#

  • Domain: ai.rt

  • Since version: 1

Quantizes the key and value payloads of selected immutable paged-cache blocks. block_indices contains unique zero-based block indices. key_scale and value_scale are positive scalar FLOAT tensors. The scalar key_zero_point and value_zero_point tensors select independent INT8, UINT8, INT4, UINT4, INT2, UINT2, FLOAT, FLOAT16 or BFLOAT16 storage formats. Integer types quantize to affine storage; floating types dequantize to dense storage. Selected dense or affine blocks are converted; unselected blocks retain their existing storage and ownership.

Inputs

  • cache (C): Input paged-cache structured value.

  • block_indices (I): Unique zero-based indices of blocks to quantize.

  • key_scale (S): Scalar FLOAT key quantization scale.

  • key_zero_point (ZK): Scalar key zero point selecting the key storage type.

  • value_scale (S): Scalar FLOAT value quantization scale.

  • value_zero_point (ZV): Scalar value zero point selecting the value storage type.

Outputs

  • quantized_cache (C): Cache with the selected blocks requantized.

Type Constraints

  • C: Constrain cache input and output to structured values. Allowed types: struct.

  • I: Constrain block indices to INT64. Allowed types: tensor(int64).

  • S: Constrain scales to FLOAT. Allowed types: tensor(float).

  • ZK: Select the key’s affine integer or dense floating storage type. Allowed types: tensor(bfloat16), tensor(float), tensor(float16), tensor(int2), tensor(int4), tensor(int8), tensor(uint2), tensor(uint4), tensor(uint8).

  • ZV: Select the value’s affine integer or dense floating storage type. Allowed types: tensor(bfloat16), tensor(float), tensor(float16), tensor(int2), tensor(int4), tensor(int8), tensor(uint2), tensor(uint4), tensor(uint8).