:nosearch: .. _op_ai_onnx_QLinearMatMul-10: QLinearMatMul - version 10 ========================== This page documents version **10** of operator **QLinearMatMul**. See :doc:`QLinearMatMul` for the latest version (since version 21). - **Domain**: ``ai.onnx`` - **Since version**: 10 Matrix product that behaves like `numpy.matmul `_. It consumes two quantized input tensors, their scales and zero points, scale and zero point of output, and computes the quantized output. The quantization formula is y = saturate((x / y_scale) + y_zero_point). For (x / y_scale), it is rounding to nearest ties to even. Refer to https://en.wikipedia.org/wiki/Rounding for details. Scale and zero point must have same shape. They must be either scalar (per tensor) or N-D tensor (per row for 'a' and per column for 'b'). Scalar refers to per tensor quantization whereas N-D refers to per row or per column quantization. If the input is 2D of shape [M, K] then zero point and scale tensor may be an M element vector [v_1, v_2, ..., v_M] for per row quantization and K element vector of shape [v_1, v_2, ..., v_K] for per column quantization. If the input is N-D tensor with shape [D1, D2, M, K] then zero point and scale tensor may have shape [D1, D2, M, 1] for per row quantization and shape [D1, D2, 1, K] for per column quantization. Production must never overflow, and accumulation may overflow if and only if in 32 bits. **Inputs** - **a** (*T1*): N-dimensional quantized matrix a - **a_scale** (*tensor(float)*): scale of quantized input a - **a_zero_point** (*T1*): zero point of quantized input a - **b** (*T2*): N-dimensional quantized matrix b - **b_scale** (*tensor(float)*): scale of quantized input b - **b_zero_point** (*T2*): zero point of quantized input b - **y_scale** (*tensor(float)*): scale of quantized output y - **y_zero_point** (*T3*): zero point of quantized output y **Outputs** - **y** (*T3*): Quantized matrix multiply results from a \* b **Type Constraints** - **T1**: Constrain input a and its zero point data type to 8-bit integer tensor. Allowed types: tensor(int8), tensor(uint8). - **T2**: Constrain input b and its zero point data type to 8-bit integer tensor. Allowed types: tensor(int8), tensor(uint8). - **T3**: Constrain output y and its zero point data type to 8-bit integer tensor. Allowed types: tensor(int8), tensor(uint8). Examples -------- **test_cc_qlinearmatmul_2D_int8_float16** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 4), dtype=int8 [[ 81, 109, -127, 111], [-124, 87, -128, -98]] a_scale: shape=(), dtype=float16 0.0066 a_zero_point: shape=(), dtype=int8 -14 b: shape=(4, 3), dtype=int8 [[ 25, -76, 117], [ -67, -101, -128], [-127, 0, 119], [ 0, 127, 120]] b_scale: shape=(), dtype=float16 0.00705 b_zero_point: shape=(), dtype=int8 -13 y_scale: shape=(), dtype=float16 0.0107 y_zero_point: shape=(), dtype=int8 -9 Outputs: y: shape=(2, 3), dtype=int8 [[ 41, -12, -9], [ 1, -75, -128]] **test_cc_qlinearmatmul_2D_int8_float32** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 4), dtype=int8 [[ 81, 109, -127, 111], [-124, 87, -128, -98]] a_scale: shape=(), dtype=float32 0.0066 a_zero_point: shape=(), dtype=int8 -14 b: shape=(4, 3), dtype=int8 [[ 25, -76, 117], [ -67, -101, -128], [-127, 0, 119], [ 0, 127, 120]] b_scale: shape=(), dtype=float32 0.00705 b_zero_point: shape=(), dtype=int8 -13 y_scale: shape=(), dtype=float32 0.0107 y_zero_point: shape=(), dtype=int8 -9 Outputs: y: shape=(2, 3), dtype=int8 [[ 41, -12, -9], [ 1, -75, -128]] **test_cc_qlinearmatmul_2D_uint8_float16** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 4), dtype=uint8 [[208, 236, 0, 238], [ 3, 214, 255, 29]] a_scale: shape=(), dtype=float16 0.0066 a_zero_point: shape=(), dtype=uint8 113 b: shape=(4, 3), dtype=uint8 [[152, 51, 244], [ 60, 26, 255], [ 0, 127, 246], [127, 254, 247]] b_scale: shape=(), dtype=float16 0.00705 b_zero_point: shape=(), dtype=uint8 114 y_scale: shape=(), dtype=float16 0.0107 y_zero_point: shape=(), dtype=uint8 118 Outputs: y: shape=(2, 3), dtype=uint8 [[168, 115, 255], [ 1, 66, 151]] **test_cc_qlinearmatmul_2D_uint8_float32** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 4), dtype=uint8 [[208, 236, 0, 238], [ 3, 214, 255, 29]] a_scale: shape=(), dtype=float32 0.0066 a_zero_point: shape=(), dtype=uint8 113 b: shape=(4, 3), dtype=uint8 [[152, 51, 244], [ 60, 26, 255], [ 0, 127, 246], [127, 254, 247]] b_scale: shape=(), dtype=float32 0.00705 b_zero_point: shape=(), dtype=uint8 114 y_scale: shape=(), dtype=float32 0.0107 y_zero_point: shape=(), dtype=uint8 118 Outputs: y: shape=(2, 3), dtype=uint8 [[168, 115, 255], [ 1, 66, 151]] **test_cc_qlinearmatmul_3D_int8_float16** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 2, 4), dtype=int8 [[[ 81, 109, -127, 111], [-124, 87, -128, -98]], [[ 81, 109, -127, 111], [-124, 87, -128, -98]]] a_scale: shape=(), dtype=float16 0.0066 a_zero_point: shape=(), dtype=int8 -14 b: shape=(2, 4, 3), dtype=int8 [[[ 25, -76, 117], [ -67, -101, -128], [-127, 0, 119], [ 0, 127, 120]], [[ 25, -76, 117], [ -67, -101, -128], [-127, 0, 119], [ 0, 127, 120]]] b_scale: shape=(), dtype=float16 0.00705 b_zero_point: shape=(), dtype=int8 114 y_scale: shape=(), dtype=float16 0.0107 y_zero_point: shape=(), dtype=int8 -9 Outputs: y: shape=(2, 2, 3), dtype=int8 [[[ -86, -128, -128], [ 115, 39, -121]], [[ -86, -128, -128], [ 115, 39, -121]]] **test_cc_qlinearmatmul_3D_int8_float32** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 2, 4), dtype=int8 [[[ 81, 109, -127, 111], [-124, 87, -128, -98]], [[ 81, 109, -127, 111], [-124, 87, -128, -98]]] a_scale: shape=(), dtype=float32 0.0066 a_zero_point: shape=(), dtype=int8 -14 b: shape=(2, 4, 3), dtype=int8 [[[ 25, -76, 117], [ -67, -101, -128], [-127, 0, 119], [ 0, 127, 120]], [[ 25, -76, 117], [ -67, -101, -128], [-127, 0, 119], [ 0, 127, 120]]] b_scale: shape=(), dtype=float32 0.00705 b_zero_point: shape=(), dtype=int8 114 y_scale: shape=(), dtype=float32 0.0107 y_zero_point: shape=(), dtype=int8 -9 Outputs: y: shape=(2, 2, 3), dtype=int8 [[[ -86, -128, -128], [ 115, 39, -121]], [[ -86, -128, -128], [ 115, 39, -121]]] **test_cc_qlinearmatmul_3D_uint8_float16** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 2, 4), dtype=uint8 [[[208, 236, 0, 238], [ 3, 214, 255, 29]], [[208, 236, 0, 238], [ 3, 214, 255, 29]]] a_scale: shape=(), dtype=float16 0.0066 a_zero_point: shape=(), dtype=uint8 113 b: shape=(2, 4, 3), dtype=uint8 [[[152, 51, 244], [ 60, 26, 255], [ 0, 127, 246], [127, 254, 247]], [[152, 51, 244], [ 60, 26, 255], [ 0, 127, 246], [127, 254, 247]]] b_scale: shape=(), dtype=float16 0.00705 b_zero_point: shape=(), dtype=uint8 114 y_scale: shape=(), dtype=float16 0.0107 y_zero_point: shape=(), dtype=uint8 118 Outputs: y: shape=(2, 2, 3), dtype=uint8 [[[168, 115, 255], [ 1, 66, 151]], [[168, 115, 255], [ 1, 66, 151]]] **test_cc_qlinearmatmul_3D_uint8_float32** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(2, 2, 4), dtype=uint8 [[[208, 236, 0, 238], [ 3, 214, 255, 29]], [[208, 236, 0, 238], [ 3, 214, 255, 29]]] a_scale: shape=(), dtype=float32 0.0066 a_zero_point: shape=(), dtype=uint8 113 b: shape=(2, 4, 3), dtype=uint8 [[[152, 51, 244], [ 60, 26, 255], [ 0, 127, 246], [127, 254, 247]], [[152, 51, 244], [ 60, 26, 255], [ 0, 127, 246], [127, 254, 247]]] b_scale: shape=(), dtype=float32 0.00705 b_zero_point: shape=(), dtype=uint8 114 y_scale: shape=(), dtype=float32 0.0107 y_zero_point: shape=(), dtype=uint8 118 Outputs: y: shape=(2, 2, 3), dtype=uint8 [[[168, 115, 255], [ 1, 66, 151]], [[168, 115, 255], [ 1, 66, 151]]] **test_cc_qlinearmatmul_overflow_int8** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(1, 1), dtype=int8 [[100]] a_scale: shape=(), dtype=float32 1. a_zero_point: shape=(), dtype=int8 0 b: shape=(1, 1), dtype=int8 [[100]] b_scale: shape=(), dtype=float32 1. b_zero_point: shape=(), dtype=int8 0 y_scale: shape=(), dtype=float32 0.5 y_zero_point: shape=(), dtype=int8 0 Outputs: y: shape=(1, 1), dtype=int8 [[127]] **test_cc_qlinearmatmul_overflow_uint8** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(1, 1), dtype=uint8 [[100]] a_scale: shape=(), dtype=float32 1. a_zero_point: shape=(), dtype=uint8 0 b: shape=(1, 1), dtype=uint8 [[100]] b_scale: shape=(), dtype=float32 1. b_zero_point: shape=(), dtype=uint8 0 y_scale: shape=(), dtype=float32 0.2 y_zero_point: shape=(), dtype=uint8 0 Outputs: y: shape=(1, 1), dtype=uint8 [[255]] **test_cc_qlinearmatmul_underflow_int8** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(1, 1), dtype=int8 [[-100]] a_scale: shape=(), dtype=float32 1. a_zero_point: shape=(), dtype=int8 0 b: shape=(1, 1), dtype=int8 [[100]] b_scale: shape=(), dtype=float32 1. b_zero_point: shape=(), dtype=int8 0 y_scale: shape=(), dtype=float32 0.5 y_zero_point: shape=(), dtype=int8 0 Outputs: y: shape=(1, 1), dtype=int8 [[-128]] **test_cc_qlinearmatmul_underflow_uint8** .. code-block:: text Node: QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y) .. code-block:: text Inputs: a: shape=(1, 1), dtype=uint8 [[0]] a_scale: shape=(), dtype=float32 1. a_zero_point: shape=(), dtype=uint8 100 b: shape=(1, 1), dtype=uint8 [[100]] b_scale: shape=(), dtype=float32 1. b_zero_point: shape=(), dtype=uint8 0 y_scale: shape=(), dtype=float32 1. y_zero_point: shape=(), dtype=uint8 0 Outputs: y: shape=(1, 1), dtype=uint8 [[0]]