QLinearMatMul - version 10

This page documents version 10 of operator QLinearMatMul. See QLinearMatMul for the latest version (since version 21).

• Domain: ai.onnx

• Since version: 10

Matrix product that behaves like numpy.matmul. It consumes two quantized input tensors, their scales and zero points, scale and zero point of output, and computes the quantized output. The quantization formula is y = saturate((x / y_scale) + y_zero_point). For (x / y_scale), it is rounding to nearest ties to even. Refer to https://en.wikipedia.org/wiki/Rounding for details. Scale and zero point must have same shape. They must be either scalar (per tensor) or N-D tensor (per row for ‘a’ and per column for ‘b’). Scalar refers to per tensor quantization whereas N-D refers to per row or per column quantization. If the input is 2D of shape [M, K] then zero point and scale tensor may be an M element vector [v_1, v_2, …, v_M] for per row quantization and K element vector of shape [v_1, v_2, …, v_K] for per column quantization. If the input is N-D tensor with shape [D1, D2, M, K] then zero point and scale tensor may have shape [D1, D2, M, 1] for per row quantization and shape [D1, D2, 1, K] for per column quantization. Production must never overflow, and accumulation may overflow if and only if in 32 bits.

Inputs

• a (T1): N-dimensional quantized matrix a

• a_scale (tensor(float)): scale of quantized input a

• a_zero_point (T1): zero point of quantized input a

• b (T2): N-dimensional quantized matrix b

• b_scale (tensor(float)): scale of quantized input b

• b_zero_point (T2): zero point of quantized input b

• y_scale (tensor(float)): scale of quantized output y

• y_zero_point (T3): zero point of quantized output y

Outputs

• y (T3): Quantized matrix multiply results from a * b

Type Constraints

• T1: Constrain input a and its zero point data type to 8-bit integer tensor. Allowed types: tensor(int8), tensor(uint8).

• T2: Constrain input b and its zero point data type to 8-bit integer tensor. Allowed types: tensor(int8), tensor(uint8).

• T3: Constrain output y and its zero point data type to 8-bit integer tensor. Allowed types: tensor(int8), tensor(uint8).

Examples

test_cc_qlinearmatmul_2D_int8_float16

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 4), dtype=int8
    [[  81,  109, -127,  111],
     [-124,   87, -128,  -98]]
  a_scale: shape=(), dtype=float16
    0.0066
  a_zero_point: shape=(), dtype=int8
    -14
  b: shape=(4, 3), dtype=int8
    [[  25,  -76,  117],
     [ -67, -101, -128],
     [-127,    0,  119],
     [   0,  127,  120]]
  b_scale: shape=(), dtype=float16
    0.00705
  b_zero_point: shape=(), dtype=int8
    -13
  y_scale: shape=(), dtype=float16
    0.0107
  y_zero_point: shape=(), dtype=int8
    -9

Outputs:
  y: shape=(2, 3), dtype=int8
    [[  41,  -12,   -9],
     [   1,  -75, -128]]

test_cc_qlinearmatmul_2D_int8_float32

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 4), dtype=int8
    [[  81,  109, -127,  111],
     [-124,   87, -128,  -98]]
  a_scale: shape=(), dtype=float32
    0.0066
  a_zero_point: shape=(), dtype=int8
    -14
  b: shape=(4, 3), dtype=int8
    [[  25,  -76,  117],
     [ -67, -101, -128],
     [-127,    0,  119],
     [   0,  127,  120]]
  b_scale: shape=(), dtype=float32
    0.00705
  b_zero_point: shape=(), dtype=int8
    -13
  y_scale: shape=(), dtype=float32
    0.0107
  y_zero_point: shape=(), dtype=int8
    -9

Outputs:
  y: shape=(2, 3), dtype=int8
    [[  41,  -12,   -9],
     [   1,  -75, -128]]

test_cc_qlinearmatmul_2D_uint8_float16

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 4), dtype=uint8
    [[208, 236,   0, 238],
     [  3, 214, 255,  29]]
  a_scale: shape=(), dtype=float16
    0.0066
  a_zero_point: shape=(), dtype=uint8
    113
  b: shape=(4, 3), dtype=uint8
    [[152,  51, 244],
     [ 60,  26, 255],
     [  0, 127, 246],
     [127, 254, 247]]
  b_scale: shape=(), dtype=float16
    0.00705
  b_zero_point: shape=(), dtype=uint8
    114
  y_scale: shape=(), dtype=float16
    0.0107
  y_zero_point: shape=(), dtype=uint8
    118

Outputs:
  y: shape=(2, 3), dtype=uint8
    [[168, 115, 255],
     [  1,  66, 151]]

test_cc_qlinearmatmul_2D_uint8_float32

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 4), dtype=uint8
    [[208, 236,   0, 238],
     [  3, 214, 255,  29]]
  a_scale: shape=(), dtype=float32
    0.0066
  a_zero_point: shape=(), dtype=uint8
    113
  b: shape=(4, 3), dtype=uint8
    [[152,  51, 244],
     [ 60,  26, 255],
     [  0, 127, 246],
     [127, 254, 247]]
  b_scale: shape=(), dtype=float32
    0.00705
  b_zero_point: shape=(), dtype=uint8
    114
  y_scale: shape=(), dtype=float32
    0.0107
  y_zero_point: shape=(), dtype=uint8
    118

Outputs:
  y: shape=(2, 3), dtype=uint8
    [[168, 115, 255],
     [  1,  66, 151]]

test_cc_qlinearmatmul_3D_int8_float16

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 2, 4), dtype=int8
    [[[  81,  109, -127,  111],
      [-124,   87, -128,  -98]],

     [[  81,  109, -127,  111],
      [-124,   87, -128,  -98]]]
  a_scale: shape=(), dtype=float16
    0.0066
  a_zero_point: shape=(), dtype=int8
    -14
  b: shape=(2, 4, 3), dtype=int8
    [[[  25,  -76,  117],
      [ -67, -101, -128],
      [-127,    0,  119],
      [   0,  127,  120]],

     [[  25,  -76,  117],
      [ -67, -101, -128],
      [-127,    0,  119],
      [   0,  127,  120]]]
  b_scale: shape=(), dtype=float16
    0.00705
  b_zero_point: shape=(), dtype=int8
    114
  y_scale: shape=(), dtype=float16
    0.0107
  y_zero_point: shape=(), dtype=int8
    -9

Outputs:
  y: shape=(2, 2, 3), dtype=int8
    [[[ -86, -128, -128],
      [ 115,   39, -121]],

     [[ -86, -128, -128],
      [ 115,   39, -121]]]

test_cc_qlinearmatmul_3D_int8_float32

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 2, 4), dtype=int8
    [[[  81,  109, -127,  111],
      [-124,   87, -128,  -98]],

     [[  81,  109, -127,  111],
      [-124,   87, -128,  -98]]]
  a_scale: shape=(), dtype=float32
    0.0066
  a_zero_point: shape=(), dtype=int8
    -14
  b: shape=(2, 4, 3), dtype=int8
    [[[  25,  -76,  117],
      [ -67, -101, -128],
      [-127,    0,  119],
      [   0,  127,  120]],

     [[  25,  -76,  117],
      [ -67, -101, -128],
      [-127,    0,  119],
      [   0,  127,  120]]]
  b_scale: shape=(), dtype=float32
    0.00705
  b_zero_point: shape=(), dtype=int8
    114
  y_scale: shape=(), dtype=float32
    0.0107
  y_zero_point: shape=(), dtype=int8
    -9

Outputs:
  y: shape=(2, 2, 3), dtype=int8
    [[[ -86, -128, -128],
      [ 115,   39, -121]],

     [[ -86, -128, -128],
      [ 115,   39, -121]]]

test_cc_qlinearmatmul_3D_uint8_float16

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 2, 4), dtype=uint8
    [[[208, 236,   0, 238],
      [  3, 214, 255,  29]],

     [[208, 236,   0, 238],
      [  3, 214, 255,  29]]]
  a_scale: shape=(), dtype=float16
    0.0066
  a_zero_point: shape=(), dtype=uint8
    113
  b: shape=(2, 4, 3), dtype=uint8
    [[[152,  51, 244],
      [ 60,  26, 255],
      [  0, 127, 246],
      [127, 254, 247]],

     [[152,  51, 244],
      [ 60,  26, 255],
      [  0, 127, 246],
      [127, 254, 247]]]
  b_scale: shape=(), dtype=float16
    0.00705
  b_zero_point: shape=(), dtype=uint8
    114
  y_scale: shape=(), dtype=float16
    0.0107
  y_zero_point: shape=(), dtype=uint8
    118

Outputs:
  y: shape=(2, 2, 3), dtype=uint8
    [[[168, 115, 255],
      [  1,  66, 151]],

     [[168, 115, 255],
      [  1,  66, 151]]]

test_cc_qlinearmatmul_3D_uint8_float32

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(2, 2, 4), dtype=uint8
    [[[208, 236,   0, 238],
      [  3, 214, 255,  29]],

     [[208, 236,   0, 238],
      [  3, 214, 255,  29]]]
  a_scale: shape=(), dtype=float32
    0.0066
  a_zero_point: shape=(), dtype=uint8
    113
  b: shape=(2, 4, 3), dtype=uint8
    [[[152,  51, 244],
      [ 60,  26, 255],
      [  0, 127, 246],
      [127, 254, 247]],

     [[152,  51, 244],
      [ 60,  26, 255],
      [  0, 127, 246],
      [127, 254, 247]]]
  b_scale: shape=(), dtype=float32
    0.00705
  b_zero_point: shape=(), dtype=uint8
    114
  y_scale: shape=(), dtype=float32
    0.0107
  y_zero_point: shape=(), dtype=uint8
    118

Outputs:
  y: shape=(2, 2, 3), dtype=uint8
    [[[168, 115, 255],
      [  1,  66, 151]],

     [[168, 115, 255],
      [  1,  66, 151]]]

test_cc_qlinearmatmul_overflow_int8

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(1, 1), dtype=int8
    [[100]]
  a_scale: shape=(), dtype=float32
    1.
  a_zero_point: shape=(), dtype=int8
    0
  b: shape=(1, 1), dtype=int8
    [[100]]
  b_scale: shape=(), dtype=float32
    1.
  b_zero_point: shape=(), dtype=int8
    0
  y_scale: shape=(), dtype=float32
    0.5
  y_zero_point: shape=(), dtype=int8
    0

Outputs:
  y: shape=(1, 1), dtype=int8
    [[127]]

test_cc_qlinearmatmul_overflow_uint8

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(1, 1), dtype=uint8
    [[100]]
  a_scale: shape=(), dtype=float32
    1.
  a_zero_point: shape=(), dtype=uint8
    0
  b: shape=(1, 1), dtype=uint8
    [[100]]
  b_scale: shape=(), dtype=float32
    1.
  b_zero_point: shape=(), dtype=uint8
    0
  y_scale: shape=(), dtype=float32
    0.2
  y_zero_point: shape=(), dtype=uint8
    0

Outputs:
  y: shape=(1, 1), dtype=uint8
    [[255]]

test_cc_qlinearmatmul_underflow_int8

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(1, 1), dtype=int8
    [[-100]]
  a_scale: shape=(), dtype=float32
    1.
  a_zero_point: shape=(), dtype=int8
    0
  b: shape=(1, 1), dtype=int8
    [[100]]
  b_scale: shape=(), dtype=float32
    1.
  b_zero_point: shape=(), dtype=int8
    0
  y_scale: shape=(), dtype=float32
    0.5
  y_zero_point: shape=(), dtype=int8
    0

Outputs:
  y: shape=(1, 1), dtype=int8
    [[-128]]

test_cc_qlinearmatmul_underflow_uint8

Node:
  QLinearMatMul(a, a_scale, a_zero_point, b, b_scale, b_zero_point, y_scale, y_zero_point) -> (y)

Inputs:
  a: shape=(1, 1), dtype=uint8
    [[0]]
  a_scale: shape=(), dtype=float32
    1.
  a_zero_point: shape=(), dtype=uint8
    100
  b: shape=(1, 1), dtype=uint8
    [[100]]
  b_scale: shape=(), dtype=float32
    1.
  b_zero_point: shape=(), dtype=uint8
    0
  y_scale: shape=(), dtype=float32
    1.
  y_zero_point: shape=(), dtype=uint8
    0

Outputs:
  y: shape=(1, 1), dtype=uint8
    [[0]]