GRU

• Domain: ai.onnx

• Since version: 22

Computes an one-layer GRU. This operator is usually supported via some custom implementation such as CuDNN.

Notations:

• X - input tensor

• z - update gate

• r - reset gate

• h - hidden gate

• t - time step (t-1 means previous time step)

• W[zrh] - W parameter weight matrix for update, reset, and hidden gates

• R[zrh] - R recurrence weight matrix for update, reset, and hidden gates

• Wb[zrh] - W bias vectors for update, reset, and hidden gates

• Rb[zrh] - R bias vectors for update, reset, and hidden gates

• WB[zrh] - W parameter weight matrix for backward update, reset, and hidden gates

• RB[zrh] - R recurrence weight matrix for backward update, reset, and hidden gates

• WBb[zrh] - W bias vectors for backward update, reset, and hidden gates

• RBb[zrh] - R bias vectors for backward update, reset, and hidden gates

• H - Hidden state

• num_directions - 2 if direction == bidirectional else 1

Activation functions:

• Relu(x) - max(0, x)

• Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})

• Sigmoid(x) - 1/(1 + e^{-x})

NOTE:

Below are optional

• Affine(x) - alpha * x + beta

• LeakyRelu(x) - x if x >= 0 else alpha * x

• ThresholdedRelu(x) - x if x >= alpha else 0

• ScaledTanh(x) - alpha * Tanh(beta * x)

• HardSigmoid(x) - min(max(alpha * x + beta, 0), 1)

• Elu(x) - x if x >= 0 else alpha * (e^x - 1)

• Softsign(x) - x/(1 + |x|)

• Softplus(x) - log(1 + e^x)

Equations (Default: f=Sigmoid, g=Tanh):

• zt = f(Xt*(Wz^T) + Ht-1*(Rz^T) + Wbz + Rbz)

• rt = f(Xt*(Wr^T) + Ht-1*(Rr^T) + Wbr + Rbr)

• ht = g(Xt*(Wh^T) + (rt (.) Ht-1)*(Rh^T) + Rbh + Wbh) # default, when linear_before_reset = 0

• ht = g(Xt*(Wh^T) + (rt (.) (Ht-1*(Rh^T) + Rbh)) + Wbh) # when linear_before_reset != 0

• Ht = (1 - zt) (.) ht + zt (.) Ht-1

Inputs

• X (T): The input sequences packed (and potentially padded) into one 3-D tensor with the shape of [seq_length, batch_size, input_size].

• W (T): The weight tensor for the gates. Concatenation of W[zrh] and WB[zrh] (if bidirectional) along dimension 0. This tensor has shape [num_directions, 3*hidden_size, input_size].

• R (T): The recurrence weight tensor. Concatenation of R[zrh] and RB[zrh] (if bidirectional) along dimension 0. This tensor has shape [num_directions, 3*hidden_size, hidden_size].

• B (T): The bias tensor for the gates. Concatenation of [Wb[zrh], Rb[zrh]] and [WBb[zrh], RBb[zrh]] (if bidirectional) along dimension 0. This tensor has shape [num_directions, 6*hidden_size]. Optional: If not specified - assumed to be 0

• sequence_lens (T1): Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length seq_length. It has shape [batch_size].

• initial_h (T): Optional initial value of the hidden. If not specified - assumed to be 0. It has shape [num_directions, batch_size, hidden_size].

Outputs

• Y (T): A tensor that concats all the intermediate output values of the hidden. It has shape [seq_length, num_directions, batch_size, hidden_size].

• Y_h (T): The last output value of the hidden. It has shape [num_directions, batch_size, hidden_size].

Type Constraints

• T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

• T1: Constrain seq_lens to integer tensor. Allowed types: tensor(int32).

Examples

test_cc_gru_batchwise

Node:
  GRU(X, W, R) -> (Y, Y_h)
  Attributes:
    hidden_size = 6
    layout = 1

Inputs:
  X: shape=(3, 1, 2), dtype=float32
    [[[1., 2.]],

     [[3., 4.]],

     [[5., 6.]]]
  W: shape=(1, 18, 2), dtype=float32
    [[[0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2],
      [0.2, 0.2]]]
  R: shape=(1, 18, 6), dtype=float32
    [[[0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
      [0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
      [0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
      ...,
      [0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
      [0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
      [0.2, 0.2, 0.2, 0.2, 0.2, 0.2]]]

Outputs:
  Y: shape=(3, 1, 1, 6), dtype=float32
    [[[[0.19030015, 0.19030015, 0.19030015, 0.19030015, 0.19030015, 0.19030015]]],


     [[[0.17513679, 0.17513679, 0.17513679, 0.17513679, 0.17513679, 0.17513679]]],


     [[[0.09733082, 0.09733082, 0.09733082, 0.09733082, 0.09733082, 0.09733082]]]]
  Y_h: shape=(3, 1, 6), dtype=float32
    [[[0.19030015, 0.19030015, 0.19030015, 0.19030015, 0.19030015, 0.19030015]],

     [[0.17513679, 0.17513679, 0.17513679, 0.17513679, 0.17513679, 0.17513679]],

     [[0.09733082, 0.09733082, 0.09733082, 0.09733082, 0.09733082, 0.09733082]]]

test_cc_gru_defaults

Node:
  GRU(X, W, R) -> ("", Y_h)
  Attributes:
    hidden_size = 5

Inputs:
  X: shape=(1, 3, 2), dtype=float32
    [[[1., 2.],
      [3., 4.],
      [5., 6.]]]
  W: shape=(1, 15, 2), dtype=float32
    [[[0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1],
      [0.1, 0.1]]]
  R: shape=(1, 15, 5), dtype=float32
    [[[0.1, 0.1, 0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1, 0.1, 0.1],
      ...,
      [0.1, 0.1, 0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1, 0.1, 0.1]]]

Outputs:
  Y_h: shape=(1, 3, 5), dtype=float32
    [[[0.12397026, 0.12397026, 0.12397026, 0.12397026, 0.12397026],
      [0.20053661, 0.20053661, 0.20053661, 0.20053661, 0.20053661],
      [0.19991653, 0.19991653, 0.19991653, 0.19991653, 0.19991653]]]

test_cc_gru_seq_length

Node:
  GRU(X, W, R, B) -> ("", Y_h)
  Attributes:
    hidden_size = 5

Inputs:
  X: shape=(2, 3, 3), dtype=float32
    [[[ 1.,  2.,  3.],
      [ 4.,  5.,  6.],
      [ 7.,  8.,  9.]],

     [[10., 11., 12.],
      [13., 14., 15.],
      [16., 17., 18.]]]
  W: shape=(1, 15, 3), dtype=float32
    [[[-0.1       , -0.05      ,  0.        ],
      [ 0.05      ,  0.1       ,  0.15      ],
      [ 0.20000002, -0.1       , -0.05      ],
      [ 0.        ,  0.05      ,  0.1       ],
      [ 0.15      ,  0.20000002, -0.1       ],
      [-0.05      ,  0.        ,  0.05      ],
      [ 0.1       ,  0.15      ,  0.20000002],
      [-0.1       , -0.05      ,  0.        ],
      [ 0.05      ,  0.1       ,  0.15      ],
      [ 0.20000002, -0.1       , -0.05      ],
      [ 0.        ,  0.05      ,  0.1       ],
      [ 0.15      ,  0.20000002, -0.1       ],
      [-0.05      ,  0.        ,  0.05      ],
      [ 0.1       ,  0.15      ,  0.20000002],
      [-0.1       , -0.05      ,  0.        ]]]
  R: shape=(1, 15, 5), dtype=float32
    [[[-0.15      , -0.11000001, -0.07000001, -0.03000001,  0.00999999],
      [ 0.04999998,  0.08999999,  0.13      ,  0.16999999, -0.15      ],
      [-0.11000001, -0.07000001, -0.03000001,  0.00999999,  0.04999998],
      ...,
      [ 0.08999999,  0.13      ,  0.16999999, -0.15      , -0.11000001],
      [-0.07000001, -0.03000001,  0.00999999,  0.04999998,  0.08999999],
      [ 0.13      ,  0.16999999, -0.15      , -0.11000001, -0.07000001]]]
  B: shape=(1, 30), dtype=float32
    [[-0.05      , -0.03      , -0.01      ,  0.01      ,  0.03      ,  0.04999999,
       0.06999999,  0.09      ,  0.11      ,  0.13      ,  0.14999999,  0.17      ,
       0.19      ,  0.21      ,  0.23      ,  0.24999999,  0.26999998,  0.29      ,
       0.30999997,  0.32999998,  0.34999996,  0.36999997,  0.39      ,  0.40999997,
       0.42999998,  0.45      ,  0.46999997,  0.48999995,  0.51      ,  0.53      ]]

Outputs:
  Y_h: shape=(1, 3, 5), dtype=float32
    [[[ 8.7044084e-01,  2.1826701e-01,  4.3450546e-01,  3.8154891e-01,
        1.3391793e-01],
      [ 9.3663889e-01,  1.3535304e-01,  4.1170382e-01,  2.8888240e-01,
       -8.0563687e-04],
      [ 9.7050184e-01,  6.3298374e-02,  3.8574046e-01,  2.0248368e-01,
       -4.7147907e-02]]]

test_cc_gru_with_initial_bias

Node:
  GRU(X, W, R, B) -> ("", Y_h)
  Attributes:
    hidden_size = 3

Inputs:
  X: shape=(1, 3, 3), dtype=float32
    [[[1., 2., 3.],
      [4., 5., 6.],
      [7., 8., 9.]]]
  W: shape=(1, 9, 3), dtype=float32
    [[[0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1]]]
  R: shape=(1, 9, 3), dtype=float32
    [[[0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1],
      [0.1, 0.1, 0.1]]]
  B: shape=(1, 18), dtype=float32
    [[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0. , 0. , 0. , 0. , 0. , 0. ,
      0. , 0. , 0. ]]

Outputs:
  Y_h: shape=(1, 3, 3), dtype=float32
    [[[0.20053661, 0.20053661, 0.20053661],
      [0.15482338, 0.15482338, 0.15482338],
      [0.07484276, 0.07484276, 0.07484276]]]

Differences with previous version (14)

SchemaDiff: GRU (domain 'ai.onnx')

• old version: 14

• new version: 22

• breaking: no

Type constraints:

• changed ‘T’: added types: [‘tensor(bfloat16)’]

Version History

• Version 14

• Version 7

• Version 3

• Version 1