RNN - version 7#

This page documents version 7 of operator RNN. See RNN for the latest version (since version 22).

  • Domain: ai.onnx

  • Since version: 7

Computes an one-layer simple RNN. This operator is usually supported via some custom implementation such as CuDNN.

Notations:

  • X - input tensor

  • i - input gate

  • t - time step (t-1 means previous time step)

  • Wi - W parameter weight matrix for input gate

  • Ri - R recurrence weight matrix for input gate

  • Wbi - W parameter bias vector for input gate

  • Rbi - R parameter bias vector for input gate

  • WBi - W parameter weight matrix for backward input gate

  • RBi - R recurrence weight matrix for backward input gate

  • WBbi - WR bias vectors for backward input gate

  • RBbi - RR bias vectors for backward input gate

  • H - Hidden state

  • num_directions - 2 if direction == bidirectional else 1

Activation functions:

  • Relu(x) - max(0, x)

  • Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})

  • Sigmoid(x) - 1/(1 + e^{-x})

NOTE: Below are optional

  • Affine(x) - alpha*x + beta

  • LeakyRelu(x) - x if x >= 0 else alpha * x

  • ThresholdedRelu(x) - x if x >= alpha else 0

  • ScaledTanh(x) - alpha*Tanh(beta*x)

  • HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)

  • Elu(x) - x if x >= 0 else alpha*(e^x - 1)

  • Softsign(x) - x/(1 + |x|)

  • Softplus(x) - log(1 + e^x)

Equations (Default: f=Tanh):

  • Ht = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Wbi + Rbi)

Inputs

  • X (T): The input sequences packed (and potentially padded) into one 3-D tensor with the shape of [seq_length, batch_size, input_size].

  • W (T): The weight tensor for input gate. Concatenation of Wi and WBi (if bidirectional). The tensor has shape [num_directions, hidden_size, input_size].

  • R (T): The recurrence weight tensor. Concatenation of Ri and RBi (if bidirectional). The tensor has shape [num_directions, hidden_size, hidden_size].

  • B (T): The bias tensor for input gate. Concatenation of [Wbi, Rbi] and [WBbi, RBbi] (if bidirectional). The tensor has shape [num_directions, 2*hidden_size]. Optional: If not specified - assumed to be 0.

  • sequence_lens (T1): Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length seq_length. It has shape [batch_size].

  • initial_h (T): Optional initial value of the hidden. If not specified - assumed to be 0. It has shape [num_directions, batch_size, hidden_size].

Outputs

  • Y (T): A tensor that concats all the intermediate output values of the hidden. It has shape [seq_length, num_directions, batch_size, hidden_size].

  • Y_h (T): The last output value of the hidden. It has shape [num_directions, batch_size, hidden_size].

Type Constraints

  • T: Constrain input and output types to float tensors. Allowed types: tensor(double), tensor(float), tensor(float16).

  • T1: Constrain seq_lens to integer tensor. Allowed types: tensor(int32).

Differences with previous version (1)#

SchemaDiff: RNN (domain 'ai.onnx')

  • old version: 1

  • new version: 7

  • breaking: no