RNN

RNN - 14

Version

• name: RNN (GitHub)

• domain: main

• since_version: 14

• function: False

• support_level: SupportType.COMMON

• shape inference: True

This version of the operator has been available since version 14.

Summary

Computes an one-layer simple RNN. This operator is usually supported via some custom implementation such as CuDNN.

Notations:

X - input tensor

i - input gate

t - time step (t-1 means previous time step)

Wi - W parameter weight matrix for input gate

Ri - R recurrence weight matrix for input gate

Wbi - W parameter bias vector for input gate

Rbi - R parameter bias vector for input gate

WBi - W parameter weight matrix for backward input gate

RBi - R recurrence weight matrix for backward input gate

WBbi - WR bias vectors for backward input gate

RBbi - RR bias vectors for backward input gate

H - Hidden state

num_directions - 2 if direction == bidirectional else 1

Activation functions:

Relu(x) - max(0, x)

Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})

Sigmoid(x) - 1/(1 + e^{-x})

(NOTE: Below are optional)

Affine(x) - alpha*x + beta

LeakyRelu(x) - x if x >= 0 else alpha * x

ThresholdedRelu(x) - x if x >= alpha else 0

ScaledTanh(x) - alpha*Tanh(beta*x)

HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)

Elu(x) - x if x >= 0 else alpha*(e^x - 1)

Softsign(x) - x/(1 + |x|)

Softplus(x) - log(1 + e^x)

Equations (Default: f=Tanh):

• Ht = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Wbi + Rbi)

This operator has optional inputs/outputs. See ONNX for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument’s name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.

Attributes

• activation_alpha: Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.For example with LeakyRelu, the default alpha is 0.01.

• activation_beta: Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.

• activations: One (or two if bidirectional) activation function for input gate. The activation function must be one of the activation functions specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].

• clip: Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.

• direction: Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional. Default value is 'forward'.

• hidden_size: Number of neurons in the hidden layer

• layout: The shape format of inputs X, initial_h and outputs Y, Y_h. If 0, the following shapes are expected: X.shape = [seq_length, batch_size, input_size], Y.shape = [seq_length, num_directions, batch_size, hidden_size], initial_h.shape = Y_h.shape = [num_directions, batch_size, hidden_size]. If 1, the following shapes are expected: X.shape = [batch_size, seq_length, input_size], Y.shape = [batch_size, seq_length, num_directions, hidden_size], initial_h.shape = Y_h.shape = [batch_size, num_directions, hidden_size]. Default value is 0.

Inputs

Between 3 and 6 inputs.

• X (heterogeneous) - T: The input sequences packed (and potentially padded) into one 3-D tensor with the shape of [seq_length, batch_size, input_size].

• W (heterogeneous) - T: The weight tensor for input gate. Concatenation of Wi and WBi (if bidirectional). The tensor has shape [num_directions, hidden_size, input_size].

• R (heterogeneous) - T: The recurrence weight tensor. Concatenation of Ri and RBi (if bidirectional). The tensor has shape [num_directions, hidden_size, hidden_size].

• B (optional, heterogeneous) - T: The bias tensor for input gate. Concatenation of [Wbi, Rbi] and [WBbi, RBbi] (if bidirectional). The tensor has shape [num_directions, 2*hidden_size]. Optional: If not specified - assumed to be 0.

• sequence_lens (optional, heterogeneous) - T1: Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length seq_length. It has shape [batch_size].

• initial_h (optional, heterogeneous) - T: Optional initial value of the hidden. If not specified - assumed to be 0. It has shape [num_directions, batch_size, hidden_size].

Outputs

Between 0 and 2 outputs.

• Y (optional, heterogeneous) - T: A tensor that concats all the intermediate output values of the hidden. It has shape [seq_length, num_directions, batch_size, hidden_size].

• Y_h (optional, heterogeneous) - T: The last output value of the hidden. It has shape [num_directions, batch_size, hidden_size].

Type Constraints

• T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input and output types to float tensors.

• T1 in ( tensor(int32) ): Constrain seq_lens to integer tensor.

Examples

defaults

input = np.array([[[1., 2.], [3., 4.], [5., 6.]]]).astype(np.float32)

input_size = 2
hidden_size = 4
weight_scale = 0.1

node = onnx.helper.make_node(
    'RNN',
    inputs=['X', 'W', 'R'],
    outputs=['', 'Y_h'],
    hidden_size=hidden_size
)

W = weight_scale * np.ones((1, hidden_size, input_size)).astype(np.float32)
R = weight_scale * np.ones((1, hidden_size, hidden_size)).astype(np.float32)

rnn = RNN_Helper(X=input, W=W, R=R)
_, Y_h = rnn.step()
expect(node, inputs=[input, W, R], outputs=[Y_h.astype(np.float32)], name='test_simple_rnn_defaults')

initial_bias

input = np.array([[[1., 2., 3.], [4., 5., 6.], [7., 8., 9.]]]).astype(np.float32)

input_size = 3
hidden_size = 5
custom_bias = 0.1
weight_scale = 0.1

node = onnx.helper.make_node(
    'RNN',
    inputs=['X', 'W', 'R', 'B'],
    outputs=['', 'Y_h'],
    hidden_size=hidden_size
)

W = weight_scale * np.ones((1, hidden_size, input_size)).astype(np.float32)
R = weight_scale * np.ones((1, hidden_size, hidden_size)).astype(np.float32)

# Adding custom bias
W_B = custom_bias * np.ones((1, hidden_size)).astype(np.float32)
R_B = np.zeros((1, hidden_size)).astype(np.float32)
B = np.concatenate((W_B, R_B), axis=1)

rnn = RNN_Helper(X=input, W=W, R=R, B=B)
_, Y_h = rnn.step()
expect(node, inputs=[input, W, R, B], outputs=[Y_h.astype(np.float32)],
       name='test_simple_rnn_with_initial_bias')

seq_length

input = np.array([[[1., 2., 3.], [4., 5., 6.], [7., 8., 9.]],
                  [[10., 11., 12.], [13., 14., 15.], [16., 17., 18.]]]).astype(np.float32)

input_size = 3
hidden_size = 5

node = onnx.helper.make_node(
    'RNN',
    inputs=['X', 'W', 'R', 'B'],
    outputs=['', 'Y_h'],
    hidden_size=hidden_size
)

W = np.random.randn(1, hidden_size, input_size).astype(np.float32)
R = np.random.randn(1, hidden_size, hidden_size).astype(np.float32)

# Adding custom bias
W_B = np.random.randn(1, hidden_size).astype(np.float32)
R_B = np.random.randn(1, hidden_size).astype(np.float32)
B = np.concatenate((W_B, R_B), axis=1)

rnn = RNN_Helper(X=input, W=W, R=R, B=B)
_, Y_h = rnn.step()
expect(node, inputs=[input, W, R, B], outputs=[Y_h.astype(np.float32)], name='test_rnn_seq_length')

batchwise

input = np.array([[[1., 2.]], [[3., 4.]], [[5., 6.]]]).astype(np.float32)

input_size = 2
hidden_size = 4
weight_scale = 0.5
layout = 1

node = onnx.helper.make_node(
    'RNN',
    inputs=['X', 'W', 'R'],
    outputs=['Y', 'Y_h'],
    hidden_size=hidden_size,
    layout=layout
)

W = weight_scale * np.ones((1, hidden_size, input_size)).astype(np.float32)
R = weight_scale * np.ones((1, hidden_size, hidden_size)).astype(np.float32)

rnn = RNN_Helper(X=input, W=W, R=R, layout=layout)
Y, Y_h = rnn.step()
expect(node, inputs=[input, W, R], outputs=[Y.astype(np.float32), Y_h.astype(np.float32)], name='test_simple_rnn_batchwise')

Differences

0	0	Computes an one-layer simple RNN. This operator is usually supported	Computes an one-layer simple RNN. This operator is usually supported
1	1	via some custom implementation such as CuDNN.	via some custom implementation such as CuDNN.
2	2
3	3	Notations:	Notations:
4	4
5	5	X - input tensor	X - input tensor
6	6
7	7	i - input gate	i - input gate
8	8
9	9	t - time step (t-1 means previous time step)	t - time step (t-1 means previous time step)
10	10
11	11	Wi - W parameter weight matrix for input gate	Wi - W parameter weight matrix for input gate
12	12
13	13	Ri - R recurrence weight matrix for input gate	Ri - R recurrence weight matrix for input gate
14	14
15	15	Wbi - W parameter bias vector for input gate	Wbi - W parameter bias vector for input gate
16	16
17	17	Rbi - R parameter bias vector for input gate	Rbi - R parameter bias vector for input gate
18	18
19	19	WBi - W parameter weight matrix for backward input gate	WBi - W parameter weight matrix for backward input gate
20	20
21	21	RBi - R recurrence weight matrix for backward input gate	RBi - R recurrence weight matrix for backward input gate
22	22
23	23	WBbi - WR bias vectors for backward input gate	WBbi - WR bias vectors for backward input gate
24	24
25	25	RBbi - RR bias vectors for backward input gate	RBbi - RR bias vectors for backward input gate
26	26
27	27	H - Hidden state	H - Hidden state
28	28
29	29	num_directions - 2 if direction == bidirectional else 1	num_directions - 2 if direction == bidirectional else 1
30	30
31	31	Activation functions:	Activation functions:
32	32
33	33	Relu(x) - max(0, x)	Relu(x) - max(0, x)
34	34
35	35	Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})	Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})
36	36
37	37	Sigmoid(x) - 1/(1 + e^{-x})	Sigmoid(x) - 1/(1 + e^{-x})
38	38
39	39	(NOTE: Below are optional)	(NOTE: Below are optional)
40	40
41	41	Affine(x) - alpha*x + beta	Affine(x) - alpha*x + beta
42	42
43	43	LeakyRelu(x) - x if x >= 0 else alpha * x	LeakyRelu(x) - x if x >= 0 else alpha * x
44	44
45	45	ThresholdedRelu(x) - x if x >= alpha else 0	ThresholdedRelu(x) - x if x >= alpha else 0
46	46
47	47	ScaledTanh(x) - alpha*Tanh(beta*x)	ScaledTanh(x) - alpha*Tanh(beta*x)
48	48
49	49	HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)	HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)
50	50
51	51	Elu(x) - x if x >= 0 else alpha*(e^x - 1)	Elu(x) - x if x >= 0 else alpha*(e^x - 1)
52	52
53	53	Softsign(x) - x/(1 + |x|)	Softsign(x) - x/(1 + |x|)
54	54
55	55	Softplus(x) - log(1 + e^x)	Softplus(x) - log(1 + e^x)
56	56
57	57	Equations (Default: f=Tanh):	Equations (Default: f=Tanh):
58	58
59	59	- Ht = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Wbi + Rbi)	- Ht = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Wbi + Rbi)
60	60	This operator has **optional** inputs/outputs. See ONNX _ for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument's name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.	This operator has **optional** inputs/outputs. See ONNX _ for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument's name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.
61	61
62	62	**Attributes**	**Attributes**
63	63
64	64	* **activation_alpha**:	* **activation_alpha**:
65	65	Optional scaling values used by some activation functions. The	Optional scaling values used by some activation functions. The
66	66	values are consumed in the order of activation functions, for	values are consumed in the order of activation functions, for
67	67	example (f, g, h) in LSTM. Default values are the same as of	example (f, g, h) in LSTM. Default values are the same as of
68	68	corresponding ONNX operators.For example with LeakyRelu, the default	corresponding ONNX operators.For example with LeakyRelu, the default
69	69	alpha is 0.01.	alpha is 0.01.
70	70	* **activation_beta**:	* **activation_beta**:
71	71	Optional scaling values used by some activation functions. The	Optional scaling values used by some activation functions. The
72	72	values are consumed in the order of activation functions, for	values are consumed in the order of activation functions, for
73	73	example (f, g, h) in LSTM. Default values are the same as of	example (f, g, h) in LSTM. Default values are the same as of
74	74	corresponding ONNX operators.	corresponding ONNX operators.
75	75	* **activations**:	* **activations**:
76	76	One (or two if bidirectional) activation function for input gate.	One (or two if bidirectional) activation function for input gate.
77	77	The activation function must be one of the activation functions	The activation function must be one of the activation functions
78	78	specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].	specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].
79	79	* **clip**:	* **clip**:
80	80	Cell clip threshold. Clipping bounds the elements of a tensor in the	Cell clip threshold. Clipping bounds the elements of a tensor in the
81	81	range of [-threshold, +threshold] and is applied to the input of	range of [-threshold, +threshold] and is applied to the input of
82	82	activations. No clip if not specified.	activations. No clip if not specified.
83	83	* **direction**:	* **direction**:
84	84	Specify if the RNN is forward, reverse, or bidirectional. Must be	Specify if the RNN is forward, reverse, or bidirectional. Must be
85	85	one of forward (default), reverse, or bidirectional. Default value is 'forward'.	one of forward (default), reverse, or bidirectional. Default value is 'forward'.
86	86	* **hidden_size**:	* **hidden_size**:
87	87	Number of neurons in the hidden layer	Number of neurons in the hidden layer
	88		* **layout**:
	89		The shape format of inputs X, initial_h and outputs Y, Y_h. If 0,
	90		the following shapes are expected: X.shape = [seq_length,
	91		batch_size, input_size], Y.shape = [seq_length, num_directions,
	92		batch_size, hidden_size], initial_h.shape = Y_h.shape =
	93		[num_directions, batch_size, hidden_size]. If 1, the following
	94		shapes are expected: X.shape = [batch_size, seq_length, input_size],
	95		Y.shape = [batch_size, seq_length, num_directions, hidden_size],
	96		initial_h.shape = Y_h.shape = [batch_size, num_directions,
	97		hidden_size]. Default value is 0.
88	98
89	99	**Inputs**	**Inputs**
90	100
91	101	Between 3 and 6 inputs.	Between 3 and 6 inputs.
92	102
93	103	* **X** (heterogeneous) - **T**:	* **X** (heterogeneous) - **T**:
94	104	The input sequences packed (and potentially padded) into one 3-D	The input sequences packed (and potentially padded) into one 3-D
95	105	tensor with the shape of [seq_length, batch_size, input_size].	tensor with the shape of [seq_length, batch_size, input_size].
96	106	* **W** (heterogeneous) - **T**:	* **W** (heterogeneous) - **T**:
97	107	The weight tensor for input gate. Concatenation of Wi and WBi	The weight tensor for input gate. Concatenation of Wi and WBi
98	108	(if bidirectional). The tensor has shape [num_directions,	(if bidirectional). The tensor has shape [num_directions,
99	109	hidden_size, input_size].	hidden_size, input_size].
100	110	* **R** (heterogeneous) - **T**:	* **R** (heterogeneous) - **T**:
101	111	The recurrence weight tensor. Concatenation of Ri and RBi (if	The recurrence weight tensor. Concatenation of Ri and RBi (if
102	112	bidirectional). The tensor has shape [num_directions, hidden_size,	bidirectional). The tensor has shape [num_directions, hidden_size,
103	113	hidden_size].	hidden_size].
104	114	* **B** (optional, heterogeneous) - **T**:	* **B** (optional, heterogeneous) - **T**:
105	115	The bias tensor for input gate. Concatenation of [Wbi, Rbi] and	The bias tensor for input gate. Concatenation of [Wbi, Rbi] and
106	116	[WBbi, RBbi] (if bidirectional). The tensor has shape	[WBbi, RBbi] (if bidirectional). The tensor has shape
107	117	[num_directions, 2*hidden_size]. Optional: If not specified -	[num_directions, 2*hidden_size]. Optional: If not specified -
108	118	assumed to be 0.	assumed to be 0.
109	119	* **sequence_lens** (optional, heterogeneous) - **T1**:	* **sequence_lens** (optional, heterogeneous) - **T1**:
110	120	Optional tensor specifying lengths of the sequences in a batch. If	Optional tensor specifying lengths of the sequences in a batch. If
111	121	not specified - assumed all sequences in the batch to have length	not specified - assumed all sequences in the batch to have length
112	122	seq_length. It has shape [batch_size].	seq_length. It has shape [batch_size].
113	123	* **initial_h** (optional, heterogeneous) - **T**:	* **initial_h** (optional, heterogeneous) - **T**:
114	124	Optional initial value of the hidden. If not specified - assumed to	Optional initial value of the hidden. If not specified - assumed to
115	125	be 0. It has shape [num_directions, batch_size, hidden_size].	be 0. It has shape [num_directions, batch_size, hidden_size].
116	126
117	127	**Outputs**	**Outputs**
118	128
119	129	Between 0 and 2 outputs.	Between 0 and 2 outputs.
120	130
121	131	* **Y** (optional, heterogeneous) - **T**:	* **Y** (optional, heterogeneous) - **T**:
122	132	A tensor that concats all the intermediate output values of the	A tensor that concats all the intermediate output values of the
123	133	hidden. It has shape [seq_length, num_directions, batch_size,	hidden. It has shape [seq_length, num_directions, batch_size,
124	134	hidden_size].	hidden_size].
125	135	* **Y_h** (optional, heterogeneous) - **T**:	* **Y_h** (optional, heterogeneous) - **T**:
126	136	The last output value of the hidden. It has shape [num_directions,	The last output value of the hidden. It has shape [num_directions,
127	137	batch_size, hidden_size].	batch_size, hidden_size].
128	138
129	139	**Type Constraints**	**Type Constraints**
130	140
131	141	* **T** in (	* **T** in (
132	142	tensor(double),	tensor(double),
133	143	tensor(float),	tensor(float),
134	144	tensor(float16)	tensor(float16)
135	145	):	):
136	146	Constrain input and output types to float tensors.	Constrain input and output types to float tensors.
137	147	* **T1** in (	* **T1** in (
138	148	tensor(int32)	tensor(int32)
139	149	):	):
140	150	Constrain seq_lens to integer tensor.	Constrain seq_lens to integer tensor.

RNN - 7

Version

• name: RNN (GitHub)

• domain: main

• since_version: 7

• function: False

• support_level: SupportType.COMMON

• shape inference: True

This version of the operator has been available since version 7.

Summary

Computes an one-layer simple RNN. This operator is usually supported via some custom implementation such as CuDNN.

Notations:

X - input tensor

i - input gate

t - time step (t-1 means previous time step)

Wi - W parameter weight matrix for input gate

Ri - R recurrence weight matrix for input gate

Wbi - W parameter bias vector for input gate

Rbi - R parameter bias vector for input gate

WBi - W parameter weight matrix for backward input gate

RBi - R recurrence weight matrix for backward input gate

WBbi - WR bias vectors for backward input gate

RBbi - RR bias vectors for backward input gate

H - Hidden state

num_directions - 2 if direction == bidirectional else 1

Activation functions:

Relu(x) - max(0, x)

Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})

Sigmoid(x) - 1/(1 + e^{-x})

(NOTE: Below are optional)

Affine(x) - alpha*x + beta

LeakyRelu(x) - x if x >= 0 else alpha * x

ThresholdedRelu(x) - x if x >= alpha else 0

ScaledTanh(x) - alpha*Tanh(beta*x)

HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)

Elu(x) - x if x >= 0 else alpha*(e^x - 1)

Softsign(x) - x/(1 + |x|)

Softplus(x) - log(1 + e^x)

Equations (Default: f=Tanh):

• Ht = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Wbi + Rbi)

This operator has optional inputs/outputs. See ONNX for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument’s name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.

Attributes

• activation_alpha: Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.For example with LeakyRelu, the default alpha is 0.01.

• activation_beta: Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.

• activations: One (or two if bidirectional) activation function for input gate. The activation function must be one of the activation functions specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].

• clip: Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.

• direction: Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional. Default value is 'forward'.

• hidden_size: Number of neurons in the hidden layer

Inputs

Between 3 and 6 inputs.

• X (heterogeneous) - T: The input sequences packed (and potentially padded) into one 3-D tensor with the shape of [seq_length, batch_size, input_size].

• W (heterogeneous) - T: The weight tensor for input gate. Concatenation of Wi and WBi (if bidirectional). The tensor has shape [num_directions, hidden_size, input_size].

• R (heterogeneous) - T: The recurrence weight tensor. Concatenation of Ri and RBi (if bidirectional). The tensor has shape [num_directions, hidden_size, hidden_size].

• B (optional, heterogeneous) - T: The bias tensor for input gate. Concatenation of [Wbi, Rbi] and [WBbi, RBbi] (if bidirectional). The tensor has shape [num_directions, 2*hidden_size]. Optional: If not specified - assumed to be 0.

• sequence_lens (optional, heterogeneous) - T1: Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length seq_length. It has shape [batch_size].

• initial_h (optional, heterogeneous) - T: Optional initial value of the hidden. If not specified - assumed to be 0. It has shape [num_directions, batch_size, hidden_size].

Outputs

Between 0 and 2 outputs.

• Y (optional, heterogeneous) - T: A tensor that concats all the intermediate output values of the hidden. It has shape [seq_length, num_directions, batch_size, hidden_size].

• Y_h (optional, heterogeneous) - T: The last output value of the hidden. It has shape [num_directions, batch_size, hidden_size].

Type Constraints

• T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input and output types to float tensors.

• T1 in ( tensor(int32) ): Constrain seq_lens to integer tensor.

Differences

0	0	Computes an one-layer simple RNN. This operator is usually supported	Computes an one-layer simple RNN. This operator is usually supported
1	1	via some custom implementation such as CuDNN.	via some custom implementation such as CuDNN.
2	2
3	3	Notations:	Notations:
4	4
5	5	X - input tensor	X - input tensor
6	6
7	7	i - input gate	i - input gate
8	8
9	9	t - time step (t-1 means previous time step)	t - time step (t-1 means previous time step)
10	10
11	11	Wi - W parameter weight matrix for input gate	Wi - W parameter weight matrix for input gate
12	12
13	13	Ri - R recurrence weight matrix for input gate	Ri - R recurrence weight matrix for input gate
14	14
15	15	Wbi - W parameter bias vector for input gate	Wbi - W parameter bias vector for input gate
16	16
17	17	Rbi - R parameter bias vector for input gate	Rbi - R parameter bias vector for input gate
18	18
19	19	WBi - W parameter weight matrix for backward input gate	WBi - W parameter weight matrix for backward input gate
20	20
21	21	RBi - R recurrence weight matrix for backward input gate	RBi - R recurrence weight matrix for backward input gate
22	22
23	23	WBbi - WR bias vectors for backward input gate	WBbi - WR bias vectors for backward input gate
24	24
25	25	RBbi - RR bias vectors for backward input gate	RBbi - RR bias vectors for backward input gate
26	26
27	27	H - Hidden state	H - Hidden state
28	28
29	29	num_directions - 2 if direction == bidirectional else 1	num_directions - 2 if direction == bidirectional else 1
30	30
31	31	Activation functions:	Activation functions:
32	32
33	33	Relu(x) - max(0, x)	Relu(x) - max(0, x)
34	34
35	35	Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})	Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})
36	36
37	37	Sigmoid(x) - 1/(1 + e^{-x})	Sigmoid(x) - 1/(1 + e^{-x})
38	38
39	39	(NOTE: Below are optional)	(NOTE: Below are optional)
40	40
41	41	Affine(x) - alpha*x + beta	Affine(x) - alpha*x + beta
42	42
43	43	LeakyRelu(x) - x if x >= 0 else alpha * x	LeakyRelu(x) - x if x >= 0 else alpha * x
44	44
45	45	ThresholdedRelu(x) - x if x >= alpha else 0	ThresholdedRelu(x) - x if x >= alpha else 0
46	46
47	47	ScaledTanh(x) - alpha*Tanh(beta*x)	ScaledTanh(x) - alpha*Tanh(beta*x)
48	48
49	49	HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)	HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)
50	50
51	51	Elu(x) - x if x >= 0 else alpha*(e^x - 1)	Elu(x) - x if x >= 0 else alpha*(e^x - 1)
52	52
53	53	Softsign(x) - x/(1 + |x|)	Softsign(x) - x/(1 + |x|)
54	54
55	55	Softplus(x) - log(1 + e^x)	Softplus(x) - log(1 + e^x)
56	56
57	57	Equations (Default: f=Tanh):	Equations (Default: f=Tanh):
58	58
	59		- Ht = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Wbi + Rbi)
59	60	- Ht = f(Xt*(Wi^T) + Ht-1*Ri + Wbi + Rbi)	This operator has **optional** inputs/outputs. See ONNX <https://github.com/onnx/onnx/blob/master/docs/IR.md>_ for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument's name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.
60	61
61	62	**Attributes**	**Attributes**
62	63
63	64	* **activation_alpha**:	* **activation_alpha**:
64	65	Optional scaling values used by some activation functions. The	Optional scaling values used by some activation functions. The
65	66	values are consumed in the order of activation functions, for	values are consumed in the order of activation functions, for
66	67	example (f, g, h) in LSTM. Default values are the same as of	example (f, g, h) in LSTM. Default values are the same as of
67	68	corresponding ONNX operators.For example with LeakyRelu, the default	corresponding ONNX operators.For example with LeakyRelu, the default
68	69	alpha is 0.01.	alpha is 0.01.
69	70	* **activation_beta**:	* **activation_beta**:
70	71	Optional scaling values used by some activation functions. The	Optional scaling values used by some activation functions. The
71	72	values are consumed in the order of activation functions, for	values are consumed in the order of activation functions, for
72	73	example (f, g, h) in LSTM. Default values are the same as of	example (f, g, h) in LSTM. Default values are the same as of
73	74	corresponding ONNX operators.	corresponding ONNX operators.
74	75	* **activations**:	* **activations**:
75	76	One (or two if bidirectional) activation function for input gate.	One (or two if bidirectional) activation function for input gate.
76	77	The activation function must be one of the activation functions	The activation function must be one of the activation functions
77	78	specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].	specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].
78	79	* **clip**:	* **clip**:
79	80	Cell clip threshold. Clipping bounds the elements of a tensor in the	Cell clip threshold. Clipping bounds the elements of a tensor in the
80	81	range of [-threshold, +threshold] and is applied to the input of	range of [-threshold, +threshold] and is applied to the input of
81	82	activations. No clip if not specified.	activations. No clip if not specified.
82	83	* **direction**:	* **direction**:
83	84	Specify if the RNN is forward, reverse, or bidirectional. Must be	Specify if the RNN is forward, reverse, or bidirectional. Must be
84	85	one of forward (default), reverse, or bidirectional. Default value is 'forward'.	one of forward (default), reverse, or bidirectional. Default value is 'forward'.
85	86	* **hidden_size**:	* **hidden_size**:
86	87	Number of neurons in the hidden layer	Number of neurons in the hidden layer
87		* **output_sequence**:
88		The sequence output for the hidden is optional if 0. Default 0. Default value is 0.
89	88
90	89	**Inputs**	**Inputs**
91	90
92	91	Between 3 and 6 inputs.	Between 3 and 6 inputs.
93	92
94	93	* **X** (heterogeneous) - **T**:	* **X** (heterogeneous) - **T**:
95	94	The input sequences packed (and potentially padded) into one 3-D	The input sequences packed (and potentially padded) into one 3-D
96	95	tensor with the shape of [seq_length, batch_size, input_size].	tensor with the shape of [seq_length, batch_size, input_size].
97	96	* **W** (heterogeneous) - **T**:	* **W** (heterogeneous) - **T**:
98	97	The weight tensor for input gate. Concatenation of Wi and WBi	The weight tensor for input gate. Concatenation of Wi and WBi
99	98	(if bidirectional). The tensor has shape [num_directions,	(if bidirectional). The tensor has shape [num_directions,
100	99	hidden_size, input_size].	hidden_size, input_size].
101	100	* **R** (heterogeneous) - **T**:	* **R** (heterogeneous) - **T**:
102	101	The recurrence weight tensor. Concatenation of Ri and RBi (if	The recurrence weight tensor. Concatenation of Ri and RBi (if
103	102	bidirectional). The tensor has shape [num_directions, hidden_size,	bidirectional). The tensor has shape [num_directions, hidden_size,
104	103	hidden_size].	hidden_size].
105	104	* **B** (optional, heterogeneous) - **T**:	* **B** (optional, heterogeneous) - **T**:
106	105	The bias tensor for input gate. Concatenation of [Wbi, Rbi] and	The bias tensor for input gate. Concatenation of [Wbi, Rbi] and
107	106	[WBbi, RBbi] (if bidirectional). The tensor has shape	[WBbi, RBbi] (if bidirectional). The tensor has shape
108	107	[num_directions, 2*hidden_size]. Optional: If not specified -	[num_directions, 2*hidden_size]. Optional: If not specified -
109	108	assumed to be 0.	assumed to be 0.
110	109	* **sequence_lens** (optional, heterogeneous) - **T1**:	* **sequence_lens** (optional, heterogeneous) - **T1**:
111	110	Optional tensor specifying lengths of the sequences in a batch. If	Optional tensor specifying lengths of the sequences in a batch. If
112	111	not specified - assumed all sequences in the batch to have length	not specified - assumed all sequences in the batch to have length
113	112	seq_length. It has shape [batch_size].	seq_length. It has shape [batch_size].
114	113	* **initial_h** (optional, heterogeneous) - **T**:	* **initial_h** (optional, heterogeneous) - **T**:
115	114	Optional initial value of the hidden. If not specified - assumed to	Optional initial value of the hidden. If not specified - assumed to
116	115	be 0. It has shape [num_directions, batch_size, hidden_size].	be 0. It has shape [num_directions, batch_size, hidden_size].
117	116
118	117	**Outputs**	**Outputs**
119	118
120	119	Between 0 and 2 outputs.	Between 0 and 2 outputs.
121	120
122	121	* **Y** (optional, heterogeneous) - **T**:	* **Y** (optional, heterogeneous) - **T**:
123	122	A tensor that concats all the intermediate output values of the	A tensor that concats all the intermediate output values of the
124	123	hidden. It has shape [seq_length, num_directions, batch_size,	hidden. It has shape [seq_length, num_directions, batch_size,
125	124	hidden_size]. It is optional if output_sequence is 0.	hidden_size].
126	125	* **Y_h** (optional, heterogeneous) - **T**:	* **Y_h** (optional, heterogeneous) - **T**:
127	126	The last output value of the hidden. It has shape [num_directions,	The last output value of the hidden. It has shape [num_directions,
128	127	batch_size, hidden_size].	batch_size, hidden_size].
129	128
130	129	**Type Constraints**	**Type Constraints**
131	130
132	131	* **T** in (	* **T** in (
133	132	tensor(double),	tensor(double),
134	133	tensor(float),	tensor(float),
135	134	tensor(float16)	tensor(float16)
136	135	):	):
137	136	Constrain input and output types to float tensors.	Constrain input and output types to float tensors.
138	137	* **T1** in (	* **T1** in (
139	138	tensor(int32)	tensor(int32)
140	139	):	):
141	140	Constrain seq_lens to integer tensor.	Constrain seq_lens to integer tensor.

RNN - 1

Version

• name: RNN (GitHub)

• domain: main

• since_version: 1

• function: False

• support_level: SupportType.COMMON

• shape inference: True

This version of the operator has been available since version 1.

Summary

Computes an one-layer simple RNN. This operator is usually supported via some custom implementation such as CuDNN.

Notations:

X - input tensor

i - input gate

t - time step (t-1 means previous time step)

Wi - W parameter weight matrix for input gate

Ri - R recurrence weight matrix for input gate

Wbi - W parameter bias vector for input gate

Rbi - R parameter bias vector for input gate

WBi - W parameter weight matrix for backward input gate

RBi - R recurrence weight matrix for backward input gate

WBbi - WR bias vectors for backward input gate

RBbi - RR bias vectors for backward input gate

H - Hidden state

num_directions - 2 if direction == bidirectional else 1

Activation functions:

Relu(x) - max(0, x)

Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})

Sigmoid(x) - 1/(1 + e^{-x})

(NOTE: Below are optional)

Affine(x) - alpha*x + beta

LeakyRelu(x) - x if x >= 0 else alpha * x

ThresholdedRelu(x) - x if x >= alpha else 0

ScaledTanh(x) - alpha*Tanh(beta*x)

HardSigmoid(x) - min(max(alpha*x + beta, 0), 1)

Elu(x) - x if x >= 0 else alpha*(e^x - 1)

Softsign(x) - x/(1 + |x|)

Softplus(x) - log(1 + e^x)

Equations (Default: f=Tanh):

• Ht = f(Xt*(Wi^T) + Ht-1*Ri + Wbi + Rbi)

Attributes

• activation_alpha: Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.For example with LeakyRelu, the default alpha is 0.01.

• activation_beta: Optional scaling values used by some activation functions. The values are consumed in the order of activation functions, for example (f, g, h) in LSTM. Default values are the same as of corresponding ONNX operators.

• activations: One (or two if bidirectional) activation function for input gate. The activation function must be one of the activation functions specified above. Optional: Default Tanh if not specified. Default value is [b'Tanh' b'Tanh'].

• clip: Cell clip threshold. Clipping bounds the elements of a tensor in the range of [-threshold, +threshold] and is applied to the input of activations. No clip if not specified.

• direction: Specify if the RNN is forward, reverse, or bidirectional. Must be one of forward (default), reverse, or bidirectional. Default value is 'forward'.

• hidden_size: Number of neurons in the hidden layer

• output_sequence: The sequence output for the hidden is optional if 0. Default 0. Default value is 0.

Inputs

Between 3 and 6 inputs.

• X (heterogeneous) - T: The input sequences packed (and potentially padded) into one 3-D tensor with the shape of [seq_length, batch_size, input_size].

• W (heterogeneous) - T: The weight tensor for input gate. Concatenation of Wi and WBi (if bidirectional). The tensor has shape [num_directions, hidden_size, input_size].

• R (heterogeneous) - T: The recurrence weight tensor. Concatenation of Ri and RBi (if bidirectional). The tensor has shape [num_directions, hidden_size, hidden_size].

• B (optional, heterogeneous) - T: The bias tensor for input gate. Concatenation of [Wbi, Rbi] and [WBbi, RBbi] (if bidirectional). The tensor has shape [num_directions, 2*hidden_size]. Optional: If not specified - assumed to be 0.

• sequence_lens (optional, heterogeneous) - T1: Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length seq_length. It has shape [batch_size].

• initial_h (optional, heterogeneous) - T: Optional initial value of the hidden. If not specified - assumed to be 0. It has shape [num_directions, batch_size, hidden_size].

Outputs

Between 0 and 2 outputs.

• Y (optional, heterogeneous) - T: A tensor that concats all the intermediate output values of the hidden. It has shape [seq_length, num_directions, batch_size, hidden_size]. It is optional if output_sequence is 0.

• Y_h (optional, heterogeneous) - T: The last output value of the hidden. It has shape [num_directions, batch_size, hidden_size].

Type Constraints

• T in ( tensor(double), tensor(float), tensor(float16) ): Constrain input and output types to float tensors.

• T1 in ( tensor(int32) ): Constrain seq_lens to integer tensor.

previous

QuantizeLinear

next

RandomNormal