BatchNormalization - version 14#

This page documents version 14 of operator BatchNormalization. See BatchNormalization for the latest version (since version 15).

  • Domain: ai.onnx

  • Since version: 14

Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. Depending on the mode it is being run, There are five required inputs ‘X’, ‘scale’, ‘B’, ‘input_mean’ and ‘input_var’. Note that ‘input_mean’ and ‘input_var’ are expected to be the estimated statistics in inference mode (training_mode=False, default), and the running statistics in training mode (training_mode=True). There are multiple cases for the number of outputs, which we list below:

Output case #1: Y, running_mean, running_var (training_mode=True) Output case #2: Y (training_mode=False)

When training_mode=False, extra outputs are invalid. The outputs are updated as follows when training_mode=True:

running_mean = input_mean * momentum + current_mean * (1 - momentum)
running_var = input_var * momentum + current_var * (1 - momentum)

Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B

where:

current_mean = ReduceMean(X, axis=all_except_channel_index)
current_var =  ReduceVar(X, axis=all_except_channel_index)

Notice that ReduceVar refers to the population variance, and it equals to
sum(sqrd(x_i - x_avg)) / N
where N is the population size (this formula does not use sample size N - 1).

When training_mode=False:

Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B

For previous (depreciated) non-spatial cases, implementers are suggested to flatten the input shape to (N x C * D1 * D2 * … * Dn) before a BatchNormalization Op.

Inputs

  • X (T): Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1

  • scale (T): Scale tensor of shape (C).

  • B (T): Bias tensor of shape (C).

  • input_mean (U): running (training) or estimated (testing) mean tensor of shape (C).

  • input_var (U): running (training) or estimated (testing) variance tensor of shape (C).

Outputs

  • Y (T): The output tensor of the same shape as X

  • running_mean (U): The running mean after the BatchNormalization operator.

  • running_var (U): The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1.

Type Constraints

  • T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

  • U: Constrain mean and variance types to float tensors. It allows all float type for U. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

Differences with previous version (9)#

SchemaDiff: BatchNormalization (domain 'ai.onnx')

  • old version: 9

  • new version: 14

  • breaking: yes

Breaking reasons:

  • input ‘mean’ (removed): was at position 3

  • input ‘var’ (removed): was at position 4

  • input ‘input_mean’ (added): at position 3; option=Single; type_str=’U’

  • input ‘input_var’ (added): at position 4; option=Single; type_str=’U’

  • output ‘mean’ (removed): was at position 1

  • output ‘var’ (removed): was at position 2

  • output ‘saved_mean’ (removed): was at position 3

  • output ‘saved_var’ (removed): was at position 4

  • output ‘running_mean’ (added): at position 1; option=Single; type_str=’U’

  • output ‘running_var’ (added): at position 2; option=Single; type_str=’U’

  • output ‘<arity>’ (changed): min_output 5 -> 3; max_output 5 -> 3

Inputs:

  • [BREAKING] removed ‘mean’: was at position 3

  • [BREAKING] removed ‘var’: was at position 4

  • [BREAKING] added ‘input_mean’: at position 3; option=Single; type_str=’U’

  • [BREAKING] added ‘input_var’: at position 4; option=Single; type_str=’U’

Outputs:

  • [BREAKING] removed ‘mean’: was at position 1

  • [BREAKING] removed ‘var’: was at position 2

  • [BREAKING] removed ‘saved_mean’: was at position 3

  • [BREAKING] removed ‘saved_var’: was at position 4

  • [BREAKING] added ‘running_mean’: at position 1; option=Single; type_str=’U’

  • [BREAKING] added ‘running_var’: at position 2; option=Single; type_str=’U’

  • [BREAKING] changed ‘<arity>’: min_output 5 -> 3; max_output 5 -> 3

Type constraints:

  • added ‘U’: added types: [‘tensor(bfloat16)’, ‘tensor(double)’, ‘tensor(float)’, ‘tensor(float16)’]

  • changed ‘T’: added types: [‘tensor(bfloat16)’]

Documentation:

  • line similarity: 0.24 (+33/-4 lines)

--- BatchNormalization v9
+++ BatchNormalization v14
@@ -1,10 +1,39 @@

 Carries out batch normalization as described in the paper
 https://arxiv.org/abs/1502.03167. Depending on the mode it is being run,
-there are multiple cases for the number of outputs, which we list below:
+There are five required inputs 'X', 'scale', 'B', 'input_mean' and
+'input_var'.
+Note that 'input_mean' and 'input_var' are expected to be the estimated
+statistics in inference mode (training_mode=False, default),
+and the running statistics in training mode (training_mode=True).
+There are multiple cases for the number of outputs, which we list below:

-Output case #1: Y, mean, var, saved_mean, saved_var (training mode)
-Output case #2: Y (test mode)
+Output case #1: Y, running_mean, running_var (training_mode=True)
+Output case #2: Y (training_mode=False)
+
+When training_mode=False, extra outputs are invalid.
+The outputs are updated as follows when training_mode=True:
+```
+running_mean = input_mean * momentum + current_mean * (1 - momentum)
+running_var = input_var * momentum + current_var * (1 - momentum)
+
+Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
+
+where:
+
+current_mean = ReduceMean(X, axis=all_except_channel_index)
+current_var =  ReduceVar(X, axis=all_except_channel_index)
+
+Notice that ReduceVar refers to the population variance, and it equals to
+sum(sqrd(x_i - x_avg)) / N
+where N is the population size (this formula does not use sample size N - 1).
+
+```
+
+When training_mode=False:
+```
+Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B
+```

 For previous (depreciated) non-spatial cases, implementers are suggested
-to flatten the input shape to (N x C*D1*D2 ..*Dn) before a BatchNormalization Op.
+to flatten the input shape to (N x C * D1 * D2 * ... * Dn) before a BatchNormalization Op.