:nosearch: .. _op_ai_onnx_BatchNormalization-14: BatchNormalization - version 14 =============================== This page documents version **14** of operator **BatchNormalization**. See :doc:`BatchNormalization` for the latest version (since version 15). - **Domain**: ``ai.onnx`` - **Since version**: 14 Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. Depending on the mode it is being run, There are five required inputs 'X', 'scale', 'B', 'input_mean' and 'input_var'. Note that 'input_mean' and 'input_var' are expected to be the estimated statistics in inference mode (training_mode=False, default), and the running statistics in training mode (training_mode=True). There are multiple cases for the number of outputs, which we list below: Output case #1: Y, running_mean, running_var (training_mode=True) Output case #2: Y (training_mode=False) When training_mode=False, extra outputs are invalid. The outputs are updated as follows when training_mode=True: .. code-block:: running_mean = input_mean * momentum + current_mean * (1 - momentum) running_var = input_var * momentum + current_var * (1 - momentum) Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B where: current_mean = ReduceMean(X, axis=all_except_channel_index) current_var = ReduceVar(X, axis=all_except_channel_index) Notice that ReduceVar refers to the population variance, and it equals to sum(sqrd(x_i - x_avg)) / N where N is the population size (this formula does not use sample size N - 1). When training_mode=False: .. code-block:: Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B For previous (depreciated) non-spatial cases, implementers are suggested to flatten the input shape to (N x C \* D1 \* D2 \* ... \* Dn) before a BatchNormalization Op. **Inputs** - **X** (*T*): Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 ... Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1 - **scale** (*T*): Scale tensor of shape (C). - **B** (*T*): Bias tensor of shape (C). - **input_mean** (*U*): running (training) or estimated (testing) mean tensor of shape (C). - **input_var** (*U*): running (training) or estimated (testing) variance tensor of shape (C). **Outputs** - **Y** (*T*): The output tensor of the same shape as X - **running_mean** (*U*): The running mean after the BatchNormalization operator. - **running_var** (*U*): The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1. **Type Constraints** - **T**: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16). - **U**: Constrain mean and variance types to float tensors. It allows all float type for U. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16). Differences with previous version (9) ------------------------------------- **SchemaDiff**: ``BatchNormalization`` (domain ``'ai.onnx'``) * old version: 9 * new version: 14 * breaking: **yes** **Breaking reasons:** * input 'mean' (removed): was at position 3 * input 'var' (removed): was at position 4 * input 'input_mean' (added): at position 3; option=Single; type_str='U' * input 'input_var' (added): at position 4; option=Single; type_str='U' * output 'mean' (removed): was at position 1 * output 'var' (removed): was at position 2 * output 'saved_mean' (removed): was at position 3 * output 'saved_var' (removed): was at position 4 * output 'running_mean' (added): at position 1; option=Single; type_str='U' * output 'running_var' (added): at position 2; option=Single; type_str='U' * output '' (changed): min_output 5 -> 3; max_output 5 -> 3 **Inputs:** * [BREAKING] removed 'mean': was at position 3 * [BREAKING] removed 'var': was at position 4 * [BREAKING] added 'input_mean': at position 3; option=Single; type_str='U' * [BREAKING] added 'input_var': at position 4; option=Single; type_str='U' **Outputs:** * [BREAKING] removed 'mean': was at position 1 * [BREAKING] removed 'var': was at position 2 * [BREAKING] removed 'saved_mean': was at position 3 * [BREAKING] removed 'saved_var': was at position 4 * [BREAKING] added 'running_mean': at position 1; option=Single; type_str='U' * [BREAKING] added 'running_var': at position 2; option=Single; type_str='U' * [BREAKING] changed '': min_output 5 -> 3; max_output 5 -> 3 **Type constraints:** * added 'U': added types: ['tensor(bfloat16)', 'tensor(double)', 'tensor(float)', 'tensor(float16)'] * changed 'T': added types: ['tensor(bfloat16)'] **Documentation:** * line similarity: 0.24 (+33/-4 lines) .. code-block:: diff --- BatchNormalization v9 +++ BatchNormalization v14 @@ -1,10 +1,39 @@ Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. Depending on the mode it is being run, -there are multiple cases for the number of outputs, which we list below: +There are five required inputs 'X', 'scale', 'B', 'input_mean' and +'input_var'. +Note that 'input_mean' and 'input_var' are expected to be the estimated +statistics in inference mode (training_mode=False, default), +and the running statistics in training mode (training_mode=True). +There are multiple cases for the number of outputs, which we list below: -Output case #1: Y, mean, var, saved_mean, saved_var (training mode) -Output case #2: Y (test mode) +Output case #1: Y, running_mean, running_var (training_mode=True) +Output case #2: Y (training_mode=False) + +When training_mode=False, extra outputs are invalid. +The outputs are updated as follows when training_mode=True: +``` +running_mean = input_mean * momentum + current_mean * (1 - momentum) +running_var = input_var * momentum + current_var * (1 - momentum) + +Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B + +where: + +current_mean = ReduceMean(X, axis=all_except_channel_index) +current_var = ReduceVar(X, axis=all_except_channel_index) + +Notice that ReduceVar refers to the population variance, and it equals to +sum(sqrd(x_i - x_avg)) / N +where N is the population size (this formula does not use sample size N - 1). + +``` + +When training_mode=False: +``` +Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B +``` For previous (depreciated) non-spatial cases, implementers are suggested -to flatten the input shape to (N x C*D1*D2 ..*Dn) before a BatchNormalization Op. +to flatten the input shape to (N x C * D1 * D2 * ... * Dn) before a BatchNormalization Op.