BatchNormalization#
Domain:
ai.onnxSince version: 15
Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. Depending on the mode it is being run, There are five required inputs ‘X’, ‘scale’, ‘B’, ‘input_mean’ and ‘input_var’. Note that ‘input_mean’ and ‘input_var’ are expected to be the estimated statistics in inference mode (training_mode=False, default), and the running statistics in training mode (training_mode=True). There are multiple cases for the number of outputs, which we list below:
Output case #1: Y, running_mean, running_var (training_mode=True)
Output case #2: Y (training_mode=False)
When training_mode=False, extra outputs are invalid. The outputs are updated as follows when training_mode=True:
running_mean = input_mean * momentum + current_mean * (1 - momentum)
running_var = input_var * momentum + current_var * (1 - momentum)
Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
where:
current_mean = ReduceMean(X, axis=all_except_channel_index)
current_var = ReduceVar(X, axis=all_except_channel_index)
Notice that ReduceVar refers to the population variance, and it equals to
sum(sqrd(x_i - x_avg)) / N
where N is the population size (this formula does not use sample size N - 1).
The computation of ReduceMean and ReduceVar uses float to avoid overflow for float16 inputs.
When training_mode=False:
Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B
For previous (depreciated) non-spatial cases, implementers are suggested to flatten the input shape to (N x C * D1 * D2 * … * Dn) before a BatchNormalization Op.
Inputs
X (T): Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1
scale (T1): Scale tensor of shape (C).
B (T1): Bias tensor of shape (C).
input_mean (T2): running (training) or estimated (testing) mean tensor of shape (C).
input_var (T2): running (training) or estimated (testing) variance tensor of shape (C).
Outputs
Y (T): The output tensor of the same shape as X
running_mean (T2): The running mean after the BatchNormalization operator.
running_var (T2): The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1.
Type Constraints
T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
T1: Constrain scale and bias types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
T2: Constrain mean and variance types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
Examples#
test_cc_batchnorm_epsilon
Node:
BatchNormalization(x, scale, B, input_mean, input_var) -> (y)
Attributes:
epsilon = 0.009999999776482582
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-1. , -0.9 , -0.8 , -0.7 , -0.6 ],
[-0.5 , -0.39999998, -0.3 , -0.19999999, -0.09999996],
[ 0. , 0.10000002, 0.20000005, 0.30000007, 0.39999998],
[ 0.5 , 0.6 , 0.70000005, 0.8000001 , 0.9 ]],
[[ 1. , 1.1000001 , 1.2 , 1.3 , 1.4000001 ],
[ 1.5 , 1.6000001 , 1.7 , 1.8 , 1.9000001 ],
[ 2. , 2.1000001 , 2.2 , 2.3 , 2.4 ],
[ 2.5 , 2.6000001 , 2.7 , 2.8 , 2.9 ]],
[[ 3. , 3.1 , 3.2000003 , 3.3000002 , 3.4 ],
[ 3.5 , 3.6 , 3.7000003 , 3.8000002 , 3.9 ],
[ 4. , 4.1 , 4.2000003 , 4.3 , 4.4 ],
[ 4.5 , 4.6 , 4.7000003 , 4.8 , 4.9 ]]],
[[[ 5. , 5.1 , 5.2000003 , 5.3 , 5.4 ],
[ 5.5 , 5.6 , 5.7000003 , 5.8 , 5.9 ],
[ 6. , 6.1 , 6.2000003 , 6.3 , 6.4 ],
[ 6.5 , 6.6 , 6.7000003 , 6.8 , 6.9 ]],
[[ 7. , 7.1000004 , 7.2 , 7.3 , 7.4000006 ],
[ 7.5 , 7.6000004 , 7.7 , 7.8 , 7.9000006 ],
[ 8. , 8.1 , 8.2 , 8.3 , 8.400001 ],
[ 8.5 , 8.6 , 8.7 , 8.8 , 8.900001 ]],
[[ 9. , 9.1 , 9.2 , 9.3 , 9.400001 ],
[ 9.5 , 9.6 , 9.7 , 9.8 , 9.900001 ],
[10. , 10.1 , 10.2 , 10.3 , 10.400001 ],
[10.5 , 10.6 , 10.7 , 10.8 , 10.900001 ]]]]
scale: shape=(3,), dtype=float32
[1. , 1.5, 2. ]
B: shape=(3,), dtype=float32
[ 0. , -0.5, 0.5]
input_mean: shape=(3,), dtype=float32
[ 0.5 , 1. , -0.25]
input_var: shape=(3,), dtype=float32
[0.25, 0.5 , 1. ]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-2.941742 , -2.745626 , -2.5495098 , -2.3533936 , -2.1572776 ],
[-1.9611614 , -1.7650452 , -1.5689292 , -1.372813 , -1.1766968 ],
[-0.9805807 , -0.7844645 , -0.5883483 , -0.39223212, -0.19611621],
[ 0. , 0.19611621, 0.39223242, 0.5883485 , 0.7844645 ]],
[[-0.5 , -0.28995776, -0.079916 , 0.130126 , 0.34016824],
[ 0.55021 , 0.76025224, 0.970294 , 1.180336 , 1.3903782 ],
[ 1.60042 , 1.8104625 , 2.020504 , 2.230546 , 2.440588 ],
[ 2.65063 , 2.8606725 , 3.070714 , 3.280756 , 3.490798 ]],
[[ 6.9677415 , 7.1667485 , 7.365757 , 7.564764 , 7.763771 ],
[ 7.9627786 , 8.161785 , 8.360794 , 8.559801 , 8.758808 ],
[ 8.957815 , 9.156823 , 9.355831 , 9.554838 , 9.753845 ],
[ 9.952853 , 10.15186 , 10.350868 , 10.549875 , 10.748882 ]]],
[[[ 8.825227 , 9.021342 , 9.21746 , 9.413575 , 9.609692 ],
[ 9.805807 , 10.001924 , 10.19804 , 10.394156 , 10.590272 ],
[10.786387 , 10.982504 , 11.17862 , 11.374737 , 11.570852 ],
[11.766969 , 11.963084 , 12.159202 , 12.355317 , 12.551434 ]],
[[12.10252 , 12.312563 , 12.522604 , 12.732646 , 12.942689 ],
[13.15273 , 13.362773 , 13.572813 , 13.782856 , 13.992899 ],
[14.20294 , 14.412983 , 14.623024 , 14.833067 , 15.04311 ],
[15.253149 , 15.463192 , 15.673233 , 15.883276 , 16.093319 ]],
[[18.908188 , 19.107195 , 19.306202 , 19.505209 , 19.704218 ],
[19.903225 , 20.102232 , 20.301239 , 20.500246 , 20.699255 ],
[20.898262 , 21.09727 , 21.296276 , 21.495283 , 21.694292 ],
[21.8933 , 22.092306 , 22.291313 , 22.49032 , 22.68933 ]]]]
test_cc_batchnorm_epsilon_training_mode
Node:
BatchNormalization(x, scale, B, input_mean, input_var) -> (y, output_mean, output_var)
Attributes:
epsilon = 0.009999999776482582
training_mode = 1
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-1. , -0.9 , -0.8 , -0.7 , -0.6 ],
[-0.5 , -0.39999998, -0.3 , -0.19999999, -0.09999996],
[ 0. , 0.10000002, 0.20000005, 0.30000007, 0.39999998],
[ 0.5 , 0.6 , 0.70000005, 0.8000001 , 0.9 ]],
[[ 1. , 1.1000001 , 1.2 , 1.3 , 1.4000001 ],
[ 1.5 , 1.6000001 , 1.7 , 1.8 , 1.9000001 ],
[ 2. , 2.1000001 , 2.2 , 2.3 , 2.4 ],
[ 2.5 , 2.6000001 , 2.7 , 2.8 , 2.9 ]],
[[ 3. , 3.1 , 3.2000003 , 3.3000002 , 3.4 ],
[ 3.5 , 3.6 , 3.7000003 , 3.8000002 , 3.9 ],
[ 4. , 4.1 , 4.2000003 , 4.3 , 4.4 ],
[ 4.5 , 4.6 , 4.7000003 , 4.8 , 4.9 ]]],
[[[ 5. , 5.1 , 5.2000003 , 5.3 , 5.4 ],
[ 5.5 , 5.6 , 5.7000003 , 5.8 , 5.9 ],
[ 6. , 6.1 , 6.2000003 , 6.3 , 6.4 ],
[ 6.5 , 6.6 , 6.7000003 , 6.8 , 6.9 ]],
[[ 7. , 7.1000004 , 7.2 , 7.3 , 7.4000006 ],
[ 7.5 , 7.6000004 , 7.7 , 7.8 , 7.9000006 ],
[ 8. , 8.1 , 8.2 , 8.3 , 8.400001 ],
[ 8.5 , 8.6 , 8.7 , 8.8 , 8.900001 ]],
[[ 9. , 9.1 , 9.2 , 9.3 , 9.400001 ],
[ 9.5 , 9.6 , 9.7 , 9.8 , 9.900001 ],
[10. , 10.1 , 10.2 , 10.3 , 10.400001 ],
[10.5 , 10.6 , 10.7 , 10.8 , 10.900001 ]]]]
scale: shape=(3,), dtype=float32
[1. , 1.5, 2. ]
B: shape=(3,), dtype=float32
[ 0. , -0.5, 0.5]
input_mean: shape=(3,), dtype=float32
[ 0.5 , 1. , -0.25]
input_var: shape=(3,), dtype=float32
[0.25, 0.5 , 1. ]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.2923064 , -1.2595899 , -1.2268733 , -1.1941566 , -1.16144 ],
[-1.1287234 , -1.0960068 , -1.0632901 , -1.0305735 , -0.9978569 ],
[-0.9651403 , -0.93242365, -0.899707 , -0.8669904 , -0.8342738 ],
[-0.8015572 , -0.76884055, -0.7361239 , -0.7034073 , -0.6706907 ]],
[[-2.4384599 , -2.389385 , -2.34031 , -2.2912352 , -2.24216 ],
[-2.1930852 , -2.1440103 , -2.0949354 , -2.0458605 , -1.9967855 ],
[-1.9477106 , -1.8986356 , -1.8495607 , -1.8004858 , -1.7514108 ],
[-1.702336 , -1.653261 , -1.604186 , -1.5551112 , -1.5060362 ]],
[[-2.084613 , -2.0191798 , -1.9537463 , -1.8883133 , -1.82288 ],
[-1.7574468 , -1.6920137 , -1.6265802 , -1.561147 , -1.495714 ],
[-1.4302807 , -1.3648474 , -1.2994139 , -1.2339809 , -1.1685476 ],
[-1.1031146 , -1.0376813 , -0.97224784, -0.9068146 , -0.84138155]]],
[[[ 0.6706907 , 0.70340735, 0.7361241 , 0.7688406 , 0.80155724],
[ 0.8342739 , 0.8669904 , 0.89970714, 0.9324238 , 0.9651403 ],
[ 0.9978569 , 1.0305736 , 1.0632901 , 1.0960069 , 1.1287234 ],
[ 1.1614399 , 1.1941566 , 1.2268734 , 1.2595899 , 1.2923064 ]],
[[ 0.5060358 , 0.55511093, 0.6041856 , 0.6532607 , 0.70233583],
[ 0.7514105 , 0.8004856 , 0.84956026, 0.8986354 , 0.9477105 ],
[ 0.99678516, 1.0458603 , 1.0949349 , 1.1440101 , 1.1930852 ],
[ 1.2421598 , 1.291235 , 1.3403096 , 1.3893847 , 1.4384599 ]],
[[ 1.8413811 , 1.9068146 , 1.9722476 , 2.037681 , 2.1031146 ],
[ 2.1685476 , 2.2339811 , 2.2994137 , 2.3648472 , 2.4302807 ],
[ 2.4957137 , 2.5611472 , 2.6265802 , 2.6920137 , 2.7574472 ],
[ 2.8228798 , 2.8883133 , 2.9537463 , 3.0191798 , 3.0846133 ]]]]
output_mean: shape=(3,), dtype=float32
[0.74500006, 1.3950001 , 0.47000018]
output_var: shape=(3,), dtype=float32
[1.1582502, 1.3832502, 1.8332503]
test_cc_batchnorm_example
Node:
BatchNormalization(x, scale, B, input_mean, input_var) -> (y)
Inputs:
x: shape=(1, 2, 1, 3), dtype=float32
[[[[-1., 0., 1.]],
[[ 2., 3., 4.]]]]
scale: shape=(2,), dtype=float32
[1. , 1.5]
B: shape=(2,), dtype=float32
[0., 1.]
input_mean: shape=(2,), dtype=float32
[0., 3.]
input_var: shape=(2,), dtype=float32
[1. , 1.5]
Outputs:
y: shape=(1, 2, 1, 3), dtype=float32
[[[[-0.999995 , 0. , 0.999995 ]],
[[-0.22474074, 1. , 2.2247407 ]]]]
test_cc_batchnorm_example_training_mode
Node:
BatchNormalization(x, scale, B, input_mean, input_var) -> (y, output_mean, output_var)
Attributes:
training_mode = 1
Inputs:
x: shape=(1, 2, 1, 3), dtype=float32
[[[[-1., 0., 1.]],
[[ 2., 3., 4.]]]]
scale: shape=(2,), dtype=float32
[1. , 1.5]
B: shape=(2,), dtype=float32
[0., 1.]
input_mean: shape=(2,), dtype=float32
[0., 3.]
input_var: shape=(2,), dtype=float32
[1. , 1.5]
Outputs:
y: shape=(1, 2, 1, 3), dtype=float32
[[[[-1.2247356 , 0. , 1.2247356 ]],
[[-0.83710337, 1. , 2.8371034 ]]]]
output_mean: shape=(2,), dtype=float32
[0., 3.]
output_var: shape=(2,), dtype=float32
[0.96666664, 1.4166666 ]
Differences with previous version (14)#
SchemaDiff: BatchNormalization (domain 'ai.onnx')
old version: 14
new version: 15
breaking: yes
Breaking reasons:
input ‘scale’ (changed): type_str changed ‘T’ -> ‘T1’
input ‘B’ (changed): type_str changed ‘T’ -> ‘T1’
input ‘input_mean’ (changed): type_str changed ‘U’ -> ‘T2’
input ‘input_var’ (changed): type_str changed ‘U’ -> ‘T2’
output ‘running_mean’ (changed): type_str changed ‘U’ -> ‘T2’
output ‘running_var’ (changed): type_str changed ‘U’ -> ‘T2’
type constraint ‘U’ (removed): entire constraint removed
Inputs:
[BREAKING] changed ‘scale’: type_str changed ‘T’ -> ‘T1’
[BREAKING] changed ‘B’: type_str changed ‘T’ -> ‘T1’
[BREAKING] changed ‘input_mean’: type_str changed ‘U’ -> ‘T2’
[BREAKING] changed ‘input_var’: type_str changed ‘U’ -> ‘T2’
Outputs:
[BREAKING] changed ‘running_mean’: type_str changed ‘U’ -> ‘T2’
[BREAKING] changed ‘running_var’: type_str changed ‘U’ -> ‘T2’
Type constraints:
[BREAKING] removed ‘U’: entire constraint removed
added ‘T1’: added types: [‘tensor(bfloat16)’, ‘tensor(double)’, ‘tensor(float)’, ‘tensor(float16)’]
added ‘T2’: added types: [‘tensor(bfloat16)’, ‘tensor(double)’, ‘tensor(float)’, ‘tensor(float16)’]
Documentation:
line similarity: 0.77 (+9/-9 lines)
--- BatchNormalization v14
+++ BatchNormalization v15
@@ -8,8 +8,8 @@
and the running statistics in training mode (training_mode=True).
There are multiple cases for the number of outputs, which we list below:
-Output case #1: Y, running_mean, running_var (training_mode=True)
-Output case #2: Y (training_mode=False)
+* Output case #1: Y, running_mean, running_var (training_mode=True)
+* Output case #2: Y (training_mode=False)
When training_mode=False, extra outputs are invalid.
The outputs are updated as follows when training_mode=True:
@@ -18,17 +18,17 @@
running_var = input_var * momentum + current_var * (1 - momentum)
Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
-
+```
where:
-
+```
current_mean = ReduceMean(X, axis=all_except_channel_index)
current_var = ReduceVar(X, axis=all_except_channel_index)
+```
+Notice that `ReduceVar` refers to the population variance, and it equals to
+`sum(sqrd(x_i - x_avg)) / N`
+where `N` is the population size (this formula does not use sample size `N - 1`).
-Notice that ReduceVar refers to the population variance, and it equals to
-sum(sqrd(x_i - x_avg)) / N
-where N is the population size (this formula does not use sample size N - 1).
-
-```
+The computation of ReduceMean and ReduceVar uses float to avoid overflow for float16 inputs.
When training_mode=False:
```