GroupNormalization#
Domain:
ai.onnxSince version: 21
A GroupNormalization function. Carries out group normalization as described in the paper https://arxiv.org/abs/1803.08494
This operator transforms input according to
y = scale * (x - mean) / sqrt(variance + epsilon) + bias,
where the mean and variance are computed per instance per group of channels, and
scale and bias should be specified for each channel. The number of
groups num_groups should be divisible by the number of channels so that there are
an equal number of channels per group.
The overall computation has two stages: the first stage normalizes the elements to
have zero mean and unit variance for each instance in each group, and the second
stage scales and shifts the results of the first stage. The floating-point precision
used in the first stage is determined by the stash_type attribute. For example,
if stash_type is 1, the operator casts all input variables to 32-bit float,
performs the computation, and finally casts the normalized results back to the
original type of X. The second stage does not depend on stash_type.
When the number of groups is the same as the number of channels, this operator is equivalent to InstanceNormalization. When there is only one group, this operator is equivalent to LayerNormalization.
Inputs
X (T): Input data tensor. Dimensions for image cases are
(N x C x H x W), whereNis the batch size,Cis the number of channels, andHandWare the height and width of the data. Statistics are computed for every group of channels overC,H, andW. For non-image cases, the dimensions are in the form of(N x C x D1 x D2 ... Dn).scale (T): Scale tensor of shape
(C).bias (T): Bias tensor of shape
(C).
Outputs
Y (T): The output tensor of the same shape as
X.
Attributes
epsilon (float): The epsilon value to use to avoid division by zero.
num_groups (int): The number of groups of channels. It should be a divisor of the number of channels
C.stash_type (int): The floating-point precision used in stage one of the computation.
Type Constraints
T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
Examples#
test_cc_group_normalization_epsilon
Node:
GroupNormalization(x, scale, bias) -> (y)
Attributes:
num_groups = 2
epsilon = 0.009999999776482582
Inputs:
x: shape=(3, 4, 2, 2), dtype=float32
[[[[-2. , -1.9 ],
[-1.8 , -1.7 ]],
[[-1.6 , -1.5 ],
[-1.4 , -1.3 ]],
[[-1.2 , -1.0999999 ],
[-1. , -0.9 ]],
[[-0.79999995, -0.6999999 ],
[-0.6 , -0.5 ]]],
[[[-0.39999998, -0.29999995],
[-0.19999993, -0.10000002]],
[[ 0. , 0.10000014],
[ 0.20000005, 0.29999995]],
[[ 0.4000001 , 0.5 ],
[ 0.60000014, 0.70000005]],
[[ 0.79999995, 0.9000001 ],
[ 1. , 1.1000001 ]]],
[[[ 1.2 , 1.3 ],
[ 1.4000001 , 1.5 ]],
[[ 1.6000001 , 1.7 ],
[ 1.8 , 1.9000001 ]],
[[ 2. , 2.1 ],
[ 2.2000003 , 2.3000002 ]],
[[ 2.4 , 2.5 ],
[ 2.6 , 2.7000003 ]]]]
scale: shape=(4,), dtype=float32
[0.5, 1. , 1.5, 2. ]
bias: shape=(4,), dtype=float32
[-0.25, 0. , 0.25, 0.5 ]
Outputs:
y: shape=(3, 4, 2, 2), dtype=float32
[[[[-0.95000005, -0.75 ],
[-0.54999995, -0.35000014]],
[[ 0.19999981, 0.5999999 ],
[ 1. , 1.4000001 ]],
[[-1.8500004 , -1.2499995 ],
[-0.6500001 , -0.04999971]],
[[ 0.9000001 , 1.7000003 ],
[ 2.4999995 , 3.2999997 ]]],
[[[-0.95000005, -0.75 ],
[-0.54999995, -0.3500001 ]],
[[ 0.19999987, 0.60000044],
[ 1. , 1.3999996 ]],
[[-1.8499999 , -1.2500005 ],
[-0.6499996 , -0.05000019]],
[[ 0.89999914, 1.7000003 ],
[ 2.4999995 , 3.3000007 ]]],
[[[-0.95000005, -0.75000024],
[-0.54999995, -0.35000014]],
[[ 0.20000029, 0.5999999 ],
[ 0.9999995 , 1.4000001 ]],
[[-1.8500004 , -1.250001 ],
[-0.64999866, -0.04999924]],
[[ 0.8999996 , 1.6999989 ],
[ 2.499998 , 3.3000011 ]]]]
test_cc_group_normalization_example
Node:
GroupNormalization(x, scale, bias) -> (y)
Attributes:
num_groups = 2
Inputs:
x: shape=(3, 4, 2, 2), dtype=float32
[[[[-2. , -1.9 ],
[-1.8 , -1.7 ]],
[[-1.6 , -1.5 ],
[-1.4 , -1.3 ]],
[[-1.2 , -1.0999999 ],
[-1. , -0.9 ]],
[[-0.79999995, -0.6999999 ],
[-0.6 , -0.5 ]]],
[[[-0.39999998, -0.29999995],
[-0.19999993, -0.10000002]],
[[ 0. , 0.10000014],
[ 0.20000005, 0.29999995]],
[[ 0.4000001 , 0.5 ],
[ 0.60000014, 0.70000005]],
[[ 0.79999995, 0.9000001 ],
[ 1. , 1.1000001 ]]],
[[[ 1.2 , 1.3 ],
[ 1.4000001 , 1.5 ]],
[[ 1.6000001 , 1.7 ],
[ 1.8 , 1.9000001 ]],
[[ 2. , 2.1 ],
[ 2.2000003 , 2.3000002 ]],
[[ 2.4 , 2.5 ],
[ 2.6 , 2.7000003 ]]]]
scale: shape=(4,), dtype=float32
[0.5, 1. , 1.5, 2. ]
bias: shape=(4,), dtype=float32
[-0.25, 0. , 0.25, 0.5 ]
Outputs:
y: shape=(3, 4, 2, 2), dtype=float32
[[[[-1.01369 , -0.79549265],
[-0.5772958 , -0.35909867]],
[[ 0.21819687, 0.6545911 ],
[ 1.0909853 , 1.5273795 ]],
[[-2.0410705 , -1.386478 ],
[-0.73188734, -0.07729578]],
[[ 0.9363942 , 1.8091831 ],
[ 2.6819706 , 3.5547595 ]]],
[[[-1.01369 , -0.79549277],
[-0.5772956 , -0.35909867]],
[[ 0.21819699, 0.6545919 ],
[ 1.0909858 , 1.5273795 ]],
[[-2.0410693 , -1.3864787 ],
[-0.7318864 , -0.07729578]],
[[ 0.93639326, 1.8091831 ],
[ 2.6819706 , 3.55476 ]]],
[[[-1.01369 , -0.7954929 ],
[-0.57729554, -0.35909867]],
[[ 0.21819735, 0.6545911 ],
[ 1.0909853 , 1.52738 ]],
[[-2.04107 , -1.3864794 ],
[-0.7318859 , -0.0772953 ]],
[[ 0.93639374, 1.8091812 ],
[ 2.6819687 , 3.55476 ]]]]
Differences with previous version (18)#
SchemaDiff: GroupNormalization (domain 'ai.onnx')
old version: 18
new version: 21
breaking: no
Deprecation:
deprecated True -> False
Attributes:
added ‘stash_type’: type=INT; required=False; default=1
Documentation:
line similarity: 0.75 (+9/-1 lines)
--- GroupNormalization v18
+++ GroupNormalization v21
@@ -7,9 +7,17 @@
y = scale * (x - mean) / sqrt(variance + epsilon) + bias,
```
where the mean and variance are computed per instance per group of channels, and
-`scale` and `bias` should be specified for each group of channels. The number of
+`scale` and `bias` should be specified for each channel. The number of
groups `num_groups` should be divisible by the number of channels so that there are
an equal number of channels per group.
+
+The overall computation has two stages: the first stage normalizes the elements to
+have zero mean and unit variance for each instance in each group, and the second
+stage scales and shifts the results of the first stage. The floating-point precision
+used in the first stage is determined by the `stash_type` attribute. For example,
+if `stash_type` is 1, the operator casts all input variables to 32-bit float,
+performs the computation, and finally casts the normalized results back to the
+original type of `X`. The second stage does not depend on `stash_type`.
When the number of groups is the same as the number of channels, this operator is
equivalent to InstanceNormalization. When there is only one group, this operator