LayerNormalization#
Domain:
ai.onnxSince version: 17
This is layer normalization defined in ONNX as function. The overall computation can be split into two stages. The first stage is standardization, which makes the normalized elements have zero mean and unit variances. The computation required by standardization can be described by the following equations.
Mean = ReduceMean (X)
D = Sub(X, Mean)
DD = Mul(D, D)
Var = ReduceMean (DD)
VarEps = Add(Var, epsilon)
StdDev = Sqrt(VarEps)
InvStdDev = Reciprocal(StdDev)
Normalized = Mul(D, InvStdDev)
where normalized_axes is [axis, ..., rank of X - 1].
The variables Var and StdDev stand for variance and
standard deviation, respectively. The second output is
Mean and the last one is InvStdDev.
Depending on stash_type attribute, the actual computation
must happen in different floating-point precision.
For example, if stash_type is 1, this operator casts
all input variables to 32-bit float, perform the computation, and
finally cast Normalized back to the original type of X.
The second stage then scales and shifts the outcome of the
first stage using
NormalizedScaled = Mul(Normalized, Scale)
Y = Add(NormalizedScaled, B)
The second stage doesn’t depends on stash_type.
All equations are in this syntax.
The same variable (i.e., input, output, and attribute) uses
the same name in the equations above and this operator’s definition.
Let d[i] indicate the i-th dimension of X.
If X’s shape is [d[0], ..., d[axis-1], d[axis], ..., d[rank-1]],
the shape of Mean and InvStdDev is [d[0], ..., d[axis-1], 1, ..., 1].
Y and X have the same shape. This operator supports unidirectional broadcasting
(tensors Scale and B should be unidirectional broadcastable to tensor X);
for more details please check the doc.
Inputs
X (T): Tensor to be normalized.
Scale (T): Scale tensor.
B (T): Bias tensor.
Outputs
Y (T): Normalized tensor.
Mean (U): Saved mean used during training to speed up gradient computation
InvStdDev (U): Saved inverse standard deviation used during training to speed up gradient computation.
Attributes
axis (int): The first normalization dimension. If rank(X) is r, axis’ allowed range is [-r, r). Negative value means counting dimensions from the back.
epsilon (float): The epsilon value to use to avoid division by zero.
stash_type (int): Type of Mean and InvStdDev. This also specifies stage one’s computation precision.
Type Constraints
T: Constrain input types and output Y type to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
U: Type of Mean and InvStdDev tensors. Allowed types: tensor(bfloat16), tensor(float).
Examples#
test_cc_layer_normalization_2d_axis0
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 0
Inputs:
X: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
W: shape=(3, 4), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 ],
[0.58 , 0.6 , 0.62 , 0.64 ],
[0.65999997, 0.68 , 0.7 , 0.72 ]]
B: shape=(3, 4), dtype=float32
[[-0.25 , -0.24 , -0.23 , -0.22 ],
[-0.21000001, -0.2 , -0.19 , -0.18 ],
[-0.17 , -0.16 , -0.15 , -0.14 ]]
Outputs:
Y: shape=(3, 4), dtype=float32
[[-1.0464939 , -0.9177438 , -0.7774086 , -0.6254878 ],
[-0.46198177, -0.28689027, -0.10021339, 0.09804872],
[ 0.30789632, 0.5293293 , 0.76234746, 1.0069512 ]]
Mean: shape=(1, 1), dtype=float32
[[-0.225]]
InvStdDev: shape=(1, 1), dtype=float32
[[5.7926826]]
test_cc_layer_normalization_2d_axis1
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 1
Inputs:
X: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
W: shape=(4,), dtype=float32
[0.5 , 0.52, 0.54, 0.56]
B: shape=(4,), dtype=float32
[-0.25, -0.24, -0.23, -0.22]
Outputs:
Y: shape=(3, 4), dtype=float32
[[-0.91974956, -0.47217965, 0.01110995, 0.5301198 ],
[-0.9197498 , -0.47217992, 0.01110995, 0.53011954],
[-0.9197497 , -0.47217977, 0.0111098 , 0.53011966]]
Mean: shape=(3, 1), dtype=float32
[[-0.425 ],
[-0.225 ],
[-0.02499999]]
InvStdDev: shape=(3, 1), dtype=float32
[[17.859991],
[17.859991],
[17.859991]]
test_cc_layer_normalization_2d_axis_negative_1
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -1
Inputs:
X: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
W: shape=(4,), dtype=float32
[0.5 , 0.52, 0.54, 0.56]
B: shape=(4,), dtype=float32
[-0.25, -0.24, -0.23, -0.22]
Outputs:
Y: shape=(3, 4), dtype=float32
[[-0.91974956, -0.47217965, 0.01110995, 0.5301198 ],
[-0.9197498 , -0.47217992, 0.01110995, 0.53011954],
[-0.9197497 , -0.47217977, 0.0111098 , 0.53011966]]
Mean: shape=(3, 1), dtype=float32
[[-0.425 ],
[-0.225 ],
[-0.02499999]]
InvStdDev: shape=(3, 1), dtype=float32
[[17.859991],
[17.859991],
[17.859991]]
test_cc_layer_normalization_2d_axis_negative_2
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -2
Inputs:
X: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
W: shape=(3, 4), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 ],
[0.58 , 0.6 , 0.62 , 0.64 ],
[0.65999997, 0.68 , 0.7 , 0.72 ]]
B: shape=(3, 4), dtype=float32
[[-0.25 , -0.24 , -0.23 , -0.22 ],
[-0.21000001, -0.2 , -0.19 , -0.18 ],
[-0.17 , -0.16 , -0.15 , -0.14 ]]
Outputs:
Y: shape=(3, 4), dtype=float32
[[-1.0464939 , -0.9177438 , -0.7774086 , -0.6254878 ],
[-0.46198177, -0.28689027, -0.10021339, 0.09804872],
[ 0.30789632, 0.5293293 , 0.76234746, 1.0069512 ]]
Mean: shape=(1, 1), dtype=float32
[[-0.225]]
InvStdDev: shape=(1, 1), dtype=float32
[[5.7926826]]
test_cc_layer_normalization_3d_axis0_epsilon
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 0
epsilon = 0.10000000149011612
Inputs:
X: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
W: shape=(2, 3, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]],
[[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ],
[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ]]]
B: shape=(2, 3, 5), dtype=float32
[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]],
[[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ],
[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001],
[ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999]]]
Outputs:
Y: shape=(2, 3, 5), dtype=float32
[[[-0.9263111 , -0.8948557 , -0.85966897, -0.82075083, -0.7781013 ],
[-0.73172045, -0.6816082 , -0.6277646 , -0.5701896 , -0.50888324],
[-0.4438455 , -0.3750764 , -0.3025759 , -0.22634405, -0.1463809 ]],
[[-0.06268631, 0.02473967, 0.11589703, 0.21078573, 0.3094057 ],
[ 0.41175717, 0.51784 , 0.6276542 , 0.74119973, 0.85847676],
[ 0.979485 , 1.1042248 , 1.2326956 , 1.364898 , 1.5008317 ]]]
Mean: shape=(1, 1, 1), dtype=float32
[[[0.22500001]]]
InvStdDev: shape=(1, 1, 1), dtype=float32
[[[1.8656857]]]
test_cc_layer_normalization_3d_axis1_epsilon
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 1
epsilon = 0.10000000149011612
Inputs:
X: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
W: shape=(3, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]]
B: shape=(3, 5), dtype=float32
[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]]
Outputs:
Y: shape=(2, 3, 5), dtype=float32
[[[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.43717134],
[-0.3566699 , -0.2709461 , -0.18000004, -0.08383158, 0.01755922],
[ 0.12417227, 0.23600774, 0.35306546, 0.47534555, 0.6028478 ]],
[[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.4371714 ],
[-0.3566699 , -0.27094603, -0.18 , -0.08383166, 0.01755923],
[ 0.12417224, 0.2360078 , 0.35306546, 0.47534543, 0.60284793]]]
Mean: shape=(2, 1, 1), dtype=float32
[[[-0.14999999]],
[[ 0.6 ]]]
InvStdDev: shape=(2, 1, 1), dtype=float32
[[[2.6111646]],
[[2.6111646]]]
test_cc_layer_normalization_3d_axis2_epsilon
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 2
epsilon = 0.10000000149011612
Inputs:
X: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
W: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
B: shape=(5,), dtype=float32
[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001]
Outputs:
Y: shape=(2, 3, 5), dtype=float32
[[[-0.40430334, -0.3202377 , -0.23 , -0.1335901 , -0.03100814],
[-0.40430337, -0.32023773, -0.23000003, -0.13359013, -0.03100812],
[-0.40430337, -0.32023773, -0.22999997, -0.13359007, -0.03100817]],
[[-0.40430337, -0.32023776, -0.23 , -0.1335901 , -0.03100818],
[-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808],
[-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808]]]
Mean: shape=(2, 3, 1), dtype=float32
[[[-0.4 ],
[-0.14999999],
[ 0.10000001]],
[[ 0.35000002],
[ 0.6 ],
[ 0.85 ]]]
InvStdDev: shape=(2, 3, 1), dtype=float32
[[[3.086067],
[3.086067],
[3.086067]],
[[3.086067],
[3.086067],
[3.086067]]]
test_cc_layer_normalization_3d_axis_negative_1_epsilon
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -1
epsilon = 0.10000000149011612
Inputs:
X: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
W: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
B: shape=(5,), dtype=float32
[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001]
Outputs:
Y: shape=(2, 3, 5), dtype=float32
[[[-0.40430334, -0.3202377 , -0.23 , -0.1335901 , -0.03100814],
[-0.40430337, -0.32023773, -0.23000003, -0.13359013, -0.03100812],
[-0.40430337, -0.32023773, -0.22999997, -0.13359007, -0.03100817]],
[[-0.40430337, -0.32023776, -0.23 , -0.1335901 , -0.03100818],
[-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808],
[-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808]]]
Mean: shape=(2, 3, 1), dtype=float32
[[[-0.4 ],
[-0.14999999],
[ 0.10000001]],
[[ 0.35000002],
[ 0.6 ],
[ 0.85 ]]]
InvStdDev: shape=(2, 3, 1), dtype=float32
[[[3.086067],
[3.086067],
[3.086067]],
[[3.086067],
[3.086067],
[3.086067]]]
test_cc_layer_normalization_3d_axis_negative_2_epsilon
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -2
epsilon = 0.10000000149011612
Inputs:
X: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
W: shape=(3, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]]
B: shape=(3, 5), dtype=float32
[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]]
Outputs:
Y: shape=(2, 3, 5), dtype=float32
[[[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.43717134],
[-0.3566699 , -0.2709461 , -0.18000004, -0.08383158, 0.01755922],
[ 0.12417227, 0.23600774, 0.35306546, 0.47534555, 0.6028478 ]],
[[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.4371714 ],
[-0.3566699 , -0.27094603, -0.18 , -0.08383166, 0.01755923],
[ 0.12417224, 0.2360078 , 0.35306546, 0.47534543, 0.60284793]]]
Mean: shape=(2, 1, 1), dtype=float32
[[[-0.14999999]],
[[ 0.6 ]]]
InvStdDev: shape=(2, 1, 1), dtype=float32
[[[2.6111646]],
[[2.6111646]]]
test_cc_layer_normalization_3d_axis_negative_3_epsilon
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -3
epsilon = 0.10000000149011612
Inputs:
X: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
W: shape=(2, 3, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]],
[[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ],
[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ]]]
B: shape=(2, 3, 5), dtype=float32
[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]],
[[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ],
[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001],
[ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999]]]
Outputs:
Y: shape=(2, 3, 5), dtype=float32
[[[-0.9263111 , -0.8948557 , -0.85966897, -0.82075083, -0.7781013 ],
[-0.73172045, -0.6816082 , -0.6277646 , -0.5701896 , -0.50888324],
[-0.4438455 , -0.3750764 , -0.3025759 , -0.22634405, -0.1463809 ]],
[[-0.06268631, 0.02473967, 0.11589703, 0.21078573, 0.3094057 ],
[ 0.41175717, 0.51784 , 0.6276542 , 0.74119973, 0.85847676],
[ 0.979485 , 1.1042248 , 1.2326956 , 1.364898 , 1.5008317 ]]]
Mean: shape=(1, 1, 1), dtype=float32
[[[0.22500001]]]
InvStdDev: shape=(1, 1, 1), dtype=float32
[[[1.8656857]]]
test_cc_layer_normalization_4d_axis0
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 0
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(2, 3, 4, 5), dtype=float32
[[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]],
[[[1.6999999 , 1.72 , 1.74 , 1.76 , 1.78 ],
[1.8 , 1.8199999 , 1.8399999 , 1.86 , 1.88 ],
[1.9 , 1.92 , 1.9399999 , 1.9599999 , 1.98 ],
[2. , 2.02 , 2.04 , 2.06 , 2.08 ]],
[[2.1 , 2.12 , 2.1399999 , 2.1599998 , 2.1799998 ],
[2.1999998 , 2.2199998 , 2.24 , 2.26 , 2.28 ],
[2.3 , 2.32 , 2.34 , 2.3600001 , 2.38 ],
[2.4 , 2.42 , 2.44 , 2.46 , 2.48 ]],
[[2.5 , 2.52 , 2.54 , 2.56 , 2.58 ],
[2.6 , 2.62 , 2.6399999 , 2.6599998 , 2.68 ],
[2.7 , 2.72 , 2.74 , 2.76 , 2.78 ],
[2.8 , 2.82 , 2.84 , 2.86 , 2.8799999 ]]]]
B: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ],
[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]],
[[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001],
[ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999],
[ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ],
[ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]],
[[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ],
[ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998],
[ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996],
[ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]],
[[[ 0.34999996, 0.36 , 0.37 , 0.38 , 0.39 ],
[ 0.39999998, 0.40999997, 0.41999996, 0.43 , 0.44 ],
[ 0.45 , 0.45999998, 0.46999997, 0.47999996, 0.49 ],
[ 0.5 , 0.51 , 0.52 , 0.53 , 0.53999996]],
[[ 0.54999995, 0.56 , 0.57 , 0.58 , 0.59 ],
[ 0.59999996, 0.60999995, 0.62 , 0.63 , 0.64 ],
[ 0.65 , 0.65999997, 0.66999996, 0.68 , 0.69 ],
[ 0.7 , 0.71 , 0.71999997, 0.72999996, 0.73999995]],
[[ 0.75 , 0.76 , 0.77 , 0.78 , 0.78999996],
[ 0.79999995, 0.80999994, 0.81999993, 0.8299999 , 0.84000003],
[ 0.85 , 0.86 , 0.87 , 0.88 , 0.89 ],
[ 0.9 , 0.90999997, 0.91999996, 0.92999995, 0.93999994]]]]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.1088369 , -1.1181788 , -1.126366 , -1.1333983 , -1.1392759 ],
[-1.143999 , -1.147567 , -1.1499805 , -1.1512392 , -1.1513432 ],
[-1.1502924 , -1.1480869 , -1.1447266 , -1.1402116 , -1.1345419 ],
[-1.1277175 , -1.1197383 , -1.1106044 , -1.1003157 , -1.0888722 ]],
[[-1.0762742 , -1.0625211 , -1.0476135 , -1.0315511 , -1.014334 ],
[-0.9959622 , -0.9764355 , -0.9557542 , -0.9339181 , -0.9109273 ],
[-0.8867818 , -0.8614815 , -0.8350265 , -0.8074168 , -0.7786523 ],
[-0.74873304, -0.717659 , -0.6854303 , -0.65204686, -0.6175086 ]],
[[-0.5818157 , -0.544968 , -0.5069655 , -0.4678084 , -0.42749655],
[-0.38602996, -0.34340864, -0.2996324 , -0.2547016 , -0.20861608],
[-0.16137576, -0.11298075, -0.06343079, -0.01272631, 0.03913289],
[ 0.0921469 , 0.14631562, 0.20163928, 0.25811747, 0.31575036]]],
[[[ 0.374538 , 0.43448046, 0.49557784, 0.5578297 , 0.6212363 ],
[ 0.68579763, 0.7515137 , 0.81838477, 0.8864104 , 0.9555907 ],
[ 1.0259258 , 1.0974154 , 1.1700602 , 1.2438595 , 1.3188136 ],
[ 1.3949223 , 1.4721859 , 1.5506042 , 1.6301771 , 1.7109048 ]],
[[ 1.7927872 , 1.8758247 , 1.9600163 , 2.0453632 , 2.1318648 ],
[ 2.2195206 , 2.3083317 , 2.3982973 , 2.489418 , 2.5816932 ],
[ 2.6751227 , 2.7697077 , 2.8654466 , 2.962341 , 3.0603902 ],
[ 3.1595933 , 3.2599518 , 3.3614645 , 3.4641323 , 3.5679553 ]],
[[ 3.672932 , 3.7790644 , 3.8863504 , 3.9947925 , 4.1043887 ],
[ 4.2151394 , 4.3270454 , 4.4401054 , 4.5543203 , 4.669691 ],
[ 4.7862153 , 4.903895 , 5.0227284 , 5.142718 , 5.263861 ],
[ 5.3861594 , 5.5096126 , 5.63422 , 5.7599826 , 5.886901 ]]]]
Mean: shape=(1, 1, 1, 1), dtype=float32
[[[[2.4750001]]]]
InvStdDev: shape=(1, 1, 1, 1), dtype=float32
[[[[0.57736933]]]]
test_cc_layer_normalization_4d_axis1
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 1
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(3, 4, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]]
B: shape=(3, 4, 5), dtype=float32
[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ],
[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]],
[[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001],
[ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999],
[ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ],
[ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]],
[[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ],
[ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998],
[ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996],
[ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.1017042 , -1.0957462 , -1.0874785 , -1.0769011 , -1.064014 ],
[-1.0488172 , -1.0313106 , -1.0114943 , -0.9893684 , -0.9649327 ],
[-0.93818736, -0.90913224, -0.8777675 , -0.84409297, -0.8081088 ],
[-0.76981485, -0.7292112 , -0.68629795, -0.6410749 , -0.5935422 ]],
[[-0.54369974, -0.49154752, -0.43708575, -0.38031426, -0.32123291],
[-0.259842 , -0.19614126, -0.13013093, -0.06181089, 0.00881899],
[ 0.08175841, 0.15700774, 0.23456657, 0.31443512, 0.3966136 ],
[ 0.48110154, 0.56789935, 0.65700674, 0.7484238 , 0.84215075]],
[[ 0.93818724, 1.0365335 , 1.1371896 , 1.2401552 , 1.3454306 ],
[ 1.4530158 , 1.5629103 , 1.6751153 , 1.7896295 , 1.9064533 ],
[ 2.0255868 , 2.1470299 , 2.2707834 , 2.396846 , 2.5252185 ],
[ 2.6559005 , 2.7888923 , 2.924194 , 3.0618052 , 3.201726 ]]],
[[[-1.1017044 , -1.0957463 , -1.0874785 , -1.0769011 , -1.064014 ],
[-1.0488173 , -1.0313108 , -1.0114943 , -0.9893684 , -0.9649328 ],
[-0.9381875 , -0.9091324 , -0.8777675 , -0.844093 , -0.80810887],
[-0.769815 , -0.72921145, -0.68629795, -0.641075 , -0.5935423 ]],
[[-0.54369986, -0.49154752, -0.43708602, -0.38031426, -0.3212328 ],
[-0.25984213, -0.19614126, -0.13013121, -0.06181089, 0.00881913],
[ 0.08175826, 0.15700774, 0.23456627, 0.31443512, 0.39661372],
[ 0.4811014 , 0.56789935, 0.6570064 , 0.7484238 , 0.8421509 ]],
[[ 0.93818706, 1.0365337 , 1.1371891 , 1.2401551 , 1.3454309 ],
[ 1.4530156 , 1.5629106 , 1.6751148 , 1.7896293 , 1.9064535 ],
[ 2.0255866 , 2.14703 , 2.2707827 , 2.3968456 , 2.5252185 ],
[ 2.6559005 , 2.7888925 , 2.9241936 , 3.061805 , 3.2017264 ]]]]
Mean: shape=(2, 1, 1, 1), dtype=float32
[[[[0.975 ]]],
[[[3.9750001]]]]
InvStdDev: shape=(2, 1, 1, 1), dtype=float32
[[[[1.1548532]]],
[[[1.1548532]]]]
test_cc_layer_normalization_4d_axis2
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 2
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(4, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]]
B: shape=(4, 5), dtype=float32
[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ],
[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.073705 , -1.006479 , -0.93231684, -0.8512181 , -0.7631829 ],
[-0.6682113 , -0.5663031 , -0.45745853, -0.34167743, -0.2189599 ],
[-0.08930597, 0.0472845 , 0.19081143, 0.34127483, 0.49867445],
[ 0.66301084, 0.8342837 , 1.0124929 , 1.1976386 , 1.3897207 ]],
[[-1.073705 , -1.0064789 , -0.93231684, -0.8512182 , -0.76318276],
[-0.6682113 , -0.56630296, -0.4574585 , -0.34167755, -0.21895987],
[-0.08930601, 0.0472846 , 0.1908114 , 0.3412746 , 0.49867463],
[ 0.6630107 , 0.8342837 , 1.0124928 , 1.1976383 , 1.3897207 ]],
[[-1.073705 , -1.0064791 , -0.93231666, -0.85121787, -0.76318276],
[-0.6682113 , -0.56630325, -0.45745823, -0.34167725, -0.21895987],
[-0.08930601, 0.04728431, 0.1908117 , 0.34127492, 0.49867463],
[ 0.6630107 , 0.83428335, 1.0124931 , 1.1976388 , 1.3897207 ]]],
[[[-1.0737052 , -1.0064794 , -0.93231684, -0.8512182 , -0.763183 ],
[-0.6682115 , -0.5663035 , -0.4574585 , -0.34167755, -0.21896015],
[-0.08930631, 0.04728401, 0.1908114 , 0.3412746 , 0.49867427],
[ 0.66301036, 0.8342831 , 1.0124928 , 1.1976383 , 1.3897202 ]],
[[-1.0737052 , -1.0064789 , -0.9323173 , -0.8512182 , -0.7631825 ],
[-0.6682115 , -0.56630296, -0.45745903, -0.34167755, -0.21895958],
[-0.08930631, 0.0472846 , 0.19081077, 0.3412746 , 0.49867493],
[ 0.66301036, 0.8342837 , 1.0124922 , 1.1976383 , 1.3897209 ]],
[[-1.0737047 , -1.0064785 , -0.93231684, -0.85121775, -0.76318204],
[-0.66821104, -0.5663025 , -0.4574585 , -0.341677 , -0.21895903],
[-0.08930573, 0.04728521, 0.1908114 , 0.34127522, 0.49867558],
[ 0.6630111 , 0.8342844 , 1.0124928 , 1.197639 , 1.3897219 ]]]]
Mean: shape=(2, 3, 1, 1), dtype=float32
[[[[-0.02499999]],
[[ 0.975 ]],
[[ 1.975 ]]],
[[[ 2.9750001 ]],
[[ 3.9750001 ]],
[[ 4.975 ]]]]
InvStdDev: shape=(2, 3, 1, 1), dtype=float32
[[[[3.4682312]],
[[3.4682312]],
[[3.4682312]]],
[[[3.4682312]],
[[3.4682312]],
[[3.4682312]]]]
test_cc_layer_normalization_4d_axis3
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = 3
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
B: shape=(5,), dtype=float32
[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.95640075, -0.60732824, -0.23 , 0.17558452, 0.6094248 ],
[-0.95640075, -0.6073283 , -0.23000012, 0.17558435, 0.6094248 ],
[-0.9564008 , -0.60732836, -0.22999988, 0.17558464, 0.6094247 ],
[-0.95640093, -0.6073285 , -0.23 , 0.17558452, 0.6094246 ]],
[[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507]],
[[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ],
[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]]],
[[[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]],
[[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ],
[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]],
[[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]]]]
Mean: shape=(2, 3, 4, 1), dtype=float32
[[[[-0.4 ],
[-0.14999999],
[ 0.10000001],
[ 0.35000002]],
[[ 0.6 ],
[ 0.85 ],
[ 1.1 ],
[ 1.35 ]],
[[ 1.6 ],
[ 1.85 ],
[ 2.1000001 ],
[ 2.3500001 ]]],
[[[ 2.6000001 ],
[ 2.8500001 ],
[ 3.1000001 ],
[ 3.3500001 ]],
[[ 3.6000001 ],
[ 3.8500001 ],
[ 4.1 ],
[ 4.35 ]],
[[ 4.6 ],
[ 4.85 ],
[ 5.1 ],
[ 5.35 ]]]]
InvStdDev: shape=(2, 3, 4, 1), dtype=float32
[[[[14.1280155],
[14.128014 ],
[14.1280155],
[14.1280155]],
[[14.1280155],
[14.1280155],
[14.1280155],
[14.1280155]],
[[14.128008 ],
[14.128008 ],
[14.128008 ],
[14.128008 ]]],
[[[14.128008 ],
[14.128008 ],
[14.128008 ],
[14.128008 ]],
[[14.128 ],
[14.128 ],
[14.128 ],
[14.128 ]],
[[14.128 ],
[14.128 ],
[14.128 ],
[14.128 ]]]]
test_cc_layer_normalization_4d_axis_negative_1
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -1
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
B: shape=(5,), dtype=float32
[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.95640075, -0.60732824, -0.23 , 0.17558452, 0.6094248 ],
[-0.95640075, -0.6073283 , -0.23000012, 0.17558435, 0.6094248 ],
[-0.9564008 , -0.60732836, -0.22999988, 0.17558464, 0.6094247 ],
[-0.95640093, -0.6073285 , -0.23 , 0.17558452, 0.6094246 ]],
[[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507]],
[[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ],
[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]]],
[[[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]],
[[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ],
[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]],
[[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]]]]
Mean: shape=(2, 3, 4, 1), dtype=float32
[[[[-0.4 ],
[-0.14999999],
[ 0.10000001],
[ 0.35000002]],
[[ 0.6 ],
[ 0.85 ],
[ 1.1 ],
[ 1.35 ]],
[[ 1.6 ],
[ 1.85 ],
[ 2.1000001 ],
[ 2.3500001 ]]],
[[[ 2.6000001 ],
[ 2.8500001 ],
[ 3.1000001 ],
[ 3.3500001 ]],
[[ 3.6000001 ],
[ 3.8500001 ],
[ 4.1 ],
[ 4.35 ]],
[[ 4.6 ],
[ 4.85 ],
[ 5.1 ],
[ 5.35 ]]]]
InvStdDev: shape=(2, 3, 4, 1), dtype=float32
[[[[14.1280155],
[14.128014 ],
[14.1280155],
[14.1280155]],
[[14.1280155],
[14.1280155],
[14.1280155],
[14.1280155]],
[[14.128008 ],
[14.128008 ],
[14.128008 ],
[14.128008 ]]],
[[[14.128008 ],
[14.128008 ],
[14.128008 ],
[14.128008 ]],
[[14.128 ],
[14.128 ],
[14.128 ],
[14.128 ]],
[[14.128 ],
[14.128 ],
[14.128 ],
[14.128 ]]]]
test_cc_layer_normalization_4d_axis_negative_2
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -2
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(4, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]]
B: shape=(4, 5), dtype=float32
[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ],
[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.073705 , -1.006479 , -0.93231684, -0.8512181 , -0.7631829 ],
[-0.6682113 , -0.5663031 , -0.45745853, -0.34167743, -0.2189599 ],
[-0.08930597, 0.0472845 , 0.19081143, 0.34127483, 0.49867445],
[ 0.66301084, 0.8342837 , 1.0124929 , 1.1976386 , 1.3897207 ]],
[[-1.073705 , -1.0064789 , -0.93231684, -0.8512182 , -0.76318276],
[-0.6682113 , -0.56630296, -0.4574585 , -0.34167755, -0.21895987],
[-0.08930601, 0.0472846 , 0.1908114 , 0.3412746 , 0.49867463],
[ 0.6630107 , 0.8342837 , 1.0124928 , 1.1976383 , 1.3897207 ]],
[[-1.073705 , -1.0064791 , -0.93231666, -0.85121787, -0.76318276],
[-0.6682113 , -0.56630325, -0.45745823, -0.34167725, -0.21895987],
[-0.08930601, 0.04728431, 0.1908117 , 0.34127492, 0.49867463],
[ 0.6630107 , 0.83428335, 1.0124931 , 1.1976388 , 1.3897207 ]]],
[[[-1.0737052 , -1.0064794 , -0.93231684, -0.8512182 , -0.763183 ],
[-0.6682115 , -0.5663035 , -0.4574585 , -0.34167755, -0.21896015],
[-0.08930631, 0.04728401, 0.1908114 , 0.3412746 , 0.49867427],
[ 0.66301036, 0.8342831 , 1.0124928 , 1.1976383 , 1.3897202 ]],
[[-1.0737052 , -1.0064789 , -0.9323173 , -0.8512182 , -0.7631825 ],
[-0.6682115 , -0.56630296, -0.45745903, -0.34167755, -0.21895958],
[-0.08930631, 0.0472846 , 0.19081077, 0.3412746 , 0.49867493],
[ 0.66301036, 0.8342837 , 1.0124922 , 1.1976383 , 1.3897209 ]],
[[-1.0737047 , -1.0064785 , -0.93231684, -0.85121775, -0.76318204],
[-0.66821104, -0.5663025 , -0.4574585 , -0.341677 , -0.21895903],
[-0.08930573, 0.04728521, 0.1908114 , 0.34127522, 0.49867558],
[ 0.6630111 , 0.8342844 , 1.0124928 , 1.197639 , 1.3897219 ]]]]
Mean: shape=(2, 3, 1, 1), dtype=float32
[[[[-0.02499999]],
[[ 0.975 ]],
[[ 1.975 ]]],
[[[ 2.9750001 ]],
[[ 3.9750001 ]],
[[ 4.975 ]]]]
InvStdDev: shape=(2, 3, 1, 1), dtype=float32
[[[[3.4682312]],
[[3.4682312]],
[[3.4682312]]],
[[[3.4682312]],
[[3.4682312]],
[[3.4682312]]]]
test_cc_layer_normalization_4d_axis_negative_3
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -3
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(3, 4, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]]
B: shape=(3, 4, 5), dtype=float32
[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ],
[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]],
[[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001],
[ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999],
[ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ],
[ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]],
[[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ],
[ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998],
[ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996],
[ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.1017042 , -1.0957462 , -1.0874785 , -1.0769011 , -1.064014 ],
[-1.0488172 , -1.0313106 , -1.0114943 , -0.9893684 , -0.9649327 ],
[-0.93818736, -0.90913224, -0.8777675 , -0.84409297, -0.8081088 ],
[-0.76981485, -0.7292112 , -0.68629795, -0.6410749 , -0.5935422 ]],
[[-0.54369974, -0.49154752, -0.43708575, -0.38031426, -0.32123291],
[-0.259842 , -0.19614126, -0.13013093, -0.06181089, 0.00881899],
[ 0.08175841, 0.15700774, 0.23456657, 0.31443512, 0.3966136 ],
[ 0.48110154, 0.56789935, 0.65700674, 0.7484238 , 0.84215075]],
[[ 0.93818724, 1.0365335 , 1.1371896 , 1.2401552 , 1.3454306 ],
[ 1.4530158 , 1.5629103 , 1.6751153 , 1.7896295 , 1.9064533 ],
[ 2.0255868 , 2.1470299 , 2.2707834 , 2.396846 , 2.5252185 ],
[ 2.6559005 , 2.7888923 , 2.924194 , 3.0618052 , 3.201726 ]]],
[[[-1.1017044 , -1.0957463 , -1.0874785 , -1.0769011 , -1.064014 ],
[-1.0488173 , -1.0313108 , -1.0114943 , -0.9893684 , -0.9649328 ],
[-0.9381875 , -0.9091324 , -0.8777675 , -0.844093 , -0.80810887],
[-0.769815 , -0.72921145, -0.68629795, -0.641075 , -0.5935423 ]],
[[-0.54369986, -0.49154752, -0.43708602, -0.38031426, -0.3212328 ],
[-0.25984213, -0.19614126, -0.13013121, -0.06181089, 0.00881913],
[ 0.08175826, 0.15700774, 0.23456627, 0.31443512, 0.39661372],
[ 0.4811014 , 0.56789935, 0.6570064 , 0.7484238 , 0.8421509 ]],
[[ 0.93818706, 1.0365337 , 1.1371891 , 1.2401551 , 1.3454309 ],
[ 1.4530156 , 1.5629106 , 1.6751148 , 1.7896293 , 1.9064535 ],
[ 2.0255866 , 2.14703 , 2.2707827 , 2.3968456 , 2.5252185 ],
[ 2.6559005 , 2.7888925 , 2.9241936 , 3.061805 , 3.2017264 ]]]]
Mean: shape=(2, 1, 1, 1), dtype=float32
[[[[0.975 ]]],
[[[3.9750001]]]]
InvStdDev: shape=(2, 1, 1, 1), dtype=float32
[[[[1.1548532]]],
[[[1.1548532]]]]
test_cc_layer_normalization_4d_axis_negative_4
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Attributes:
axis = -4
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(2, 3, 4, 5), dtype=float32
[[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]],
[[[1.6999999 , 1.72 , 1.74 , 1.76 , 1.78 ],
[1.8 , 1.8199999 , 1.8399999 , 1.86 , 1.88 ],
[1.9 , 1.92 , 1.9399999 , 1.9599999 , 1.98 ],
[2. , 2.02 , 2.04 , 2.06 , 2.08 ]],
[[2.1 , 2.12 , 2.1399999 , 2.1599998 , 2.1799998 ],
[2.1999998 , 2.2199998 , 2.24 , 2.26 , 2.28 ],
[2.3 , 2.32 , 2.34 , 2.3600001 , 2.38 ],
[2.4 , 2.42 , 2.44 , 2.46 , 2.48 ]],
[[2.5 , 2.52 , 2.54 , 2.56 , 2.58 ],
[2.6 , 2.62 , 2.6399999 , 2.6599998 , 2.68 ],
[2.7 , 2.72 , 2.74 , 2.76 , 2.78 ],
[2.8 , 2.82 , 2.84 , 2.86 , 2.8799999 ]]]]
B: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001],
[-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ],
[-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ],
[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]],
[[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001],
[ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999],
[ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ],
[ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]],
[[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ],
[ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998],
[ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996],
[ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]],
[[[ 0.34999996, 0.36 , 0.37 , 0.38 , 0.39 ],
[ 0.39999998, 0.40999997, 0.41999996, 0.43 , 0.44 ],
[ 0.45 , 0.45999998, 0.46999997, 0.47999996, 0.49 ],
[ 0.5 , 0.51 , 0.52 , 0.53 , 0.53999996]],
[[ 0.54999995, 0.56 , 0.57 , 0.58 , 0.59 ],
[ 0.59999996, 0.60999995, 0.62 , 0.63 , 0.64 ],
[ 0.65 , 0.65999997, 0.66999996, 0.68 , 0.69 ],
[ 0.7 , 0.71 , 0.71999997, 0.72999996, 0.73999995]],
[[ 0.75 , 0.76 , 0.77 , 0.78 , 0.78999996],
[ 0.79999995, 0.80999994, 0.81999993, 0.8299999 , 0.84000003],
[ 0.85 , 0.86 , 0.87 , 0.88 , 0.89 ],
[ 0.9 , 0.90999997, 0.91999996, 0.92999995, 0.93999994]]]]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-1.1088369 , -1.1181788 , -1.126366 , -1.1333983 , -1.1392759 ],
[-1.143999 , -1.147567 , -1.1499805 , -1.1512392 , -1.1513432 ],
[-1.1502924 , -1.1480869 , -1.1447266 , -1.1402116 , -1.1345419 ],
[-1.1277175 , -1.1197383 , -1.1106044 , -1.1003157 , -1.0888722 ]],
[[-1.0762742 , -1.0625211 , -1.0476135 , -1.0315511 , -1.014334 ],
[-0.9959622 , -0.9764355 , -0.9557542 , -0.9339181 , -0.9109273 ],
[-0.8867818 , -0.8614815 , -0.8350265 , -0.8074168 , -0.7786523 ],
[-0.74873304, -0.717659 , -0.6854303 , -0.65204686, -0.6175086 ]],
[[-0.5818157 , -0.544968 , -0.5069655 , -0.4678084 , -0.42749655],
[-0.38602996, -0.34340864, -0.2996324 , -0.2547016 , -0.20861608],
[-0.16137576, -0.11298075, -0.06343079, -0.01272631, 0.03913289],
[ 0.0921469 , 0.14631562, 0.20163928, 0.25811747, 0.31575036]]],
[[[ 0.374538 , 0.43448046, 0.49557784, 0.5578297 , 0.6212363 ],
[ 0.68579763, 0.7515137 , 0.81838477, 0.8864104 , 0.9555907 ],
[ 1.0259258 , 1.0974154 , 1.1700602 , 1.2438595 , 1.3188136 ],
[ 1.3949223 , 1.4721859 , 1.5506042 , 1.6301771 , 1.7109048 ]],
[[ 1.7927872 , 1.8758247 , 1.9600163 , 2.0453632 , 2.1318648 ],
[ 2.2195206 , 2.3083317 , 2.3982973 , 2.489418 , 2.5816932 ],
[ 2.6751227 , 2.7697077 , 2.8654466 , 2.962341 , 3.0603902 ],
[ 3.1595933 , 3.2599518 , 3.3614645 , 3.4641323 , 3.5679553 ]],
[[ 3.672932 , 3.7790644 , 3.8863504 , 3.9947925 , 4.1043887 ],
[ 4.2151394 , 4.3270454 , 4.4401054 , 4.5543203 , 4.669691 ],
[ 4.7862153 , 4.903895 , 5.0227284 , 5.142718 , 5.263861 ],
[ 5.3861594 , 5.5096126 , 5.63422 , 5.7599826 , 5.886901 ]]]]
Mean: shape=(1, 1, 1, 1), dtype=float32
[[[[2.4750001]]]]
InvStdDev: shape=(1, 1, 1, 1), dtype=float32
[[[[0.57736933]]]]
test_cc_layer_normalization_default_axis
Node:
LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev)
Inputs:
X: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
W: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
B: shape=(5,), dtype=float32
[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001]
Outputs:
Y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.95640075, -0.60732824, -0.23 , 0.17558452, 0.6094248 ],
[-0.95640075, -0.6073283 , -0.23000012, 0.17558435, 0.6094248 ],
[-0.9564008 , -0.60732836, -0.22999988, 0.17558464, 0.6094247 ],
[-0.95640093, -0.6073285 , -0.23 , 0.17558452, 0.6094246 ]],
[[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507],
[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507]],
[[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ],
[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]]],
[[[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364],
[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]],
[[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ],
[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]],
[[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ],
[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]]]]
Mean: shape=(2, 3, 4, 1), dtype=float32
[[[[-0.4 ],
[-0.14999999],
[ 0.10000001],
[ 0.35000002]],
[[ 0.6 ],
[ 0.85 ],
[ 1.1 ],
[ 1.35 ]],
[[ 1.6 ],
[ 1.85 ],
[ 2.1000001 ],
[ 2.3500001 ]]],
[[[ 2.6000001 ],
[ 2.8500001 ],
[ 3.1000001 ],
[ 3.3500001 ]],
[[ 3.6000001 ],
[ 3.8500001 ],
[ 4.1 ],
[ 4.35 ]],
[[ 4.6 ],
[ 4.85 ],
[ 5.1 ],
[ 5.35 ]]]]
InvStdDev: shape=(2, 3, 4, 1), dtype=float32
[[[[14.1280155],
[14.128014 ],
[14.1280155],
[14.1280155]],
[[14.1280155],
[14.1280155],
[14.1280155],
[14.1280155]],
[[14.128008 ],
[14.128008 ],
[14.128008 ],
[14.128008 ]]],
[[[14.128008 ],
[14.128008 ],
[14.128008 ],
[14.128008 ]],
[[14.128 ],
[14.128 ],
[14.128 ],
[14.128 ]],
[[14.128 ],
[14.128 ],
[14.128 ],
[14.128 ]]]]