RMSNormalization#
Domain:
ai.onnxSince version: 23
This is RMS normalization defined in ONNX as function as described in the paper https://arxiv.org/pdf/1910.07467. The overall computation can be split into two stages. The root mean squared norm is taken over the last D dimensions, where D is the dimension of normalized_shape. For example, if normalized_shape is (3, 5) (a 2-dimensional shape), the rms norm is computed over the last 2 dimensions of the input. The computation required by standardization can be described by the following equations.
XSquared = Mul(X, X)
XSquaredMean = ReduceMean (XSquared)
MeanSquareEpsilon = Add(XSquaredMean, epsilon)
RMS = Sqrt(MeanSquareEpsilon)
Normalized = Div(X, RMS)
where normalized_axes is [axis, ..., rank of X - 1]. The variables RMS stand for root mean square,
Depending on stash_type attribute, the actual computation
must happen in different floating-point precision.
For example, if stash_type is 1, this operator casts
all input variables to 32-bit float, perform the computation, and
finally cast Normalized back to the original type of X.
The second stage then scales the outcome of the first stage using:
Y= Mul(Normalized, Scale)
Let d[i] indicate the i-th dimension of X.
If X’s shape is [d[0], ..., d[axis-1], d[axis], ..., d[rank-1]],
the shape of RMS is [d[0], ..., d[axis-1], 1, ..., 1].
Y and X have the same shape. This operator supports unidirectional broadcasting
(Scale should be unidirectional broadcastable to tensor X);
for more details please check the doc.
Inputs
X (T): The input tensor to be normalized. In general, the shape is (D1, D2, … , Dn) for n-dimensional data, where the root mean squared norm is taken over the last D dimensions, D is determined by the axis attribute.
scale (V): Scale tensor. Scale tensor shape should be broadcastable to the normalized shape.
Outputs
Y (V): Output data tensor. Same shape as X
Attributes
axis (int): The first normalization dimension. If rank(X) is r, axis’ allowed range is [-r, r). Negative value means counting dimensions from the back.
epsilon (float): The epsilon value to use to avoid division by zero.
stash_type (int): The floating-point precision used in stage one of the computation.
Type Constraints
T: Constrain input X type to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
V: Constrain output Y and scale type to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).
Examples#
test_cc_rms_normalization_2d_axis0
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 0
Inputs:
x: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
scale: shape=(3, 4), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 ],
[0.58 , 0.6 , 0.62 , 0.64 ],
[0.65999997, 0.68 , 0.7 , 0.72 ]]
Outputs:
y: shape=(3, 4), dtype=float32
[[-0.8815359 , -0.8251175 , -0.7616471 , -0.6911241 ],
[-0.613549 , -0.52892154, -0.4372418 , -0.33850977],
[-0.23272546, -0.11988883, 0. , 0.1269412 ]]
test_cc_rms_normalization_2d_axis1
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 1
Inputs:
x: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
scale: shape=(4,), dtype=float32
[0.5 , 0.52, 0.54, 0.56]
Outputs:
y: shape=(3, 4), dtype=float32
[[-0.583196 , -0.5458714 , -0.50388134, -0.45722565],
[-0.6469365 , -0.56067824, -0.4657942 , -0.36228442],
[-0.8154101 , -0.4240131 , 0. , 0.45662978]]
test_cc_rms_normalization_2d_axis_negative_1
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -1
Inputs:
x: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
scale: shape=(4,), dtype=float32
[0.5 , 0.52, 0.54, 0.56]
Outputs:
y: shape=(3, 4), dtype=float32
[[-0.583196 , -0.5458714 , -0.50388134, -0.45722565],
[-0.6469365 , -0.56067824, -0.4657942 , -0.36228442],
[-0.8154101 , -0.4240131 , 0. , 0.45662978]]
test_cc_rms_normalization_2d_axis_negative_2
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -2
Inputs:
x: shape=(3, 4), dtype=float32
[[-0.5 , -0.45 , -0.4 , -0.35 ],
[-0.3 , -0.25 , -0.19999999, -0.15 ],
[-0.09999999, -0.04999998, 0. , 0.05000001]]
scale: shape=(3, 4), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 ],
[0.58 , 0.6 , 0.62 , 0.64 ],
[0.65999997, 0.68 , 0.7 , 0.72 ]]
Outputs:
y: shape=(3, 4), dtype=float32
[[-0.8815359 , -0.8251175 , -0.7616471 , -0.6911241 ],
[-0.613549 , -0.52892154, -0.4372418 , -0.33850977],
[-0.23272546, -0.11988883, 0. , 0.1269412 ]]
test_cc_rms_normalization_3d_axis0_epsilon
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 0
epsilon = 0.10000000149011612
Inputs:
x: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
scale: shape=(2, 3, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]],
[[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ],
[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ]]]
Outputs:
y: shape=(2, 3, 5), dtype=float32
[[[-0.4300661 , -0.40254185, -0.3715771 , -0.33717182, -0.29932603],
[-0.25803968, -0.21331279, -0.1651454 , -0.11353745, -0.05848897],
[ 0. , 0.06192954, 0.1272996 , 0.19611017, 0.26836124]],
[[ 0.34405288, 0.42318508, 0.50575775, 0.591771 , 0.6812247 ],
[ 0.77411896, 0.87045383, 0.9702292 , 1.073445 , 1.1801014 ],
[ 1.2901983 , 1.4037359 , 1.5207138 , 1.6411321 , 1.7649912 ]]]
test_cc_rms_normalization_3d_axis1_epsilon
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 1
epsilon = 0.10000000149011612
Inputs:
x: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
scale: shape=(3, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]]
Outputs:
y: shape=(2, 3, 5), dtype=float32
[[[-0.6078307 , -0.5689295 , -0.52516574, -0.47653928, -0.42305017],
[-0.36469844, -0.30148402, -0.23340699, -0.16046728, -0.08266494],
[ 0. , 0.08752765, 0.17991793, 0.27717087, 0.37928632]],
[[ 0.17560984, 0.21916108, 0.26552212, 0.31469285, 0.36667332],
[ 0.42146364, 0.4790637 , 0.5394735 , 0.60269296, 0.66872233],
[ 0.73756135, 0.80921024, 0.8836688 , 0.960937 , 1.0410151 ]]]
test_cc_rms_normalization_3d_axis2_epsilon
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 2
epsilon = 0.10000000149011612
Inputs:
x: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
scale: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
Outputs:
y: shape=(2, 3, 5), dtype=float32
[[[-0.4856429 , -0.4545617 , -0.4195955 , -0.38074404, -0.33800748],
[-0.35007 , -0.2912582 , -0.22684538, -0.15683134, -0.08121621],
[ 0. , 0.07666983, 0.15923735, 0.24770254, 0.34206527]],
[[ 0.2620712 , 0.32706484, 0.39625168, 0.4696316 , 0.5472046 ],
[ 0.3666178 , 0.4194108 , 0.4751367 , 0.5337955 , 0.59538734],
[ 0.4122373 , 0.4573086 , 0.5045785 , 0.55404687, 0.605714 ]]]
test_cc_rms_normalization_3d_axis_negative_1_epsilon
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -1
epsilon = 0.10000000149011612
Inputs:
x: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
scale: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
Outputs:
y: shape=(2, 3, 5), dtype=float32
[[[-0.4856429 , -0.4545617 , -0.4195955 , -0.38074404, -0.33800748],
[-0.35007 , -0.2912582 , -0.22684538, -0.15683134, -0.08121621],
[ 0. , 0.07666983, 0.15923735, 0.24770254, 0.34206527]],
[[ 0.2620712 , 0.32706484, 0.39625168, 0.4696316 , 0.5472046 ],
[ 0.3666178 , 0.4194108 , 0.4751367 , 0.5337955 , 0.59538734],
[ 0.4122373 , 0.4573086 , 0.5045785 , 0.55404687, 0.605714 ]]]
test_cc_rms_normalization_3d_axis_negative_2_epsilon
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -2
epsilon = 0.10000000149011612
Inputs:
x: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
scale: shape=(3, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]]
Outputs:
y: shape=(2, 3, 5), dtype=float32
[[[-0.6078307 , -0.5689295 , -0.52516574, -0.47653928, -0.42305017],
[-0.36469844, -0.30148402, -0.23340699, -0.16046728, -0.08266494],
[ 0. , 0.08752765, 0.17991793, 0.27717087, 0.37928632]],
[[ 0.17560984, 0.21916108, 0.26552212, 0.31469285, 0.36667332],
[ 0.42146364, 0.4790637 , 0.5394735 , 0.60269296, 0.66872233],
[ 0.73756135, 0.80921024, 0.8836688 , 0.960937 , 1.0410151 ]]]
test_cc_rms_normalization_3d_axis_negative_3_epsilon
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -3
epsilon = 0.10000000149011612
Inputs:
x: shape=(2, 3, 5), dtype=float32
[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]],
[[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ],
[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]]
scale: shape=(2, 3, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]],
[[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ],
[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ]]]
Outputs:
y: shape=(2, 3, 5), dtype=float32
[[[-0.4300661 , -0.40254185, -0.3715771 , -0.33717182, -0.29932603],
[-0.25803968, -0.21331279, -0.1651454 , -0.11353745, -0.05848897],
[ 0. , 0.06192954, 0.1272996 , 0.19611017, 0.26836124]],
[[ 0.34405288, 0.42318508, 0.50575775, 0.591771 , 0.6812247 ],
[ 0.77411896, 0.87045383, 0.9702292 , 1.073445 , 1.1801014 ],
[ 1.2901983 , 1.4037359 , 1.5207138 , 1.6411321 , 1.7649912 ]]]
test_cc_rms_normalization_4d_axis0
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 0
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(2, 3, 4, 5), dtype=float32
[[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]],
[[[1.6999999 , 1.72 , 1.74 , 1.76 , 1.78 ],
[1.8 , 1.8199999 , 1.8399999 , 1.86 , 1.88 ],
[1.9 , 1.92 , 1.9399999 , 1.9599999 , 1.98 ],
[2. , 2.02 , 2.04 , 2.06 , 2.08 ]],
[[2.1 , 2.12 , 2.1399999 , 2.1599998 , 2.1799998 ],
[2.1999998 , 2.2199998 , 2.24 , 2.26 , 2.28 ],
[2.3 , 2.32 , 2.34 , 2.3600001 , 2.38 ],
[2.4 , 2.42 , 2.44 , 2.46 , 2.48 ]],
[[2.5 , 2.52 , 2.54 , 2.56 , 2.58 ],
[2.6 , 2.62 , 2.6399999 , 2.6599998 , 2.68 ],
[2.7 , 2.72 , 2.74 , 2.76 , 2.78 ],
[2.8 , 2.82 , 2.84 , 2.86 , 2.8799999 ]]]]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.08275866, -0.0774621 , -0.07150348, -0.06488279, -0.05760003],
[-0.0496552 , -0.04104829, -0.03177932, -0.02184828, -0.01125517],
[ 0. , 0.01191725, 0.02449657, 0.03773796, 0.0516414 ],
[ 0.06620692, 0.08143452, 0.09732419, 0.11387593, 0.13108972]],
[[ 0.14896558, 0.16750354, 0.18670353, 0.20656559, 0.22708978],
[ 0.24827597, 0.2701243 , 0.2926346 , 0.315807 , 0.33964154],
[ 0.36413807, 0.38929677, 0.4151174 , 0.4416002 , 0.46874508],
[ 0.49655196, 0.52502096, 0.554152 , 0.58394504, 0.61440027]],
[[ 0.64551747, 0.6772968 , 0.70973825, 0.7428417 , 0.7766072 ],
[ 0.8110348 , 0.8461245 , 0.88187635, 0.9182902 , 0.95536596],
[ 0.99310386, 1.0315039 , 1.070566 , 1.1102902 , 1.1506764 ],
[ 1.1917247 , 1.233435 , 1.2758076 , 1.3188419 , 1.3625386 ]]],
[[[ 1.4068972 , 1.4519179 , 1.4976007 , 1.5439456 , 1.5909524 ],
[ 1.6386214 , 1.6869524 , 1.7359456 , 1.7856009 , 1.8359181 ],
[ 1.8868973 , 1.9385387 , 1.9908423 , 2.0438077 , 2.0974355 ],
[ 2.151725 , 2.2066767 , 2.2622907 , 2.3185666 , 2.3755045 ]],
[[ 2.4331043 , 2.4913666 , 2.5502906 , 2.6098769 , 2.6701252 ],
[ 2.7310355 , 2.792608 , 2.8548427 , 2.9177392 , 2.9812982 ],
[ 3.0455186 , 3.1104016 , 3.1759462 , 3.2421534 , 3.3090227 ],
[ 3.3765533 , 3.4447465 , 3.5136018 , 3.583119 , 3.6532986 ]],
[[ 3.7241397 , 3.795643 , 3.8678086 , 3.9406362 , 4.0141263 ],
[ 4.088278 , 4.163091 , 4.2385674 , 4.3147054 , 4.3915057 ],
[ 4.4689674 , 4.547092 , 4.625878 , 4.705326 , 4.7854366 ],
[ 4.866209 , 4.9476438 , 5.02974 , 5.1124988 , 5.1959195 ]]]]
test_cc_rms_normalization_4d_axis1
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 1
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(3, 4, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.19171718, -0.17944726, -0.16564365, -0.15030625, -0.13343516],
[-0.11503031, -0.09509172, -0.0736194 , -0.05061333, -0.02607353],
[ 0. , 0.02760728, 0.0567483 , 0.08742305, 0.11963151],
[ 0.15337373, 0.18864971, 0.22545943, 0.2638029 , 0.30367997]],
[[ 0.3450909 , 0.3880356 , 0.43251398, 0.47852606, 0.52607197],
[ 0.57515156, 0.62576497, 0.67791194, 0.73159266, 0.7868073 ],
[ 0.8435555 , 0.90183765, 0.9616534 , 1.0230029 , 1.0858862 ],
[ 1.1503031 , 1.2162539 , 1.2837384 , 1.3527564 , 1.4233084 ]],
[[ 1.495394 , 1.5690132 , 1.6441666 , 1.7208533 , 1.799074 ],
[ 1.8788284 , 1.9601163 , 2.0429385 , 2.127294 , 2.2131832 ],
[ 2.3006063 , 2.3895628 , 2.4800534 , 2.5720778 , 2.6656356 ],
[ 2.7607274 , 2.8573527 , 2.955512 , 3.055205 , 3.1564317 ]]],
[[[ 0.30725956, 0.3259409 , 0.34511396, 0.36477858, 0.38493478],
[ 0.40558264, 0.42672208, 0.44835314, 0.47047585, 0.49309015],
[ 0.5161961 , 0.5397936 , 0.56388277, 0.58846354, 0.6135359 ],
[ 0.63909984, 0.6651555 , 0.6917028 , 0.7187416 , 0.746272 ]],
[[ 0.7742941 , 0.80280775, 0.83181304, 0.86131 , 0.89129865],
[ 0.9217787 , 0.95275044, 0.9842137 , 1.0161688 , 1.0486155 ],
[ 1.0815536 , 1.1149836 , 1.1489049 , 1.183318 , 1.2182229 ],
[ 1.2536191 , 1.289507 , 1.3258864 , 1.3627577 , 1.4001204 ]],
[[ 1.4379747 , 1.4763207 , 1.5151582 , 1.5544875 , 1.5943084 ],
[ 1.6346208 , 1.675425 , 1.7167206 , 1.7585081 , 1.8007869 ],
[ 1.8435574 , 1.8868196 , 1.9305731 , 1.9748186 , 2.0195556 ],
[ 2.0647843 , 2.1105046 , 2.156716 , 2.2034197 , 2.250615 ]]]]
test_cc_rms_normalization_4d_axis2
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 2
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(4, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.8638168 , -0.8085325 , -0.7463378 , -0.6772324 , -0.6012165 ],
[-0.5182901 , -0.42845312, -0.33170566, -0.22804761, -0.11747905],
[ 0. , 0.12438966, 0.25568986, 0.39390057, 0.5390216 ],
[ 0.6910534 , 0.84999573, 1.0158486 , 1.1886121 , 1.3682858 ]],
[[ 0.24588391, 0.28129122, 0.31866556, 0.35800695, 0.3993155 ],
[ 0.44259104, 0.4878337 , 0.5350434 , 0.5842202 , 0.63536406],
[ 0.68847495, 0.743553 , 0.8005981 , 0.85961014, 0.9205893 ],
[ 0.9835356 , 1.048449 , 1.1153294 , 1.1841769 , 1.2549915 ]],
[[ 0.3757636 , 0.40382057, 0.43287972, 0.46294078, 0.49400386],
[ 0.52606905, 0.5591362 , 0.59320545, 0.6282767 , 0.66435003],
[ 0.7014253 , 0.7395027 , 0.7785822 , 0.8186636 , 0.85974705],
[ 0.9018325 , 0.9449201 , 0.9890098 , 1.0341014 , 1.0801951 ]]],
[[[ 0.41820854, 0.4436356 , 0.4697319 , 0.4964972 , 0.5239316 ],
[ 0.5520353 , 0.58080804, 0.61024994, 0.64036095, 0.6711411 ],
[ 0.70259035, 0.7347087 , 0.76749635, 0.80095303, 0.8350788 ],
[ 0.86987376, 0.9053378 , 0.94147116, 0.97827345, 1.0157449 ]],
[[ 0.43909788, 0.46318555, 0.48777503, 0.5128663 , 0.5384595 ],
[ 0.56455445, 0.59115124, 0.6182498 , 0.64585024, 0.6739526 ],
[ 0.7025566 , 0.73166263, 0.7612703 , 0.79137987, 0.8219913 ],
[ 0.8531045 , 0.88471955, 0.91683644, 0.94945514, 0.9825757 ]],
[[ 0.45150366, 0.4747812 , 0.49846005, 0.5225402 , 0.5470218 ],
[ 0.57190466, 0.5971889 , 0.6228744 , 0.64896125, 0.6754495 ],
[ 0.702339 , 0.72963 , 0.75732213, 0.78541565, 0.8139106 ],
[ 0.84280676, 0.87210447, 0.9018034 , 0.93190354, 0.96240515]]]]
test_cc_rms_normalization_4d_axis3
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = 3
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.6154388 , -0.5760507 , -0.5317392 , -0.48250404, -0.4283454 ],
[-0.75364137, -0.6270296 , -0.48835963, -0.33763131, -0.17484474],
[ 0. , 0.21221842, 0.44076136, 0.6856287 , 0.9468203 ],
[ 0.3500563 , 0.43687028, 0.5292852 , 0.6273009 , 0.7309175 ]],
[[ 0.41379726, 0.4733841 , 0.5362813 , 0.60248876, 0.6720068 ],
[ 0.43965474, 0.48772368, 0.53813744, 0.59089595, 0.6459994 ],
[ 0.45360735, 0.49533924, 0.53888553, 0.5842463 , 0.63142145],
[ 0.46232793, 0.5000539 , 0.5392594 , 0.5799442 , 0.62210846]],
[[ 0.46829197, 0.50325775, 0.5394724 , 0.57693577, 0.61564785],
[ 0.47262716, 0.505576 , 0.53960526, 0.5747147 , 0.6109044 ],
[ 0.47592017, 0.50733083, 0.53969353, 0.5730079 , 0.6072741 ],
[ 0.47850642, 0.50870544, 0.5397553 , 0.5716557 , 0.6044068 ]]],
[[[ 0.48059118, 0.5098111 , 0.53980005, 0.57055783, 0.60208464],
[ 0.48230743, 0.5107197 , 0.5398336 , 0.569649 , 0.60016584],
[ 0.4837449 , 0.51147956, 0.53985935, 0.56888396, 0.59855366],
[ 0.48496643, 0.51212454, 0.53987956, 0.5682314 , 0.5971802 ]],
[[ 0.48601717, 0.5126787 , 0.53989565, 0.5676681 , 0.5959959 ],
[ 0.48693067, 0.51316 , 0.53990877, 0.5671769 , 0.5949644 ],
[ 0.48773223, 0.51358205, 0.5399196 , 0.56674486, 0.59405786],
[ 0.48844108, 0.51395494, 0.53992856, 0.56636184, 0.5932548 ]],
[[ 0.48907256, 0.514287 , 0.5399361 , 0.56601995, 0.5925386 ],
[ 0.4896386 , 0.51458436, 0.5399425 , 0.5657129 , 0.59189576],
[ 0.4901489 , 0.5148524 , 0.53994805, 0.56543577, 0.5913157 ],
[ 0.49061126, 0.51509506, 0.53995275, 0.5651842 , 0.59078944]]]]
test_cc_rms_normalization_4d_axis_negative_1
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -1
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.6154388 , -0.5760507 , -0.5317392 , -0.48250404, -0.4283454 ],
[-0.75364137, -0.6270296 , -0.48835963, -0.33763131, -0.17484474],
[ 0. , 0.21221842, 0.44076136, 0.6856287 , 0.9468203 ],
[ 0.3500563 , 0.43687028, 0.5292852 , 0.6273009 , 0.7309175 ]],
[[ 0.41379726, 0.4733841 , 0.5362813 , 0.60248876, 0.6720068 ],
[ 0.43965474, 0.48772368, 0.53813744, 0.59089595, 0.6459994 ],
[ 0.45360735, 0.49533924, 0.53888553, 0.5842463 , 0.63142145],
[ 0.46232793, 0.5000539 , 0.5392594 , 0.5799442 , 0.62210846]],
[[ 0.46829197, 0.50325775, 0.5394724 , 0.57693577, 0.61564785],
[ 0.47262716, 0.505576 , 0.53960526, 0.5747147 , 0.6109044 ],
[ 0.47592017, 0.50733083, 0.53969353, 0.5730079 , 0.6072741 ],
[ 0.47850642, 0.50870544, 0.5397553 , 0.5716557 , 0.6044068 ]]],
[[[ 0.48059118, 0.5098111 , 0.53980005, 0.57055783, 0.60208464],
[ 0.48230743, 0.5107197 , 0.5398336 , 0.569649 , 0.60016584],
[ 0.4837449 , 0.51147956, 0.53985935, 0.56888396, 0.59855366],
[ 0.48496643, 0.51212454, 0.53987956, 0.5682314 , 0.5971802 ]],
[[ 0.48601717, 0.5126787 , 0.53989565, 0.5676681 , 0.5959959 ],
[ 0.48693067, 0.51316 , 0.53990877, 0.5671769 , 0.5949644 ],
[ 0.48773223, 0.51358205, 0.5399196 , 0.56674486, 0.59405786],
[ 0.48844108, 0.51395494, 0.53992856, 0.56636184, 0.5932548 ]],
[[ 0.48907256, 0.514287 , 0.5399361 , 0.56601995, 0.5925386 ],
[ 0.4896386 , 0.51458436, 0.5399425 , 0.5657129 , 0.59189576],
[ 0.4901489 , 0.5148524 , 0.53994805, 0.56543577, 0.5913157 ],
[ 0.49061126, 0.51509506, 0.53995275, 0.5651842 , 0.59078944]]]]
test_cc_rms_normalization_4d_axis_negative_2
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -2
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(4, 5), dtype=float32
[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.8638168 , -0.8085325 , -0.7463378 , -0.6772324 , -0.6012165 ],
[-0.5182901 , -0.42845312, -0.33170566, -0.22804761, -0.11747905],
[ 0. , 0.12438966, 0.25568986, 0.39390057, 0.5390216 ],
[ 0.6910534 , 0.84999573, 1.0158486 , 1.1886121 , 1.3682858 ]],
[[ 0.24588391, 0.28129122, 0.31866556, 0.35800695, 0.3993155 ],
[ 0.44259104, 0.4878337 , 0.5350434 , 0.5842202 , 0.63536406],
[ 0.68847495, 0.743553 , 0.8005981 , 0.85961014, 0.9205893 ],
[ 0.9835356 , 1.048449 , 1.1153294 , 1.1841769 , 1.2549915 ]],
[[ 0.3757636 , 0.40382057, 0.43287972, 0.46294078, 0.49400386],
[ 0.52606905, 0.5591362 , 0.59320545, 0.6282767 , 0.66435003],
[ 0.7014253 , 0.7395027 , 0.7785822 , 0.8186636 , 0.85974705],
[ 0.9018325 , 0.9449201 , 0.9890098 , 1.0341014 , 1.0801951 ]]],
[[[ 0.41820854, 0.4436356 , 0.4697319 , 0.4964972 , 0.5239316 ],
[ 0.5520353 , 0.58080804, 0.61024994, 0.64036095, 0.6711411 ],
[ 0.70259035, 0.7347087 , 0.76749635, 0.80095303, 0.8350788 ],
[ 0.86987376, 0.9053378 , 0.94147116, 0.97827345, 1.0157449 ]],
[[ 0.43909788, 0.46318555, 0.48777503, 0.5128663 , 0.5384595 ],
[ 0.56455445, 0.59115124, 0.6182498 , 0.64585024, 0.6739526 ],
[ 0.7025566 , 0.73166263, 0.7612703 , 0.79137987, 0.8219913 ],
[ 0.8531045 , 0.88471955, 0.91683644, 0.94945514, 0.9825757 ]],
[[ 0.45150366, 0.4747812 , 0.49846005, 0.5225402 , 0.5470218 ],
[ 0.57190466, 0.5971889 , 0.6228744 , 0.64896125, 0.6754495 ],
[ 0.702339 , 0.72963 , 0.75732213, 0.78541565, 0.8139106 ],
[ 0.84280676, 0.87210447, 0.9018034 , 0.93190354, 0.96240515]]]]
test_cc_rms_normalization_4d_axis_negative_3
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -3
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(3, 4, 5), dtype=float32
[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.19171718, -0.17944726, -0.16564365, -0.15030625, -0.13343516],
[-0.11503031, -0.09509172, -0.0736194 , -0.05061333, -0.02607353],
[ 0. , 0.02760728, 0.0567483 , 0.08742305, 0.11963151],
[ 0.15337373, 0.18864971, 0.22545943, 0.2638029 , 0.30367997]],
[[ 0.3450909 , 0.3880356 , 0.43251398, 0.47852606, 0.52607197],
[ 0.57515156, 0.62576497, 0.67791194, 0.73159266, 0.7868073 ],
[ 0.8435555 , 0.90183765, 0.9616534 , 1.0230029 , 1.0858862 ],
[ 1.1503031 , 1.2162539 , 1.2837384 , 1.3527564 , 1.4233084 ]],
[[ 1.495394 , 1.5690132 , 1.6441666 , 1.7208533 , 1.799074 ],
[ 1.8788284 , 1.9601163 , 2.0429385 , 2.127294 , 2.2131832 ],
[ 2.3006063 , 2.3895628 , 2.4800534 , 2.5720778 , 2.6656356 ],
[ 2.7607274 , 2.8573527 , 2.955512 , 3.055205 , 3.1564317 ]]],
[[[ 0.30725956, 0.3259409 , 0.34511396, 0.36477858, 0.38493478],
[ 0.40558264, 0.42672208, 0.44835314, 0.47047585, 0.49309015],
[ 0.5161961 , 0.5397936 , 0.56388277, 0.58846354, 0.6135359 ],
[ 0.63909984, 0.6651555 , 0.6917028 , 0.7187416 , 0.746272 ]],
[[ 0.7742941 , 0.80280775, 0.83181304, 0.86131 , 0.89129865],
[ 0.9217787 , 0.95275044, 0.9842137 , 1.0161688 , 1.0486155 ],
[ 1.0815536 , 1.1149836 , 1.1489049 , 1.183318 , 1.2182229 ],
[ 1.2536191 , 1.289507 , 1.3258864 , 1.3627577 , 1.4001204 ]],
[[ 1.4379747 , 1.4763207 , 1.5151582 , 1.5544875 , 1.5943084 ],
[ 1.6346208 , 1.675425 , 1.7167206 , 1.7585081 , 1.8007869 ],
[ 1.8435574 , 1.8868196 , 1.9305731 , 1.9748186 , 2.0195556 ],
[ 2.0647843 , 2.1105046 , 2.156716 , 2.2034197 , 2.250615 ]]]]
test_cc_rms_normalization_4d_axis_negative_4
Node:
RMSNormalization(x, scale) -> (y)
Attributes:
axis = -4
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(2, 3, 4, 5), dtype=float32
[[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ],
[0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ],
[0.7 , 0.72 , 0.74 , 0.76 , 0.78 ],
[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]],
[[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ],
[1. , 1.02 , 1.04 , 1.06 , 1.0799999 ],
[1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ],
[1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]],
[[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ],
[1.4 , 1.42 , 1.44 , 1.46 , 1.48 ],
[1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ],
[1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]],
[[[1.6999999 , 1.72 , 1.74 , 1.76 , 1.78 ],
[1.8 , 1.8199999 , 1.8399999 , 1.86 , 1.88 ],
[1.9 , 1.92 , 1.9399999 , 1.9599999 , 1.98 ],
[2. , 2.02 , 2.04 , 2.06 , 2.08 ]],
[[2.1 , 2.12 , 2.1399999 , 2.1599998 , 2.1799998 ],
[2.1999998 , 2.2199998 , 2.24 , 2.26 , 2.28 ],
[2.3 , 2.32 , 2.34 , 2.3600001 , 2.38 ],
[2.4 , 2.42 , 2.44 , 2.46 , 2.48 ]],
[[2.5 , 2.52 , 2.54 , 2.56 , 2.58 ],
[2.6 , 2.62 , 2.6399999 , 2.6599998 , 2.68 ],
[2.7 , 2.72 , 2.74 , 2.76 , 2.78 ],
[2.8 , 2.82 , 2.84 , 2.86 , 2.8799999 ]]]]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.08275866, -0.0774621 , -0.07150348, -0.06488279, -0.05760003],
[-0.0496552 , -0.04104829, -0.03177932, -0.02184828, -0.01125517],
[ 0. , 0.01191725, 0.02449657, 0.03773796, 0.0516414 ],
[ 0.06620692, 0.08143452, 0.09732419, 0.11387593, 0.13108972]],
[[ 0.14896558, 0.16750354, 0.18670353, 0.20656559, 0.22708978],
[ 0.24827597, 0.2701243 , 0.2926346 , 0.315807 , 0.33964154],
[ 0.36413807, 0.38929677, 0.4151174 , 0.4416002 , 0.46874508],
[ 0.49655196, 0.52502096, 0.554152 , 0.58394504, 0.61440027]],
[[ 0.64551747, 0.6772968 , 0.70973825, 0.7428417 , 0.7766072 ],
[ 0.8110348 , 0.8461245 , 0.88187635, 0.9182902 , 0.95536596],
[ 0.99310386, 1.0315039 , 1.070566 , 1.1102902 , 1.1506764 ],
[ 1.1917247 , 1.233435 , 1.2758076 , 1.3188419 , 1.3625386 ]]],
[[[ 1.4068972 , 1.4519179 , 1.4976007 , 1.5439456 , 1.5909524 ],
[ 1.6386214 , 1.6869524 , 1.7359456 , 1.7856009 , 1.8359181 ],
[ 1.8868973 , 1.9385387 , 1.9908423 , 2.0438077 , 2.0974355 ],
[ 2.151725 , 2.2066767 , 2.2622907 , 2.3185666 , 2.3755045 ]],
[[ 2.4331043 , 2.4913666 , 2.5502906 , 2.6098769 , 2.6701252 ],
[ 2.7310355 , 2.792608 , 2.8548427 , 2.9177392 , 2.9812982 ],
[ 3.0455186 , 3.1104016 , 3.1759462 , 3.2421534 , 3.3090227 ],
[ 3.3765533 , 3.4447465 , 3.5136018 , 3.583119 , 3.6532986 ]],
[[ 3.7241397 , 3.795643 , 3.8678086 , 3.9406362 , 4.0141263 ],
[ 4.088278 , 4.163091 , 4.2385674 , 4.3147054 , 4.3915057 ],
[ 4.4689674 , 4.547092 , 4.625878 , 4.705326 , 4.7854366 ],
[ 4.866209 , 4.9476438 , 5.02974 , 5.1124988 , 5.1959195 ]]]]
test_cc_rms_normalization_default_axis
Node:
RMSNormalization(x, scale) -> (y)
Inputs:
x: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ],
[-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998],
[ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999],
[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]],
[[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005],
[ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005],
[ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ],
[ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]],
[[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ],
[ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ],
[ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ],
[ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]],
[[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ],
[ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ],
[ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ],
[ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]],
[[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ],
[ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ],
[ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ],
[ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]],
[[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ],
[ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ],
[ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ],
[ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]]
scale: shape=(5,), dtype=float32
[0.5 , 0.52, 0.54, 0.56, 0.58]
Outputs:
y: shape=(2, 3, 4, 5), dtype=float32
[[[[-0.6154388 , -0.5760507 , -0.5317392 , -0.48250404, -0.4283454 ],
[-0.75364137, -0.6270296 , -0.48835963, -0.33763131, -0.17484474],
[ 0. , 0.21221842, 0.44076136, 0.6856287 , 0.9468203 ],
[ 0.3500563 , 0.43687028, 0.5292852 , 0.6273009 , 0.7309175 ]],
[[ 0.41379726, 0.4733841 , 0.5362813 , 0.60248876, 0.6720068 ],
[ 0.43965474, 0.48772368, 0.53813744, 0.59089595, 0.6459994 ],
[ 0.45360735, 0.49533924, 0.53888553, 0.5842463 , 0.63142145],
[ 0.46232793, 0.5000539 , 0.5392594 , 0.5799442 , 0.62210846]],
[[ 0.46829197, 0.50325775, 0.5394724 , 0.57693577, 0.61564785],
[ 0.47262716, 0.505576 , 0.53960526, 0.5747147 , 0.6109044 ],
[ 0.47592017, 0.50733083, 0.53969353, 0.5730079 , 0.6072741 ],
[ 0.47850642, 0.50870544, 0.5397553 , 0.5716557 , 0.6044068 ]]],
[[[ 0.48059118, 0.5098111 , 0.53980005, 0.57055783, 0.60208464],
[ 0.48230743, 0.5107197 , 0.5398336 , 0.569649 , 0.60016584],
[ 0.4837449 , 0.51147956, 0.53985935, 0.56888396, 0.59855366],
[ 0.48496643, 0.51212454, 0.53987956, 0.5682314 , 0.5971802 ]],
[[ 0.48601717, 0.5126787 , 0.53989565, 0.5676681 , 0.5959959 ],
[ 0.48693067, 0.51316 , 0.53990877, 0.5671769 , 0.5949644 ],
[ 0.48773223, 0.51358205, 0.5399196 , 0.56674486, 0.59405786],
[ 0.48844108, 0.51395494, 0.53992856, 0.56636184, 0.5932548 ]],
[[ 0.48907256, 0.514287 , 0.5399361 , 0.56601995, 0.5925386 ],
[ 0.4896386 , 0.51458436, 0.5399425 , 0.5657129 , 0.59189576],
[ 0.4901489 , 0.5148524 , 0.53994805, 0.56543577, 0.5913157 ],
[ 0.49061126, 0.51509506, 0.53995275, 0.5651842 , 0.59078944]]]]
test_cc_shape_inference_tiny_llm
Node:
RMSNormalization(hidden, input_layernorm.weight) -> (normed1)