.. _op_ai_onnx_LayerNormalization: LayerNormalization ================== - **Domain**: ``ai.onnx`` - **Since version**: 17 This is layer normalization defined in ONNX as function. The overall computation can be split into two stages. The first stage is standardization, which makes the normalized elements have zero mean and unit variances. The computation required by standardization can be described by the following equations. .. code-block:: Mean = ReduceMean (X) D = Sub(X, Mean) DD = Mul(D, D) Var = ReduceMean (DD) VarEps = Add(Var, epsilon) StdDev = Sqrt(VarEps) InvStdDev = Reciprocal(StdDev) Normalized = Mul(D, InvStdDev) where ``normalized_axes`` is ``[axis, ..., rank of X - 1]``. The variables ``Var`` and ``StdDev`` stand for variance and standard deviation, respectively. The second output is ``Mean`` and the last one is ``InvStdDev``. Depending on ``stash_type`` attribute, the actual computation must happen in different floating-point precision. For example, if ``stash_type`` is 1, this operator casts all input variables to 32-bit float, perform the computation, and finally cast ``Normalized`` back to the original type of ``X``. The second stage then scales and shifts the outcome of the first stage using .. code-block:: NormalizedScaled = Mul(Normalized, Scale) Y = Add(NormalizedScaled, B) The second stage doesn't depends on ``stash_type``. All equations are in `this syntax `_. The same variable (i.e., input, output, and attribute) uses the same name in the equations above and this operator's definition. Let ``d[i]`` indicate the i-th dimension of ``X``. If ``X``'s shape is ``[d[0], ..., d[axis-1], d[axis], ..., d[rank-1]]``, the shape of ``Mean`` and ``InvStdDev`` is ``[d[0], ..., d[axis-1], 1, ..., 1]``. ``Y`` and ``X`` have the same shape. This operator supports unidirectional broadcasting (tensors ``Scale`` and ``B`` should be unidirectional broadcastable to tensor ``X``); for more details please check `the doc `_. **Inputs** - **X** (*T*): Tensor to be normalized. - **Scale** (*T*): Scale tensor. - **B** (*T*): Bias tensor. **Outputs** - **Y** (*T*): Normalized tensor. - **Mean** (*U*): Saved mean used during training to speed up gradient computation - **InvStdDev** (*U*): Saved inverse standard deviation used during training to speed up gradient computation. **Attributes** - **axis** (*int*): The first normalization dimension. If rank(X) is r, axis' allowed range is [-r, r). Negative value means counting dimensions from the back. - **epsilon** (*float*): The epsilon value to use to avoid division by zero. - **stash_type** (*int*): Type of Mean and InvStdDev. This also specifies stage one's computation precision. **Type Constraints** - **T**: Constrain input types and output Y type to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16). - **U**: Type of Mean and InvStdDev tensors. Allowed types: tensor(bfloat16), tensor(float). Examples -------- **test_cc_layer_normalization_2d_axis0** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 0 .. code-block:: text Inputs: X: shape=(3, 4), dtype=float32 [[-0.5 , -0.45 , -0.4 , -0.35 ], [-0.3 , -0.25 , -0.19999999, -0.15 ], [-0.09999999, -0.04999998, 0. , 0.05000001]] W: shape=(3, 4), dtype=float32 [[0.5 , 0.52 , 0.54 , 0.56 ], [0.58 , 0.6 , 0.62 , 0.64 ], [0.65999997, 0.68 , 0.7 , 0.72 ]] B: shape=(3, 4), dtype=float32 [[-0.25 , -0.24 , -0.23 , -0.22 ], [-0.21000001, -0.2 , -0.19 , -0.18 ], [-0.17 , -0.16 , -0.15 , -0.14 ]] Outputs: Y: shape=(3, 4), dtype=float32 [[-1.0464939 , -0.9177438 , -0.7774086 , -0.6254878 ], [-0.46198177, -0.28689027, -0.10021339, 0.09804872], [ 0.30789632, 0.5293293 , 0.76234746, 1.0069512 ]] Mean: shape=(1, 1), dtype=float32 [[-0.225]] InvStdDev: shape=(1, 1), dtype=float32 [[5.7926826]] **test_cc_layer_normalization_2d_axis1** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 1 .. code-block:: text Inputs: X: shape=(3, 4), dtype=float32 [[-0.5 , -0.45 , -0.4 , -0.35 ], [-0.3 , -0.25 , -0.19999999, -0.15 ], [-0.09999999, -0.04999998, 0. , 0.05000001]] W: shape=(4,), dtype=float32 [0.5 , 0.52, 0.54, 0.56] B: shape=(4,), dtype=float32 [-0.25, -0.24, -0.23, -0.22] Outputs: Y: shape=(3, 4), dtype=float32 [[-0.91974956, -0.47217965, 0.01110995, 0.5301198 ], [-0.9197498 , -0.47217992, 0.01110995, 0.53011954], [-0.9197497 , -0.47217977, 0.0111098 , 0.53011966]] Mean: shape=(3, 1), dtype=float32 [[-0.425 ], [-0.225 ], [-0.02499999]] InvStdDev: shape=(3, 1), dtype=float32 [[17.859991], [17.859991], [17.859991]] **test_cc_layer_normalization_2d_axis_negative_1** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -1 .. code-block:: text Inputs: X: shape=(3, 4), dtype=float32 [[-0.5 , -0.45 , -0.4 , -0.35 ], [-0.3 , -0.25 , -0.19999999, -0.15 ], [-0.09999999, -0.04999998, 0. , 0.05000001]] W: shape=(4,), dtype=float32 [0.5 , 0.52, 0.54, 0.56] B: shape=(4,), dtype=float32 [-0.25, -0.24, -0.23, -0.22] Outputs: Y: shape=(3, 4), dtype=float32 [[-0.91974956, -0.47217965, 0.01110995, 0.5301198 ], [-0.9197498 , -0.47217992, 0.01110995, 0.53011954], [-0.9197497 , -0.47217977, 0.0111098 , 0.53011966]] Mean: shape=(3, 1), dtype=float32 [[-0.425 ], [-0.225 ], [-0.02499999]] InvStdDev: shape=(3, 1), dtype=float32 [[17.859991], [17.859991], [17.859991]] **test_cc_layer_normalization_2d_axis_negative_2** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -2 .. code-block:: text Inputs: X: shape=(3, 4), dtype=float32 [[-0.5 , -0.45 , -0.4 , -0.35 ], [-0.3 , -0.25 , -0.19999999, -0.15 ], [-0.09999999, -0.04999998, 0. , 0.05000001]] W: shape=(3, 4), dtype=float32 [[0.5 , 0.52 , 0.54 , 0.56 ], [0.58 , 0.6 , 0.62 , 0.64 ], [0.65999997, 0.68 , 0.7 , 0.72 ]] B: shape=(3, 4), dtype=float32 [[-0.25 , -0.24 , -0.23 , -0.22 ], [-0.21000001, -0.2 , -0.19 , -0.18 ], [-0.17 , -0.16 , -0.15 , -0.14 ]] Outputs: Y: shape=(3, 4), dtype=float32 [[-1.0464939 , -0.9177438 , -0.7774086 , -0.6254878 ], [-0.46198177, -0.28689027, -0.10021339, 0.09804872], [ 0.30789632, 0.5293293 , 0.76234746, 1.0069512 ]] Mean: shape=(1, 1), dtype=float32 [[-0.225]] InvStdDev: shape=(1, 1), dtype=float32 [[5.7926826]] **test_cc_layer_normalization_3d_axis0_epsilon** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 0 epsilon = 0.10000000149011612 .. code-block:: text Inputs: X: shape=(2, 3, 5), dtype=float32 [[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]], [[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ], [ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]] W: shape=(2, 3, 5), dtype=float32 [[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]], [[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ], [0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ], [1. , 1.02 , 1.04 , 1.06 , 1.0799999 ]]] B: shape=(2, 3, 5), dtype=float32 [[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]], [[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ], [-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001], [ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999]]] Outputs: Y: shape=(2, 3, 5), dtype=float32 [[[-0.9263111 , -0.8948557 , -0.85966897, -0.82075083, -0.7781013 ], [-0.73172045, -0.6816082 , -0.6277646 , -0.5701896 , -0.50888324], [-0.4438455 , -0.3750764 , -0.3025759 , -0.22634405, -0.1463809 ]], [[-0.06268631, 0.02473967, 0.11589703, 0.21078573, 0.3094057 ], [ 0.41175717, 0.51784 , 0.6276542 , 0.74119973, 0.85847676], [ 0.979485 , 1.1042248 , 1.2326956 , 1.364898 , 1.5008317 ]]] Mean: shape=(1, 1, 1), dtype=float32 [[[0.22500001]]] InvStdDev: shape=(1, 1, 1), dtype=float32 [[[1.8656857]]] **test_cc_layer_normalization_3d_axis1_epsilon** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 1 epsilon = 0.10000000149011612 .. code-block:: text Inputs: X: shape=(2, 3, 5), dtype=float32 [[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]], [[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ], [ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]] W: shape=(3, 5), dtype=float32 [[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]] B: shape=(3, 5), dtype=float32 [[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]] Outputs: Y: shape=(2, 3, 5), dtype=float32 [[[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.43717134], [-0.3566699 , -0.2709461 , -0.18000004, -0.08383158, 0.01755922], [ 0.12417227, 0.23600774, 0.35306546, 0.47534555, 0.6028478 ]], [[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.4371714 ], [-0.3566699 , -0.27094603, -0.18 , -0.08383166, 0.01755923], [ 0.12417224, 0.2360078 , 0.35306546, 0.47534543, 0.60284793]]] Mean: shape=(2, 1, 1), dtype=float32 [[[-0.14999999]], [[ 0.6 ]]] InvStdDev: shape=(2, 1, 1), dtype=float32 [[[2.6111646]], [[2.6111646]]] **test_cc_layer_normalization_3d_axis2_epsilon** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 2 epsilon = 0.10000000149011612 .. code-block:: text Inputs: X: shape=(2, 3, 5), dtype=float32 [[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]], [[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ], [ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]] W: shape=(5,), dtype=float32 [0.5 , 0.52, 0.54, 0.56, 0.58] B: shape=(5,), dtype=float32 [-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001] Outputs: Y: shape=(2, 3, 5), dtype=float32 [[[-0.40430334, -0.3202377 , -0.23 , -0.1335901 , -0.03100814], [-0.40430337, -0.32023773, -0.23000003, -0.13359013, -0.03100812], [-0.40430337, -0.32023773, -0.22999997, -0.13359007, -0.03100817]], [[-0.40430337, -0.32023776, -0.23 , -0.1335901 , -0.03100818], [-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808], [-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808]]] Mean: shape=(2, 3, 1), dtype=float32 [[[-0.4 ], [-0.14999999], [ 0.10000001]], [[ 0.35000002], [ 0.6 ], [ 0.85 ]]] InvStdDev: shape=(2, 3, 1), dtype=float32 [[[3.086067], [3.086067], [3.086067]], [[3.086067], [3.086067], [3.086067]]] **test_cc_layer_normalization_3d_axis_negative_1_epsilon** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -1 epsilon = 0.10000000149011612 .. code-block:: text Inputs: X: shape=(2, 3, 5), dtype=float32 [[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]], [[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ], [ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]] W: shape=(5,), dtype=float32 [0.5 , 0.52, 0.54, 0.56, 0.58] B: shape=(5,), dtype=float32 [-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001] Outputs: Y: shape=(2, 3, 5), dtype=float32 [[[-0.40430334, -0.3202377 , -0.23 , -0.1335901 , -0.03100814], [-0.40430337, -0.32023773, -0.23000003, -0.13359013, -0.03100812], [-0.40430337, -0.32023773, -0.22999997, -0.13359007, -0.03100817]], [[-0.40430337, -0.32023776, -0.23 , -0.1335901 , -0.03100818], [-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808], [-0.40430337, -0.32023767, -0.23 , -0.13359019, -0.03100808]]] Mean: shape=(2, 3, 1), dtype=float32 [[[-0.4 ], [-0.14999999], [ 0.10000001]], [[ 0.35000002], [ 0.6 ], [ 0.85 ]]] InvStdDev: shape=(2, 3, 1), dtype=float32 [[[3.086067], [3.086067], [3.086067]], [[3.086067], [3.086067], [3.086067]]] **test_cc_layer_normalization_3d_axis_negative_2_epsilon** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -2 epsilon = 0.10000000149011612 .. code-block:: text Inputs: X: shape=(2, 3, 5), dtype=float32 [[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]], [[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ], [ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]] W: shape=(3, 5), dtype=float32 [[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]] B: shape=(3, 5), dtype=float32 [[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]] Outputs: Y: shape=(2, 3, 5), dtype=float32 [[[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.43717134], [-0.3566699 , -0.2709461 , -0.18000004, -0.08383158, 0.01755922], [ 0.12417227, 0.23600774, 0.35306546, 0.47534555, 0.6028478 ]], [[-0.7069538 , -0.64734167, -0.58250725, -0.51245046, -0.4371714 ], [-0.3566699 , -0.27094603, -0.18 , -0.08383166, 0.01755923], [ 0.12417224, 0.2360078 , 0.35306546, 0.47534543, 0.60284793]]] Mean: shape=(2, 1, 1), dtype=float32 [[[-0.14999999]], [[ 0.6 ]]] InvStdDev: shape=(2, 1, 1), dtype=float32 [[[2.6111646]], [[2.6111646]]] **test_cc_layer_normalization_3d_axis_negative_3_epsilon** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -3 epsilon = 0.10000000149011612 .. code-block:: text Inputs: X: shape=(2, 3, 5), dtype=float32 [[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999]], [[ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ], [ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005]]] W: shape=(2, 3, 5), dtype=float32 [[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ]], [[0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ], [0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ], [1. , 1.02 , 1.04 , 1.06 , 1.0799999 ]]] B: shape=(2, 3, 5), dtype=float32 [[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ]], [[-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ], [-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001], [ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999]]] Outputs: Y: shape=(2, 3, 5), dtype=float32 [[[-0.9263111 , -0.8948557 , -0.85966897, -0.82075083, -0.7781013 ], [-0.73172045, -0.6816082 , -0.6277646 , -0.5701896 , -0.50888324], [-0.4438455 , -0.3750764 , -0.3025759 , -0.22634405, -0.1463809 ]], [[-0.06268631, 0.02473967, 0.11589703, 0.21078573, 0.3094057 ], [ 0.41175717, 0.51784 , 0.6276542 , 0.74119973, 0.85847676], [ 0.979485 , 1.1042248 , 1.2326956 , 1.364898 , 1.5008317 ]]] Mean: shape=(1, 1, 1), dtype=float32 [[[0.22500001]]] InvStdDev: shape=(1, 1, 1), dtype=float32 [[[1.8656857]]] **test_cc_layer_normalization_4d_axis0** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 0 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(2, 3, 4, 5), dtype=float32 [[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ], [0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]], [[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ], [1. , 1.02 , 1.04 , 1.06 , 1.0799999 ], [1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ], [1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]], [[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ], [1.4 , 1.42 , 1.44 , 1.46 , 1.48 ], [1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ], [1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]], [[[1.6999999 , 1.72 , 1.74 , 1.76 , 1.78 ], [1.8 , 1.8199999 , 1.8399999 , 1.86 , 1.88 ], [1.9 , 1.92 , 1.9399999 , 1.9599999 , 1.98 ], [2. , 2.02 , 2.04 , 2.06 , 2.08 ]], [[2.1 , 2.12 , 2.1399999 , 2.1599998 , 2.1799998 ], [2.1999998 , 2.2199998 , 2.24 , 2.26 , 2.28 ], [2.3 , 2.32 , 2.34 , 2.3600001 , 2.38 ], [2.4 , 2.42 , 2.44 , 2.46 , 2.48 ]], [[2.5 , 2.52 , 2.54 , 2.56 , 2.58 ], [2.6 , 2.62 , 2.6399999 , 2.6599998 , 2.68 ], [2.7 , 2.72 , 2.74 , 2.76 , 2.78 ], [2.8 , 2.82 , 2.84 , 2.86 , 2.8799999 ]]]] B: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ], [-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]], [[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001], [ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999], [ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ], [ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]], [[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ], [ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998], [ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996], [ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]], [[[ 0.34999996, 0.36 , 0.37 , 0.38 , 0.39 ], [ 0.39999998, 0.40999997, 0.41999996, 0.43 , 0.44 ], [ 0.45 , 0.45999998, 0.46999997, 0.47999996, 0.49 ], [ 0.5 , 0.51 , 0.52 , 0.53 , 0.53999996]], [[ 0.54999995, 0.56 , 0.57 , 0.58 , 0.59 ], [ 0.59999996, 0.60999995, 0.62 , 0.63 , 0.64 ], [ 0.65 , 0.65999997, 0.66999996, 0.68 , 0.69 ], [ 0.7 , 0.71 , 0.71999997, 0.72999996, 0.73999995]], [[ 0.75 , 0.76 , 0.77 , 0.78 , 0.78999996], [ 0.79999995, 0.80999994, 0.81999993, 0.8299999 , 0.84000003], [ 0.85 , 0.86 , 0.87 , 0.88 , 0.89 ], [ 0.9 , 0.90999997, 0.91999996, 0.92999995, 0.93999994]]]] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-1.1088369 , -1.1181788 , -1.126366 , -1.1333983 , -1.1392759 ], [-1.143999 , -1.147567 , -1.1499805 , -1.1512392 , -1.1513432 ], [-1.1502924 , -1.1480869 , -1.1447266 , -1.1402116 , -1.1345419 ], [-1.1277175 , -1.1197383 , -1.1106044 , -1.1003157 , -1.0888722 ]], [[-1.0762742 , -1.0625211 , -1.0476135 , -1.0315511 , -1.014334 ], [-0.9959622 , -0.9764355 , -0.9557542 , -0.9339181 , -0.9109273 ], [-0.8867818 , -0.8614815 , -0.8350265 , -0.8074168 , -0.7786523 ], [-0.74873304, -0.717659 , -0.6854303 , -0.65204686, -0.6175086 ]], [[-0.5818157 , -0.544968 , -0.5069655 , -0.4678084 , -0.42749655], [-0.38602996, -0.34340864, -0.2996324 , -0.2547016 , -0.20861608], [-0.16137576, -0.11298075, -0.06343079, -0.01272631, 0.03913289], [ 0.0921469 , 0.14631562, 0.20163928, 0.25811747, 0.31575036]]], [[[ 0.374538 , 0.43448046, 0.49557784, 0.5578297 , 0.6212363 ], [ 0.68579763, 0.7515137 , 0.81838477, 0.8864104 , 0.9555907 ], [ 1.0259258 , 1.0974154 , 1.1700602 , 1.2438595 , 1.3188136 ], [ 1.3949223 , 1.4721859 , 1.5506042 , 1.6301771 , 1.7109048 ]], [[ 1.7927872 , 1.8758247 , 1.9600163 , 2.0453632 , 2.1318648 ], [ 2.2195206 , 2.3083317 , 2.3982973 , 2.489418 , 2.5816932 ], [ 2.6751227 , 2.7697077 , 2.8654466 , 2.962341 , 3.0603902 ], [ 3.1595933 , 3.2599518 , 3.3614645 , 3.4641323 , 3.5679553 ]], [[ 3.672932 , 3.7790644 , 3.8863504 , 3.9947925 , 4.1043887 ], [ 4.2151394 , 4.3270454 , 4.4401054 , 4.5543203 , 4.669691 ], [ 4.7862153 , 4.903895 , 5.0227284 , 5.142718 , 5.263861 ], [ 5.3861594 , 5.5096126 , 5.63422 , 5.7599826 , 5.886901 ]]]] Mean: shape=(1, 1, 1, 1), dtype=float32 [[[[2.4750001]]]] InvStdDev: shape=(1, 1, 1, 1), dtype=float32 [[[[0.57736933]]]] **test_cc_layer_normalization_4d_axis1** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 1 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(3, 4, 5), dtype=float32 [[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ], [0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]], [[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ], [1. , 1.02 , 1.04 , 1.06 , 1.0799999 ], [1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ], [1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]], [[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ], [1.4 , 1.42 , 1.44 , 1.46 , 1.48 ], [1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ], [1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]] B: shape=(3, 4, 5), dtype=float32 [[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ], [-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]], [[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001], [ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999], [ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ], [ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]], [[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ], [ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998], [ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996], [ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-1.1017042 , -1.0957462 , -1.0874785 , -1.0769011 , -1.064014 ], [-1.0488172 , -1.0313106 , -1.0114943 , -0.9893684 , -0.9649327 ], [-0.93818736, -0.90913224, -0.8777675 , -0.84409297, -0.8081088 ], [-0.76981485, -0.7292112 , -0.68629795, -0.6410749 , -0.5935422 ]], [[-0.54369974, -0.49154752, -0.43708575, -0.38031426, -0.32123291], [-0.259842 , -0.19614126, -0.13013093, -0.06181089, 0.00881899], [ 0.08175841, 0.15700774, 0.23456657, 0.31443512, 0.3966136 ], [ 0.48110154, 0.56789935, 0.65700674, 0.7484238 , 0.84215075]], [[ 0.93818724, 1.0365335 , 1.1371896 , 1.2401552 , 1.3454306 ], [ 1.4530158 , 1.5629103 , 1.6751153 , 1.7896295 , 1.9064533 ], [ 2.0255868 , 2.1470299 , 2.2707834 , 2.396846 , 2.5252185 ], [ 2.6559005 , 2.7888923 , 2.924194 , 3.0618052 , 3.201726 ]]], [[[-1.1017044 , -1.0957463 , -1.0874785 , -1.0769011 , -1.064014 ], [-1.0488173 , -1.0313108 , -1.0114943 , -0.9893684 , -0.9649328 ], [-0.9381875 , -0.9091324 , -0.8777675 , -0.844093 , -0.80810887], [-0.769815 , -0.72921145, -0.68629795, -0.641075 , -0.5935423 ]], [[-0.54369986, -0.49154752, -0.43708602, -0.38031426, -0.3212328 ], [-0.25984213, -0.19614126, -0.13013121, -0.06181089, 0.00881913], [ 0.08175826, 0.15700774, 0.23456627, 0.31443512, 0.39661372], [ 0.4811014 , 0.56789935, 0.6570064 , 0.7484238 , 0.8421509 ]], [[ 0.93818706, 1.0365337 , 1.1371891 , 1.2401551 , 1.3454309 ], [ 1.4530156 , 1.5629106 , 1.6751148 , 1.7896293 , 1.9064535 ], [ 2.0255866 , 2.14703 , 2.2707827 , 2.3968456 , 2.5252185 ], [ 2.6559005 , 2.7888925 , 2.9241936 , 3.061805 , 3.2017264 ]]]] Mean: shape=(2, 1, 1, 1), dtype=float32 [[[[0.975 ]]], [[[3.9750001]]]] InvStdDev: shape=(2, 1, 1, 1), dtype=float32 [[[[1.1548532]]], [[[1.1548532]]]] **test_cc_layer_normalization_4d_axis2** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 2 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(4, 5), dtype=float32 [[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ], [0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]] B: shape=(4, 5), dtype=float32 [[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ], [-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-1.073705 , -1.006479 , -0.93231684, -0.8512181 , -0.7631829 ], [-0.6682113 , -0.5663031 , -0.45745853, -0.34167743, -0.2189599 ], [-0.08930597, 0.0472845 , 0.19081143, 0.34127483, 0.49867445], [ 0.66301084, 0.8342837 , 1.0124929 , 1.1976386 , 1.3897207 ]], [[-1.073705 , -1.0064789 , -0.93231684, -0.8512182 , -0.76318276], [-0.6682113 , -0.56630296, -0.4574585 , -0.34167755, -0.21895987], [-0.08930601, 0.0472846 , 0.1908114 , 0.3412746 , 0.49867463], [ 0.6630107 , 0.8342837 , 1.0124928 , 1.1976383 , 1.3897207 ]], [[-1.073705 , -1.0064791 , -0.93231666, -0.85121787, -0.76318276], [-0.6682113 , -0.56630325, -0.45745823, -0.34167725, -0.21895987], [-0.08930601, 0.04728431, 0.1908117 , 0.34127492, 0.49867463], [ 0.6630107 , 0.83428335, 1.0124931 , 1.1976388 , 1.3897207 ]]], [[[-1.0737052 , -1.0064794 , -0.93231684, -0.8512182 , -0.763183 ], [-0.6682115 , -0.5663035 , -0.4574585 , -0.34167755, -0.21896015], [-0.08930631, 0.04728401, 0.1908114 , 0.3412746 , 0.49867427], [ 0.66301036, 0.8342831 , 1.0124928 , 1.1976383 , 1.3897202 ]], [[-1.0737052 , -1.0064789 , -0.9323173 , -0.8512182 , -0.7631825 ], [-0.6682115 , -0.56630296, -0.45745903, -0.34167755, -0.21895958], [-0.08930631, 0.0472846 , 0.19081077, 0.3412746 , 0.49867493], [ 0.66301036, 0.8342837 , 1.0124922 , 1.1976383 , 1.3897209 ]], [[-1.0737047 , -1.0064785 , -0.93231684, -0.85121775, -0.76318204], [-0.66821104, -0.5663025 , -0.4574585 , -0.341677 , -0.21895903], [-0.08930573, 0.04728521, 0.1908114 , 0.34127522, 0.49867558], [ 0.6630111 , 0.8342844 , 1.0124928 , 1.197639 , 1.3897219 ]]]] Mean: shape=(2, 3, 1, 1), dtype=float32 [[[[-0.02499999]], [[ 0.975 ]], [[ 1.975 ]]], [[[ 2.9750001 ]], [[ 3.9750001 ]], [[ 4.975 ]]]] InvStdDev: shape=(2, 3, 1, 1), dtype=float32 [[[[3.4682312]], [[3.4682312]], [[3.4682312]]], [[[3.4682312]], [[3.4682312]], [[3.4682312]]]] **test_cc_layer_normalization_4d_axis3** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = 3 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(5,), dtype=float32 [0.5 , 0.52, 0.54, 0.56, 0.58] B: shape=(5,), dtype=float32 [-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.95640075, -0.60732824, -0.23 , 0.17558452, 0.6094248 ], [-0.95640075, -0.6073283 , -0.23000012, 0.17558435, 0.6094248 ], [-0.9564008 , -0.60732836, -0.22999988, 0.17558464, 0.6094247 ], [-0.95640093, -0.6073285 , -0.23 , 0.17558452, 0.6094246 ]], [[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507]], [[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ], [-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]]], [[[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]], [[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ], [-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]], [[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]]]] Mean: shape=(2, 3, 4, 1), dtype=float32 [[[[-0.4 ], [-0.14999999], [ 0.10000001], [ 0.35000002]], [[ 0.6 ], [ 0.85 ], [ 1.1 ], [ 1.35 ]], [[ 1.6 ], [ 1.85 ], [ 2.1000001 ], [ 2.3500001 ]]], [[[ 2.6000001 ], [ 2.8500001 ], [ 3.1000001 ], [ 3.3500001 ]], [[ 3.6000001 ], [ 3.8500001 ], [ 4.1 ], [ 4.35 ]], [[ 4.6 ], [ 4.85 ], [ 5.1 ], [ 5.35 ]]]] InvStdDev: shape=(2, 3, 4, 1), dtype=float32 [[[[14.1280155], [14.128014 ], [14.1280155], [14.1280155]], [[14.1280155], [14.1280155], [14.1280155], [14.1280155]], [[14.128008 ], [14.128008 ], [14.128008 ], [14.128008 ]]], [[[14.128008 ], [14.128008 ], [14.128008 ], [14.128008 ]], [[14.128 ], [14.128 ], [14.128 ], [14.128 ]], [[14.128 ], [14.128 ], [14.128 ], [14.128 ]]]] **test_cc_layer_normalization_4d_axis_negative_1** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -1 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(5,), dtype=float32 [0.5 , 0.52, 0.54, 0.56, 0.58] B: shape=(5,), dtype=float32 [-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.95640075, -0.60732824, -0.23 , 0.17558452, 0.6094248 ], [-0.95640075, -0.6073283 , -0.23000012, 0.17558435, 0.6094248 ], [-0.9564008 , -0.60732836, -0.22999988, 0.17558464, 0.6094247 ], [-0.95640093, -0.6073285 , -0.23 , 0.17558452, 0.6094246 ]], [[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507]], [[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ], [-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]]], [[[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]], [[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ], [-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]], [[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]]]] Mean: shape=(2, 3, 4, 1), dtype=float32 [[[[-0.4 ], [-0.14999999], [ 0.10000001], [ 0.35000002]], [[ 0.6 ], [ 0.85 ], [ 1.1 ], [ 1.35 ]], [[ 1.6 ], [ 1.85 ], [ 2.1000001 ], [ 2.3500001 ]]], [[[ 2.6000001 ], [ 2.8500001 ], [ 3.1000001 ], [ 3.3500001 ]], [[ 3.6000001 ], [ 3.8500001 ], [ 4.1 ], [ 4.35 ]], [[ 4.6 ], [ 4.85 ], [ 5.1 ], [ 5.35 ]]]] InvStdDev: shape=(2, 3, 4, 1), dtype=float32 [[[[14.1280155], [14.128014 ], [14.1280155], [14.1280155]], [[14.1280155], [14.1280155], [14.1280155], [14.1280155]], [[14.128008 ], [14.128008 ], [14.128008 ], [14.128008 ]]], [[[14.128008 ], [14.128008 ], [14.128008 ], [14.128008 ]], [[14.128 ], [14.128 ], [14.128 ], [14.128 ]], [[14.128 ], [14.128 ], [14.128 ], [14.128 ]]]] **test_cc_layer_normalization_4d_axis_negative_2** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -2 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(4, 5), dtype=float32 [[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ], [0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]] B: shape=(4, 5), dtype=float32 [[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ], [-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-1.073705 , -1.006479 , -0.93231684, -0.8512181 , -0.7631829 ], [-0.6682113 , -0.5663031 , -0.45745853, -0.34167743, -0.2189599 ], [-0.08930597, 0.0472845 , 0.19081143, 0.34127483, 0.49867445], [ 0.66301084, 0.8342837 , 1.0124929 , 1.1976386 , 1.3897207 ]], [[-1.073705 , -1.0064789 , -0.93231684, -0.8512182 , -0.76318276], [-0.6682113 , -0.56630296, -0.4574585 , -0.34167755, -0.21895987], [-0.08930601, 0.0472846 , 0.1908114 , 0.3412746 , 0.49867463], [ 0.6630107 , 0.8342837 , 1.0124928 , 1.1976383 , 1.3897207 ]], [[-1.073705 , -1.0064791 , -0.93231666, -0.85121787, -0.76318276], [-0.6682113 , -0.56630325, -0.45745823, -0.34167725, -0.21895987], [-0.08930601, 0.04728431, 0.1908117 , 0.34127492, 0.49867463], [ 0.6630107 , 0.83428335, 1.0124931 , 1.1976388 , 1.3897207 ]]], [[[-1.0737052 , -1.0064794 , -0.93231684, -0.8512182 , -0.763183 ], [-0.6682115 , -0.5663035 , -0.4574585 , -0.34167755, -0.21896015], [-0.08930631, 0.04728401, 0.1908114 , 0.3412746 , 0.49867427], [ 0.66301036, 0.8342831 , 1.0124928 , 1.1976383 , 1.3897202 ]], [[-1.0737052 , -1.0064789 , -0.9323173 , -0.8512182 , -0.7631825 ], [-0.6682115 , -0.56630296, -0.45745903, -0.34167755, -0.21895958], [-0.08930631, 0.0472846 , 0.19081077, 0.3412746 , 0.49867493], [ 0.66301036, 0.8342837 , 1.0124922 , 1.1976383 , 1.3897209 ]], [[-1.0737047 , -1.0064785 , -0.93231684, -0.85121775, -0.76318204], [-0.66821104, -0.5663025 , -0.4574585 , -0.341677 , -0.21895903], [-0.08930573, 0.04728521, 0.1908114 , 0.34127522, 0.49867558], [ 0.6630111 , 0.8342844 , 1.0124928 , 1.197639 , 1.3897219 ]]]] Mean: shape=(2, 3, 1, 1), dtype=float32 [[[[-0.02499999]], [[ 0.975 ]], [[ 1.975 ]]], [[[ 2.9750001 ]], [[ 3.9750001 ]], [[ 4.975 ]]]] InvStdDev: shape=(2, 3, 1, 1), dtype=float32 [[[[3.4682312]], [[3.4682312]], [[3.4682312]]], [[[3.4682312]], [[3.4682312]], [[3.4682312]]]] **test_cc_layer_normalization_4d_axis_negative_3** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -3 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(3, 4, 5), dtype=float32 [[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ], [0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]], [[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ], [1. , 1.02 , 1.04 , 1.06 , 1.0799999 ], [1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ], [1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]], [[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ], [1.4 , 1.42 , 1.44 , 1.46 , 1.48 ], [1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ], [1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]] B: shape=(3, 4, 5), dtype=float32 [[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ], [-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]], [[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001], [ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999], [ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ], [ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]], [[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ], [ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998], [ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996], [ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-1.1017042 , -1.0957462 , -1.0874785 , -1.0769011 , -1.064014 ], [-1.0488172 , -1.0313106 , -1.0114943 , -0.9893684 , -0.9649327 ], [-0.93818736, -0.90913224, -0.8777675 , -0.84409297, -0.8081088 ], [-0.76981485, -0.7292112 , -0.68629795, -0.6410749 , -0.5935422 ]], [[-0.54369974, -0.49154752, -0.43708575, -0.38031426, -0.32123291], [-0.259842 , -0.19614126, -0.13013093, -0.06181089, 0.00881899], [ 0.08175841, 0.15700774, 0.23456657, 0.31443512, 0.3966136 ], [ 0.48110154, 0.56789935, 0.65700674, 0.7484238 , 0.84215075]], [[ 0.93818724, 1.0365335 , 1.1371896 , 1.2401552 , 1.3454306 ], [ 1.4530158 , 1.5629103 , 1.6751153 , 1.7896295 , 1.9064533 ], [ 2.0255868 , 2.1470299 , 2.2707834 , 2.396846 , 2.5252185 ], [ 2.6559005 , 2.7888923 , 2.924194 , 3.0618052 , 3.201726 ]]], [[[-1.1017044 , -1.0957463 , -1.0874785 , -1.0769011 , -1.064014 ], [-1.0488173 , -1.0313108 , -1.0114943 , -0.9893684 , -0.9649328 ], [-0.9381875 , -0.9091324 , -0.8777675 , -0.844093 , -0.80810887], [-0.769815 , -0.72921145, -0.68629795, -0.641075 , -0.5935423 ]], [[-0.54369986, -0.49154752, -0.43708602, -0.38031426, -0.3212328 ], [-0.25984213, -0.19614126, -0.13013121, -0.06181089, 0.00881913], [ 0.08175826, 0.15700774, 0.23456627, 0.31443512, 0.39661372], [ 0.4811014 , 0.56789935, 0.6570064 , 0.7484238 , 0.8421509 ]], [[ 0.93818706, 1.0365337 , 1.1371891 , 1.2401551 , 1.3454309 ], [ 1.4530156 , 1.5629106 , 1.6751148 , 1.7896293 , 1.9064535 ], [ 2.0255866 , 2.14703 , 2.2707827 , 2.3968456 , 2.5252185 ], [ 2.6559005 , 2.7888925 , 2.9241936 , 3.061805 , 3.2017264 ]]]] Mean: shape=(2, 1, 1, 1), dtype=float32 [[[[0.975 ]]], [[[3.9750001]]]] InvStdDev: shape=(2, 1, 1, 1), dtype=float32 [[[[1.1548532]]], [[[1.1548532]]]] **test_cc_layer_normalization_4d_axis_negative_4** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) Attributes: axis = -4 .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(2, 3, 4, 5), dtype=float32 [[[[0.5 , 0.52 , 0.54 , 0.56 , 0.58 ], [0.6 , 0.62 , 0.64 , 0.65999997, 0.68 ], [0.7 , 0.72 , 0.74 , 0.76 , 0.78 ], [0.79999995, 0.82 , 0.84000003, 0.86 , 0.88 ]], [[0.9 , 0.91999996, 0.94 , 0.96 , 0.98 ], [1. , 1.02 , 1.04 , 1.06 , 1.0799999 ], [1.0999999 , 1.12 , 1.14 , 1.16 , 1.1800001 ], [1.2 , 1.22 , 1.24 , 1.26 , 1.28 ]], [[1.3 , 1.3199999 , 1.3399999 , 1.3599999 , 1.38 ], [1.4 , 1.42 , 1.44 , 1.46 , 1.48 ], [1.5 , 1.52 , 1.54 , 1.56 , 1.5799999 ], [1.6 , 1.62 , 1.64 , 1.66 , 1.68 ]]], [[[1.6999999 , 1.72 , 1.74 , 1.76 , 1.78 ], [1.8 , 1.8199999 , 1.8399999 , 1.86 , 1.88 ], [1.9 , 1.92 , 1.9399999 , 1.9599999 , 1.98 ], [2. , 2.02 , 2.04 , 2.06 , 2.08 ]], [[2.1 , 2.12 , 2.1399999 , 2.1599998 , 2.1799998 ], [2.1999998 , 2.2199998 , 2.24 , 2.26 , 2.28 ], [2.3 , 2.32 , 2.34 , 2.3600001 , 2.38 ], [2.4 , 2.42 , 2.44 , 2.46 , 2.48 ]], [[2.5 , 2.52 , 2.54 , 2.56 , 2.58 ], [2.6 , 2.62 , 2.6399999 , 2.6599998 , 2.68 ], [2.7 , 2.72 , 2.74 , 2.76 , 2.78 ], [2.8 , 2.82 , 2.84 , 2.86 , 2.8799999 ]]]] B: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001], [-0.2 , -0.19 , -0.18 , -0.17 , -0.16 ], [-0.15 , -0.14 , -0.13 , -0.12 , -0.11 ], [-0.10000001, -0.09 , -0.08 , -0.07000001, -0.06 ]], [[-0.05000001, -0.04000001, -0.03 , -0.02000001, -0.01000001], [ 0. , 0.00999999, 0.01999998, 0.03 , 0.03999999], [ 0.04999998, 0.06 , 0.06999999, 0.07999998, 0.09 ], [ 0.09999999, 0.10999998, 0.12 , 0.13 , 0.13999999]], [[ 0.14999998, 0.16 , 0.16999999, 0.17999998, 0.19 ], [ 0.19999999, 0.20999998, 0.22 , 0.22999999, 0.23999998], [ 0.25 , 0.26 , 0.26999998, 0.27999997, 0.28999996], [ 0.3 , 0.31 , 0.32 , 0.32999998, 0.33999997]]], [[[ 0.34999996, 0.36 , 0.37 , 0.38 , 0.39 ], [ 0.39999998, 0.40999997, 0.41999996, 0.43 , 0.44 ], [ 0.45 , 0.45999998, 0.46999997, 0.47999996, 0.49 ], [ 0.5 , 0.51 , 0.52 , 0.53 , 0.53999996]], [[ 0.54999995, 0.56 , 0.57 , 0.58 , 0.59 ], [ 0.59999996, 0.60999995, 0.62 , 0.63 , 0.64 ], [ 0.65 , 0.65999997, 0.66999996, 0.68 , 0.69 ], [ 0.7 , 0.71 , 0.71999997, 0.72999996, 0.73999995]], [[ 0.75 , 0.76 , 0.77 , 0.78 , 0.78999996], [ 0.79999995, 0.80999994, 0.81999993, 0.8299999 , 0.84000003], [ 0.85 , 0.86 , 0.87 , 0.88 , 0.89 ], [ 0.9 , 0.90999997, 0.91999996, 0.92999995, 0.93999994]]]] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-1.1088369 , -1.1181788 , -1.126366 , -1.1333983 , -1.1392759 ], [-1.143999 , -1.147567 , -1.1499805 , -1.1512392 , -1.1513432 ], [-1.1502924 , -1.1480869 , -1.1447266 , -1.1402116 , -1.1345419 ], [-1.1277175 , -1.1197383 , -1.1106044 , -1.1003157 , -1.0888722 ]], [[-1.0762742 , -1.0625211 , -1.0476135 , -1.0315511 , -1.014334 ], [-0.9959622 , -0.9764355 , -0.9557542 , -0.9339181 , -0.9109273 ], [-0.8867818 , -0.8614815 , -0.8350265 , -0.8074168 , -0.7786523 ], [-0.74873304, -0.717659 , -0.6854303 , -0.65204686, -0.6175086 ]], [[-0.5818157 , -0.544968 , -0.5069655 , -0.4678084 , -0.42749655], [-0.38602996, -0.34340864, -0.2996324 , -0.2547016 , -0.20861608], [-0.16137576, -0.11298075, -0.06343079, -0.01272631, 0.03913289], [ 0.0921469 , 0.14631562, 0.20163928, 0.25811747, 0.31575036]]], [[[ 0.374538 , 0.43448046, 0.49557784, 0.5578297 , 0.6212363 ], [ 0.68579763, 0.7515137 , 0.81838477, 0.8864104 , 0.9555907 ], [ 1.0259258 , 1.0974154 , 1.1700602 , 1.2438595 , 1.3188136 ], [ 1.3949223 , 1.4721859 , 1.5506042 , 1.6301771 , 1.7109048 ]], [[ 1.7927872 , 1.8758247 , 1.9600163 , 2.0453632 , 2.1318648 ], [ 2.2195206 , 2.3083317 , 2.3982973 , 2.489418 , 2.5816932 ], [ 2.6751227 , 2.7697077 , 2.8654466 , 2.962341 , 3.0603902 ], [ 3.1595933 , 3.2599518 , 3.3614645 , 3.4641323 , 3.5679553 ]], [[ 3.672932 , 3.7790644 , 3.8863504 , 3.9947925 , 4.1043887 ], [ 4.2151394 , 4.3270454 , 4.4401054 , 4.5543203 , 4.669691 ], [ 4.7862153 , 4.903895 , 5.0227284 , 5.142718 , 5.263861 ], [ 5.3861594 , 5.5096126 , 5.63422 , 5.7599826 , 5.886901 ]]]] Mean: shape=(1, 1, 1, 1), dtype=float32 [[[[2.4750001]]]] InvStdDev: shape=(1, 1, 1, 1), dtype=float32 [[[[0.57736933]]]] **test_cc_layer_normalization_default_axis** .. code-block:: text Node: LayerNormalization(X, W, B) -> (Y, Mean, InvStdDev) .. code-block:: text Inputs: X: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.5 , -0.45 , -0.4 , -0.35 , -0.3 ], [-0.25 , -0.19999999, -0.15 , -0.09999999, -0.04999998], [ 0. , 0.05000001, 0.10000002, 0.15000004, 0.19999999], [ 0.25 , 0.3 , 0.35000002, 0.40000004, 0.45 ]], [[ 0.5 , 0.5500001 , 0.6 , 0.65 , 0.70000005], [ 0.75 , 0.8000001 , 0.85 , 0.9 , 0.95000005], [ 1. , 1.0500001 , 1.1 , 1.15 , 1.2 ], [ 1.25 , 1.3000001 , 1.35 , 1.4 , 1.45 ]], [[ 1.5 , 1.55 , 1.6000001 , 1.6500001 , 1.7 ], [ 1.75 , 1.8 , 1.8500001 , 1.9000001 , 1.95 ], [ 2. , 2.05 , 2.1000001 , 2.15 , 2.2 ], [ 2.25 , 2.3 , 2.3500001 , 2.4 , 2.45 ]]], [[[ 2.5 , 2.55 , 2.6000001 , 2.65 , 2.7 ], [ 2.75 , 2.8 , 2.8500001 , 2.9 , 2.95 ], [ 3. , 3.05 , 3.1000001 , 3.15 , 3.2 ], [ 3.25 , 3.3 , 3.3500001 , 3.4 , 3.45 ]], [[ 3.5 , 3.5500002 , 3.6 , 3.65 , 3.7000003 ], [ 3.75 , 3.8000002 , 3.85 , 3.9 , 3.9500003 ], [ 4. , 4.05 , 4.1 , 4.15 , 4.2000003 ], [ 4.25 , 4.3 , 4.35 , 4.4 , 4.4500003 ]], [[ 4.5 , 4.55 , 4.6 , 4.65 , 4.7000003 ], [ 4.75 , 4.8 , 4.85 , 4.9 , 4.9500003 ], [ 5. , 5.05 , 5.1 , 5.15 , 5.2000003 ], [ 5.25 , 5.3 , 5.35 , 5.4 , 5.4500003 ]]]] W: shape=(5,), dtype=float32 [0.5 , 0.52, 0.54, 0.56, 0.58] B: shape=(5,), dtype=float32 [-0.25 , -0.24 , -0.23 , -0.22 , -0.21000001] Outputs: Y: shape=(2, 3, 4, 5), dtype=float32 [[[[-0.95640075, -0.60732824, -0.23 , 0.17558452, 0.6094248 ], [-0.95640075, -0.6073283 , -0.23000012, 0.17558435, 0.6094248 ], [-0.9564008 , -0.60732836, -0.22999988, 0.17558464, 0.6094247 ], [-0.95640093, -0.6073285 , -0.23 , 0.17558452, 0.6094246 ]], [[-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507], [-0.95640093, -0.60732806, -0.23 , 0.17558405, 0.60942507]], [[-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ], [-0.9564006 , -0.6073287 , -0.2299991 , 0.1755848 , 0.6094246 ], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]]], [[[-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364], [-0.9564014 , -0.6073296 , -0.23 , 0.17558387, 0.60942364]], [[-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ], [-0.95640105, -0.60732764, -0.23000182, 0.17558363, 0.6094252 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]], [[-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ], [-0.9563993 , -0.6073259 , -0.23 , 0.17558554, 0.6094271 ]]]] Mean: shape=(2, 3, 4, 1), dtype=float32 [[[[-0.4 ], [-0.14999999], [ 0.10000001], [ 0.35000002]], [[ 0.6 ], [ 0.85 ], [ 1.1 ], [ 1.35 ]], [[ 1.6 ], [ 1.85 ], [ 2.1000001 ], [ 2.3500001 ]]], [[[ 2.6000001 ], [ 2.8500001 ], [ 3.1000001 ], [ 3.3500001 ]], [[ 3.6000001 ], [ 3.8500001 ], [ 4.1 ], [ 4.35 ]], [[ 4.6 ], [ 4.85 ], [ 5.1 ], [ 5.35 ]]]] InvStdDev: shape=(2, 3, 4, 1), dtype=float32 [[[[14.1280155], [14.128014 ], [14.1280155], [14.1280155]], [[14.1280155], [14.1280155], [14.1280155], [14.1280155]], [[14.128008 ], [14.128008 ], [14.128008 ], [14.128008 ]]], [[[14.128008 ], [14.128008 ], [14.128008 ], [14.128008 ]], [[14.128 ], [14.128 ], [14.128 ], [14.128 ]], [[14.128 ], [14.128 ], [14.128 ], [14.128 ]]]]