SkipLayerNormalization (CPU)#

  • Domain: com.microsoft

  • Device: CPU

  • Kernel: onnx_light_cpu::SkipLayerNormalization

  • Supported types: FLOAT, FLOAT16, BFLOAT16

  • Opset bounds: since opset 1

Operator support#

Capability

Implementation

Shape inference

onnx_light_cpu::ComputeShapeSkipLayerNormalization()

Peak memory

onnx_light_cpu::ComputePeakMemorySkipLayerNormalization()

Fusion patterns

None

Gradient

Not provided

LightOpSchema#

Version: 1

Adds input, skip, and optional bias, then applies last-axis layer normalization, gamma scaling, and optional beta. Supports rank-2 and rank-3 inputs with matching FLOAT, FLOAT16, or BFLOAT16 types. For rank-3 input, skip may omit the batch dimension or have batch size one. Optional FLOAT statistics contain the mean and inverse standard deviation. Optional output slot 3 contains the residual sum in the input type.

Inputs

  • input (T): Input tensor with shape (S, H) or (B, S, H), with 0 < H <= INT_MAX.

  • skip (T): Residual tensor matching input, or (S, H)/(1, S, H) for rank-3 input.

  • gamma (T): Rank-1 scale tensor with length H.

  • beta (T): Optional rank-1 shift tensor with length H.

  • bias (T): Optional rank-1 bias added to the residual, with length H.

Outputs

  • output (T): Normalized result with the input shape and type.

  • mean (U): Optional FLOAT mean with input shape and final dimension one.

  • inv_std_var (U): Optional FLOAT inverse standard deviation with input shape and final dimension one.

  • input_skip_bias_sum (T): Optional residual sum with the input shape and type.

Attributes

  • epsilon (FLOAT): Finite nonnegative value added to the last-axis variance. (optional, default: 1e-12)

Type constraints

  • T: Matching FLOAT, FLOAT16, or BFLOAT16 tensors. Allowed types: tensor(bfloat16), tensor(float), tensor(float16).

  • U: FLOAT saved-statistics tensors. Allowed types: tensor(float).