SwiGLU#

Domain: ai.onnx
Since version: 28

SwiGLU is a gated activation that takes two inputs, a gate A and a linear (value) input B, and produces one output Y. It applies the Swish activation to the gate and multiplies the result elementwise by the linear input:

Y = Swish_alpha(A) * B

The gate activation Swish_alpha is exactly the Swish operator with the same alpha, i.e. Swish_alpha(a) = a * Sigmoid(alpha * a). Inputs A and B must have identical shapes; broadcasting is not applied and the output Y has the same shape as the inputs.

Exporters typically produce A and B in one of two ways: for the common two-projection form (e.g. Llama’s gate_proj/up_proj) wire the two projection outputs directly to A (gate) and B (value); for a fused/packed single projection, split it upstream into A and B with Split (contiguous layout) or Slice/Gather (interleaved layout).

Inputs

A (T): Gate input tensor
B (T): Linear (value) input tensor

Outputs

Y (T): Output tensor

Attributes

alpha (float): Coefficient that scales the gate input inside the sigmoid of the Swish activation. The default value is 1.0.

Type Constraints

T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

Examples#

test_cc_swiglu

Node:
  SwiGLU(A, B) -> (Y)

Inputs:
  A: shape=(2, 4), dtype=float32
    [[ 1. , -2. ,  3. ,  4. ],
     [-1. ,  2. , -3. ,  0.5]]
  B: shape=(2, 4), dtype=float32
    [[ 0.5,  1. , -1. ,  2. ],
     [ 2. , -1. ,  0.5,  1. ]]

Outputs:
  Y: shape=(2, 4), dtype=float32
    [[ 0.3655293 , -0.23840584, -2.8577223 ,  7.85611   ],
     [-0.53788286, -1.761594  , -0.07113881,  0.31122968]]

test_cc_swiglu_alpha

Node:
  SwiGLU(A, B) -> (Y)
  Attributes:
    alpha = 0.5

Inputs:
  A: shape=(2, 4), dtype=float32
    [[ 1. , -2. ,  3. ,  4. ],
     [-1. ,  2. , -3. ,  0.5]]
  B: shape=(2, 4), dtype=float32
    [[ 0.5,  1. , -1. ,  2. ],
     [ 2. , -1. ,  0.5,  1. ]]

Outputs:
  Y: shape=(2, 4), dtype=float32
    [[ 0.31122968, -0.53788286, -2.4527233 ,  7.046376  ],
     [-0.75508136, -1.4621172 , -0.2736383 ,  0.28108826]]

test_cc_swiglu_bfloat16

Node:
  SwiGLU(A, B) -> (Y)

Inputs:
  A: shape=(2, 4), dtype=bfloat16
    [[1, -2, 3, 4],
     [-1, 2, -3, 0.5]]
  B: shape=(2, 4), dtype=bfloat16
    [[0.5, 1, -1, 2],
     [2, -1, 0.5, 1]]

Outputs:
  Y: shape=(2, 4), dtype=bfloat16
    [[0.365234, -0.238281, -2.85938, 7.84375],
     [-0.539062, -1.75781, -0.0712891, 0.310547]]

test_cc_swiglu_float16

Node:
  SwiGLU(A, B) -> (Y)

Inputs:
  A: shape=(2, 4), dtype=float16
    [[ 1. , -2. ,  3. ,  4. ],
     [-1. ,  2. , -3. ,  0.5]]
  B: shape=(2, 4), dtype=float16
    [[ 0.5,  1. , -1. ,  2. ],
     [ 2. , -1. ,  0.5,  1. ]]

Outputs:
  Y: shape=(2, 4), dtype=float16
    [[ 0.3655 , -0.2384 , -2.857  ,  7.855  ],
     [-0.538  , -1.762  , -0.07117,  0.3113 ]]