.. _op_ai_onnx_SwiGLU:

SwiGLU
======

- **Domain**: ``ai.onnx``
- **Since version**: 28

SwiGLU is a gated activation that takes two inputs, a gate ``A`` and a linear (value)
input ``B``, and produces one output ``Y``. It applies the Swish activation to the gate
and multiplies the result elementwise by the linear input:

.. code-block::

    Y = Swish_alpha(A) * B


The gate activation ``Swish_alpha`` is exactly the ``Swish`` operator with the same
``alpha``, i.e. ``Swish_alpha(a) = a * Sigmoid(alpha * a)``. Inputs ``A`` and ``B`` must
have identical shapes; broadcasting is not applied and the output ``Y`` has the same
shape as the inputs.

Exporters typically produce ``A`` and ``B`` in one of two ways: for the common
two-projection form (e.g. Llama's ``gate_proj``/``up_proj``) wire the two projection
outputs directly to ``A`` (gate) and ``B`` (value); for a fused/packed single
projection, split it upstream into ``A`` and ``B`` with ``Split`` (contiguous layout)
or ``Slice``/``Gather`` (interleaved layout).

**Inputs**

- **A** (*T*): Gate input tensor
- **B** (*T*): Linear (value) input tensor

**Outputs**

- **Y** (*T*): Output tensor

**Attributes**

- **alpha** (*float*): Coefficient that scales the gate input inside the sigmoid of the Swish activation. The default value is 1.0.

**Type Constraints**

- **T**: Constrain input and output types to float tensors.
  Allowed types: tensor(bfloat16), tensor(double), tensor(float), tensor(float16).

Examples
--------

**test_cc_swiglu**

.. code-block:: text

    Node:
      SwiGLU(A, B) -> (Y)

.. code-block:: text

    Inputs:
      A: shape=(2, 4), dtype=float32
        [[ 1. , -2. ,  3. ,  4. ],
         [-1. ,  2. , -3. ,  0.5]]
      B: shape=(2, 4), dtype=float32
        [[ 0.5,  1. , -1. ,  2. ],
         [ 2. , -1. ,  0.5,  1. ]]

    Outputs:
      Y: shape=(2, 4), dtype=float32
        [[ 0.3655293 , -0.23840584, -2.8577223 ,  7.85611   ],
         [-0.53788286, -1.761594  , -0.07113881,  0.31122968]]

**test_cc_swiglu_alpha**

.. code-block:: text

    Node:
      SwiGLU(A, B) -> (Y)
      Attributes:
        alpha = 0.5

.. code-block:: text

    Inputs:
      A: shape=(2, 4), dtype=float32
        [[ 1. , -2. ,  3. ,  4. ],
         [-1. ,  2. , -3. ,  0.5]]
      B: shape=(2, 4), dtype=float32
        [[ 0.5,  1. , -1. ,  2. ],
         [ 2. , -1. ,  0.5,  1. ]]

    Outputs:
      Y: shape=(2, 4), dtype=float32
        [[ 0.31122968, -0.53788286, -2.4527233 ,  7.046376  ],
         [-0.75508136, -1.4621172 , -0.2736383 ,  0.28108826]]

**test_cc_swiglu_bfloat16**

.. code-block:: text

    Node:
      SwiGLU(A, B) -> (Y)

.. code-block:: text

    Inputs:
      A: shape=(2, 4), dtype=bfloat16
        [[1, -2, 3, 4],
         [-1, 2, -3, 0.5]]
      B: shape=(2, 4), dtype=bfloat16
        [[0.5, 1, -1, 2],
         [2, -1, 0.5, 1]]

    Outputs:
      Y: shape=(2, 4), dtype=bfloat16
        [[0.365234, -0.238281, -2.85938, 7.84375],
         [-0.539062, -1.75781, -0.0712891, 0.310547]]

**test_cc_swiglu_float16**

.. code-block:: text

    Node:
      SwiGLU(A, B) -> (Y)

.. code-block:: text

    Inputs:
      A: shape=(2, 4), dtype=float16
        [[ 1. , -2. ,  3. ,  4. ],
         [-1. ,  2. , -3. ,  0.5]]
      B: shape=(2, 4), dtype=float16
        [[ 0.5,  1. , -1. ,  2. ],
         [ 2. , -1. ,  0.5,  1. ]]

    Outputs:
      Y: shape=(2, 4), dtype=float16
        [[ 0.3655 , -0.2384 , -2.857  ,  7.855  ],
         [-0.538  , -1.762  , -0.07117,  0.3113 ]]