.. _op_ai_onnx_CausalConvWithState:

CausalConvWithState
===================

- **Domain**: ``ai.onnx``
- **Since version**: 27

Stateful causal 1D depthwise convolution.

Used by Gated DeltaNet (Qwen3.5) and Mamba (Jamba, FalconMamba) as a preprocessing step.
Replaces the 3-op pattern (Concat + Conv + Slice) with a single fused operation.

The convolution is causal (looks only at current and past positions) and depthwise
(each channel is convolved independently with its own kernel).

The input, weight, past_state, output, and present_state tensors are rank-3 with
shape (batch_size, channels, length). The optional bias input is rank-1 with
shape (channels). For higher-dimensional data, use Reshape nodes before and
after this operator to pack extra dimensions into the batch or channel axis.

Weight layout: (channels, 1, k) for depthwise convolution.
The carry state stores the last (k-1) positions for incremental decode.

The optional activation attribute supports fused SiLU/Swish activation.

Mathematical definition
~~~~~~~~~~~~~~~~~~~~~~~

Let ```X```` be the input of shape ````(B, C, L)````, ````W``` the weight of shape
```(C, 1, k)````, ````b```` the optional bias of shape ````(C)````, and ````S_past``` the
optional carry state of shape ```(B, C, k - 1)```. Define the padded sequence
```Xpad```` of shape ````(B, C, L + k - 1)``` as the concatenation along the last
axis of the carry state (or zero-padding when ```past_state``` is absent) and
the input::

.. code-block:: text

    Xpad[b, c, t] = S_past[b, c, t]          if 0 <= t < k - 1 and past_state is present
    Xpad[b, c, t] = 0                        if 0 <= t < k - 1 and past_state is absent
    Xpad[b, c, t] = X[b, c, t - (k - 1)]     if k - 1 <= t < L + k - 1

The convolution output is then::

.. code-block:: text

    Y[b, c, l] = sum_{j=0}^{k-1} W[c, 0, j] * Xpad[b, c, l + j]    for 0 <= l < L

with the per-channel bias added when provided::

.. code-block:: text

    Y[b, c, l] += b[c]

When ```activation```` is ````"silu"```` or ````"swish"``` the fused activation is
applied element-wise::

.. code-block:: text

    Y[b, c, l] = Y[b, c, l] * sigmoid(Y[b, c, l])      where sigmoid(x) = 1 / (1 + exp(-x))

The updated carry state collects the last ```k - 1``` positions of the padded
sequence so it can be fed back as ```past_state``` on the next call::

.. code-block:: text

    present_state[b, c, j] = Xpad[b, c, L + j]         for 0 <= j < k - 1


**Inputs**

- **input** (*T*): Input tensor with shape (batch_size, channels, length). Channels-first layout.
- **weight** (*T*): Depthwise convolution kernel with shape (channels, 1, k) where k is the kernel size. The middle dim of size 1 follows the ONNX ``Conv`` weight layout ``(M, C/group, k1, ..., kn)``: since this op is always depthwise, ``group = channels``, so ``C/group = 1``. Keeping this layout makes the weight tensor a drop-in for a depthwise ``Conv(group=channels)`` weight, so ``Conv``   ``CausalConvWithState`` rewrites require no reshape.
- **bias** (*T*): Optional per-channel bias with shape (channels).
- **past_state** (*T*): Carry state from previous step with shape (batch_size, channels, k - 1). If not provided, padding is zero.

**Outputs**

- **output** (*T*): Convolution output with same shape as input.
- **present_state** (*T*): Updated carry state with shape (batch_size, channels, k - 1). Contains the last (k - 1) values of the effective padded/concatenated sequence along the causal axis, including any values from past_state or zero-padding when the current input is shorter than k - 1.

**Attributes**

- **activation** (*string*): Fused activation function. One of: 'silu', 'swish', 'none'. Default is 'none'.

**Type Constraints**

- **T**: Constrain input and output types to float tensors.
  Allowed types: tensor(bfloat16), tensor(float), tensor(float16).

Examples
--------

**test_cc_causal_conv_with_state_b1_c1_degenerate**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight, "", past_state) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 1, 1), dtype=float32
        [[[0.3]]]
      weight: shape=(1, 1, 2), dtype=float32
        [[[ 0.5 , -0.25]]]
      past_state: shape=(1, 1, 1), dtype=float32
        [[[0.7]]]

    Outputs:
      output: shape=(1, 1, 1), dtype=float32
        [[[0.27499998]]]
      present_state: shape=(1, 1, 1), dtype=float32
        [[[0.3]]]

**test_cc_causal_conv_with_state_basic**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[0.       , 0.0125   , 0.       , 0.0375   ],
          [0.25     , 0.775    , 1.8499999, 2.025    ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_decode_step**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight, "", past_state) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 1), dtype=float32
        [[[0.4],
          [1.4]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]
      past_state: shape=(1, 2, 2), dtype=float32
        [[[ 0.5,  0.6],
          [-0.5, -0.6]]]

    Outputs:
      output: shape=(1, 2, 1), dtype=float32
        [[[ 0.14999999],
          [-0.45000002]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[ 0.6,  0.4],
          [-0.6,  1.4]]]

**test_cc_causal_conv_with_state_fp16**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float16
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float16
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float16
        [[[0.    , 0.0125, 0.    , 0.0375],
          [0.25  , 0.775 , 1.85  , 2.025 ]]]
      present_state: shape=(1, 2, 2), dtype=float16
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_kernel_size_one**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 1), dtype=float32
        [[[ 0.5]],
        
         [[-1. ]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[ 0.  ,  0.05,  0.1 ,  0.15],
          [-1.  , -1.1 , -1.2 , -1.3 ]]]
      present_state: shape=(1, 2, 0), dtype=float32
        []

**test_cc_causal_conv_with_state_short_input_no_past_state**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 1), dtype=float32
        [[[0.4],
          [1.4]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]

    Outputs:
      output: shape=(1, 2, 1), dtype=float32
        [[[0.05],
          [0.35]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0. , 0.4],
          [0. , 1.4]]]

**test_cc_causal_conv_with_state_silu**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)
      Attributes:
        activation = "silu"

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[0.        , 0.00628906, 0.        , 0.01910152],
          [0.14054413, 0.53056616, 1.5986351 , 1.7888793 ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_silu_fp16**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)
      Attributes:
        activation = "silu"

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float16
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float16
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float16
        [[[0.      , 0.006287, 0.      , 0.0191  ],
          [0.1405  , 0.5303  , 1.599   , 1.788   ]]]
      present_state: shape=(1, 2, 2), dtype=float16
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_silu_with_past_state**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight, "", past_state) -> (output, present_state)
      Attributes:
        activation = "silu"

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]
      past_state: shape=(1, 2, 2), dtype=float32
        [[[ 0.5,  0.6],
          [-0.5, -0.6]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[ 0.05249792,  0.18046731,  0.        ,  0.01910152],
          [-0.20122544,  0.09513676,  1.5986351 ,  1.7888793 ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_swish_alias**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight) -> (output, present_state)
      Attributes:
        activation = "swish"

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[0.        , 0.00628906, 0.        , 0.01910152],
          [0.14054413, 0.53056616, 1.5986351 , 1.7888793 ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_with_bias**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight, bias) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]
      bias: shape=(2,), dtype=float32
        [ 0.1, -0.2]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[0.1       , 0.1125    , 0.1       , 0.13750002],
          [0.05      , 0.57500005, 1.6500001 , 1.825     ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_with_bias_and_past_state**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight, bias, past_state) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]
      bias: shape=(2,), dtype=float32
        [ 0.1, -0.2]
      past_state: shape=(1, 2, 2), dtype=float32
        [[[ 0.5,  0.6],
          [-0.5, -0.6]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[ 0.19999999,  0.4125    ,  0.1       ,  0.13750002],
          [-0.75      , -0.02500001,  1.6500001 ,  1.825     ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]

**test_cc_causal_conv_with_state_with_past_state**

.. code-block:: text

    Node:
      CausalConvWithState(input, weight, "", past_state) -> (output, present_state)

.. code-block:: text

    Inputs:
      input: shape=(1, 2, 4), dtype=float32
        [[[0. , 0.1, 0.2, 0.3],
          [1. , 1.1, 1.2, 1.3]]]
      weight: shape=(2, 1, 3), dtype=float32
        [[[ 0.5  , -0.25 ,  0.125]],
        
         [[ 1.   ,  0.5  ,  0.25 ]]]
      past_state: shape=(1, 2, 2), dtype=float32
        [[[ 0.5,  0.6],
          [-0.5, -0.6]]]

    Outputs:
      output: shape=(1, 2, 4), dtype=float32
        [[[ 0.09999999,  0.3125    ,  0.        ,  0.0375    ],
          [-0.55      ,  0.17499998,  1.8499999 ,  2.025     ]]]
      present_state: shape=(1, 2, 2), dtype=float32
        [[[0.2, 0.3],
          [1.2, 1.3]]]