CausalConvWithState#
Domain:
ai.onnxSince version: 27
Stateful causal 1D depthwise convolution.
Used by Gated DeltaNet (Qwen3.5) and Mamba (Jamba, FalconMamba) as a preprocessing step. Replaces the 3-op pattern (Concat + Conv + Slice) with a single fused operation.
The convolution is causal (looks only at current and past positions) and depthwise (each channel is convolved independently with its own kernel).
The input, weight, past_state, output, and present_state tensors are rank-3 with shape (batch_size, channels, length). The optional bias input is rank-1 with shape (channels). For higher-dimensional data, use Reshape nodes before and after this operator to pack extra dimensions into the batch or channel axis.
Weight layout: (channels, 1, k) for depthwise convolution. The carry state stores the last (k-1) positions for incremental decode.
The optional activation attribute supports fused SiLU/Swish activation.
Mathematical definition#
Let `X`` be the input of shape ``(B, C, L)``, ``W` the weight of shape
`(C, 1, k)``, ``b`` the optional bias of shape ``(C)``, and ``S_past` the
optional carry state of shape `(B, C, k - 1)`. Define the padded sequence
`Xpad`` of shape ``(B, C, L + k - 1)` as the concatenation along the last
axis of the carry state (or zero-padding when `past_state` is absent) and
the input:
.. code-block:: text
Xpad[b, c, t] = S_past[b, c, t] if 0 <= t < k - 1 and past_state is present Xpad[b, c, t] = 0 if 0 <= t < k - 1 and past_state is absent Xpad[b, c, t] = X[b, c, t - (k - 1)] if k - 1 <= t < L + k - 1
The convolution output is then:
.. code-block:: text
Y[b, c, l] = sum_{j=0}^{k-1} W[c, 0, j] * Xpad[b, c, l + j] for 0 <= l < L
with the per-channel bias added when provided:
.. code-block:: text
Y[b, c, l] += b[c]
When `activation`` is ``"silu"`` or ``"swish"` the fused activation is
applied element-wise:
.. code-block:: text
Y[b, c, l] = Y[b, c, l] * sigmoid(Y[b, c, l]) where sigmoid(x) = 1 / (1 + exp(-x))
The updated carry state collects the last `k - 1` positions of the padded
sequence so it can be fed back as `past_state` on the next call:
.. code-block:: text
present_state[b, c, j] = Xpad[b, c, L + j] for 0 <= j < k - 1
Inputs
input (T): Input tensor with shape (batch_size, channels, length). Channels-first layout.
weight (T): Depthwise convolution kernel with shape (channels, 1, k) where k is the kernel size. The middle dim of size 1 follows the ONNX
Convweight layout(M, C/group, k1, ..., kn): since this op is always depthwise,group = channels, soC/group = 1. Keeping this layout makes the weight tensor a drop-in for a depthwiseConv(group=channels)weight, soConvCausalConvWithStaterewrites require no reshape.bias (T): Optional per-channel bias with shape (channels).
past_state (T): Carry state from previous step with shape (batch_size, channels, k - 1). If not provided, padding is zero.
Outputs
output (T): Convolution output with same shape as input.
present_state (T): Updated carry state with shape (batch_size, channels, k - 1). Contains the last (k - 1) values of the effective padded/concatenated sequence along the causal axis, including any values from past_state or zero-padding when the current input is shorter than k - 1.
Attributes
activation (string): Fused activation function. One of: ‘silu’, ‘swish’, ‘none’. Default is ‘none’.
Type Constraints
T: Constrain input and output types to float tensors. Allowed types: tensor(bfloat16), tensor(float), tensor(float16).
Examples#
test_cc_causal_conv_with_state_b1_c1_degenerate
Node:
CausalConvWithState(input, weight, "", past_state) -> (output, present_state)
Inputs:
input: shape=(1, 1, 1), dtype=float32
[[[0.3]]]
weight: shape=(1, 1, 2), dtype=float32
[[[ 0.5 , -0.25]]]
past_state: shape=(1, 1, 1), dtype=float32
[[[0.7]]]
Outputs:
output: shape=(1, 1, 1), dtype=float32
[[[0.27499998]]]
present_state: shape=(1, 1, 1), dtype=float32
[[[0.3]]]
test_cc_causal_conv_with_state_basic
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[0. , 0.0125 , 0. , 0.0375 ],
[0.25 , 0.775 , 1.8499999, 2.025 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_decode_step
Node:
CausalConvWithState(input, weight, "", past_state) -> (output, present_state)
Inputs:
input: shape=(1, 2, 1), dtype=float32
[[[0.4],
[1.4]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
past_state: shape=(1, 2, 2), dtype=float32
[[[ 0.5, 0.6],
[-0.5, -0.6]]]
Outputs:
output: shape=(1, 2, 1), dtype=float32
[[[ 0.14999999],
[-0.45000002]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[ 0.6, 0.4],
[-0.6, 1.4]]]
test_cc_causal_conv_with_state_fp16
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Inputs:
input: shape=(1, 2, 4), dtype=float16
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float16
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
Outputs:
output: shape=(1, 2, 4), dtype=float16
[[[0. , 0.0125, 0. , 0.0375],
[0.25 , 0.775 , 1.85 , 2.025 ]]]
present_state: shape=(1, 2, 2), dtype=float16
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_kernel_size_one
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 1), dtype=float32
[[[ 0.5]],
[[-1. ]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[ 0. , 0.05, 0.1 , 0.15],
[-1. , -1.1 , -1.2 , -1.3 ]]]
present_state: shape=(1, 2, 0), dtype=float32
[]
test_cc_causal_conv_with_state_short_input_no_past_state
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Inputs:
input: shape=(1, 2, 1), dtype=float32
[[[0.4],
[1.4]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
Outputs:
output: shape=(1, 2, 1), dtype=float32
[[[0.05],
[0.35]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0. , 0.4],
[0. , 1.4]]]
test_cc_causal_conv_with_state_silu
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Attributes:
activation = "silu"
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[0. , 0.00628906, 0. , 0.01910152],
[0.14054413, 0.53056616, 1.5986351 , 1.7888793 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_silu_fp16
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Attributes:
activation = "silu"
Inputs:
input: shape=(1, 2, 4), dtype=float16
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float16
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
Outputs:
output: shape=(1, 2, 4), dtype=float16
[[[0. , 0.006287, 0. , 0.0191 ],
[0.1405 , 0.5303 , 1.599 , 1.788 ]]]
present_state: shape=(1, 2, 2), dtype=float16
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_silu_with_past_state
Node:
CausalConvWithState(input, weight, "", past_state) -> (output, present_state)
Attributes:
activation = "silu"
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
past_state: shape=(1, 2, 2), dtype=float32
[[[ 0.5, 0.6],
[-0.5, -0.6]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[ 0.05249792, 0.18046731, 0. , 0.01910152],
[-0.20122544, 0.09513676, 1.5986351 , 1.7888793 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_swish_alias
Node:
CausalConvWithState(input, weight) -> (output, present_state)
Attributes:
activation = "swish"
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[0. , 0.00628906, 0. , 0.01910152],
[0.14054413, 0.53056616, 1.5986351 , 1.7888793 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_with_bias
Node:
CausalConvWithState(input, weight, bias) -> (output, present_state)
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
bias: shape=(2,), dtype=float32
[ 0.1, -0.2]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[0.1 , 0.1125 , 0.1 , 0.13750002],
[0.05 , 0.57500005, 1.6500001 , 1.825 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_with_bias_and_past_state
Node:
CausalConvWithState(input, weight, bias, past_state) -> (output, present_state)
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
bias: shape=(2,), dtype=float32
[ 0.1, -0.2]
past_state: shape=(1, 2, 2), dtype=float32
[[[ 0.5, 0.6],
[-0.5, -0.6]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[ 0.19999999, 0.4125 , 0.1 , 0.13750002],
[-0.75 , -0.02500001, 1.6500001 , 1.825 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]
test_cc_causal_conv_with_state_with_past_state
Node:
CausalConvWithState(input, weight, "", past_state) -> (output, present_state)
Inputs:
input: shape=(1, 2, 4), dtype=float32
[[[0. , 0.1, 0.2, 0.3],
[1. , 1.1, 1.2, 1.3]]]
weight: shape=(2, 1, 3), dtype=float32
[[[ 0.5 , -0.25 , 0.125]],
[[ 1. , 0.5 , 0.25 ]]]
past_state: shape=(1, 2, 2), dtype=float32
[[[ 0.5, 0.6],
[-0.5, -0.6]]]
Outputs:
output: shape=(1, 2, 4), dtype=float32
[[[ 0.09999999, 0.3125 , 0. , 0.0375 ],
[-0.55 , 0.17499998, 1.8499999 , 2.025 ]]]
present_state: shape=(1, 2, 2), dtype=float32
[[[0.2, 0.3],
[1.2, 1.3]]]