Runtime test coverage (onnxruntime and shape_inference)#

This page reports, for every backend test case collected by onnx_light.backend.test.case.base.collect_test_case(), the outcome of three independent scenarios:

  • onnxruntime (CPU) — the model is executed with onnxruntime on the CPU execution provider. The cell shows the maximum absolute discrepancy between the reference outputs and the ORT outputs, colored green when it stays within the test-case atol/rtol tolerances and red otherwise (n/a when ORT cannot load or run the model — typically when the model uses an op from a domain ORT does not register, such as ai.onnx.preview or ai.onnx.preview.training).

  • static shape — the model is passed as-is to onnx_light.onnx_lib.shape_inference.infer_shapes(). A yes indicates shape inference completes without raising.

  • dynamic_shapes — every numeric input dimension is first replaced with a symbolic dim_param (identical numeric values share the same symbol), then shape inference is run on the resulting symbolic model.

Summary#

Global pass rates across all collected test cases:

Scenario

Passed

Total

Pass rate

onnxruntime (CPU)

97

105

92.4%

Static shape inference

105

105

100.0%

Dynamic shape inference

105

105

100.0%

Per-domain pass rates:

Domain

Tests

onnxruntime (CPU)

Static shape

Dynamic shapes

ai.onnx (default)

95

94 (98.9%)

95 (100.0%)

95 (100.0%)

ai.onnx.ml

4

3 (75.0%)

4 (100.0%)

4 (100.0%)

ai.onnx.preview

2

0 (0.0%)

2 (100.0%)

2 (100.0%)

ai.onnx.preview.training

4

0 (0.0%)

4 (100.0%)

4 (100.0%)

Per-test-case status#

Each tab below lists every backend test case from one domain. The tables are rendered with the sphinx-datatables extension, so they are interactive: use the search box to filter by op or test name and click on a column header to sort.

test op

test name

discrepancies (onnxruntime CPU)

static shape

dynamic_shapes

Abs

test_cc_abs

0.00e+00

yes

yes

Acos

test_cc_acos

1.19e-07

yes

yes

Acosh

test_cc_acosh

0.00e+00

yes

yes

Add

test_cc_add

0.00e+00

yes

yes

Add

test_cc_add_bcast

0.00e+00

yes

yes

And

test_and2d

0.00e+00

yes

yes

And

test_and3d

0.00e+00

yes

yes

And

test_and4d

0.00e+00

yes

yes

And

test_and_bcast3v1d

0.00e+00

yes

yes

And

test_and_bcast3v2d

0.00e+00

yes

yes

And

test_and_bcast4v2d

0.00e+00

yes

yes

And

test_and_bcast4v3d

0.00e+00

yes

yes

And

test_and_bcast4v4d

0.00e+00

yes

yes

And

test_cc_and

0.00e+00

yes

yes

And

test_cc_and_bcast

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_1d_default

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_ceil

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_ceil_last_window_starts_on_pad

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_default

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_pads

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_pads_count_include_pad

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_precomputed_pads

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_precomputed_pads_count_include_pad

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_precomputed_strides

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_2d_strides

0.00e+00

yes

yes

AveragePool

test_cc_averagepool_3d_default

0.00e+00

yes

yes

BlackmanWindow

test_cc_blackmanwindow

1.49e-08

yes

yes

BlackmanWindow

test_cc_blackmanwindow_symmetric

2.98e-08

yes

yes

Cast

test_cc_cast_DOUBLE_to_FLOAT

0.00e+00

yes

yes

Cast

test_cc_cast_FLOAT_to_DOUBLE

0.00e+00

yes

yes

Cast

test_cc_cast_FLOAT_to_INT32

0.00e+00

yes

yes

Cast

test_cc_cast_INT64_to_FLOAT

0.00e+00

yes

yes

Concat

test_cc_concat_2d_axis_0

0.00e+00

yes

yes

Concat

test_cc_concat_2d_axis_negative

0.00e+00

yes

yes

Constant

test_cc_constant

0.00e+00

yes

yes

Div

test_cc_div

0.00e+00

yes

yes

Div

test_cc_div_bcast

0.00e+00

yes

yes

Div

test_div

0.00e+00

yes

yes

Div

test_div_bcast

0.00e+00

yes

yes

Div

test_div_example

0.00e+00

yes

yes

Greater

test_cc_greater

0.00e+00

yes

yes

Greater

test_cc_greater_bcast

0.00e+00

yes

yes

Greater

test_greater

0.00e+00

yes

yes

Greater

test_greater_bcast

0.00e+00

yes

yes

If

test_cc_if

0.00e+00

yes

yes

If

test_cc_if_else

0.00e+00

yes

yes

Less

test_cc_less

0.00e+00

yes

yes

Less

test_cc_less_bcast

0.00e+00

yes

yes

Less

test_less

0.00e+00

yes

yes

Less

test_less_bcast

0.00e+00

yes

yes

Mul

test_cc_mul

0.00e+00

yes

yes

Mul

test_cc_mul_bcast

0.00e+00

yes

yes

Mul

test_mul

0.00e+00

yes

yes

Mul

test_mul_bcast

0.00e+00

yes

yes

Mul

test_mul_example

0.00e+00

yes

yes

Optional

test_cc_optional

0.00e+00

yes

yes

Or

test_cc_or

0.00e+00

yes

yes

Or

test_cc_or_bcast

0.00e+00

yes

yes

Or

test_or2d

0.00e+00

yes

yes

Or

test_or3d

0.00e+00

yes

yes

Or

test_or4d

0.00e+00

yes

yes

Or

test_or_bcast3v1d

0.00e+00

yes

yes

Or

test_or_bcast3v2d

0.00e+00

yes

yes

Or

test_or_bcast4v2d

0.00e+00

yes

yes

Or

test_or_bcast4v3d

0.00e+00

yes

yes

Or

test_or_bcast4v4d

0.00e+00

yes

yes

QuantizeLinear

test_cc_quantizelinear

0.00e+00

yes

yes

QuantizeLinear

test_cc_quantizelinear_int8

0.00e+00

yes

yes

ReduceSum

test_cc_reducesum_default_axes_keepdims

0.00e+00

yes

yes

ReduceSum

test_cc_reducesum_do_not_keepdims

0.00e+00

yes

yes

ReduceSum

test_cc_reducesum_negative_axes_keepdims

0.00e+00

yes

yes

RoiAlign

test_cc_roialign

0.00e+00

yes

yes

RoiAlign

test_cc_roialign_max

5.69e-01

yes

yes

SequenceConstruct

test_cc_sequence_construct

0.00e+00

yes

yes

SequenceConstruct

test_cc_sequence_construct_int64_single

0.00e+00

yes

yes

StringConcat

test_cc_string_concat

0.00e+00

yes

yes

StringConcat

test_cc_string_concat_bcast

0.00e+00

yes

yes

StringConcat

test_cc_string_concat_chained_3in_1out

0.00e+00

yes

yes

StringConcat

test_cc_string_concat_utf8

0.00e+00

yes

yes

StringConcat

test_cc_string_concat_zero_dimensional

0.00e+00

yes

yes

Sub

test_cc_sub

0.00e+00

yes

yes

Sub

test_cc_sub_bcast

0.00e+00

yes

yes

Sub

test_sub

0.00e+00

yes

yes

Sub

test_sub_bcast

0.00e+00

yes

yes

Sub

test_sub_example

0.00e+00

yes

yes

Xor

test_cc_xor

0.00e+00

yes

yes

Xor

test_cc_xor_bcast

0.00e+00

yes

yes

Xor

test_xor2d

0.00e+00

yes

yes

Xor

test_xor3d

0.00e+00

yes

yes

Xor

test_xor4d

0.00e+00

yes

yes

Xor

test_xor_bcast3v1d

0.00e+00

yes

yes

Xor

test_xor_bcast3v2d

0.00e+00

yes

yes

Xor

test_xor_bcast4v2d

0.00e+00

yes

yes

Xor

test_xor_bcast4v3d

0.00e+00

yes

yes

Xor

test_xor_bcast4v4d

0.00e+00

yes

yes

test op

test name

discrepancies (onnxruntime CPU)

static shape

dynamic_shapes

Binarizer

test_cc_binarizer_float

0.00e+00

yes

yes

Binarizer

test_cc_binarizer_int64

n/a

yes

yes

LabelEncoder

test_cc_label_encoder_float_to_int64

0.00e+00

yes

yes

LabelEncoder

test_cc_label_encoder_int64_to_float

0.00e+00

yes

yes

test op

test name

discrepancies (onnxruntime CPU)

static shape

dynamic_shapes

FlexAttention

test_cc_flex_attention_basic

n/a

yes

yes

FlexAttention

test_cc_flex_attention_gqa

n/a

yes

yes

test op

test name

discrepancies (onnxruntime CPU)

static shape

dynamic_shapes

Adam

test_adam

n/a

yes

yes

Adam

test_adam_multiple

n/a

yes

yes

Adam

test_cc_adam_multiple

n/a

yes

yes

Adam

test_cc_adam_single

n/a

yes

yes

See also#