onnx_light.backend.runtime_coverage#

Runtime coverage report for the backend test cases.

This module compares every backend test case collected by onnx_light.backend.test.case.base.collect_test_case() against three independent runtime / static-analysis scenarios:

  • onnxruntime CPU — the model is executed with onnxruntime on the CPU execution provider and the maximum absolute discrepancy between the reference outputs and the ORT outputs is recorded;

  • static shape — the model is passed to onnx_light.onnx_lib.shape_inference.infer_shapes() with the original (numeric) input shapes;

  • dynamic shapes — every numeric input dimension is replaced with a symbolic dim_param (so identical numeric values share the same symbol) and shape inference is run on the resulting symbolic model.

The result is a RuntimeCoverageReport containing one TestCaseStatus per backend test case, plus aggregate statistics per domain and globally.

class onnx_light.backend.runtime_coverage.DomainSummary(domain: str, total: int = 0, onnxruntime_ok: int = 0, static_shape_ok: int = 0, dynamic_shapes_ok: int = 0)#

Aggregated pass counts for a single domain.

class onnx_light.backend.runtime_coverage.RuntimeCoverageReport(statuses: list[TestCaseStatus] = <factory>, summaries: dict[str, ~onnx_light.backend.runtime_coverage.DomainSummary]=<factory>, overall: DomainSummary = <factory>)#

Per-test-case runtime coverage report.

The statuses list contains one TestCaseStatus per collected test case. summaries groups the statuses by domain and counts how many test cases pass each scenario, and overall is the same aggregation across every test case.

class onnx_light.backend.runtime_coverage.TestCaseStatus(name: str, op_type: str, domain: str, onnxruntime_cpu: float | None, onnxruntime_error: str | None, static_shape: bool, static_shape_error: str | None, dynamic_shapes: bool, dynamic_shapes_error: str | None)#

Outcome of the three runtime scenarios for one backend test case.

dynamic_shapes: bool#

True if infer_shapes() succeeds on the symbolic-shape model.

onnxruntime_cpu: float | None#

Maximum absolute discrepancy between reference and ORT outputs, or None when ORT could not load / run the model (registered op missing, unsupported domain, …).

onnxruntime_error: str | None#

Error message returned by ORT when onnxruntime_cpu is None.

property onnxruntime_ok: bool#

True when ORT ran and outputs matched within tolerances.

static_shape: bool#

True if infer_shapes() succeeds on the original model.

onnx_light.backend.runtime_coverage.compute_runtime_coverage(test_cases: Iterable[TestCase] | None = None) → RuntimeCoverageReport#

Builds the runtime coverage report for every backend test case.

Parameters:

test_cases – Optional iterable of TestCase to evaluate. When None, collect_test_case() is invoked to gather every available test case.

Returns:

A populated RuntimeCoverageReport.

onnx_light.backend.runtime_coverage.render_rst_domain_summary(report: RuntimeCoverageReport) → str#

Renders one row per domain with its individual pass percentages.

onnx_light.backend.runtime_coverage.render_rst_domain_tabs(report: RuntimeCoverageReport, css_class: str | None = 'sphinx-datatable') → str#

Renders one tab-item per domain, each containing a coverage table.

onnx_light.backend.runtime_coverage.render_rst_summary(report: RuntimeCoverageReport) → str#

Renders the global pass-percentage summary as a reST list-table.

onnx_light.backend.runtime_coverage.render_rst_table_for_domain(report: RuntimeCoverageReport, domain: str, css_class: str | None = None, indent: str = '') → str#

Renders the per-test-case table for one domain as a reST list-table.

Parameters:

  • report – The runtime coverage report.

  • domain – ONNX operator domain to filter on (e.g. "" for the default ai.onnx domain).

  • css_class – When provided, a :class: option is emitted on the list-table directive. Pass "sphinx-datatable" to opt into the interactive DataTables widget.

  • indent – Optional whitespace prefix prepended to every output line — useful to inline the table inside a tab-item directive.