onnx_light.backend.coverage#

Test-case coverage report against the light ONNX operator schemas.

This module measures how well the backend test cases collected by onnx_light.backend.test.case.base.collect_test_case() exercise the ONNX operators described by the lightweight schemas exposed in the C++ onnx_op extension (LightOpSchema).

The baseline is the set of light op schemas: for every supported operator and each tensor type it accepts, one signature ((domain, op_name, type)) is counted. A signature is considered covered when at least one collected test case instantiates a model whose single node has that op_type and uses that ONNX type for one of its (typed) graph inputs or outputs.

The result is reported by compute_test_case_coverage() as a CoverageReport:

  • CoverageReport.total_signatures — total number of (op, type) signatures to cover (the baseline);

  • CoverageReport.covered_signatures — signatures actually covered by a test case;

  • CoverageReport.ratio — coverage ratio in [0, 1];

  • CoverageReport.uncovered_operators — list of (domain, op_name) pairs for which no test case is registered at all;

  • CoverageReport.operator_coverages — per-operator breakdown (OperatorCoverage).

class onnx_light.backend.coverage.CoverageReport(total_signatures: int, covered_signatures: int, operator_coverages: list[OperatorCoverage], uncovered_operators: list[tuple[str, str]])#

Aggregated test-case coverage report.

Parameters:

  • total_signatures – Total number of (op, type) signatures from the baseline (sum of supported_types over every operator).

  • covered_signatures – Number of signatures covered by at least one test case.

  • operator_coverages – Per-operator breakdown, sorted by (domain, name).

  • uncovered_operators – (domain, op_name) pairs of operators that have no test case at all (a strict subset of operator_coverages entries whose covered == 0).

property ratio: float#

Coverage ratio in [0, 1] (1 if total_signatures is 0).

class onnx_light.backend.coverage.OperatorCoverage(domain: str, name: str, supported_types: list[str] = <factory>, covered_types: list[str] = <factory>)#

Per-operator coverage entry of a CoverageReport.

Parameters:

  • domain – Operator domain (e.g. "ai.onnx").

  • name – Operator name (e.g. "Add").

  • supported_types – ONNX type strings the operator accepts (union of all its type constraints in the latest schema version).

  • covered_types – Subset of supported_types for which at least one test case uses that type on a graph input or output.

property covered: int#

Number of supported types covered by at least one test case.

property missing_types: list[str]#

ONNX type strings supported by the operator but never exercised.

property ratio: float#

Coverage ratio for this operator in [0, 1] (1 if total is 0).

property total: int#

Number of supported types (baseline for this operator).

onnx_light.backend.coverage.compute_test_case_coverage(test_cases: Mapping[str, Any] | Iterable[Any] | None = None) → CoverageReport#

Computes the backend test-case coverage report.

The baseline is the set of light ONNX operator schemas (one per operator, latest opset version). For every operator, each of the tensor types it accepts contributes one signature to the baseline. The coverage is the number of those (operator, type) signatures for which at least one collected test case exists that uses the operator and that type.

Parameters:

test_cases – Optional override for the test cases to score. Accepts the mapping returned by onnx_light.backend.test.case.base.collect_test_case() or any iterable of TestCase objects. When None (the default) the test cases are collected via collect_test_case().

Returns:

A CoverageReport describing the coverage.