Backend test-case coverage#

This page describes how the test-case coverage of onnx-light is measured against the lightweight ONNX operator schemas (LightOpSchema).

What is measured#

The baseline is the set of lightweight operator schemas exposed by the C++ onnx_op extension (one schema per operator, latest opset version). For each operator, the union of allowed tensor types across all its type constraints defines a list of supported types. Every (domain, operator_name, type) triple is called a signature.

The covered signatures are the signatures for which at least one backend test case — collected via onnx_light.backend.test.case.base.collect_test_case() — exists whose single-node model uses that operator and that ONNX type on one of its typed graph inputs or outputs.

The coverage ratio is the number of covered signatures divided by the total number of signatures.

Computing the report#

The function onnx_light.backend.coverage.compute_test_case_coverage() collects every available test case (Python-registered cases from Base subclasses and the canonical C++ cases from lib_onnx_backend_test) and produces a CoverageReport:

<<<

from onnx_light.backend.coverage import compute_test_case_coverage

report = compute_test_case_coverage()
print(report)
print(f"ratio = {report.ratio:.3f}")
print(f"operators with no test case: {len(report.uncovered_operators)}")
for domain, name in report.uncovered_operators[:10]:
    print(f"  - {domain}::{name}")

>>>

    CoverageReport(total=887, covered=45, ratio=0.051, uncovered_operators=45)
    ratio = 0.051
    operators with no test case: 45
      - ai.onnx::ArgMax
      - ai.onnx::ArgMin
      - ai.onnx::Asin
      - ai.onnx::Asinh
      - ai.onnx::ConcatFromSequence
      - ai.onnx::Cos
      - ai.onnx::Cosh
      - ai.onnx::Equal
      - ai.onnx::GRU
      - ai.onnx::Gemm

The report exposes:

  • total_signatures — total number of (operator, type) signatures derived from the baseline schemas;

  • covered_signatures — number of those signatures covered by at least one test case;

  • ratio — coverage ratio in [0, 1];

  • operator_coverages — per-operator breakdown including supported_types, covered_types and missing_types;

  • uncovered_operators — list of (domain, operator_name) pairs that have no test case at all.

Listing operators without a test case#

The uncovered_operators attribute makes it straightforward to drive the prioritization of new backend test cases. It only contains operators that have zero test cases — operators partially covered (for example Abs for which only tensor(float) is exercised) are listed in operator_coverages with a non-empty missing_types property.

See also#