Standalone C++ example: run every backend test through onnxruntime#

This page documents examples/run_backend_test_ort, a self-contained CMake project that demonstrates how to run every C++-generated backend test case (the same registry exposed by lib_onnx_backend_test and used by the Python reference backend tests) through a downloaded onnxruntime release, and verify that the runtime outputs match the expected outputs declared by each test case.

It is the C++ counterpart of unittests/backend/test_backend_with_onnxruntime.py and reuses a similar exclusion list for cases that ORT cannot run as-is (test_cc_roialign_max, test_cc_flex_attention_*, test_cc_adam_*), plus a few extras specific to this simplified harness: test_cc_acos / test_cc_acosh (stamped with opset 22, beyond ORT 1.19.x’s supported max of opset 21) and test_cc_sequence_construct* (model output is an Ort::Value sequence rather than a tensor, which the small comparator in main.cc does not handle).

Models stamped by lib_onnx_backend_test use the latest ONNX IR version (13). The downloaded onnxruntime release pins a lower maximum IR version (10 in 1.19.x), so the example downgrades model.ir_version to that maximum before serializing – the IR version is the serialization-format identifier and does not change the semantics of the embedded opsets. Pass -DONNX_LIGHT_RUN_BACKEND_TEST_ORT_IR_VERSION=<n> at CMake configure time to override the cap if you point the example at a newer onnxruntime release.

Why a pre-built onnxruntime release?#

Building onnxruntime from source is heavy (gigabytes of dependencies, long compile times, Python tooling) and not the point of this example. The official pre-built CPU release archives published at microsoft/onnxruntime already ship the C++ headers and the dynamic library, so this example just consumes one of those archives directly.

Step 1 – Install the onnx_light C++ library#

From the onnx-light repository root, build and install the static library and its public headers (the Python extension is not needed):

cmake -S . -B build-install \
      -DCMAKE_BUILD_TYPE=Release \
      -DONNX_LIGHT_BUILD_PYTHON=OFF \
      -DCMAKE_INSTALL_PREFIX=/usr/local
cmake --build  build-install
cmake --install build-install

Step 2 – Obtain a pre-built onnxruntime CPU release#

You have two options:

  • Option A – let CMake download it for you. The example ships with cmake/FindOrt.cmake (a trimmed copy of onnx-extended’s FindOrt.cmake) that downloads the prebuilt release via FetchContent. Pick the version via -DORT_VERSION=<x.y.z> at configure time (default: 1.19.2). Skip to Step 3 – Option B.

  • Option B – point at an existing extracted release. Pick the archive that matches your platform on microsoft/onnxruntime, extract it, and remember the path – for example /opt/onnxruntime-linux-x64-1.19.2. The directory must contain include/onnxruntime_cxx_api.h and lib/libonnxruntime.so.* (onnxruntime.dll / onnxruntime.lib on Windows, libonnxruntime.dylib on macOS).

Step 3 – Build the example#

Option A – point at an existing extracted release:

cmake -S examples/run_backend_test_ort -B build-run-backend-test-ort \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_PREFIX_PATH=/usr/local \
      -DONNXRUNTIME_ROOT_DIR=/opt/onnxruntime-linux-x64-1.19.2
cmake --build build-run-backend-test-ort

Option B – let CMake download the release:

cmake -S examples/run_backend_test_ort -B build-run-backend-test-ort \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_PREFIX_PATH=/usr/local \
      -DORT_VERSION=1.19.2
cmake --build build-run-backend-test-ort

Step 4 – Run the example#

LD_LIBRARY_PATH=/opt/onnxruntime-linux-x64-1.19.2/lib \
    ./build-run-backend-test-ort/run_backend_test_ort

The binary accepts an optional positional argument that is treated as a regular expression filter on the test case name – handy for narrowing down a single case:

LD_LIBRARY_PATH=/opt/onnxruntime-linux-x64-1.19.2/lib \
    ./build-run-backend-test-ort/run_backend_test_ort '^test_cc_add'

One-shot script#

To download onnxruntime, install onnx_light, and build the example in one go:

bash examples/run_backend_test_ort/build.sh

On Windows:

examples\run_backend_test_ort\build.bat

Set ONNXRUNTIME_VERSION=<x.y.z> to pick a release tag, or ONNXRUNTIME_ROOT_DIR=/path/to/extracted/release to reuse an already downloaded archive.