ONNX CI Pipelines

  • CI pipelines matrix:

    When it runs

    Config

    Test

    Linux-CI

    Every PR

    • Ubuntu-18.04
    • DEBUG=1 or 0
    • ONNX_USE_LITE_PROTO=ON or OFF
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_BUILD_TESTS=1
    • ONNX_WERROR=ON
    • ONNX_ML=1 or 0

    • ONNX C++ tests
    • Test flake8
    • Test doc generation
    • Test proto generation
    • mypy typecheck
    • Verify backend node data
    • Verify node test generation

    Windows-CI

    Every PR

    • vs2017-win2016
    • ONNX_USE_LITE_PROTO=ON
    • ONNX_USE_PROTOBUF_SHARED_LIBS=ON
    • ONNX_BUILD_TESTS=1
    • ONNX_WERROR=ON
    • ONNX_ML=1 or 0

    • Test building ONNX in conda environment
    • Test doc generation
    • Test proto generation
    • mypy typecheck

    Mac-CI

    Every PR

    • macOS-10.14
    • DEBUG=1
    • ONNX_USE_LITE_PROTO=ON or OFF
    • ONNX_ML=1 or 0
    • ONNX_BUILD_TESTS=1
    • ONNX_WERROR=ON

    • ONNX C++ tests
    • Test flake8
    • Test doc generation
    • Test proto generation

    Windows_No_Exception CI

    Every PR

    • vs2019-winlatest
    • ONNX_DISABLE_EXCEPTIONS=ON
    • ONNX_USE_LITE_PROTO=ON
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_MSVC_STATIC_RUNTIME=ON
    • ONNX_DISABLE_STATIC_REGISTRATION=ON or OFF

    • Only ONNX C++ tests
    • Test selective schema loading

    WindowsRelease

    • Main branch
    • Release branch
    • Weekly(1)

    • Latest Windows
    • x86 and x64
    • ONNX_USE_LITE_PROTO=ON
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1
    • ONNX_USE_MSVC_STATIC_RUNTIME=OFF

    • Release Windows wheel
    • Release onnx-weekly package
    • Verify backend node data
    • Verify node test generation
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version(2)
    • Verify ONNX with the latest ort-nightly(3).

    LinuxRelease_aarch64

    • Main branch
    • Release branch
    • Weekly

    • Latest manylinux2014_aarch64
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1

    • Release Linux aarch64 wheel
    • Release onnx-weekly package
    • Verify backend node data
    • Verify node test generation
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version
    • Verify ONNX with the latest ort-nightly.

    LinuxRelease_i686

    • Main branch
    • Release branch
    • Weekly

    • Latest manylinux2010_x86_64
    • Release Linux i686 wheel
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1

    • Release onnx-weekly package
    • Verify backend node data
    • Verify node test generation
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version

    LinuxRelease_x86_64

    • Main branch
    • Release branch
    • Weekly

    • Latest manylinux2014_aarch64
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1

    • Release Linux x86_64 wheel
    • Release onnx-weekly package
    • Test TEST_HUB=1(4)
    • Verify backend node data
    • Verify node test generation
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version
    • Verify ONNX with the latest ort-nightly.

    MacRelease

    • Main branch
    • Release branch
    • Weekly

    • macos-10.15
    • MACOSX_DEPLOYMENT_TARGET=10.12(5)
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1

    • Release Mac wheel
    • Release onnx-weekly package
    • Verify backend node data
    • Verify node test generation
    • Verify with different dependency versions - latest numpy version, latest and min supported protobuf version
    • Verify ONNX with the latest ort-nightly.
    • Test source distribution generation
    • Test build with source distribution
    • Release onnx-weekly source distribution

    Weekly CI with latest onnx.checker

    weekly(6)

    • macos-latest
    • MACOSX_DEPLOYMENT_TARGET=10.15
    • ONNX_USE_PROTOBUF_SHARED_LIBS=OFF
    • ONNX_ML=1

    • Test latest ONNX checker
    • Test latest ONNX shape inference
    • With all models from onnx/models(7)

    • (1) When the release CIs will run:

      • After a PR has been merged into main/rel-* branch

      • Run weekly (Sunday midnight) and release Python wheel to onnx-weekly package on TestPyPI.

      • Any PR targeting rel-* branch

      • To manually run them, add a PR label “run release CIs” (only maintainers have permission).

    • (2) Minimum supported versions are listed here.

    • (3) Test ONNX Python wheel with onnxruntime.InferenceSession from latest ONNXRuntime. Please note that ort-nightly does not support Linux-i686 and Windows-x86 thus their verification are skipped.

    • (4) TEST_HUB=1 will test onnx.hub by using this API to download an ONNX model from onnx/models. This test is restricted to only 1 pipeline for saving quota usage.

    • (5) Although the build envioronment is macos-10.15, use MACOSX_DEPLOYMENT_TARGET=10.12 and -p macosx_10_12_x86_64 to force the wheel to support 10.12+.

    • (6):

      • The ONNX Model Zoo test will run weekly (Sunday midnight)

      • To manually trigger it, add a PR label “test ONNX Model Zoo” (only maintainers have permission). Please note that it will need a lot of download bandwidth from onnx/models so use it with caution.

    • (7) Some old deprecated models (opset-1) are skipped.