CPU benchmarks are sensitive to persistent worker pools, allocator reuse, lazy preparation, and runtime caches. The onnx-light dashboards therefore separate competing runtimes into global phases instead of alternating them for each test.
Models and inputs are prepared first. Every plain onnx-light
case runs before the process-wide onnx-light-cpu kernels are
registered. Every onnx-light-cpu case then runs before any
ONNX Runtime session is constructed. Sessions are released and garbage
collection runs between phases.
Each runtime keeps its documented default worker-spin policy. Changing the spin policy would change the runtime being measured; phase separation prevents a live session from one runtime perturbing another runtime's timed phase.
Warm-up calls are untimed. The benchmark requests two warm-up repetitions and ten measured repetitions per logical CPU. Each phase stops when its requested repetition count is reached or after one second of cumulative execution, whichever comes first. A call already in progress finishes normally. The dashboard records the actual sample count and reports the existing trimmed mean or median defined by each recorder.
Models and logical inputs are identical across backends. Published dashboard results come from shared CI machines and are diagnostic rather than stable performance gates. Runtime versions, SIMD selection, dates, and raw timing ranges should be considered before comparing snapshots.