onnx_io_policy.h

Adaptive external-data I/O policy: lazy worker creation, calibrated block size and worker count, and a lightweight trace of the resolved policy.

This is Native PR01 of :ref:l-next-steps-native-fast-loading-completion. Parallel reads/writes of external tensor data already create their worker threads lazily (see :cpp:class:ONNX_LIGHT_NAMESPACE::utils::ThreadPool, which only spawns OS threads on the first submitted task), so a metadata-only parse (ParseOptions::skip_raw_data) or a wholly borrowed/mapped parse (ParseOptions::no_copy or TwoFilesStream::use_mmap_weights) never pays for a worker pool because no delayed block is ever submitted. What was missing is (1) a minimum block size below which a read/write is kept on the calling thread instead of being queued (ParseOptions::min_parallel_block_size and SerializeOptions::min_parallel_block_size were declared but never enforced), and (2) an automatic worker-count/block-size choice that adapts to the storage kind instead of a blind std::thread::hardware_concurrency() guess. This header adds both.

namespace onnx_light

Alias that makes onnx-light headers compatible with code that references ONNX_LIGHT_NAMESPACE (the macro used in the standard onnx package).

Set to ONNX_LIGHT_NAMESPACE so both names resolve to the same namespace.

Symbol-visibility attribute for the public onnx-light C++ API.

Maps the upstream compatibility macro to onnx-light’s explicit proto ABI annotation. This keeps declarations from vendored ONNX headers visible when lib_onnx_proto uses hidden visibility by default.

Namespace alias so that ONNX C++ code (and consumers such as onnxruntime) that refers to the literal onnx namespace — rather than the ONNX_NAMESPACE macro — resolves to the onnx-light namespace. The standard onnx package lives in namespace onnx; onnx-light uses onnx_light (via ONNX_LIGHT_NAMESPACE), so this alias keeps onnx-light a true drop-in. It is only introduced when the onnx-light namespace differs from onnx.

namespace utils

Enums

enum class IOStorageKind : int32_t

Coarse classification of the storage backing an external-data read, used to calibrate the adaptive I/O policy.

Values:

enumerator kMmap

The tensor bytes are borrowed from a memory-mapped file (no_copy or file_load_mode=MMAP): bytes are resolved by lazy page faults, not by a worker pool.

enumerator kWarmPageCache

The file is resident (or mostly resident) in the OS page cache: reads are memory-bandwidth bound rather than I/O-latency bound, so more, smaller-block workers help.

enumerator kBufferedReads

The file’s residency could not be determined (unsupported platform, probe failure, or a stream that is not backed by a real file); a moderate, conservative default is used.

enumerator kColdStorage

The file is not resident in the OS page cache: reads are disk-latency bound, so fewer workers with larger blocks reduce seek overhead.

Functions

ONNX_LIGHT_PROTO_API IOPolicy ResolveIOPolicy (IOStorageKind kind, int64_t total_bytes, int32_t requested_num_threads, int64_t requested_min_block_size)

Resolves the worker count and minimum block size to use for one parallel I/O operation.

Parameters:

• kind – Storage kind backing the operation (see :cpp:enum:IOStorageKind).

• total_bytes – Total number of bytes the operation may transfer; used to avoid requesting more workers than there is useful work for.

• requested_num_threads – Caller request, using the same convention as ParseOptions::num_threads / SerializeOptions::num_threads: 1 forces serial execution, > 1 forces exactly that many workers, and <= 0 asks for an automatic, storage-aware choice.

• requested_min_block_size – Caller-provided minimum block size, or 0 to use the storage-aware default.

Returns:

The resolved :cpp:class:IOPolicy. kind == kMmap always resolves to zero workers: mapped bytes are resolved through page faults, not a worker pool.

ONNX_LIGHT_PROTO_API IOStorageKind DetectIOStorageKind (const std::string &file_path)

Best-effort classification of the storage backing file_path as warm (resident in the OS page cache) or cold. Samples at most the first 64 MiB of the file with a PROT_NONE mapping and queries per-page residency (mincore); only implemented on Linux today.

Returns:

:cpp:enumerator:IOStorageKind::kWarmPageCache or :cpp:enumerator:IOStorageKind::kColdStorage when residency could be sampled, or :cpp:enumerator:IOStorageKind::kBufferedReads when the file could not be opened, probed (non-Linux platform), or is empty.

struct IOPolicy

#include <onnx_io_policy.h>

Resolved worker count and minimum block size for one parallel I/O operation.

Public Members

int32_t workers = 1

Number of worker threads to request from the stream’s thread pool. 0 means “do not start a thread pool”; every read/write happens on the calling thread.

int64_t min_block_size = 0

Minimum block size (bytes) a single read/write must reach to be submitted to the thread pool; smaller blocks are processed on the calling thread to avoid thread-pool overhead.

struct IOPolicyTrace

#include <onnx_io_policy.h>

Trace of the policy actually applied to one parse or serialize call, plus the observed byte/fault counters. Left at its default (all zero) when policy tracing was not requested.

Public Members

IOStorageKind storage_kind = IOStorageKind::kBufferedReads

Storage kind detected (or assumed) when the policy was resolved.

int32_t resolved_workers = 0

Worker count actually used (0 means no thread pool was started).

int64_t resolved_min_block_size = 0

Minimum block size actually enforced.

int64_t physical_bytes = 0

Bytes physically present in the external-data source examined to resolve the policy (typically the weights file size).

int64_t bytes_in_flight = 0

Total bytes submitted to the thread pool as delayed blocks and awaited together, i.e. the peak number of bytes outstanding at once under the current submit-then-wait model.

uint64_t page_faults = 0

Number of memory pages touched by :cpp:var:ParseOptions::_touch_raw_data_pages, i.e. an upper bound on the lazy page faults a wholly mapped load can trigger. Zero when page touching was not requested.