ORT flatbuffer format: parallelization and alignment#

onnxruntime defines a compact FlatBuffers serialization (.ort) of an ONNX model. Unlike the protobuf wire format used by .onnx files (see Protobuf format applied to ONNX), a flatbuffer is a single, flat, contiguous byte buffer in which every table and vector is addressed through relative offsets, so it can be memory-mapped and read without an up-front parsing pass.

The C++ reader and writer for SerializeFormat::kOrtFlatbuffers are not implemented yet (calls raise RuntimeError — see html_theme.sidebar_secondary.remove). Before implementing them, two properties that onnx_light already provides for the .onnx path had to be checked against the format itself:

  • parallelization — can the large tensor payloads be read or written by a thread pool, the way the protobuf StringStream parser parallelizes large raw_data blocks (ParseOptions::is_parallel, StartThreadPool / WaitForDelayedBlock)?

  • alignment — can a tensor’s bytes be placed on a power-of-two boundary (element-size, SIMD, or page) so that a downstream consumer can mmap or zero-copy them, the way SerializeOptions.alignment aligns external weights for .onnx (see ModelProto creation and no-copy ownership and How to re-align external weights without loading them in memory)?

The relevant subset of the onnxruntime schema (ort.fbs) is:

table Tensor {
  name:string;
  doc_string:string;
  dims:[int64];
  data_type:TensorDataType;
  raw_data:[uint8];
  string_data:[string];
  // offset into an external data file so that data >2GB can be handled;
  // -1 when the bytes are stored inline in raw_data.
  external_data_offset:int64 = -1;
}

The bulk of a model’s bytes live in Tensor.raw_data (or, for tensors routed to a companion file, at Tensor.external_data_offset).

Parallelization#

Reading is parallelizable. A flatbuffer never has to be parsed: the table/vtable graph is walked through offsets, which is cheap and copies no payload. Each raw_data vector is an independent, length-prefixed, contiguous region at a known offset in the buffer. Once the offsets of the initializers have been collected — a lightweight walk — the large blocks are mutually disjoint and can be dispatched to a thread pool to be copied (or, in a no-copy load, page-touched) in parallel. This mirrors exactly how the protobuf parser submits large raw_data LEN fields to its worker pool, so the future ORT reader can reuse the same ParseOptions::is_parallel / StartThreadPool / WaitForDelayedBlock machinery. A pure zero-copy / mmap load needs no threads at all, because the bytes are already in memory at their final addresses.

Writing is fundamentally single-threaded. flatbuffers::FlatBufferBuilder builds the buffer bottom-up through a single growing cursor: every child (vector or table) must be finished before the parent table that references it is started. There is no API to append two independent raw_data vectors to the same buffer from two threads. The parallelism available to the writer is therefore limited to:

  • preparing each tensor’s bytes (consolidation, dtype conversion, copying) on worker threads and then feeding the already-materialized spans to the single-threaded builder; and

  • routing large tensors to the companion external file through external_data_offset, where the external writer can stream bytes in parallel exactly like the .onnx external-data path.

Alignment#

Inline ``raw_data`` is not aligned beyond 4 bytes. FlatBuffers aligns a vector to the size of its element type; raw_data is declared [uint8], so its elements are 1-byte aligned and the vector’s data start is only guaranteed to follow its 32-bit length prefix (4-byte alignment). The schema does not apply the force_align attribute to raw_data, so float or double weights stored inline are not aligned to their natural element size, let alone to a SIMD (e.g. 64-byte) or page (e.g. 4096-byte) boundary. A consumer that needs aligned access to inline raw_data must therefore tolerate unaligned reads or copy the bytes out; the inline path cannot offer the mmap-friendly zero-copy guarantee that .onnx external data provides.

The external-data path can be aligned. Tensor.external_data_offset lets a tensor’s bytes live in a companion file at a writer-chosen offset. That offset is controlled entirely by the external writer, which can zero-pad to any power-of-two boundary — the same technique SerializeOptions.alignment and onnx_light.onnx.align_external_data_streaming() already use for .onnx external weights. Alignment for .ort is therefore achievable, but only through external_data_offset, not for bytes embedded inside the flatbuffer.

Summary#

Property

Inline raw_data

External (external_data_offset)

Parallel read

Yes — disjoint offset-addressed blocks, reuse the thread pool

Yes — independent file regions

Parallel write

No — FlatBufferBuilder has a single cursor

Yes — external writer can stream in parallel

Alignment

No — [uint8] is 1-byte aligned, no force_align

Yes — writer pads each offset to the requested boundary

Consequences for the (future) onnx-light implementation:

  • the reader can reuse the existing parallel-block machinery for raw_data and offers true zero-copy / mmap loads;

  • the writer is single-threaded at the flatbuffer-assembly step, so any parallelism and any alignment guarantee (to honour SerializeOptions.alignment) must come from routing large or alignment-sensitive tensors through external_data_offset, mirroring the .onnx external-data path.

See also#