How to save a model in the ORT flatbuffer format#

onnxruntime defines a compact flatbuffer serialization (.ort) commonly used in size-constrained deployments because it can be memory-mapped directly into the runtime and avoids the protobuf parsing step.

onnx-light exposes the format through onnx_light.onnx.SerializeFormat (value ORT_FLATBUFFERS). The reader and writer for that format are not implemented in the C++ core yet: calls today raise RuntimeError. Until they land, use onnxruntime itself to produce the .ort file.

Convert an .onnx file to .ort with onnxruntime#

import onnxruntime as ort

so = ort.SessionOptions()
# Keep the graph structurally identical to the input model.
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL
so.optimized_model_filepath = "model.ort"
so.add_session_config_entry("session.save_model_format", "ORT")

# Creating the session triggers the optimized-model dump in ORT format.
ort.InferenceSession("model.onnx", so, providers=["CPUExecutionProvider"])

Planned onnx-light API#

Once the C++ writer ships, the one-liner is the same in Python and C++: flip SerializeOptions::format to ORT_FLATBUFFERS and serialize as usual.

import onnx_light.onnx as onnxl

sopts = onnxl.SerializeOptions()
sopts.format = onnxl.SerializeFormat.ORT_FLATBUFFERS
model.SerializeToFile("model.ort", sopts)

#include "onnx.h"
#include "onnx_helper.h"
#include "stream.h"

onnx::SerializeOptions options;
options.format = onnx::SerializeFormat::kOrtFlatbuffers;
onnx::utils::FileWriteStream stream("model.ort");
onnx::SerializeModelProtoToStream(model, stream, options);

See also#