CompiledTensorProto#

Date:

2026-08

discussion

Motivation#

A runtime may transform an initializer into a device-specific packed representation. Persisting that representation avoids repeating an expensive prepacking step when the same model is loaded again.

A compiled tensor is a cache, not a new tensor semantics. The graph continues to reference the original initializer, which remains the portable fallback. The cached bytes use Structured views over byte buffers and are ignored when the current runtime or device is incompatible.

Stable contract#

message CompiledTensorProto {
    string source_name = 1;        // original graph initializer
    StructProto value = 2;   // prepacked physical representation
    int32 device = 3;              // index into ModelProto.devices
    bytes source_digest = 4;       // digest of the canonical source tensor
    string digest_algorithm = 5;   // for example "blake3"
    repeated StringStringEntryProto metadata_props = 6;
}

value is always an exact structured value: it uses either a non-negative model-level type index or an inline concrete struct_type. An unconstrained static type is not valid here.

source_name refers to one initializer in ModelProto.graph. The first version does not address initializers owned by nested subgraphs. source_digest prevents stale compiled data from being used after that initializer changes. The digest algorithm is explicit so the cache format does not depend on one hard-coded hash implementation. The digest covers the element type, dimensions, and canonical tensor content, but not the tensor name, external-data location, or storage method.

DeviceProto#

The device descriptor identifies compatibility, not only a device ordinal:

message DeviceProto {
    string type = 1;              // "cpu", "cuda", "rocm", ...
    optional int32 index = 2;     // exact ordinal only when required
    string architecture = 3;      // "x86_64-avx2", "sm_80", ...
    string runtime = 4;           // producer/runtime domain
    string runtime_version = 5;   // compatible runtime version
    repeated StringStringEntryProto metadata_props = 6;
}

architecture records the instruction-set or accelerator compatibility needed by the packed representation. runtime and runtime_version identify the implementation ABI that interprets the type. Additional compatibility keys may be stored in metadata_props.

ModelProto extension#

message ModelProto {
    ...
    repeated StructTypeProto struct_types = <N>;
    repeated DeviceProto devices = <N+1>;
    repeated CompiledTensorProto compiled_tensors = <N+2>;
}

Several compiled entries may reference the same source_name for different architectures, runtimes, or packing strategies. Their StructTypeProto.name and metadata distinguish the physical formats.

Loading rules#

A runtime uses a compiled entry only when all of the following hold:

  • source_name resolves to exactly one initializer;

  • source_digest matches the canonical initializer content;

  • device is an in-range model-level index;

  • device type, architecture, runtime, version, and required metadata are compatible;

  • value and its concrete StructTypeProto pass structural and payload-size validation;

  • the runtime recognizes that physical type and compiled-format version.

If a compatibility condition or digest comparison fails, the runtime treats the entry as a cache miss and rebuilds it from the original initializer. Invalid compiled data must never change graph results or make an otherwise valid portable model unloadable. Malformed indices, payloads, or digest declarations are checker errors; ordinary incompatibility or a stale digest is only a cache miss.

Quantized and tiled tensors#

No dependency on QuantizedTensorProto is needed. A quantized, tiled, or otherwise packed cache entry is represented by the same StructProto mechanism:

CompiledTensorProto {
    source_name: "decoder.layers.0.attn.weight"
    value: StructProto {
        type: 3                    // model-level packed CUDA type
        raw_data: ...
    }
    device: 1                      // e.g. CUDA sm_80 + runtime ABI
    source_digest: ...
    digest_algorithm: "blake3"
}

The referenced structured type describes the complete byte layout. Its optional decoder describes portable interpretation for inspectable formats. A runtime-specific prepack may omit the decoder when only the named runtime can consume it; the original initializer still guarantees portability.

Validation#

A checker validates:

  • unique device descriptors and valid device indices;

  • unique (source_name, device, physical type) cache keys;

  • source initializer existence;

  • non-empty digest and algorithm fields;

  • exact structured type resolution and payload size;

  • absence of unconstrained static structured types;

  • metadata keys are unique.

Digest comparison and runtime compatibility may be deferred until load time, but structural errors are rejected independently of hardware availability.

Relationship to other proposals#

CompiledTensorProto depends only on the stable physical representation in StructProto. The specialized hierarchy in Quantization may remain a format catalogue, but it is not a storage dependency. Proto inheritance is likewise unnecessary because the compiled value is composed from a StructProto rather than derived from it.