GraphBuilder#
yobx.xbuilder.GraphBuilder simplifies the programmatic construction
and optimization of ONNX graphs. It is the primary tool used to convert a
torch.fx.Graph into a onnx.ModelProto, but it can equally
be used standalone to build or transform any ONNX graph from scratch.
Class Hierarchy#
GraphBuilder is composed of three
cooperative base classes:
_BuilderRuntime— evaluates small constant sub-expressions (e.g. the[0, 0, -1]passed to aReshapenode) so the builder can resolve-1to the correct symbolic formula and fold constants early._ShapeRuntime— handles value-as-shape tracking needed by operators such asShape,Gather,Concat, andSlicewhen their outputs feed directly into aReshape._InferenceRuntime— walks the graph node by node, dispatching each node to the matching per-operator handler inyobx.xshape.shape_type_computeso that shapes and types are tracked for every intermediate result.
Two helper classes round out the public API:
FunctionOptions— controls whether (and how) a sub-graph is exported as a reusable ONNX local function.OptimizationOptions— selects which optimization passes run insideto_onnx.
Protocol API#
GraphBuilder satisfies a hierarchy of
protocols defined in yobx.typing. Callers that do not need the
full concrete class should type-annotate against the narrowest protocol that
covers the methods they actually use:
Protocol |
Scope |
|---|---|
Core construction API: inputs, outputs, initializers, nodes,
opset registration, type/shape/sequence accessors, and
|
|
Extends the core protocol with |
|
Extends |
|
The read-only view of the graph exposed to pattern-optimization authors
inside |
The op property returns an
object that satisfies OpsetProtocol,
which resolves g.op.Add(x, y)-style attribute-access dispatch to ONNX
node creation.
Building a graph from scratch#
The simplest workflow is:
Construct a
GraphBuilderwith an opset version.Call
make_tensor_inputto declare each graph input.Call
make_node(or the short-handg.op.<OpType>(…)syntax) to add operators.Call
make_tensor_outputto declare each graph output.Call
to_onnxto obtain aonnx.ModelProto.
<<<
import numpy as np
import onnx
from yobx.helpers.onnx_helper import pretty_onnx
from yobx.xbuilder import GraphBuilder
TFLOAT = onnx.TensorProto.FLOAT
# 1. create builder targeting opset 18
g = GraphBuilder(18, ir_version=10)
# 2. declare inputs
g.make_tensor_input("X", TFLOAT, ("batch", "seq", 64))
g.make_tensor_input("W", TFLOAT, (64, 32))
# 3. add a MatMul node via the short-hand op accessor
result = g.op.MatMul("X", "W")
# 4. declare the output and export
g.make_tensor_output(
result, elem_type=TFLOAT, shape=("batch", "seq", 32), indexed=False
)
model = g.to_onnx()
print(f"nodes : {len(model.graph.node)}")
print(f"opset : {model.opset_import[0].version}")
print(f"output : {model.graph.output[0].name}")
print(pretty_onnx(model))
>>>
nodes : 1
opset : 18
output : _onx_matmul_X
opset: domain='' version=18
input: name='X' type=dtype('float32') shape=['batch', 'seq', 64]
input: name='W' type=dtype('float32') shape=[64, 32]
MatMul(X, W) -> _onx_matmul_X
output: name='_onx_matmul_X' type=dtype('float32') shape=['batch', 'seq', 32]
Loading an existing model#
Passing an existing onnx.ModelProto to the constructor loads it into
the builder so its nodes and initializers can be inspected, modified, or
re-optimized.
<<<
import onnx
import onnx.helper as oh
from yobx.xbuilder import GraphBuilder
TFLOAT = onnx.TensorProto.FLOAT
model = oh.make_model(
oh.make_graph(
[
oh.make_node("Add", ["X", "Y"], ["T"]),
oh.make_node("Relu", ["T"], ["Z"]),
],
"add_relu",
[
oh.make_tensor_value_info("X", TFLOAT, ["batch", 4]),
oh.make_tensor_value_info("Y", TFLOAT, ["batch", 4]),
],
[oh.make_tensor_value_info("Z", TFLOAT, ["batch", 4])],
),
opset_imports=[oh.make_opsetid("", 18)],
ir_version=10,
)
g = GraphBuilder(model)
print("input shapes:", {n: g.get_shape(n) for n in g.input_names})
print("nodes :", [node.op_type for node in g.nodes])
>>>
input shapes: {'X': ('batch', 4), 'Y': ('batch', 4)}
nodes : ['Add', 'Relu']
Initializers#
Initializers (model weights and constants) are added with
make_initializer.
The builder deduplicates small integer arrays automatically: if the same
value is added twice it returns the name of the first occurrence rather than
creating a duplicate node.
<<<
import numpy as np
import onnx
from yobx.xbuilder import GraphBuilder
TFLOAT = onnx.TensorProto.FLOAT
g = GraphBuilder(18, ir_version=10)
g.make_tensor_input("X", TFLOAT, ("batch", 64))
# Add a weight matrix as an initializer
W = np.random.randn(64, 32).astype(np.float32)
w_name = g.make_initializer("W", W, source="example")
result = g.op.MatMul("X", w_name)
g.make_tensor_output(result, elem_type=TFLOAT, shape=("batch", 32), indexed=False)
model = g.to_onnx()
print("initializer name :", list(g.initializers_dict)[0])
print("initializer shape:", list(g.initializers_dict.values())[0].shape)
>>>
initializer name : W
initializer shape: (64, 32)
Shape and type tracking#
GraphBuilder inherits the full
ShapeBuilder interface. Shapes and types
are registered for every intermediate result as nodes are added, and are used
during optimization and for populating value_info in the exported proto.
See Expected API.
Dynamic shapes#
When some input dimensions are unknown at graph-construction time, they are
represented as strings (e.g. "batch", "seq"). For graphs that are
later exported for dynamic-shape inference with torch.export, the builder
accepts a dynamic_shapes dictionary that maps input names to per-axis
dimension objects (torch.export.Dim or WrapDim).
register_dynamic_objects_from_shape
registers any string dimension names encountered in a shape so that they are
tracked as symbolic dimensions.
<<<
import onnx
from yobx.xbuilder import GraphBuilder
TFLOAT = onnx.TensorProto.FLOAT
g = GraphBuilder(18, ir_version=10)
g.make_tensor_input("X", TFLOAT, ("batch", "seq", 64))
g.make_tensor_input("Y", TFLOAT, ("batch", "seq", 64))
# symbolic dimensions are tracked automatically once shapes are set
result = g.op.Add("X", "Y")
g.make_tensor_output(
result, elem_type=TFLOAT, shape=("batch", "seq", 64), indexed=False
)
model = g.to_onnx()
out = model.graph.output[0]
dims = [
d.dim_param if d.dim_param else d.dim_value for d in out.type.tensor_type.shape.dim
]
print("output shape:", dims)
>>>
output shape: ['batch', 'seq', 64]
Optimizations#
to_onnx runs a sequence of
optimization passes by default. The set of passes is controlled by
OptimizationOptions.
Default passes (in order):
Pass |
Effect |
|---|---|
|
Remove nodes whose outputs are never consumed. |
|
Evaluate operators such as |
|
Remove |
|
Merge identical constant initializers into a single tensor, removing redundant copies. |
|
Apply user-supplied or built-in fusion patterns (e.g.
|
|
Reorder nodes to reduce peak memory by moving each |
<<<
import onnx
import onnx.helper as oh
from yobx.xbuilder import GraphBuilder, OptimizationOptions
TFLOAT = onnx.TensorProto.FLOAT
model = oh.make_model(
oh.make_graph(
[
oh.make_node("Identity", ["X"], ["X2"]),
oh.make_node("Relu", ["X2"], ["Z"]),
],
"id_relu",
[oh.make_tensor_value_info("X", TFLOAT, [None, 4])],
[oh.make_tensor_value_info("Z", TFLOAT, [None, 4])],
),
opset_imports=[oh.make_opsetid("", 18)],
ir_version=10,
)
opts = OptimizationOptions(remove_identity=True)
g = GraphBuilder(model, optimization_options=opts)
optimized = g.to_onnx()
print("nodes before:", len(model.graph.node))
print("nodes after :", len(optimized.graph.node))
>>>
nodes before: 2
nodes after : 1
Optimization report#
Passing return_optimize_report=True to
to_onnx makes the method return
a (model, stats) tuple instead of just the model. stats is a list of
dictionaries — one entry per optimization pass — that records how many nodes
were added or removed and how long each pass took.
Key |
Description |
|---|---|
|
Name of the optimization pass (e.g. |
|
Number of nodes added by this pass. |
|
Number of nodes removed by this pass. |
|
Wall-clock time spent in this pass (seconds). |
|
Iteration number (only for pattern-based passes). |
|
Sequential index of the match within the iteration (pattern passes). |
|
Number of times the pattern was matched (pattern passes). |
The list can be converted to a pandas.DataFrame for quick
exploration:
<<<
import pandas
import onnx
import onnx.helper as oh
from yobx.xbuilder import GraphBuilder, OptimizationOptions
TFLOAT = onnx.TensorProto.FLOAT
model = oh.make_model(
oh.make_graph(
[
oh.make_node("Identity", ["X"], ["X2"]),
oh.make_node("Transpose", ["X2"], ["T"], perm=[1, 0]),
oh.make_node("Transpose", ["T"], ["Z"], perm=[1, 0]),
],
"demo",
[oh.make_tensor_value_info("X", TFLOAT, [3, 4])],
[oh.make_tensor_value_info("Z", TFLOAT, [3, 4])],
),
opset_imports=[oh.make_opsetid("", 18)],
ir_version=10,
)
opts = OptimizationOptions(patterns="default")
g = GraphBuilder(model, infer_shapes_options=True, optimization_options=opts)
optimized = g.to_onnx(return_optimize_report=True)
df = pandas.DataFrame(optimized.report.stats)
# keep only rows that have numeric added/removed counts
df["added"] = df["added"].fillna(0).astype(int)
df["removed"] = df["removed"].fillna(0).astype(int)
print(df[["pattern", "added", "removed", "time_in"]].to_string(index=False))
print(f"\nnodes before: {len(model.graph.node)}")
print(f"nodes after : {len(optimized.graph.node)}")
>>>
pattern added removed time_in
dynamic_dimension_naming 0 0 1.304900e-05
check_A-dynamic_dimension_naming 0 0 8.503000e-06
check_A-opt-sub 0 0 7.572000e-06
remove_identity 1 2 2.535800e-05
check_remove_identity-0 0 0 5.828000e-06
remove_unused 0 0 1.261800e-05
check_remove_unused-1 0 0 5.058000e-06
constant_folding 0 0 6.219000e-06
apply_constant_folding_new_inits 0 0 NaN
check_constant_folding-2 0 0 4.727000e-06
remove_unused 0 0 9.654000e-06
check_remove_unused-3 0 0 4.676000e-06
patterns 0 1 3.597401e-03
check_pattern_00 0 0 8.633000e-06
match_BatchNormalizationPattern 0 0 4.657000e-06
match_BatchNormalizationTrainingPattern 0 0 2.814000e-06
match_CastPattern 0 0 2.063000e-06
match_CastCastPattern 0 0 1.993000e-06
match_ConcatGatherPattern 0 0 2.424000e-06
match_ConcatReshapePattern 0 0 3.075000e-06
match_ConvBiasNullPattern 0 0 2.003000e-06
match_PadConvPattern 0 0 1.842000e-06
match_ExpandPattern 0 0 1.823000e-06
match_ExpandUnsqueezeExpandPattern 0 0 2.263000e-06
match_GatherConcatPattern 0 0 2.023000e-06
match_GatherGatherPattern 0 0 1.772000e-06
match_GatherShapePattern 0 0 2.053000e-06
match_GeluPattern 0 0 8.910000e-07
match_IdentityPattern 0 0 1.659500e-05
match_LeakyReluPattern 0 0 7.263400e-04
match_MulUnsqueezeUnsqueezePattern 0 0 3.716000e-06
match_ReshapePattern 0 0 3.105000e-06
match_ReshapeSqueezePattern 0 0 3.175000e-06
match_ShapeBasedReshapeIsSqueezePattern 0 0 3.335000e-06
match_ShapeBasedStaticExpandPattern 0 0 2.133000e-06
match_ShapeBasedEditDistanceReshapePattern 0 0 2.163000e-06
match_ShapeBasedIdentityPattern 0 0 5.878000e-06
match_ShapedBasedReshapePattern 0 0 2.504000e-06
match_ShapeBasedSameChildrenPattern 0 0 2.324000e-06
match_ShapeBasedShapeShapeAddPattern 0 0 2.313000e-06
match_ShapeTransposePattern 0 0 2.023000e-06
match_UnsqueezeShapePattern 0 0 2.203000e-06
match_ReshapeReshapePattern 0 0 2.434000e-06
match_SameChildrenPattern 0 0 5.589000e-06
match_SameChildrenFromInputPattern 0 0 4.667000e-06
match_SoftmaxCrossEntropyLossCastPattern 0 0 1.247649e-03
match_SqueezeAddPattern 0 0 3.886000e-06
match_SqueezeBinaryUnsqueezePattern 0 0 2.374000e-06
match_SqueezeUnsqueezePattern 0 0 2.433000e-06
match_StaticConcatReshapePattern 0 0 2.724000e-06
match_SwapExpandReshapePattern 0 0 2.063000e-06
match_SwapExpandUnsqueezePattern 0 0 1.983000e-06
match_SwapUnaryPattern 0 0 1.417100e-05
match_SwapUnsqueezeTransposePattern 0 0 6.640000e-06
match_TransposeGatherPattern 0 0 2.694000e-06
match_TransposeReshapeTransposePattern 0 0 4.968000e-06
match_TransposeTransposePattern 0 0 2.442600e-05
match_UnsqueezeOrSqueezeReshapePattern 0 0 3.074000e-06
match_UnsqueezeReshapePattern 0 0 2.834000e-06
match_UnsqueezeUnsqueezePattern 0 0 2.434000e-06
match_FunctionAttentionPattern 0 0 2.413000e-06
match_FunctionAttentionGQAPattern 0 0 3.365000e-06
insert_and_remove_nodes 0 0 5.543100e-05
apply_TransposeTransposePattern 1 2 1.027620e-04
check_pattern_A10 0 0 7.210000e-07
check_pattern_A20 0 0 6.179000e-06
remove_duplicated_shape 0 0 1.803000e-06
check_pattern_BD0 0 0 4.446000e-06
remove_identity_nodes 0 0 1.027500e-05
check_pattern_BI0 0 0 3.785000e-06
remove_unused 0 0 7.812000e-06
check_pattern_BUS0 0 0 3.405000e-06
build_graph_for_pattern 0 0 6.610000e-06
iteration_0 0 0 2.372456e-03
match_BatchNormalizationPattern 0 0 2.283000e-06
match_BatchNormalizationTrainingPattern 0 0 1.783000e-06
match_CastPattern 0 0 1.542000e-06
match_CastCastPattern 0 0 1.332000e-06
match_ConcatGatherPattern 0 0 1.412000e-06
match_ConcatReshapePattern 0 0 1.963000e-06
match_ConvBiasNullPattern 0 0 1.522000e-06
match_PadConvPattern 0 0 1.452000e-06
match_ExpandPattern 0 0 1.281000e-06
match_ExpandUnsqueezeExpandPattern 0 0 1.452000e-06
match_GatherConcatPattern 0 0 1.382000e-06
match_GatherGatherPattern 0 0 1.542000e-06
match_GatherShapePattern 0 0 1.422000e-06
match_GeluPattern 0 0 6.109999e-07
match_IdentityPattern 0 0 1.842000e-06
match_LeakyReluPattern 0 0 3.925000e-06
match_MulUnsqueezeUnsqueezePattern 0 0 1.432000e-06
match_ReshapePattern 0 0 1.472000e-06
match_ReshapeSqueezePattern 0 0 1.653000e-06
match_ShapeBasedReshapeIsSqueezePattern 0 0 1.833000e-06
match_ShapeBasedStaticExpandPattern 0 0 1.532000e-06
match_ShapeBasedEditDistanceReshapePattern 0 0 1.372000e-06
match_ShapeBasedIdentityPattern 0 0 1.332000e-06
match_ShapedBasedReshapePattern 0 0 1.512000e-06
match_ShapeBasedSameChildrenPattern 0 0 1.452000e-06
match_ShapeBasedShapeShapeAddPattern 0 0 1.252000e-06
match_ShapeTransposePattern 0 0 1.312000e-06
match_UnsqueezeShapePattern 0 0 1.422000e-06
match_ReshapeReshapePattern 0 0 2.023000e-06
match_SameChildrenPattern 0 0 2.724000e-06
match_SameChildrenFromInputPattern 0 0 2.965000e-06
match_SoftmaxCrossEntropyLossCastPattern 0 0 2.974000e-06
match_SqueezeAddPattern 0 0 1.352000e-06
match_SqueezeBinaryUnsqueezePattern 0 0 1.632000e-06
match_SqueezeUnsqueezePattern 0 0 1.322000e-06
match_StaticConcatReshapePattern 0 0 1.502000e-06
match_SwapExpandReshapePattern 0 0 1.642000e-06
match_SwapExpandUnsqueezePattern 0 0 1.262000e-06
match_SwapUnaryPattern 0 0 1.222000e-06
match_SwapUnsqueezeTransposePattern 0 0 1.301000e-06
match_TransposeGatherPattern 0 0 1.322000e-06
match_TransposeReshapeTransposePattern 0 0 1.322000e-06
match_TransposeTransposePattern 0 0 1.372000e-06
match_UnsqueezeOrSqueezeReshapePattern 0 0 1.412000e-06
match_UnsqueezeReshapePattern 0 0 1.823000e-06
match_UnsqueezeUnsqueezePattern 0 0 1.563000e-06
match_FunctionAttentionPattern 0 0 1.402000e-06
match_FunctionAttentionGQAPattern 0 0 1.903000e-06
check_pattern_A20 0 0 5.008000e-06
remove_duplicated_shape 0 0 1.222000e-06
check_pattern_BD0 0 0 3.655000e-06
remove_identity_nodes 0 0 1.000500e-05
check_pattern_BI0 0 0 3.435000e-06
remove_unused 0 0 7.160000e-06
check_pattern_BUS0 0 0 3.335000e-06
build_graph_for_pattern 0 0 5.458000e-06
iteration_1 0 0 1.778920e-04
match_BatchNormalizationPattern 0 0 1.873000e-06
match_BatchNormalizationTrainingPattern 0 0 1.572000e-06
match_CastLayerNormalizationCastPattern 0 0 1.803000e-06
match_CastPattern 0 0 1.662000e-06
match_CastCastBinaryPattern 0 0 1.692000e-06
match_CastCastPattern 0 0 1.332000e-06
match_CastOpCastPattern 0 0 2.033000e-06
match_ClipClipPattern 0 0 1.953000e-06
match_ConcatEmptyPattern 0 0 1.803000e-06
match_ConcatGatherPattern 0 0 1.443000e-06
match_ConcatReshapePattern 0 0 1.552000e-06
match_ConcatTwiceUnaryPattern 0 0 2.373000e-06
match_ConstantToInitializerPattern 0 0 1.552000e-06
match_ConvBiasNullPattern 0 0 1.372000e-06
match_PadConvPattern 0 0 1.382000e-06
match_DropoutPattern 0 0 1.472000e-06
match_ExpandPattern 0 0 1.572000e-06
match_ExpandBroadcastPattern 0 0 1.643000e-06
match_ExpandSwapPattern 0 0 2.033000e-06
match_ExpandUnsqueezeExpandPattern 0 0 1.432000e-06
match_GatherConcatPattern 0 0 1.422000e-06
match_GatherGatherPattern 0 0 1.452000e-06
match_GathersSplitPattern 0 0 7.135500e-05
match_GatherShapePattern 0 0 1.672000e-06
match_GeluPattern 0 0 6.310000e-07
match_IdentityPattern 0 0 1.683000e-06
match_LayerNormalizationPattern 0 0 1.682000e-06
match_LayerNormalizationScalePattern 0 0 1.362000e-06
match_LeakyReluPattern 0 0 3.085000e-06
match_MaxReluPattern 0 0 1.603000e-06
match_MulMulMulScalarPattern 0 0 1.963000e-06
match_MulUnsqueezeUnsqueezePattern 0 0 1.272000e-06
match_NotNotPattern 0 0 1.442000e-06
match_NotWherePattern 0 0 1.582000e-06
match_ReduceArgTopKPattern 0 0 1.903000e-06
match_ReduceReshapePattern 0 0 1.823000e-06
match_ReduceSumNormalizePattern 0 0 1.552000e-06
match_ReshapePattern 0 0 1.592000e-06
match_ReshapeMatMulReshapePattern 0 0 1.752000e-06
match_Reshape2Of3Pattern 0 0 1.503000e-06
match_ReshapeReshapeBinaryPattern 0 0 1.732000e-06
match_ReshapeSqueezePattern 0 0 1.553000e-06
match_GemmTransposePattern 0 0 1.442000e-06
match_MatMulReshape2Of3Pattern 0 0 1.663000e-06
match_MulMulMatMulPattern 0 0 1.382000e-06
match_ShapeBasedReshapeIsSqueezePattern 0 0 1.452000e-06
match_ShapeBasedStaticExpandPattern 0 0 1.242000e-06
match_ShapeBasedConcatExpandPattern 0 0 1.873000e-06
match_ShapeBasedEditDistanceReshapePattern 0 0 1.552000e-06
match_ShapeBasedIdentityPattern 0 0 1.472000e-06
match_ShapeBasedExpandBroadcastPattern 0 0 1.833000e-06
match_ShapeBasedExpandBroadcastMatMulPattern 0 0 1.562000e-06
match_ShapeBasedExpandCastWhereSwapPattern 0 0 1.683000e-06
match_ShapeBasedExpandSwapPattern 0 0 1.953000e-06
match_ShapeBasedMatMulToMulPattern 0 0 1.713000e-06
match_ShapedBasedReshapePattern 0 0 1.603000e-06
match_ShapeBasedSameChildrenPattern 0 0 1.773000e-06
match_ShapeBasedShapeShapeAddPattern 0 0 1.272000e-06
match_ShapeTransposePattern 0 0 1.382000e-06
match_UnsqueezeShapePattern 0 0 1.322000e-06
match_ReshapeReshapePattern 0 0 1.562000e-06
match_RotaryEmbeddingPattern 0 0 1.723000e-06
match_SameChildrenPattern 0 0 2.513000e-06
match_SameChildrenFromInputPattern 0 0 2.954000e-06
match_SequenceConstructAtPattern 0 0 1.813000e-06
match_SplitToSequenceSequenceAtPattern 0 0 1.763000e-06
match_SliceSlicePattern 0 0 1.963000e-06
match_SlicesSplitPattern 0 0 1.362000e-06
match_SoftmaxCrossEntropyLossCastPattern 0 0 2.804000e-06
match_SplitConcatPattern 0 0 1.953000e-06
match_SqueezeAddPattern 0 0 1.382000e-06
match_SqueezeBinaryUnsqueezePattern 0 0 1.322000e-06
match_SqueezeUnsqueezePattern 0 0 1.472000e-06
match_StaticConcatReshapePattern 0 0 2.023000e-06
match_Sub1MulPattern 0 0 2.694000e-06
match_SwapExpandReshapePattern 0 0 1.292000e-06
match_SwapExpandUnsqueezePattern 0 0 1.302000e-06
match_SwapRangeAddScalarPattern 0 0 1.653000e-06
match_SwapUnaryPattern 0 0 1.322000e-06
match_SwapUnsqueezeTransposePattern 0 0 1.272000e-06
match_SwitchOrderBinaryPattern 0 0 1.683000e-06
match_SwitchReshapeActivationPattern 0 0 2.022000e-06
match_TransposeEqualReshapePattern 0 0 1.702000e-06
match_TransposeGatherPattern 0 0 1.493000e-06
match_TransposeMatMulPattern 0 0 1.702000e-06
match_TransposeReshapeMatMulPattern 0 0 1.372000e-06
match_TransposeReshapeTransposePattern 0 0 1.442000e-06
match_TransposeTransposePattern 0 0 1.402000e-06
match_UnsqueezeEqualPattern 0 0 1.472000e-06
match_UnsqueezeOrSqueezeReshapePattern 0 0 1.592000e-06
match_UnsqueezeReshapePattern 0 0 1.583000e-06
match_UnsqueezeUnsqueezePattern 0 0 1.382000e-06
match_WhereAddPattern 0 0 1.362000e-06
match_RotaryConcatPartPattern 0 0 1.863000e-06
match_FunctionAttentionPattern 0 0 1.543000e-06
match_FunctionAttentionGQAPattern 0 0 1.893000e-06
match_FunctionCausalMaskPattern 0 0 1.663000e-06
match_FunctionCausalMaskMulAddPattern 0 0 1.472000e-06
match_FunctionCosSinCachePattern 0 0 1.723000e-06
match_FunctionHalfRotaryEmbeddingPattern 0 0 1.592000e-06
match_RMSNormalizationPattern 0 0 1.593000e-06
match_RMSNormalizationMulPattern 0 0 1.572000e-06
check_pattern_A20 0 0 5.138000e-06
remove_duplicated_shape 0 0 1.281000e-06
check_pattern_BD0 0 0 3.846000e-06
remove_identity_nodes 0 0 9.394000e-06
check_pattern_BI0 0 0 3.234000e-06
remove_unused 0 0 7.221000e-06
check_pattern_BUS0 0 0 3.395000e-06
build_graph_for_pattern 0 0 5.508000e-06
iteration_2 0 0 3.788180e-04
match_BatchNormalizationPattern 0 0 1.843000e-06
match_BatchNormalizationTrainingPattern 0 0 1.602000e-06
match_CastLayerNormalizationCastPattern 0 0 1.422000e-06
match_CastPattern 0 0 1.432000e-06
match_CastCastBinaryPattern 0 0 1.553000e-06
match_CastCastPattern 0 0 1.342000e-06
match_CastOpCastPattern 0 0 1.713000e-06
match_ClipClipPattern 0 0 1.682000e-06
match_ConcatEmptyPattern 0 0 1.362000e-06
match_ConcatGatherPattern 0 0 1.622000e-06
match_ConcatReshapePattern 0 0 1.612000e-06
match_ConcatTwiceUnaryPattern 0 0 1.733000e-06
match_ConstantToInitializerPattern 0 0 1.442000e-06
match_ConvBiasNullPattern 0 0 1.362000e-06
match_PadConvPattern 0 0 1.362000e-06
match_DropoutPattern 0 0 1.362000e-06
match_ExpandPattern 0 0 1.352000e-06
match_ExpandBroadcastPattern 0 0 1.342000e-06
match_ExpandSwapPattern 0 0 1.322000e-06
match_ExpandUnsqueezeExpandPattern 0 0 1.392000e-06
match_GatherConcatPattern 0 0 1.392000e-06
match_GatherGatherPattern 0 0 1.332000e-06
match_GathersSplitPattern 0 0 1.302000e-06
match_GatherShapePattern 0 0 1.332000e-06
match_GeluPattern 0 0 7.310000e-07
match_IdentityPattern 0 0 1.472000e-06
match_LayerNormalizationPattern 0 0 1.392000e-06
match_LayerNormalizationScalePattern 0 0 1.202000e-06
match_LeakyReluPattern 0 0 3.314000e-06
match_MaxReluPattern 0 0 1.432000e-06
match_MulMulMulScalarPattern 0 0 1.492000e-06
match_MulUnsqueezeUnsqueezePattern 0 0 1.372000e-06
match_NotNotPattern 0 0 1.301000e-06
match_NotWherePattern 0 0 1.322000e-06
match_ReduceArgTopKPattern 0 0 1.512000e-06
match_ReduceReshapePattern 0 0 1.552000e-06
match_ReduceSumNormalizePattern 0 0 1.251000e-06
match_ReshapePattern 0 0 1.382000e-06
match_ReshapeMatMulReshapePattern 0 0 1.332000e-06
match_Reshape2Of3Pattern 0 0 1.342000e-06
match_ReshapeReshapeBinaryPattern 0 0 1.462000e-06
match_ReshapeSqueezePattern 0 0 1.643000e-06
match_GemmTransposePattern 0 0 1.482000e-06
match_MatMulReshape2Of3Pattern 0 0 1.402000e-06
match_MulMulMatMulPattern 0 0 1.382000e-06
match_ShapeBasedReshapeIsSqueezePattern 0 0 1.442000e-06
match_ShapeBasedStaticExpandPattern 0 0 1.382000e-06
match_ShapeBasedConcatExpandPattern 0 0 1.422000e-06
match_ShapeBasedEditDistanceReshapePattern 0 0 1.502000e-06
match_ShapeBasedIdentityPattern 0 0 1.452000e-06
match_ShapeBasedExpandBroadcastPattern 0 0 1.432000e-06
match_ShapeBasedExpandBroadcastMatMulPattern 0 0 1.292000e-06
match_ShapeBasedExpandCastWhereSwapPattern 0 0 1.342000e-06
match_ShapeBasedExpandSwapPattern 0 0 1.412000e-06
match_ShapeBasedMatMulToMulPattern 0 0 1.502000e-06
match_ShapedBasedReshapePattern 0 0 1.813000e-06
match_ShapeBasedSameChildrenPattern 0 0 1.422000e-06
match_ShapeBasedShapeShapeAddPattern 0 0 1.352000e-06
match_ShapeTransposePattern 0 0 1.342000e-06
match_UnsqueezeShapePattern 0 0 1.352000e-06
match_ReshapeReshapePattern 0 0 1.722000e-06
match_RotaryEmbeddingPattern 0 0 1.492000e-06
match_SameChildrenPattern 0 0 2.394000e-06
match_SameChildrenFromInputPattern 0 0 2.674000e-06
match_SequenceConstructAtPattern 0 0 1.472000e-06
match_SplitToSequenceSequenceAtPattern 0 0 1.312000e-06
match_SliceSlicePattern 0 0 1.262000e-06
match_SlicesSplitPattern 0 0 1.372000e-06
match_SoftmaxCrossEntropyLossCastPattern 0 0 2.714000e-06
match_SplitConcatPattern 0 0 1.402000e-06
match_SqueezeAddPattern 0 0 3.435000e-06
match_SqueezeBinaryUnsqueezePattern 0 0 1.352000e-06
match_SqueezeUnsqueezePattern 0 0 1.522000e-06
match_StaticConcatReshapePattern 0 0 1.442000e-06
match_Sub1MulPattern 0 0 1.602000e-06
match_SwapExpandReshapePattern 0 0 1.362000e-06
match_SwapExpandUnsqueezePattern 0 0 1.302000e-06
match_SwapRangeAddScalarPattern 0 0 1.302000e-06
match_SwapUnaryPattern 0 0 1.372000e-06
match_SwapUnsqueezeTransposePattern 0 0 1.292000e-06
match_SwitchOrderBinaryPattern 0 0 1.542000e-06
match_SwitchReshapeActivationPattern 0 0 1.472000e-06
match_TransposeEqualReshapePattern 0 0 1.452000e-06
match_TransposeGatherPattern 0 0 1.332000e-06
match_TransposeMatMulPattern 0 0 1.412000e-06
match_TransposeReshapeMatMulPattern 0 0 1.392000e-06
match_TransposeReshapeTransposePattern 0 0 1.402000e-06
match_TransposeTransposePattern 0 0 1.372000e-06
match_UnsqueezeEqualPattern 0 0 1.342000e-06
match_UnsqueezeOrSqueezeReshapePattern 0 0 1.492000e-06
match_UnsqueezeReshapePattern 0 0 1.443000e-06
match_UnsqueezeUnsqueezePattern 0 0 1.302000e-06
match_WhereAddPattern 0 0 1.292000e-06
match_RotaryConcatPartPattern 0 0 1.462000e-06
match_FunctionAttentionPattern 0 0 1.442000e-06
match_FunctionAttentionGQAPattern 0 0 1.793000e-06
match_FunctionCausalMaskPattern 0 0 1.372000e-06
match_FunctionCausalMaskMulAddPattern 0 0 1.382000e-06
match_FunctionCosSinCachePattern 0 0 1.472000e-06
match_FunctionHalfRotaryEmbeddingPattern 0 0 1.352000e-06
match_RMSNormalizationPattern 0 0 1.403000e-06
match_RMSNormalizationMulPattern 0 0 1.422000e-06
match_AttentionGQAPattern 0 0 1.602000e-06
check_pattern_A20 0 0 4.407000e-06
remove_duplicated_shape 0 0 1.032000e-06
check_pattern_BD0 0 0 3.205000e-06
remove_identity_nodes 0 0 8.442000e-06
check_pattern_BI0 0 0 3.245000e-06
remove_unused 0 0 6.640000e-06
check_pattern_BUS0 0 0 3.335000e-06
build_graph_for_pattern 0 0 4.877000e-06
iteration_3 0 0 2.989300e-04
match_BatchNormalizationPattern 0 0 1.482000e-06
match_BatchNormalizationTrainingPattern 0 0 1.312000e-06
match_CastLayerNormalizationCastPattern 0 0 1.362000e-06
match_CastPattern 0 0 1.292000e-06
match_CastCastBinaryPattern 0 0 1.342000e-06
match_CastCastPattern 0 0 1.322000e-06
match_CastOpCastPattern 0 0 1.512000e-06
match_ClipClipPattern 0 0 1.452000e-06
match_ConcatEmptyPattern 0 0 1.252000e-06
match_ConcatGatherPattern 0 0 1.502000e-06
match_ConcatReshapePattern 0 0 1.562000e-06
match_ConcatTwiceUnaryPattern 0 0 1.593000e-06
match_ConstantToInitializerPattern 0 0 1.392000e-06
match_ConvBiasNullPattern 0 0 1.342000e-06
match_PadConvPattern 0 0 1.392000e-06
match_DropoutPattern 0 0 1.302000e-06
match_ExpandPattern 0 0 1.332000e-06
match_ExpandBroadcastPattern 0 0 1.322000e-06
match_ExpandSwapPattern 0 0 1.292000e-06
match_ExpandUnsqueezeExpandPattern 0 0 1.272000e-06
match_GatherConcatPattern 0 0 1.332000e-06
match_GatherGatherPattern 0 0 1.292000e-06
match_GathersSplitPattern 0 0 1.402000e-06
match_GatherShapePattern 0 0 1.342000e-06
match_GeluPattern 0 0 6.010000e-07
match_IdentityPattern 0 0 1.422000e-06
match_LayerNormalizationPattern 0 0 1.272000e-06
match_LayerNormalizationScalePattern 0 0 1.181000e-06
match_LeakyReluPattern 0 0 2.764000e-06
match_MaxReluPattern 0 0 1.341000e-06
match_MulMulMulScalarPattern 0 0 1.442000e-06
match_MulUnsqueezeUnsqueezePattern 0 0 1.302000e-06
match_NotNotPattern 0 0 1.292000e-06
match_NotWherePattern 0 0 1.262000e-06
match_ReduceArgTopKPattern 0 0 1.432000e-06
match_ReduceReshapePattern 0 0 1.402000e-06
match_ReduceSumNormalizePattern 0 0 1.322000e-06
match_ReshapePattern 0 0 1.442000e-06
match_ReshapeMatMulReshapePattern 0 0 1.272000e-06
match_Reshape2Of3Pattern 0 0 1.412000e-06
match_ReshapeReshapeBinaryPattern 0 0 1.362000e-06
match_ReshapeSqueezePattern 0 0 1.582000e-06
match_MatMulAddPattern 0 0 1.743000e-06
match_GemmTransposePattern 0 0 1.272000e-06
match_MatMulReshape2Of3Pattern 0 0 1.392000e-06
match_MulMulMatMulPattern 0 0 1.402000e-06
match_ShapeBasedReshapeIsSqueezePattern 0 0 1.513000e-06
match_ShapeBasedStaticExpandPattern 0 0 1.281000e-06
match_ShapeBasedConcatExpandPattern 0 0 1.473000e-06
match_ShapeBasedEditDistanceReshapePattern 0 0 1.462000e-06
match_ShapeBasedIdentityPattern 0 0 1.342000e-06
match_ShapeBasedExpandBroadcastPattern 0 0 1.472000e-06
match_ShapeBasedExpandBroadcastMatMulPattern 0 0 1.352000e-06
match_ShapeBasedExpandCastWhereSwapPattern 0 0 1.372000e-06
match_ShapeBasedExpandSwapPattern 0 0 1.352000e-06
match_ShapeBasedMatMulToMulPattern 0 0 1.362000e-06
match_ShapedBasedReshapePattern 0 0 1.602000e-06
match_ShapeBasedSameChildrenPattern 0 0 3.776000e-06
match_ShapeBasedShapeShapeAddPattern 0 0 1.382000e-06
match_ShapeTransposePattern 0 0 1.402000e-06
match_UnsqueezeShapePattern 0 0 1.372000e-06
match_ReshapeReshapePattern 0 0 1.572000e-06
match_RotaryEmbeddingPattern 0 0 1.412000e-06
match_SameChildrenPattern 0 0 2.484000e-06
match_SameChildrenFromInputPattern 0 0 2.524000e-06
match_SequenceConstructAtPattern 0 0 1.402000e-06
match_SplitToSequenceSequenceAtPattern 0 0 1.422000e-06
match_SliceSlicePattern 0 0 1.352000e-06
match_SlicesSplitPattern 0 0 1.382000e-06
match_SoftmaxCrossEntropyLossCastPattern 0 0 2.623000e-06
match_SplitConcatPattern 0 0 1.271000e-06
match_SqueezeAddPattern 0 0 1.281000e-06
match_SqueezeBinaryUnsqueezePattern 0 0 1.331000e-06
match_SqueezeUnsqueezePattern 0 0 1.352000e-06
match_StaticConcatReshapePattern 0 0 1.432000e-06
match_Sub1MulPattern 0 0 1.332000e-06
match_SwapExpandReshapePattern 0 0 1.312000e-06
match_SwapExpandUnsqueezePattern 0 0 1.272000e-06
match_SwapRangeAddScalarPattern 0 0 1.301000e-06
match_SwapUnaryPattern 0 0 1.341000e-06
match_SwapUnsqueezeTransposePattern 0 0 1.252000e-06
match_SwitchOrderBinaryPattern 0 0 1.442000e-06
match_SwitchReshapeActivationPattern 0 0 1.412000e-06
match_TransposeEqualReshapePattern 0 0 1.282000e-06
match_TransposeGatherPattern 0 0 1.342000e-06
match_TransposeMatMulPattern 0 0 1.442000e-06
match_TransposeReshapeMatMulPattern 0 0 1.382000e-06
match_TransposeReshapeTransposePattern 0 0 1.292000e-06
match_TransposeTransposePattern 0 0 1.352000e-06
match_UnsqueezeEqualPattern 0 0 1.261000e-06
match_UnsqueezeOrSqueezeReshapePattern 0 0 1.382000e-06
match_UnsqueezeReshapePattern 0 0 1.402000e-06
match_UnsqueezeUnsqueezePattern 0 0 1.292000e-06
match_WhereAddPattern 0 0 1.292000e-06
match_RotaryConcatPartPattern 0 0 1.322000e-06
match_FunctionAttentionPattern 0 0 1.382000e-06
match_FunctionAttentionGQAPattern 0 0 1.572000e-06
match_FunctionCausalMaskPattern 0 0 1.372000e-06
match_FunctionCausalMaskMulAddPattern 0 0 1.362000e-06
match_FunctionCosSinCachePattern 0 0 1.392000e-06
match_FunctionHalfRotaryEmbeddingPattern 0 0 1.502000e-06
match_RMSNormalizationPattern 0 0 1.442000e-06
match_RMSNormalizationMulPattern 0 0 1.282000e-06
match_AttentionGQAPattern 0 0 1.372000e-06
check_pattern_A20 0 0 3.855000e-06
remove_duplicated_shape 0 0 1.061000e-06
check_pattern_BD0 0 0 3.305000e-06
remove_identity_nodes 0 0 8.002000e-06
check_pattern_BI0 0 0 3.305000e-06
remove_unused 0 0 6.369000e-06
check_pattern_BUS0 0 0 3.164000e-06
build_graph_for_pattern 0 0 7.180000e-06
check_patterns-4 0 0 5.067000e-06
remove_unused 0 0 7.201000e-06
check_remove_unused-5 0 0 3.365000e-06
remove_identity 0 0 7.832000e-06
check_remove_identity-6 0 0 3.124000e-06
constant_folding 0 0 5.518000e-06
apply_constant_folding_new_inits 0 0 NaN
check_constant_folding-7 0 0 3.145000e-06
remove_unused 0 0 5.859000e-06
check_remove_unused-8 0 0 2.925000e-06
remove_duplicated_initializer 0 0 8.910000e-07
check_remove_duplicated_initializer-9 0 0 2.874000e-06
remove_identity 0 0 7.130000e-06
check_remove_identity-10 0 0 2.884000e-06
remove_unused 0 0 5.338000e-06
check_remove_unused-11 0 0 2.865000e-06
order 0 0 2.533800e-05
check_orderA 0 0 5.018000e-06
check_orderL 0 0 3.275000e-06
shape_order 0 0 1.379100e-05
order 0 0 NaN
check_order-12 0 0 3.164000e-06
optimization 0 2 3.829002e-03
nodes before: 3
nodes after : 1
The report can be aggregated by pass name:
<<<
import pandas
import onnx
import onnx.helper as oh
from yobx.xbuilder import GraphBuilder, OptimizationOptions
TFLOAT = onnx.TensorProto.FLOAT
model = oh.make_model(
oh.make_graph(
[
oh.make_node("Identity", ["X"], ["X2"]),
oh.make_node("Transpose", ["X2"], ["T"], perm=[1, 0]),
oh.make_node("Transpose", ["T"], ["Z"], perm=[1, 0]),
],
"demo",
[oh.make_tensor_value_info("X", TFLOAT, [3, 4])],
[oh.make_tensor_value_info("Z", TFLOAT, [3, 4])],
),
opset_imports=[oh.make_opsetid("", 18)],
ir_version=10,
)
opts = OptimizationOptions(patterns="default")
g = GraphBuilder(model, infer_shapes_options=True, optimization_options=opts)
art = g.to_onnx(return_optimize_report=True)
df = pandas.DataFrame(art.report.stats)
for c in ["added", "removed"]:
df[c] = df[c].fillna(0).astype(int)
agg = df.groupby("pattern")[["added", "removed", "time_in"]].sum()
agg = agg[(agg["added"] > 0) | (agg["removed"] > 0)].sort_values(
"removed", ascending=False
)
print(agg.to_string())
>>>
added removed time_in
pattern
apply_TransposeTransposePattern 1 2 0.000095
optimization 0 2 0.003748
remove_identity 1 2 0.000040
patterns 0 1 0.003521
Local functions#
A sub-graph can be exported as a reusable ONNX local function (a
FunctionProto) by passing a FunctionOptions instance to
to_onnx.
<<<
import onnx
from yobx.xbuilder import GraphBuilder, FunctionOptions
TFLOAT = onnx.TensorProto.FLOAT
g = GraphBuilder(18, ir_version=10, as_function=True)
g.make_tensor_input("X", TFLOAT, ("batch", 64))
r = g.op.Relu("X")
g.make_tensor_output(r, indexed=False)
func = g.to_onnx(
function_options=FunctionOptions(
export_as_function=True,
name="MyRelu",
domain="my.domain",
),
inline=False,
)
proto = func.proto
print(type(proto).__name__)
print("function name :", proto.name)
print("function domain:", proto.domain)
>>>
FunctionProto
function name : MyRelu
function domain: my.domain
Debugging GraphBuilder with Environment Variables#
GraphBuilder respects several
environment variables that help narrow down construction or optimization
problems:
Environment variable |
Effect |
|---|---|
|
Raises an exception the moment result |
|
Raises an exception the moment result |
|
Raises an exception the moment result |
|
Raises an exception the moment a node produces output |
|
Prints extra information for shape-as-value tracking (e.g. inputs
to |
|
Prints which constant is being evaluated. |
|
Prints details when nodes from a local function domain are added. |
|
Raises an exception when a shape is missing for a result that should have one. |
|
Raises an exception as soon as a null/empty shape is encountered. |
|
Prints a message every time dynamic dimension |
|
Prints a message every time a node producing |
In addition,
get_debug_msg
returns a detailed text dump of the builder’s internal state (known shapes,
types, ranks, constants, and node list) which can be printed or logged whenever
an assertion fails.
pretty_text returns a
human-readable representation of the whole graph (inputs, initializers, nodes,
outputs) and is useful for quick visual inspection:
<<<
import onnx
import onnx.helper as oh
from yobx.xbuilder import GraphBuilder
TFLOAT = onnx.TensorProto.FLOAT
model = oh.make_model(
oh.make_graph(
[
oh.make_node("Add", ["X", "Y"], ["T"]),
oh.make_node("Relu", ["T"], ["Z"]),
],
"add_relu",
[
oh.make_tensor_value_info("X", TFLOAT, ["batch", 4]),
oh.make_tensor_value_info("Y", TFLOAT, ["batch", 4]),
],
[oh.make_tensor_value_info("Z", TFLOAT, ["batch", 4])],
),
opset_imports=[oh.make_opsetid("", 18)],
ir_version=10,
)
g = GraphBuilder(model)
print(g.pretty_text())
>>>
dyn---: batch -> WrapSym(batch)
dynrev: batch -> [('batch', SymInt(batch))]
dynsrc: batch -> [{batch:('input_name', 'X'), batch:('axis', 0)}, {batch:('input_name', 'Y'), batch:('axis', 0)}, {batch:('input_name', 'Z'), batch:('axis', 0)}]
opset: : 18
input:: X |T1: batch x 4
input:: Y |T1: batch x 4
Add: X, Y -> T |T1: batch x 4
Relu: T -> Z |T1: batch x 4
output:: Z |T1: batch x 4
See also
GraphBuilderProtocol,
GraphBuilderExtendedProtocol,
GraphBuilderTorchProtocol,
OpsetProtocol —
formal protocol interfaces satisfied by
GraphBuilder.
GraphBuilderPatternOptimizationProtocol
— the read-only API exposed to pattern authors; satisfied by
GraphBuilderPatternOptimization.
Expected API — the full list of methods and attributes
every graph builder must expose for use with to_onnx().
Pattern Optimizer — how the pattern optimizer uses
GraphBuilderPatternOptimization.