.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples_core/plot_evaluator_comparison.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_core_plot_evaluator_comparison.py: .. _l-plot-evaluator-comparison: Comparing the three evaluators ================================ ``yobx`` ships three evaluators that share the same ``run(outputs, feeds)`` interface but differ in their backend: * :class:`ExtendedReferenceEvaluator ` — pure Python / NumPy, extends :class:`onnx.reference.ReferenceEvaluator` with contrib-op kernels. No ONNX Runtime dependency. * :class:`OnnxruntimeEvaluator ` — runs each node individually through :class:`onnxruntime.InferenceSession`. Allows full intermediate-result inspection. * :class:`TorchReferenceEvaluator ` — hand-written PyTorch kernels; inputs and outputs are :class:`torch.Tensor`; supports CUDA. This example runs the same model through all three and verifies the outputs agree. .. GENERATED FROM PYTHON SOURCE LINES 26-37 .. code-block:: Python import numpy as np import onnx import onnx.helper as oh import torch from yobx.reference import ExtendedReferenceEvaluator from yobx.reference.onnxruntime_evaluator import OnnxruntimeEvaluator from yobx.reference.torch_evaluator import TorchReferenceEvaluator TFLOAT = onnx.TensorProto.FLOAT .. GENERATED FROM PYTHON SOURCE LINES 38-42 Build a small model -------------------- The model computes ``Z = Tanh(X + Y)``. .. GENERATED FROM PYTHON SOURCE LINES 42-60 .. code-block:: Python model = oh.make_model( oh.make_graph( [oh.make_node("Add", ["X", "Y"], ["T"]), oh.make_node("Tanh", ["T"], ["Z"])], "add_tanh", [ oh.make_tensor_value_info("X", TFLOAT, [None, None]), oh.make_tensor_value_info("Y", TFLOAT, [None, None]), ], [oh.make_tensor_value_info("Z", TFLOAT, [None, None])], ), opset_imports=[oh.make_opsetid("", 18)], ir_version=10, ) x = np.array([[1.0, -2.0], [3.0, -4.0]], dtype=np.float32) y = np.array([[0.5, 0.5], [-0.5, -0.5]], dtype=np.float32) .. GENERATED FROM PYTHON SOURCE LINES 61-66 1. ExtendedReferenceEvaluator ------------------------------ Pure Python / NumPy. A drop-in replacement for :class:`onnx.reference.ReferenceEvaluator` that also handles contrib ops. .. GENERATED FROM PYTHON SOURCE LINES 66-71 .. code-block:: Python ref = ExtendedReferenceEvaluator(model) (result_ref,) = ref.run(None, {"X": x, "Y": y}) print("ExtendedReferenceEvaluator:", result_ref) .. rst-class:: sphx-glr-script-out .. code-block:: none ExtendedReferenceEvaluator: [[ 0.9051482 -0.9051482 ] [ 0.9866143 -0.99975324]] .. GENERATED FROM PYTHON SOURCE LINES 72-78 2. OnnxruntimeEvaluator ------------------------ Executes each node via ONNX Runtime. Setting ``intermediate=True`` returns a dictionary with *all* intermediate results, which is very handy when debugging a model. .. GENERATED FROM PYTHON SOURCE LINES 78-83 .. code-block:: Python ort_eval = OnnxruntimeEvaluator(model) (result_ort,) = ort_eval.run(None, {"X": x, "Y": y}) print("OnnxruntimeEvaluator:", result_ort) .. rst-class:: sphx-glr-script-out .. code-block:: none OnnxruntimeEvaluator: [[ 0.9051482 -0.9051482 ] [ 0.98661435 -0.9997532 ]] .. GENERATED FROM PYTHON SOURCE LINES 84-85 Retrieve intermediate results: .. GENERATED FROM PYTHON SOURCE LINES 85-89 .. code-block:: Python all_intermediates = ort_eval.run(None, {"X": x, "Y": y}, intermediate=True) for name, value in sorted(all_intermediates.items()): print(f" {name}: {value}") .. rst-class:: sphx-glr-script-out .. code-block:: none T: [[ 1.5 -1.5] [ 2.5 -4.5]] X: [[ 1. -2.] [ 3. -4.]] Y: [[ 0.5 0.5] [-0.5 -0.5]] Z: [[ 0.9051482 -0.9051482 ] [ 0.98661435 -0.9997532 ]] .. GENERATED FROM PYTHON SOURCE LINES 90-95 3. TorchReferenceEvaluator --------------------------- All computation uses PyTorch tensors. The same model can run on CUDA by passing ``providers=["CUDAExecutionProvider"]``. .. GENERATED FROM PYTHON SOURCE LINES 95-102 .. code-block:: Python torch_eval = TorchReferenceEvaluator(model) x_t = torch.from_numpy(x) y_t = torch.from_numpy(y) (result_torch,) = torch_eval.run(None, {"X": x_t, "Y": y_t}) print("TorchReferenceEvaluator:", result_torch) .. rst-class:: sphx-glr-script-out .. code-block:: none TorchReferenceEvaluator: tensor([[ 0.9051, -0.9051], [ 0.9866, -0.9998]]) .. GENERATED FROM PYTHON SOURCE LINES 103-105 Verify all three evaluators agree ----------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 105-110 .. code-block:: Python assert np.allclose(result_ref, result_ort), "ExtendedRef vs ORT mismatch" assert np.allclose(result_ref, result_torch.numpy()), "ExtendedRef vs Torch mismatch" print("All three evaluators produce the same result ✓") .. rst-class:: sphx-glr-script-out .. code-block:: none All three evaluators produce the same result ✓ .. GENERATED FROM PYTHON SOURCE LINES 111-123 Summary ------- +---------------------------------+--------------------+--------------------------------+ | Evaluator | Input/output type | Highlights | +=================================+====================+================================+ | ExtendedReferenceEvaluator | NumPy ndarray | No ORT required; contrib ops | +---------------------------------+--------------------+--------------------------------+ | OnnxruntimeEvaluator | NumPy or PyTorch | intermediate=True; ORT backend | +---------------------------------+--------------------+--------------------------------+ | TorchReferenceEvaluator | torch.Tensor | CUDA support; no round-trip | +---------------------------------+--------------------+--------------------------------+ .. GENERATED FROM PYTHON SOURCE LINES 125-130 Plot: outputs from all three evaluators ----------------------------------------- The heat-maps below show the ``Tanh(X + Y)`` output produced by each evaluator. All three panels are identical, confirming the results agree. .. GENERATED FROM PYTHON SOURCE LINES 130-144 .. code-block:: Python import matplotlib.pyplot as plt # noqa: E402 fig, axes = plt.subplots(1, 3, figsize=(9, 3), sharey=True) labels = ["ExtendedRef", "OnnxruntimeEval", "TorchEval"] results_all = [result_ref, result_ort, result_torch.numpy()] for ax, label, res in zip(axes, labels, results_all): im = ax.imshow(res, cmap="RdBu", vmin=-1, vmax=1, aspect="auto") ax.set_title(label, fontsize=9) ax.set_xlabel("col") axes[0].set_ylabel("row") fig.colorbar(im, ax=axes.tolist(), shrink=0.8, label="Tanh(X+Y)") fig.suptitle("Outputs from all three evaluators (Tanh(X + Y))") plt.show() .. image-sg:: /auto_examples_core/images/sphx_glr_plot_evaluator_comparison_001.png :alt: Outputs from all three evaluators (Tanh(X + Y)), ExtendedRef, OnnxruntimeEval, TorchEval :srcset: /auto_examples_core/images/sphx_glr_plot_evaluator_comparison_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 0.126 seconds) .. _sphx_glr_download_auto_examples_core_plot_evaluator_comparison.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_evaluator_comparison.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_evaluator_comparison.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_evaluator_comparison.zip ` .. include:: plot_evaluator_comparison.recommendations .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_