:orphan:

Runtime, Backend Test, and Kernel Examples
==========================================

Examples showing how to retrieve, inspect or run ONNX backend test cases
exposed by :mod:`onnx_light.onnx.backend`.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example compares the built-in Abs kernel in onnx-light with onnxruntime and numpy.abs for vectors ranging from one hundred to one hundred million elements. The comparison is repeated for three element types: float32, float16 and bfloat16. The bfloat16 values rely on the ml_dtypes NumPy extension, which provides a native bfloat16 dtype.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_abs_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_abs_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark Abs: onnxruntime vs onnx-light</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Before a model can be run, both onnxruntime and onnx_light go through an initialization phase that turns a ModelProto into an object ready for inference. For onnxruntime this happens when the onnxruntime.InferenceSession is created: the graph is optimized, every node&#x27;s kernel is resolved and the weights are loaded. For onnx_light the equivalent step is the construction of ReferenceEvaluator, which prepares the persistent RuntimeContext used by every subsequent run call.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_qwen3_init_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_qwen3_init_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark the initialization steps: onnxruntime vs onnx-light</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="ReferenceEvaluator dispatches every NodeProto against the static C++ KernelDispatchTable. Any operator that is not built in — typically an operator from a user-defined domain, or a temporary stand-in for an op that is not yet implemented — would otherwise fail with unsupported op_type.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_custom_kernel_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_custom_kernel`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Extend ReferenceEvaluator with a custom kernel</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example uses onnx_light.kernel_tuning to discover every tuning parameter used by one exact kernel, compare its portable and local values, write a validated local profile, and run a bounded calibration.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_kernel_tuning_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_kernel_tuning`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Inspect, change, and calibrate kernel tuning from Python</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="When a RuntimeContext runs a model with event logging enabled, it records a RuntimeEvent for every tensor map mutation and every node dispatch. Attaching a SimpleRawBufferAllocator to the context makes each event additionally carry the allocator&#x27;s live footprint (``allocated_bytes``, the total size of every buffer alive at that moment) and its peak footprint (``peak_bytes``, the maximum ever reached). Together with the wall-clock duration_ns of each node this turns the event log into a per-node time-and-memory profile.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_runtime_memory_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_runtime_memory`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Profile the runtime memory of a model with the event log</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This is the Python counterpart of the standalone C++ example examples/register_custom_kernel (see l-cpp-register-custom-kernel-example). The C++ example writes a brand-new :cpp~onnx_light::core::runtime::KernelBase subclass for the existing Abs operator, installs it into the shared dispatch table and checks — via a run counter — that the custom kernel, not the built-in one, executed the node.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_register_custom_kernel_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_register_custom_kernel`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Replace a built-in kernel with a Python one and prove it ran</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="The ONNX backend test suite shipped with onnx-light is exposed as a small C++ data model bound to Python under onnx_light.onnx.backend. Each entry is a TestCase made of a ModelProto plus one or more DataSet (lists of reference input / output Tensor instances).">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_backend_test_case_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_backend_test_case`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Retrieve a backend test case and display its model and data</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="onnx_light ships a C++ kernel dispatcher exposed in Python through ReferenceEvaluator. After each run call the evaluator retains the RuntimeContext used internally; calling events returns an append-only log of every tensor map mutation: graph initializers seeded on entry, inputs injected by the caller, intermediate values produced by each node kernel and outputs propagated back to the caller.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_runtime_events_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_runtime_events`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Run a model with the runtime and inspect intermediate results</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example shows how to take one backend test case from the suite shipped with onnx-light (onnx_light.onnx.backend), run its ONNX model with the reference runtime, and then re-run the same case as a backend test.">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_run_cast_to_int2_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_run_cast_to_int2`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Run an ONNX model casting a float tensor into an int2 tensor</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="ReferenceEvaluator supports three kinds of ONNX values at the graph boundary:">

.. only:: html

  .. image:: /auto_examples_runtime/images/thumb/sphx_glr_plot_run_with_tensor_sequence_dict_thumb.png
    :alt:

  :doc:`/auto_examples_runtime/plot_run_with_tensor_sequence_dict`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Run the reference evaluator with tensor, sequence and dictionary inputs/outputs</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /auto_examples_runtime/plot_abs_benchmark
   /auto_examples_runtime/plot_qwen3_init_benchmark
   /auto_examples_runtime/plot_custom_kernel
   /auto_examples_runtime/plot_kernel_tuning
   /auto_examples_runtime/plot_runtime_memory
   /auto_examples_runtime/plot_register_custom_kernel
   /auto_examples_runtime/plot_backend_test_case
   /auto_examples_runtime/plot_runtime_events
   /auto_examples_runtime/plot_run_cast_to_int2
   /auto_examples_runtime/plot_run_with_tensor_sequence_dict


.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-gallery

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download all examples in Python source code: auto_examples_runtime_python.zip </auto_examples_runtime/auto_examples_runtime_python.zip>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download all examples in Jupyter notebooks: auto_examples_runtime_jupyter.zip </auto_examples_runtime/auto_examples_runtime_jupyter.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_