:orphan:

.. _benchmarks-gallery:

Benchmarks
==========

A gallery of benchmarks comparing the SIMD-accelerated CPU kernels provided by
``onnx-light-cpu`` against other back-ends such as ``numpy``, ``onnxruntime``
and ``onnx-light``'s built-in reference kernels.

The Gemm and Attention corpora used by the
:doc:`Gemm and MatMul roadmap </next_steps/2026/2026_08_gemm_matmul>` are implemented
as C++ backend cases in ``TestMode::BENCHMARK``. This is the benchmark framework
provided by ``onnx-light``: cases are generated lazily in C++, exposed through
``CollectTestCases``, and consumed by the common benchmark recorder. The Gemm
cases live in
``onnx_light_cpu/backend_test/cases/math/cases_gemm.cc``; the Attention cases
live in ``onnx-light``'s C++ backend-test registry.

The Gemm corpus contains shape-forced cases for every prepared algorithm:
``direct`` (small K), ``skinny_m``, ``skinny_n``, ``split_k`` (large K with a
small output), and square/transformer shapes (general five-loop). Every shape
is registered for each element type the ``GemmKernel`` implements -- ``float32``,
``float16`` and ``bfloat16`` -- so the corpus also measures the fp16/bf16
widen/round-trip overhead. Standalone recorders run on their calling thread.
Integration benchmarks compare participant counts through the ``onnx-light``
session ``cpu_execution`` policy, which owns thread count, affinity, and spin
behavior. ``onnx-light-cpu`` does not create workers or read thread-control
environment variables.

The unary backend corpus samples both sides of the ``Exp`` and ``Log``
scheduling thresholds. ``plot_exp_log_benchmark.py`` visualizes those
transitions. ``plot_tree_ensemble_benchmark.py`` visualizes representative
cases from the maintained TreeEnsemble parity runner; it does not claim
backend-test kernel coverage because TreeEnsemble is not registered there.

``plot_backend_cases_benchmark.py`` walks a subset of the ``TestMode::BENCHMARK``
``test_cpu_*`` backend test cases -- covering every operator with an
onnx-light-cpu backend test registration (``Abs``, ``Exp``, ``Log``, ``Gemm``
and ``Not``) -- and times each one through onnx-light (with onnx-light-cpu's
accelerated kernels registered) and through ONNX Runtime, using the exact same
generated model and inputs for both.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This benchmark compares the portable onnx-light-cpu implementations of com.microsoft.CDist and com.microsoft.BiasGelu with ONNX Runtime CPU.">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_custom_operators_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_custom_operators_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark custom operators against ONNX Runtime</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example compares ONNX Runtime, the built-in onnx-light kernels, the onnx-light-cpu kernels, and NumPy around the scheduling thresholds used by the Exp and Log CPU kernels. Each runtime is measured in a separate phase so that persistent worker pools do not perturb measurements of another candidate.">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_exp_log_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_exp_log_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark Exp and Log parallel scheduling</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example measures TreeEnsemble-5 regression forests while varying the number of trees, input features, and batch size. Batch size 1 is represented explicitly because ONNX Runtime uses a specialized execution path for that case. The largest forest contains 10,000 trees and the widest input contains 4,096 features, both representative of production models. Those largest configurations are enabled by --big, which defaults to True on machines with more than 64 cores and can be disabled with --no-big.">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_tree_ensemble_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_tree_ensemble_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark TreeEnsemble scheduling scenarios</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example compares four ways of computing a float32 general matrix multiplication Y = A @ B for square matrices of increasing size:">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_gemm_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_gemm_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark Gemm: numpy vs onnxruntime vs onnx-light vs onnx-light-cpu</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example compares up to four ways of computing the elementwise absolute value of a float32 array across a range of input sizes:">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_abs_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_abs_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark Abs: onnxruntime vs onnx-light + onnx-light-cpu</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="onnx-light-cpu&#x27;s Gemm kernel picks between several internal code paths depending on the shape of A/``B`` (see ../../design/kernels/gemm_kernel_design for the full decision tree):">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_gemm_dtype_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_gemm_dtype_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark Gemm: float32 vs float16 vs bfloat16 across kernel code paths</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="onnx-light-cpu ships its own ONNX backend test cases -- named test_cpu_ -- in a dedicated C++ registration library (``lib_onnx_light_cpu_backend_test``, see onnx_light_cpu.register_backend_test_cases). Every case that has an accelerated kernel also has a TestMode.BENCHMARK variant: the same operator and attributes, but with inputs large enough that a single evaluation takes a measurable amount of time. This example collects every test_cpu__benchmark case -- across every operator and element type onnx-light-cpu ships one for -- and times each one through onnx-light (with onnx-light-cpu&#x27;s accelerated kernel registered) and through ONNX Runtime, using the exact same generated model and inputs for both.">

.. only:: html

  .. image:: /auto_examples/benchmarks/images/thumb/sphx_glr_plot_backend_cases_benchmark_thumb.png
    :alt:

  :doc:`/auto_examples/benchmarks/plot_backend_cases_benchmark`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Benchmark backend test cases against ONNX Runtime</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /auto_examples/benchmarks/plot_custom_operators_benchmark
   /auto_examples/benchmarks/plot_exp_log_benchmark
   /auto_examples/benchmarks/plot_tree_ensemble_benchmark
   /auto_examples/benchmarks/plot_gemm_benchmark
   /auto_examples/benchmarks/plot_abs_benchmark
   /auto_examples/benchmarks/plot_gemm_dtype_benchmark
   /auto_examples/benchmarks/plot_backend_cases_benchmark


.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-gallery

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download all examples in Python source code: benchmarks_python.zip </auto_examples/benchmarks/benchmarks_python.zip>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download all examples in Jupyter notebooks: benchmarks_jupyter.zip </auto_examples/benchmarks/benchmarks_jupyter.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_