Benchmark inference for scikit-learn models

This short code compares the execution of a couple of runtime for inference including onnxruntime. It uses examples Measure ONNX runtime performances. It is an automated process to compare the performance of a model against scikit-learn. This model is a simple model taken from all implemented by scikit-learn.

Linear Regression

from pandas import read_csv
from mlprodict.cli import validate_runtime
from mlprodict.plotting.plotting import plot_validate_benchmark

res = validate_runtime(
    verbose=1,
    out_raw="data.csv", out_summary="summary.csv",
    benchmark=True, dump_folder="dump_errors",
    runtime=['python', 'onnxruntime1'],
    models=['LinearRegression'],
    skip_models=['LinearRegression[m-reg]'],
    n_features=[10, 50], dtype="32",
    out_graph="bench.png",
    opset_min=15, opset_max=15,
    time_kwargs={
        1: {"number": 50, "repeat": 50},
        10: {"number": 25, "repeat": 25},
        100: {"number": 20, "repeat": 20},
        1000: {"number": 20, "repeat": 20},
        10000: {"number": 10, "repeat": 10},
    }
)

results = read_csv('summary.csv')
results

Out:

time_kwargs={1: {'number': 50, 'repeat': 50}, 10: {'number': 25, 'repeat': 25}, 100: {'number': 20, 'repeat': 20}, 1000: {'number': 20, 'repeat': 20}, 10000: {'number': 10, 'repeat': 10}}
[enumerate_validated_operator_opsets] opset in [15, 15].

  0%|          | 0/1 [00:00<?, ?it/s]
LinearRegression            :   0%|          | 0/1 [00:00<?, ?it/s][enumerate_compatible_opset] opset in [15, 15].

LinearRegression            : 100%|##########| 1/1 [00:40<00:00, 40.77s/it]
LinearRegression            : 100%|##########| 1/1 [00:40<00:00, 40.78s/it]
Saving raw_data into 'data.csv'.
Saving summary into 'summary.csv'.
Saving graph into 'bench.png'.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXGeneral'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXNonUnicode'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeOneSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeTwoSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeThreeSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFourSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['STIXSizeFiveSym'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmsy10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmr10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmtt10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmmi10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmb10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmss10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['cmex10'] not found. Falling back to DejaVu Sans.
findfont: Font family ['DejaVu Sans Display'] not found. Falling back to DejaVu Sans.

	name	problem	scenario	optim	method_name	output_index	conv_options	inst	n_features	runtime	skl_version	skl_nop	skl_ncoef	skl_nlin	onx_size	onx_nnodes	onx_ninits	onx_producer_name	onx_producer_version	onx_ai.onnx.ml	onx_size_optim	onx_nnodes_optim	onx_ninits_optim	opset15	RT/SKL-N=1	N=10	N=100	N=1000	N=10000	RT/SKL-N=1-min	RT/SKL-N=1-max	N=10-min	N=10-max	N=100-min	N=100-max	N=1000-min	N=1000-max	N=10000-min	N=10000-max
0	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	10	onnxruntime1	1.0.2	1	10	1	288	1	0	skl2onnx	1.11	1	288	1	0	OK 15/1	0.700963	0.713135	0.720735	1.007996	2.071347	0.646313	0.858745	0.684352	1.286767	0.713765	0.759072	0.990734	1.015034	2.017907	2.100248
1	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	10	python	1.0.2	1	10	1	288	1	0	skl2onnx	1.11	1	288	1	0	OK 15/1	0.412429	0.409111	0.414968	0.442972	0.554032	0.396090	0.421306	0.407047	0.410657	0.411942	0.418201	0.432090	0.467094	0.541209	0.562949
2	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	50	onnxruntime1	1.0.2	1	50	1	489	1	0	skl2onnx	1.11	1	489	1	0	OK 15/1	0.708293	0.688762	0.795733	2.117726	2.658230	0.677977	0.835150	0.658404	1.226313	0.787257	0.834253	2.076018	2.148842	2.622809	2.690103
3	LinearRegression	b-reg	default	NaN	predict	0	{}	NaN	50	python	1.0.2	1	50	1	489	1	0	skl2onnx	1.11	1	489	1	0	OK 15/1	0.402319	0.399174	0.409233	0.441200	0.539927	0.397949	0.403915	0.397709	0.400972	0.406222	0.412706	0.430927	0.448560	0.529116	0.554031

Graph.

_, ax = plot_validate_benchmark(results)
ax

# import matplotlib.pyplot as plt
# plt.show()

Out:

array([<AxesSubplot:title={'center':'RT/SKL-N=1'}>,
       <AxesSubplot:title={'center':'N=10'}>,
       <AxesSubplot:title={'center':'N=100'}>,
       <AxesSubplot:title={'center':'N=1000'}>,
       <AxesSubplot:title={'center':'N=10000'}>], dtype=object)

previous

Benchmark onnxruntime API: run or …

next

Benchmark inference for a linear regression