Benchmarks#
Validates a runtime against scikit-learn#
This reuse the example Measure ONNX runtime performances. The goal is to compare different implementation of an operator.
Step 1: create a virtual environment
python -m virtualenv baseline
All other steps are executed within the local environment.
Step 2: install the packages
pip install numpy scikit-learn onnx onnxruntime skl2onnx pyquickhelper matplotlib mlprodict threadpoolctl lightgbm xgboost
Step 3: run the benchmark
export model="RandomForestRegressor"
python -m mlprodict validate_runtime --n_features 4,50 -nu 2 -re 2 -o 11 -op 11 -v 1 --out_raw data$model.csv --out_summary summary$model.csv -b 1 --dump_folder dump_errors --runtime python_compiled,onnxruntime1 --models $model --out_graph bench_png$model --dtype 32
A full example is available on the following page.
Compares two different onnxruntime#
The benchmark is done using asv. This section assumes there exist a local pypi server with the different tested versions. It can be done with module pypiserver.
Step 0: pypiserver
Copy all the necessary packages in a local folder and starts a local pypiserver.
python -m pypiserver -u -p 8067 --disable-fallback local_pypi_server
Step 1: virtual environment
python -m virtualenv pyenv
Everything else is done from the virtual environment.
pip install numpy scikit-learn matplotlib pandas cython pybind11 lightgbm xgboost virtualenv pyquickhelper
The module asv may work, otherwise a modified version is necessary:
pip install git+https://github.com/sdpython/asv.git@jenkins
And the local packages:
pip install --upgrade --index http://localhost:8067/simple/ skl2onnx onnxconverter_common mlprodict onnxruntime onnx --extra-index-url=https://pypi.python.org/simple/
Step 2: create the benchmark
Versions must be verified.
python -m mlprodict asv_bench --location . --models "RandomForestRegressor" --build "build" -dt 32 -ma "{\"onnxruntime\":[\"1.1.2\", \"http://localhost:8067/simple/\"],\"onnx\":[\"1.6.0\"],\"scikit-learn\":[\"0.22.2.post1\"]}" -n 4 -o 11 -op 11 -r scikit-learn,python_compiled,onnxruntime1
Step 3: run the benchmak
python -m asv run --show-stderr --config=asv.conf.json
Step 4: publish the bechmark
python -m asv publish -o html --config=asv.conf.json
Step 5: export into ASV format
python -m mlprodict asv2csv -f . -o asv_benchmark.csv -b skl -c asv.conf.json