{"cells": [{"cell_type": "markdown", "metadata": {}, "source": ["# Discrepencies with ONNX\n", "\n", "The notebook shows one example where the conversion leads with discrepencies if default options are used. It converts a pipeline with two steps, a scaler followed by a tree."]}, {"cell_type": "markdown", "metadata": {}, "source": ["The bug this notebook is tracking does not always appear, it has a better chance to happen with integer features but that's not always the case. The notebook must be run again in that case."]}, {"cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [{"data": {"text/html": ["
run previous cell, wait for 2 seconds
\n", ""], "text/plain": [""]}, "execution_count": 2, "metadata": {}, "output_type": "execute_result"}], "source": ["from jyquickhelper import add_notebook_menu\n", "add_notebook_menu()"]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": ["%matplotlib inline"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Data and first model\n", "\n", "We take a random datasets with mostly integers."]}, {"cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": ["import math\n", "import numpy\n", "from sklearn.datasets import make_regression\n", "from sklearn.model_selection import train_test_split\n", "\n", "X, y = make_regression(10000, 10)\n", "X_train, X_test, y_train, y_test = train_test_split(X, y)\n", "\n", "Xi_train, yi_train = X_train.copy(), y_train.copy()\n", "Xi_test, yi_test = X_test.copy(), y_test.copy()\n", "for i in range(X.shape[1]):\n", " Xi_train[:, i] = (Xi_train[:, i] * math.pi * 2 ** i).astype(numpy.int64)\n", " Xi_test[:, i] = (Xi_test[:, i] * math.pi * 2 ** i).astype(numpy.int64)"]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [{"data": {"text/plain": ["Pipeline(steps=[('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=10))])"]}, "execution_count": 5, "metadata": {}, "output_type": "execute_result"}], "source": ["from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.tree import DecisionTreeRegressor\n", "\n", "max_depth = 10\n", "\n", "model = Pipeline([\n", " ('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "\n", "model.fit(Xi_train, yi_train)"]}, {"cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [{"data": {"text/plain": ["array([-283.03708629, 263.17931397, -160.34784206, -126.59514441,\n", " -150.1963714 ])"]}, "execution_count": 6, "metadata": {}, "output_type": "execute_result"}], "source": ["model.predict(Xi_test[:5])"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Other models:"]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": ["model2 = Pipeline([\n", " ('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "model3 = Pipeline([\n", " ('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=3))\n", "])\n", "\n", "\n", "models = [\n", " ('bug', Xi_test.astype(numpy.float32), model),\n", " ('no scaler', Xi_test.astype(numpy.float32), \n", " DecisionTreeRegressor(max_depth=max_depth).fit(Xi_train, yi_train)),\n", " ('float', X_test.astype(numpy.float32),\n", " model2.fit(X_train, y_train)),\n", " ('max_depth=3', X_test.astype(numpy.float32),\n", " model3.fit(X_train, y_train))\n", "]"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Conversion to ONNX"]}, {"cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": ["import numpy\n", "from mlprodict.onnx_conv import to_onnx\n", "\n", "onx = to_onnx(model, X_train[:1].astype(numpy.float32))"]}, {"cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["OnnxInference(...)\n", " def compiled_run(dict_inputs):\n", " # inputs\n", " X = dict_inputs['X']\n", " (variable1, ) = n0_scaler(X)\n", " (variable, ) = n1_treeensembleregressor(variable1)\n", " return {\n", " 'variable': variable,\n", " }\n"]}], "source": ["from mlprodict.onnxrt import OnnxInference\n", "\n", "oinfpy = OnnxInference(onx, runtime=\"python_compiled\")\n", "print(oinfpy)"]}, {"cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffv[1583]
0sklearn0.000000-439.590635
1python133.641599-305.949036
2python_compiled133.641599-305.949036
3onnxruntime1133.641599-305.949036
\n", "
"], "text/plain": [" runtime diff v[1583]\n", "0 sklearn 0.000000 -439.590635\n", "1 python 133.641599 -305.949036\n", "2 python_compiled 133.641599 -305.949036\n", "3 onnxruntime1 133.641599 -305.949036"]}, "execution_count": 10, "metadata": {}, "output_type": "execute_result"}], "source": ["import pandas\n", "\n", "X32 = Xi_test.astype(numpy.float32)\n", "y_skl = model.predict(X32)\n", "\n", "obs = [dict(runtime='sklearn', diff=0)]\n", "for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " oinf = OnnxInference(onx, runtime=runtime)\n", " y_onx = oinf.run({'X': X32})['variable']\n", " delta = numpy.abs(y_skl - y_onx.ravel())\n", " am = delta.argmax()\n", " obs.append(dict(runtime=runtime, diff=delta.max()))\n", " obs[-1]['v[%d]' % am] = y_onx.ravel()[am]\n", " obs[0]['v[%d]' % am] = y_skl.ravel()[am]\n", "\n", "pandas.DataFrame(obs)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The pipeline shows huge discrepencies. They appear for a pipeline *StandardScaler* + *DecisionTreeRegressor* applied in integer features. They disappear if floats are used, or if the scaler is removed. The bug also disappear if the tree is not big enough (max_depth=4 instread of 5)."]}, {"cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffnamev[1583]v[1109]v[19]v[4]
0sklearn0.000000sklearn-439.590635516.084502-549.753386-97.726497
1python133.641599bug-305.949036NaNNaNNaN
2python_compiled133.641599bug-305.949036NaNNaNNaN
3onnxruntime1133.641599bug-305.949036NaNNaNNaN
4python0.000029no scalerNaN516.084473NaNNaN
5python_compiled0.000029no scalerNaN516.084473NaNNaN
6onnxruntime10.000029no scalerNaN516.084473NaNNaN
7python0.000029floatNaNNaN-549.753357NaN
8python_compiled0.000029floatNaNNaN-549.753357NaN
9onnxruntime10.000029floatNaNNaN-549.753357NaN
10python0.000003max_depth=3NaNNaNNaN-97.726494
11python_compiled0.000003max_depth=3NaNNaNNaN-97.726494
12onnxruntime10.000003max_depth=3NaNNaNNaN-97.726494
\n", "
"], "text/plain": [" runtime diff name v[1583] v[1109] \\\n", "0 sklearn 0.000000 sklearn -439.590635 516.084502 \n", "1 python 133.641599 bug -305.949036 NaN \n", "2 python_compiled 133.641599 bug -305.949036 NaN \n", "3 onnxruntime1 133.641599 bug -305.949036 NaN \n", "4 python 0.000029 no scaler NaN 516.084473 \n", "5 python_compiled 0.000029 no scaler NaN 516.084473 \n", "6 onnxruntime1 0.000029 no scaler NaN 516.084473 \n", "7 python 0.000029 float NaN NaN \n", "8 python_compiled 0.000029 float NaN NaN \n", "9 onnxruntime1 0.000029 float NaN NaN \n", "10 python 0.000003 max_depth=3 NaN NaN \n", "11 python_compiled 0.000003 max_depth=3 NaN NaN \n", "12 onnxruntime1 0.000003 max_depth=3 NaN NaN \n", "\n", " v[19] v[4] \n", "0 -549.753386 -97.726497 \n", "1 NaN NaN \n", "2 NaN NaN \n", "3 NaN NaN \n", "4 NaN NaN \n", "5 NaN NaN \n", "6 NaN NaN \n", "7 -549.753357 NaN \n", "8 -549.753357 NaN \n", "9 -549.753357 NaN \n", "10 NaN -97.726494 \n", "11 NaN -97.726494 \n", "12 NaN -97.726494 "]}, "execution_count": 11, "metadata": {}, "output_type": "execute_result"}], "source": ["obs = [dict(runtime='sklearn', diff=0, name='sklearn')]\n", "for name, x32, mod in models:\n", " for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " lonx = to_onnx(mod, x32[:1])\n", " loinf = OnnxInference(lonx, runtime=runtime)\n", " y_skl = mod.predict(X32)\n", " y_onx = loinf.run({'X': X32})['variable']\n", " delta = numpy.abs(y_skl - y_onx.ravel())\n", " am = delta.argmax()\n", " obs.append(dict(runtime=runtime, diff=delta.max(), name=name))\n", " obs[-1]['v[%d]' % am] = y_onx.ravel()[am]\n", " obs[0]['v[%d]' % am] = y_skl.ravel()[am]\n", "\n", "df = pandas.DataFrame(obs)\n", "df"]}, {"cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
namebugfloatmax_depth=3no scalersklearn
runtime
onnxruntime1133.6415990.0000290.0000030.000029NaN
python133.6415990.0000290.0000030.000029NaN
python_compiled133.6415990.0000290.0000030.000029NaN
sklearnNaNNaNNaNNaN0.0
\n", "
"], "text/plain": ["name bug float max_depth=3 no scaler sklearn\n", "runtime \n", "onnxruntime1 133.641599 0.000029 0.000003 0.000029 NaN\n", "python 133.641599 0.000029 0.000003 0.000029 NaN\n", "python_compiled 133.641599 0.000029 0.000003 0.000029 NaN\n", "sklearn NaN NaN NaN NaN 0.0"]}, "execution_count": 12, "metadata": {}, "output_type": "execute_result"}], "source": ["df.pivot(\"runtime\", \"name\", \"diff\")"]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Other way to convert\n", "\n", "ONNX does not support double for TreeEnsembleRegressor but that a new operator TreeEnsembleRegressorDouble was implemented into *mlprodict*. We need to update the conversion."]}, {"cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": ["%load_ext mlprodict"]}, {"cell_type": "code", "execution_count": 13, "metadata": {"scrolled": false}, "outputs": [{"data": {"text/html": ["
\n", ""], "text/plain": [""]}, "execution_count": 14, "metadata": {}, "output_type": "execute_result"}], "source": ["onx32 = to_onnx(model, X_train[:1].astype(numpy.float32))\n", "onx64 = to_onnx(model, X_train[:1].astype(numpy.float64), \n", " rewrite_ops=True)\n", "%onnxview onx64"]}, {"cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffv[1583]v[0]realerror
0sklearn0.000000-439.590635-283.037086NaNNaN
1python133.641599-305.949036NaNfloatNaN
2python0.000000NaN-283.037086doubleNaN
3python_compiled133.641599-305.949036NaNfloatNaN
4python_compiled0.000000NaN-283.037086doubleNaN
5onnxruntime1133.641599-305.949036NaNfloatNaN
6onnxruntime1NaNNaNNaNdoubleUnable to create InferenceSession due to '[ONN...
\n", "
"], "text/plain": [" runtime diff v[1583] v[0] real \\\n", "0 sklearn 0.000000 -439.590635 -283.037086 NaN \n", "1 python 133.641599 -305.949036 NaN float \n", "2 python 0.000000 NaN -283.037086 double \n", "3 python_compiled 133.641599 -305.949036 NaN float \n", "4 python_compiled 0.000000 NaN -283.037086 double \n", "5 onnxruntime1 133.641599 -305.949036 NaN float \n", "6 onnxruntime1 NaN NaN NaN double \n", "\n", " error \n", "0 NaN \n", "1 NaN \n", "2 NaN \n", "3 NaN \n", "4 NaN \n", "5 NaN \n", "6 Unable to create InferenceSession due to '[ONN... "]}, "execution_count": 15, "metadata": {}, "output_type": "execute_result"}], "source": ["X32 = Xi_test.astype(numpy.float32)\n", "X64 = Xi_test.astype(numpy.float64)\n", "\n", "obs = [dict(runtime='sklearn', diff=0)]\n", "for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " for name, onx, xr in [('float', onx32, X32), ('double', onx64, X64)]:\n", " try:\n", " oinf = OnnxInference(onx, runtime=runtime)\n", " except Exception as e:\n", " obs.append(dict(runtime=runtime, error=str(e), real=name))\n", " continue\n", " y_skl = model.predict(xr)\n", " y_onx = oinf.run({'X': xr})['variable']\n", " delta = numpy.abs(y_skl - y_onx.ravel())\n", " am = delta.argmax()\n", " obs.append(dict(runtime=runtime, diff=delta.max(), real=name))\n", " obs[-1]['v[%d]' % am] = y_onx.ravel()[am]\n", " obs[0]['v[%d]' % am] = y_skl.ravel()[am]\n", "\n", "pandas.DataFrame(obs)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["We see that the use of double removes the discrepencies."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## OnnxPipeline\n", "\n", "Another way to reduce the number of discrepencies is to use a pipeline which converts every steps into ONNX before training the next one. That way, every steps is either trained on the inputs, either trained on the outputs produced by ONNX. Let's see how it works."]}, {"cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["C:\\xavierdupre\\__home_\\github_fork\\scikit-learn\\sklearn\\base.py:209: FutureWarning: From version 0.24, get_params will raise an AttributeError if a parameter cannot be retrieved as an instance attribute. Previously it would return None.\n", " FutureWarning)\n"]}, {"data": {"text/plain": ["OnnxPipeline(steps=[('scaler',\n", " OnnxTransformer(onnx_bytes=b'\\x08\\x06\\x12\\x08skl2onnx\\x1a\\x081.7.1076\"\\x07ai.onnx(\\x002\\x00:\\xf6\\x01\\n\\xa6\\x01\\n\\x01X\\x12\\x08variable\\x1a\\x06Scaler\"\\x06Scaler*=\\n\\x06offset=>\\xc3.;=+=\\xc0;=|\\xf2\\xb0<=\\xcd`\\xf9>=\\x89\\xad3\\xbd=RL\\xab\\xbf=V\\xc4V\\xbe=6<\\x9d\\xc0=B>\\xa0@=\\xbb\\x93\\xea@\\xa0\\x01\\x06*<\\n\\x05scale=ik\\xb7>=\\xe8\\x17,>=)\\xb5\\xa9==\\xa7\\xd5#==Q\\x9e\\xa1<=\\xf5)$<=\\x90<\\xa2;=(D%;=a\\xa8\\xa1:= \\x9f$:\\xa0\\x01\\x06:\\nai.onnx.ml\\x12\\x1emlprodict_ONNX(StandardScaler)Z\\x11\\n\\x01X\\x12\\x0c\\n\\n\\x08\\x01\\x12\\x06\\n\\x00\\n\\x02\\x08\\nb\\x18\\n\\x08variable\\x12\\x0c\\n\\n\\x08\\x01\\x12\\x06\\n\\x00\\n\\x02\\x08\\nB\\x0e\\n\\nai.onnx.ml\\x10\\x01')),\n", " ('dt', DecisionTreeRegressor(max_depth=10))])"]}, "execution_count": 16, "metadata": {}, "output_type": "execute_result"}], "source": ["from mlprodict.sklapi import OnnxPipeline\n", "\n", "model_onx = OnnxPipeline([\n", " ('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "model_onx.fit(Xi_train, yi_train)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["We see that the first steps was replaced by an object *OnnxTransformer* which wraps an ONNX file into a transformer following the *scikit-learn* API. The initial steps are still available."]}, {"cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [{"data": {"text/plain": ["[('scaler', StandardScaler()), ('dt', DecisionTreeRegressor(max_depth=10))]"]}, "execution_count": 17, "metadata": {}, "output_type": "execute_result"}], "source": ["model_onx.raw_steps_"]}, {"cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": ["models = [\n", " ('bug', Xi_test.astype(numpy.float32), model),\n", " ('OnnxPipeline', Xi_test.astype(numpy.float32), model_onx),\n", "]"]}, {"cell_type": "code", "execution_count": 18, "metadata": {"scrolled": false}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffnamev[2276]v[1109]
0sklearn0.000000sklearn272.784708516.084502
1python234.930666bug37.854042NaN
2python_compiled234.930666bug37.854042NaN
3onnxruntime1234.930666bug37.854042NaN
4python0.000029OnnxPipelineNaN516.084473
5python_compiled0.000029OnnxPipelineNaN516.084473
6onnxruntime10.000029OnnxPipelineNaN516.084473
\n", "
"], "text/plain": [" runtime diff name v[2276] v[1109]\n", "0 sklearn 0.000000 sklearn 272.784708 516.084502\n", "1 python 234.930666 bug 37.854042 NaN\n", "2 python_compiled 234.930666 bug 37.854042 NaN\n", "3 onnxruntime1 234.930666 bug 37.854042 NaN\n", "4 python 0.000029 OnnxPipeline NaN 516.084473\n", "5 python_compiled 0.000029 OnnxPipeline NaN 516.084473\n", "6 onnxruntime1 0.000029 OnnxPipeline NaN 516.084473"]}, "execution_count": 19, "metadata": {}, "output_type": "execute_result"}], "source": ["obs = [dict(runtime='sklearn', diff=0, name='sklearn')]\n", "for name, x32, mod in models:\n", " for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " lonx = to_onnx(mod, x32[:1])\n", " loinf = OnnxInference(lonx, runtime=runtime)\n", " y_skl = model_onx.predict(X32) # model_onx is the new baseline\n", " y_onx = loinf.run({'X': X32})['variable']\n", " delta = numpy.abs(y_skl - y_onx.ravel())\n", " am = delta.argmax()\n", " obs.append(dict(runtime=runtime, diff=delta.max(), name=name))\n", " obs[-1]['v[%d]' % am] = y_onx.ravel()[am]\n", " obs[0]['v[%d]' % am] = y_skl.ravel()[am]\n", "\n", "df = pandas.DataFrame(obs)\n", "df"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Training the next steps based on ONNX outputs is better. This is not completely satisfactory... Let's check the accuracy."]}, {"cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [{"data": {"text/plain": ["(0.6492778377907853, 0.6536515451871481)"]}, "execution_count": 20, "metadata": {}, "output_type": "execute_result"}], "source": ["model.score(Xi_test, yi_test), model_onx.score(Xi_test, yi_test)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Pretty close."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Final explanation: StandardScalerFloat\n", "\n", "We proposed two ways to have an ONNX pipeline which produces the same prediction as *scikit-learn*. Let's now replace the StandardScaler by a new one which outputs float and not double. It turns out that class *StandardScaler* computes ``X /= self.scale_`` but ONNX does ``X *= self.scale_inv_``. We need to implement this exact same operator with float32 to remove all discrepencies."]}, {"cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [{"data": {"text/plain": ["Pipeline(steps=[('scaler', StandardScalerFloat()),\n", " ('dt', DecisionTreeRegressor(max_depth=10))])"]}, "execution_count": 21, "metadata": {}, "output_type": "execute_result"}], "source": ["class StandardScalerFloat(StandardScaler):\n", " \n", " def __init__(self, with_mean=True, with_std=True):\n", " StandardScaler.__init__(self, with_mean=with_mean, with_std=with_std)\n", " \n", " def fit(self, X, y=None):\n", " StandardScaler.fit(self, X, y)\n", " if self.scale_ is not None:\n", " self.scale_inv_ = (1. / self.scale_).astype(numpy.float32)\n", " return self\n", " \n", " def transform(self, X):\n", " X = X.copy()\n", " if self.with_mean:\n", " X -= self.mean_\n", " if self.with_std:\n", " X *= self.scale_inv_\n", " return X\n", "\n", " \n", "model_float = Pipeline([\n", " ('scaler', StandardScalerFloat()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "\n", "model_float.fit(Xi_train.astype(numpy.float32), yi_train.astype(numpy.float32))"]}, {"cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Unable to find a shape calculator for type ''.\n", "It usually means the pipeline being converted contains a\n", "transformer or a predictor with no corresponding converter\n", "implemented in sklearn-onnx. If the converted is implemented\n", "in another library, you need to register\n", "the converted so that it can be used by sklearn-onnx (function\n", "update_registered_converter). If the model is not yet covered\n", "by sklearn-onnx, you may raise an issue to\n", "https://github.com/onnx/sklearn-onnx/issues\n", "to get the converter implemented or even contribute to the\n", "project. If the model is a custom model, a new converter must\n", "be implemented. Examples can be found in the gallery.\n", "\n"]}], "source": ["try:\n", " onx_float = to_onnx(model_float, Xi_test[:1].astype(numpy.float))\n", "except RuntimeError as e:\n", " print(e)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["We need to register a new converter so that *sklearn-onnx* knows how to convert the new scaler. We reuse the existing converters."]}, {"cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": ["from skl2onnx import update_registered_converter\n", "from skl2onnx.operator_converters.scaler_op import convert_sklearn_scaler\n", "from skl2onnx.shape_calculators.scaler import calculate_sklearn_scaler_output_shapes\n", "\n", "\n", "update_registered_converter(\n", " StandardScalerFloat, \"SklearnStandardScalerFloat\",\n", " calculate_sklearn_scaler_output_shapes,\n", " convert_sklearn_scaler,\n", " options={'div': ['std', 'div', 'div_cast']})"]}, {"cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": ["models = [\n", " ('bug', Xi_test.astype(numpy.float32), model),\n", " ('FloatPipeline', Xi_test.astype(numpy.float32), model_float),\n", "]"]}, {"cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffnamev[1489]v[1109]
0sklearn0.000000sklearn378.038116516.084493
1python273.322334bug104.715782NaN
2python_compiled273.322334bug104.715782NaN
3onnxruntime1273.322334bug104.715782NaN
4python0.000020FloatPipelineNaN516.084473
5python_compiled0.000020FloatPipelineNaN516.084473
6onnxruntime10.000020FloatPipelineNaN516.084473
\n", "
"], "text/plain": [" runtime diff name v[1489] v[1109]\n", "0 sklearn 0.000000 sklearn 378.038116 516.084493\n", "1 python 273.322334 bug 104.715782 NaN\n", "2 python_compiled 273.322334 bug 104.715782 NaN\n", "3 onnxruntime1 273.322334 bug 104.715782 NaN\n", "4 python 0.000020 FloatPipeline NaN 516.084473\n", "5 python_compiled 0.000020 FloatPipeline NaN 516.084473\n", "6 onnxruntime1 0.000020 FloatPipeline NaN 516.084473"]}, "execution_count": 25, "metadata": {}, "output_type": "execute_result"}], "source": ["obs = [dict(runtime='sklearn', diff=0, name='sklearn')]\n", "for name, x32, mod in models:\n", " for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " lonx = to_onnx(mod, x32[:1])\n", " loinf = OnnxInference(lonx, runtime=runtime)\n", " y_skl = model_float.predict(X32) # we use model_float as a baseline\n", " y_onx = loinf.run({'X': X32})['variable']\n", " delta = numpy.abs(y_skl - y_onx.ravel())\n", " am = delta.argmax()\n", " obs.append(dict(runtime=runtime, diff=delta.max(), name=name))\n", " obs[-1]['v[%d]' % am] = y_onx.ravel()[am]\n", " obs[0]['v[%d]' % am] = y_skl.ravel()[am]\n", "\n", "df = pandas.DataFrame(obs)\n", "df"]}, {"cell_type": "markdown", "metadata": {}, "source": ["That means than the differences between ``float32(X / Y)`` and ``float32(X) * float32(1 / Y)`` are big enough to select a different path in the decision tree. ``float32(X) / float32(Y)`` and ``float32(X) * float32(1 / Y)`` are also different enough to trigger a different path. Let's illustrate that on example:"]}, {"cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["0 1.9073486e-06 7.105427357601002e-15\n", "1 3.7252903e-09 3.469446951953614e-18\n", "2 0.00390625 7.275957614183426e-12\n"]}], "source": ["a1 = numpy.random.randn(100, 2) * 10\n", "a2 = a1.copy()\n", "a2[:, 1] *= 1000\n", "a3 = a1.copy()\n", "a3[:, 0] *= 1000\n", "\n", "for i, a in enumerate([a1, a2, a3]):\n", " a = a.astype(numpy.float32)\n", " max_diff32 = numpy.max([\n", " numpy.abs(numpy.float32(x[0]) / numpy.float32(x[1]) - \n", " numpy.float32(x[0]) * (numpy.float32(1) / numpy.float32(x[1])))\n", " for x in a])\n", " max_diff64 = numpy.max([\n", " numpy.abs(numpy.float64(x[0]) / numpy.float64(x[1]) - \n", " numpy.float64(x[0]) * (numpy.float64(1) / numpy.float64(x[1])))\n", " for x in a])\n", " print(i, max_diff32, max_diff64)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The last random set shows very big differences, obviously big enough to trigger a different path in the graph. The difference for double could probably be significant in some cases, not enough on this example."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Change the conversion with option *div*\n", "\n", "Option ``'div'`` was added to the converter for *StandardScaler* to change the way the scaler is converted."]}, {"cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [{"data": {"text/plain": ["Pipeline(steps=[('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=10))])"]}, "execution_count": 27, "metadata": {}, "output_type": "execute_result"}], "source": ["model = Pipeline([\n", " ('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "model.fit(Xi_train, yi_train)"]}, {"cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", ""], "text/plain": [""]}, "execution_count": 28, "metadata": {}, "output_type": "execute_result"}], "source": ["onx_std = to_onnx(model, Xi_train[:1].astype(numpy.float32))\n", "\n", "%onnxview onx_std"]}, {"cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", ""], "text/plain": [""]}, "execution_count": 29, "metadata": {}, "output_type": "execute_result"}], "source": ["onx_div = to_onnx(model, Xi_train[:1].astype(numpy.float32),\n", " options={StandardScaler: {'div': 'div'}})\n", "%onnxview onx_div "]}, {"cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", ""], "text/plain": [""]}, "execution_count": 30, "metadata": {}, "output_type": "execute_result"}], "source": ["onx_div_cast = to_onnx(model, Xi_train[:1].astype(numpy.float32),\n", " options={StandardScaler: {'div': 'div_cast'}})\n", "%onnxview onx_div_cast"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The ONNX graph is different and using division. Let's measure the discrepencies."]}, {"cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffnamev32[1583]v64[1246]v32[1246]v64[2080]v64[1109]diff32diff64
0sklearn0.0sklearn-439.590635-364.555875-203.438616171.604023516.084502NaNNaN
1pythonNaNbug-305.949036-203.438614NaNNaNNaN133.641599161.117261
2python_compiledNaNbug-305.949036-203.438614NaNNaNNaN133.641599161.117261
3onnxruntime1NaNbug-305.949036-203.438614NaNNaNNaN133.641599161.117261
4pythonNaNdivNaNNaN-364.555878329.592377NaN161.117261157.988354
5python_compiledNaNdivNaNNaN-364.555878329.592377NaN161.117261157.988354
6onnxruntime1NaNdivNaNNaN-364.555878329.592377NaN161.117261157.988354
7pythonNaNdiv_castNaNNaN-364.555878NaN516.084473161.1172610.000029
8python_compiledNaNdiv_castNaNNaN-364.555878NaN516.084473161.1172610.000029
9onnxruntime1NaNdiv_castNaNNaN-364.555878NaN516.084473161.1172610.000029
\n", "
"], "text/plain": [" runtime diff name v32[1583] v64[1246] v32[1246] \\\n", "0 sklearn 0.0 sklearn -439.590635 -364.555875 -203.438616 \n", "1 python NaN bug -305.949036 -203.438614 NaN \n", "2 python_compiled NaN bug -305.949036 -203.438614 NaN \n", "3 onnxruntime1 NaN bug -305.949036 -203.438614 NaN \n", "4 python NaN div NaN NaN -364.555878 \n", "5 python_compiled NaN div NaN NaN -364.555878 \n", "6 onnxruntime1 NaN div NaN NaN -364.555878 \n", "7 python NaN div_cast NaN NaN -364.555878 \n", "8 python_compiled NaN div_cast NaN NaN -364.555878 \n", "9 onnxruntime1 NaN div_cast NaN NaN -364.555878 \n", "\n", " v64[2080] v64[1109] diff32 diff64 \n", "0 171.604023 516.084502 NaN NaN \n", "1 NaN NaN 133.641599 161.117261 \n", "2 NaN NaN 133.641599 161.117261 \n", "3 NaN NaN 133.641599 161.117261 \n", "4 329.592377 NaN 161.117261 157.988354 \n", "5 329.592377 NaN 161.117261 157.988354 \n", "6 329.592377 NaN 161.117261 157.988354 \n", "7 NaN 516.084473 161.117261 0.000029 \n", "8 NaN 516.084473 161.117261 0.000029 \n", "9 NaN 516.084473 161.117261 0.000029 "]}, "execution_count": 31, "metadata": {}, "output_type": "execute_result"}], "source": ["X32 = Xi_test.astype(numpy.float32)\n", "X64 = Xi_test.astype(numpy.float64)\n", "models = [('bug', model, onx_std),\n", " ('div', model, onx_div),\n", " ('div_cast', model, onx_div_cast),]\n", "\n", "obs = [dict(runtime='sklearn', diff=0, name='sklearn')]\n", "for name, mod, onx in models:\n", " for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " oinf = OnnxInference(onx, runtime=runtime)\n", " y_skl32 = mod.predict(X32)\n", " y_skl64 = mod.predict(X64)\n", " y_onx = oinf.run({'X': X32})['variable']\n", "\n", " delta32 = numpy.abs(y_skl32 - y_onx.ravel())\n", " am32 = delta32.argmax()\n", " delta64 = numpy.abs(y_skl64 - y_onx.ravel())\n", " am64 = delta64.argmax()\n", "\n", " obs.append(dict(runtime=runtime, diff32=delta32.max(), \n", " diff64=delta64.max(), name=name))\n", " obs[0]['v32[%d]' % am32] = y_skl32.ravel()[am32]\n", " obs[0]['v64[%d]' % am64] = y_skl64.ravel()[am64]\n", " obs[-1]['v32[%d]' % am32] = y_onx.ravel()[am32]\n", " obs[-1]['v64[%d]' % am64] = y_onx.ravel()[am64]\n", "\n", "df = pandas.DataFrame(obs)\n", "df"]}, {"cell_type": "markdown", "metadata": {}, "source": ["The only combination which works is the model converted with option *div_cast* (use of division in double precision), float input for ONNX, double input for *scikit-learn*."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Explanation in practice\n", "\n", "Based on previous sections, the following example buids a case where discreprencies are significant."]}, {"cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": ["std = StandardScaler()\n", "std.fit(Xi_train)\n", "xt32 = Xi_test.astype(numpy.float32)\n", "xt64 = Xi_test.astype(numpy.float64)\n", "pred = std.transform(xt32)"]}, {"cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [{"data": {"text/plain": ["2.3841858e-07"]}, "execution_count": 33, "metadata": {}, "output_type": "execute_result"}], "source": ["from onnxruntime import InferenceSession\n", "\n", "onx32 = to_onnx(std, Xi_train[:1].astype(numpy.float32))\n", "sess32 = InferenceSession(onx32.SerializeToString())\n", "got32 = sess32.run(0, {'X': xt32})[0]\n", "d32 = numpy.max(numpy.abs(pred.ravel() - got32.ravel()))\n", "d32"]}, {"cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [{"data": {"text/plain": ["2.3841858e-07"]}, "execution_count": 34, "metadata": {}, "output_type": "execute_result"}], "source": ["oinf32 = OnnxInference(onx32.SerializeToString())\n", "gotpy32 = oinf32.run({'X': xt32})['variable']\n", "dpy32 = numpy.max(numpy.abs(pred.ravel() - gotpy32.ravel()))\n", "dpy32"]}, {"cell_type": "markdown", "metadata": {}, "source": ["We tried to cast float into double before applying the normalisation and to cast back into single float. It does not help much."]}, {"cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [{"data": {"text/plain": ["2.3841858e-07"]}, "execution_count": 35, "metadata": {}, "output_type": "execute_result"}], "source": ["onx64 = to_onnx(std, Xi_train[:1].astype(numpy.float32),\n", " options={id(std): {'div': 'div'}}) \n", "sess64 = InferenceSession(onx64.SerializeToString())\n", "got64 = sess64.run(0, {'X': xt32})[0]\n", "d64 = numpy.max(numpy.abs(pred.ravel() - got64.ravel()))\n", "d64"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Last experiment, we try to use double all along."]}, {"cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["[ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. Error in Node:Scaler : Mismatched attribute type in 'Scaler : offset'\n"]}], "source": ["from onnxruntime.capi.onnxruntime_pybind11_state import InvalidGraph\n", "\n", "onx64_2 = to_onnx(std, Xi_train[:1].astype(numpy.float64))\n", "try:\n", " sess64_2 = InferenceSession(onx64_2.SerializeToString())\n", "except InvalidGraph as e:\n", " print(e)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["*onnxruntime* does not support this. Let's switch to *mlprodict*."]}, {"cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [{"data": {"text/plain": ["4.440892098500626e-16"]}, "execution_count": 37, "metadata": {}, "output_type": "execute_result"}], "source": ["onx64_2 = to_onnx(std, Xi_train[:1].astype(numpy.float64))\n", "sess64_2 = OnnxInference(onx64_2, runtime=\"python\")\n", "pred64 = std.transform(xt64)\n", "got64_2 = sess64_2.run({'X': xt64})['variable']\n", "d64_2 = numpy.max(numpy.abs(pred64.ravel() - got64_2.ravel()))\n", "d64_2"]}, {"cell_type": "markdown", "metadata": {}, "source": ["Differences are lower if every operator is done with double."]}, {"cell_type": "markdown", "metadata": {}, "source": ["## Conclusion\n", "\n", "Maybe the best option is just to introduce a transform which just cast inputs into floats."]}, {"cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [{"data": {"text/plain": ["Pipeline(steps=[('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=10))])"]}, "execution_count": 38, "metadata": {}, "output_type": "execute_result"}], "source": ["model1 = Pipeline([\n", " ('scaler', StandardScaler()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "\n", "model1.fit(Xi_train, yi_train)"]}, {"cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [{"data": {"text/plain": ["Pipeline(steps=[('cast64', CastTransformer(dtype=)),\n", " ('scaler', StandardScaler()), ('cast', CastTransformer()),\n", " ('dt', DecisionTreeRegressor(max_depth=10))])"]}, "execution_count": 39, "metadata": {}, "output_type": "execute_result"}], "source": ["from skl2onnx.sklapi import CastTransformer\n", "\n", "model2 = Pipeline([\n", " ('cast64', CastTransformer(dtype=numpy.float64)),\n", " ('scaler', StandardScaler()),\n", " ('cast', CastTransformer()),\n", " ('dt', DecisionTreeRegressor(max_depth=max_depth))\n", "])\n", "\n", "model2.fit(Xi_train, yi_train)"]}, {"cell_type": "code", "execution_count": 39, "metadata": {"scrolled": false}, "outputs": [{"data": {"text/html": ["
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
runtimediffnamev[1583]v[1246]v[1109]optionserr
0sklearn0.000000model1-439.590635-162.952888516.084502NaNNaN
1sklearn0.000000model2-439.590635-364.555875516.084502NaNNaN
2python133.641599model1-305.949036NaNNaN-NaN
3python_compiled133.641599model1-305.949036NaNNaN-NaN
4onnxruntime1133.641599model1-305.949036NaNNaN-NaN
5python201.602989model1NaN-364.555878NaNdiv_castNaN
6python_compiled201.602989model1NaN-364.555878NaNdiv_castNaN
7onnxruntime1201.602989model1NaN-364.555878NaNdiv_castNaN
8python0.000029model2NaNNaN516.084473-NaN
9python_compiled0.000029model2NaNNaN516.084473-NaN
10onnxruntime1NaNmodel2NaNNaNNaN-Unable to create InferenceSession due to '[ONN...
11python0.000029model2NaNNaN516.084473div_castNaN
12python_compiled0.000029model2NaNNaN516.084473div_castNaN
13onnxruntime10.000029model2NaNNaN516.084473div_castNaN
\n", "
"], "text/plain": [" runtime diff name v[1583] v[1246] v[1109] \\\n", "0 sklearn 0.000000 model1 -439.590635 -162.952888 516.084502 \n", "1 sklearn 0.000000 model2 -439.590635 -364.555875 516.084502 \n", "2 python 133.641599 model1 -305.949036 NaN NaN \n", "3 python_compiled 133.641599 model1 -305.949036 NaN NaN \n", "4 onnxruntime1 133.641599 model1 -305.949036 NaN NaN \n", "5 python 201.602989 model1 NaN -364.555878 NaN \n", "6 python_compiled 201.602989 model1 NaN -364.555878 NaN \n", "7 onnxruntime1 201.602989 model1 NaN -364.555878 NaN \n", "8 python 0.000029 model2 NaN NaN 516.084473 \n", "9 python_compiled 0.000029 model2 NaN NaN 516.084473 \n", "10 onnxruntime1 NaN model2 NaN NaN NaN \n", "11 python 0.000029 model2 NaN NaN 516.084473 \n", "12 python_compiled 0.000029 model2 NaN NaN 516.084473 \n", "13 onnxruntime1 0.000029 model2 NaN NaN 516.084473 \n", "\n", " options err \n", "0 NaN NaN \n", "1 NaN NaN \n", "2 - NaN \n", "3 - NaN \n", "4 - NaN \n", "5 div_cast NaN \n", "6 div_cast NaN \n", "7 div_cast NaN \n", "8 - NaN \n", "9 - NaN \n", "10 - Unable to create InferenceSession due to '[ONN... \n", "11 div_cast NaN \n", "12 div_cast NaN \n", "13 div_cast NaN "]}, "execution_count": 40, "metadata": {}, "output_type": "execute_result"}], "source": ["X32 = Xi_test.astype(numpy.float32)\n", "models = [('model1', model1, X32), ('model2', model2, X32)]\n", "options = [('-', None),\n", " ('div_cast', {StandardScaler: {'div': 'div_cast'}})]\n", "\n", "obs = [dict(runtime='sklearn', diff=0, name='model1'),\n", " dict(runtime='sklearn', diff=0, name='model2')]\n", "for name, mod, x32 in models:\n", " for no, opts in options:\n", " onx = to_onnx(mod, Xi_train[:1].astype(numpy.float32),\n", " options=opts)\n", " for runtime in ['python', 'python_compiled', 'onnxruntime1']:\n", " try:\n", " oinf = OnnxInference(onx, runtime=runtime)\n", " except Exception as e:\n", " obs.append(dict(runtime=runtime, err=str(e),\n", " name=name, options=no))\n", " continue\n", " \n", " y_skl = mod.predict(x32)\n", " try:\n", " y_onx = oinf.run({'X': x32})['variable']\n", " except Exception as e:\n", " obs.append(dict(runtime=runtime, err=str(e),\n", " name=name, options=no))\n", " continue\n", "\n", " delta = numpy.abs(y_skl - y_onx.ravel())\n", " am = delta.argmax()\n", "\n", " obs.append(dict(runtime=runtime, diff=delta.max(),\n", " name=name, options=no))\n", " obs[-1]['v[%d]' % am] = y_onx.ravel()[am]\n", " if name == 'model1':\n", " obs[0]['v[%d]' % am] = y_skl.ravel()[am]\n", " obs[1]['v[%d]' % am] = model2.predict(Xi_test).ravel()[am]\n", " elif name == 'model2':\n", " obs[0]['v[%d]' % am] = model1.predict(Xi_test).ravel()[am]\n", " obs[1]['v[%d]' % am] = y_skl.ravel()[am]\n", "\n", "df = pandas.DataFrame(obs)\n", "df"]}, {"cell_type": "markdown", "metadata": {}, "source": ["It seems to work that way."]}, {"cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": []}], "metadata": {"kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.2"}}, "nbformat": 4, "nbformat_minor": 4}