diff --git a/modules/modelops/attachments/BYOM_v6.ipynb b/modules/modelops/attachments/BYOM_v6.ipynb deleted file mode 100644 index 40c186975..000000000 --- a/modules/modelops/attachments/BYOM_v6.ipynb +++ /dev/null @@ -1,1088 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## BYOM In-Vantage Scoring with PMML and ONNX\n", - "\n", - "In this notebook, we will show you how to work with the Bring Your Own Model (BYOM) pattern and BYOM In-Vantage Scoring. This pattern allows you to use whatever data science platform you want to perform model development and experimentation. You can use the vast majority of popular data science libraries and transformations. The only constraint is that you can convert it to one of the following open formats\n", - "\n", - "- ONNX\n", - "- PMML\n", - "- H2O (MOJO)\n", - "- H2O (Driverless AI)\n", - "\n", - "ONNX is become more popular by the day. It is a very efficient model format which was created and is maintained by Microsoft and its adoption by other companies and libraries as the standard open format is incresingly rapidly. While the name suggests it is primarily related to neural networks, it can be used with most sklearn libraries and algorithms. \n", - "\n", - "\n", - "In this example, we will show you how you can develop in a notebook or other third-party tooling, produce a model and convert it to both `onnx` and `pmml` formats for deploying in Vantage with ModelOps." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import pandas as pd\n", - "import getpass\n", - "\n", - "from teradataml import (\n", - " create_context, \n", - " remove_context,\n", - " get_context,\n", - " get_connection,\n", - " DataFrame,\n", - " retrieve_byom,\n", - " PMMLPredict,\n", - " configure)" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Host: tdprd.td.teradata.com\n", - "Username: wf250003\n", - "Password: ········\n", - "VAL DB: TRNG_XSP\n", - "BYOM DB: TRNG_BYOM\n" - ] - }, - { - "data": { - "text/plain": [ - "Engine(teradatasql://wf250003:***@tdprd.td.teradata.com/?LOGDATA=%2A%2A%2A&LOGMECH=%2A%2A%2A)" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "host = input(\"Host: \")\n", - "username = input(\"Username: \")\n", - "password = getpass.getpass(\"Password: \")\n", - "val_db = input(\"VAL DB: \")\n", - "byom_db = input(\"BYOM DB: \")\n", - "\n", - "# configure byom/val installation\n", - "configure.val_install_location = val_db\n", - "configure.byom_install_location = byom_db\n", - "\n", - "# by default we assume your are using your user database. change as required\n", - "database = username\n", - "\n", - "create_context(host=host, username=username, password=password, logmech=\"TDNEGO\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "Pipeline(steps=[('scaler', MinMaxScaler()),\n", - " ('xgb',\n", - " XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,\n", - " colsample_bylevel=1, colsample_bynode=1,\n", - " colsample_bytree=1, early_stopping_rounds=None,\n", - " enable_categorical=False, eta=0.2,\n", - " eval_metric=None, gamma=0, gpu_id=-1,\n", - " grow_policy='depthwise', importance_type=None,\n", - " interaction_constraints='',\n", - " learning_rate=0.200000003, max_bin=256,\n", - " max_cat_to_onehot=4, max_delta_step=0,\n", - " max_depth=6, max_leaves=0, min_child_weight=1,\n", - " missing=nan, monotone_constraints='()',\n", - " n_estimators=100, n_jobs=0, num_parallel_tree=1,\n", - " predictor='auto', random_state=0, reg_alpha=0, ...))])" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from xgboost import XGBClassifier\n", - "from sklearn.preprocessing import MinMaxScaler\n", - "from sklearn.pipeline import Pipeline\n", - "\n", - "\n", - "train_pdf = DataFrame.from_query(\"\"\"\n", - "SELECT \n", - " F.*, D.hasdiabetes \n", - "FROM pima_patient_features F\n", - "JOIN pima_patient_diagnoses D\n", - " ON F.patientid = D.patientid \n", - " WHERE F.patientid MOD 5 <> 0\n", - "\"\"\").to_pandas(all_rows=True)\n", - "\n", - "features = [\"NumTimesPrg\", \"Age\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\"]\n", - "target = \"HasDiabetes\"\n", - "\n", - "# split data into X and y\n", - "X_train = train_pdf[features]\n", - "y_train = train_pdf[target]\n", - "\n", - "model = Pipeline([('scaler', MinMaxScaler()),\n", - " ('xgb', XGBClassifier(eta=0.2, max_depth=6))])\n", - "\n", - "model.fit(X_train, y_train)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Convert the model to PMML\n", - "\n", - "You can use the sklearn2pmml or the nyoka python libraries to convert to pmml. The nyoka is a python only package and so it is preferrable. " - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "from nyoka import xgboost_to_pmml\n", - "\n", - "xgboost_to_pmml(pipeline=model, col_names=features, target_name=target, pmml_f_name=\"model.pmml\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Convert the model to ONNX\n", - "\n", - "We can also convert the model to onnx format. This is a bit more involved as the client libraries for converting from sklearn/xgboost to onnx are not yet as mature.\n", - "\n", - "```\n", - "pip install onnx==1.10.2 skl2onnx==1.11.2 onnxruntime==1.9.0 protobuf==3.20.1 onnxmltools==1.7.0\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from skl2onnx import to_onnx\n", - "from skl2onnx import convert_sklearn, to_onnx, update_registered_converter\n", - "from skl2onnx.common.shape_calculator import (\n", - " calculate_linear_classifier_output_shapes,\n", - " calculate_linear_regressor_output_shapes)\n", - "from onnxmltools.convert.xgboost.operator_converters.XGBoost import convert_xgboost\n", - "from onnxmltools.convert import convert_xgboost as convert_xgboost_booster\n", - "\n", - "update_registered_converter(\n", - " XGBClassifier, 'XGBoostXGBClassifier',\n", - " calculate_linear_classifier_output_shapes, convert_xgboost,\n", - " options={'nocl': [True, False], 'zipmap': [True, False, 'columns']})\n", - "\n", - "\n", - "model_onnx = to_onnx(model, X_train.astype(np.float32), target_opset=15)\n", - "with open(\"model.onnx\", \"wb\") as f:\n", - " f.write(model_onnx.SerializeToString())\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Import into ModelOps to Operationalize\n", - "\n", - "Go to the ModelOps UI and import this as a new model version. Then follow the workflow to deploy. Note that you can also import programatically via the ModelOps Python SDK. \n", - "\n", - "You may be wondering why you can't just directly insert the onnx or pmml model directly into the database table. And the answer is you can. However, with ModelOps, you get full governance around this model deployment, including data drift and model monitoring and alerting. \n", - "\n", - "\n", - "### View Published Models\n", - "\n", - "Once deployed via ModelOps, we can view the models published to vantage by querying the table they are published to. Note this information is available via the AOA APIs also.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
model_versionmodel_idmodel_typeproject_iddeployed_atmodel
09de00f0d-060c-4737-b0a3-531768363ced2354a903-601b-5f72-b014-8983306005b4PMML414bec4e-c677-4f2e-a370-0076e57918ea2022-07-20 11:02:35.470b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ...
15761d5c1-bf57-456b-8076-c3062be0b5442354a903-601b-5f72-b014-8983306005b4PMML414bec4e-c677-4f2e-a370-0076e57918ea2022-07-18 07:04:19.430b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ...
\n", - "
" - ], - "text/plain": [ - " model_version model_id \\\n", - "0 9de00f0d-060c-4737-b0a3-531768363ced 2354a903-601b-5f72-b014-8983306005b4 \n", - "1 5761d5c1-bf57-456b-8076-c3062be0b544 2354a903-601b-5f72-b014-8983306005b4 \n", - "\n", - " model_type project_id deployed_at \\\n", - "0 PMML 414bec4e-c677-4f2e-a370-0076e57918ea 2022-07-20 11:02:35.470 \n", - "1 PMML 414bec4e-c677-4f2e-a370-0076e57918ea 2022-07-18 07:04:19.430 \n", - "\n", - " model \n", - "0 b'\\n\\n
\\n \\n ... \n", - "1 b'\\n\\n
\\n \\n ... " - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "pd.options.display.max_colwidth = 250\n", - "pd.read_sql(\"SELECT TOP 2 * FROM aoa_byom_models\", get_connection())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## On-Demand Scoring" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PatientIdpredictionjson_report
05451{\"probability_0\":0.015597303259093032,\"probability_1\":0.984402696740907,\"predicted_HasDiabetes\":1}
12650{\"probability_0\":0.8621357683534957,\"probability_1\":0.1378642316465043,\"predicted_HasDiabetes\":0}
2400{\"probability_0\":0.574482742227689,\"probability_1\":0.425517257772311,\"predicted_HasDiabetes\":0}
33850{\"probability_0\":0.9902272802038437,\"probability_1\":0.009772719796156322,\"predicted_HasDiabetes\":0}
401{\"probability_0\":0.0950635688096706,\"probability_1\":0.9049364311903294,\"predicted_HasDiabetes\":1}
56000{\"probability_0\":0.9499693357624215,\"probability_1\":0.050030664237578494,\"predicted_HasDiabetes\":0}
65300{\"probability_0\":0.9729698349887085,\"probability_1\":0.02703016501129154,\"predicted_HasDiabetes\":0}
71201{\"probability_0\":0.11393098981414929,\"probability_1\":0.8860690101858507,\"predicted_HasDiabetes\":1}
86500{\"probability_0\":0.9820752256818963,\"probability_1\":0.0179247743181037,\"predicted_HasDiabetes\":0}
9800{\"probability_0\":0.9819309856605802,\"probability_1\":0.01806901433941982,\"predicted_HasDiabetes\":0}
\n", - "
" - ], - "text/plain": [ - " PatientId prediction \\\n", - "0 545 1 \n", - "1 265 0 \n", - "2 40 0 \n", - "3 385 0 \n", - "4 0 1 \n", - "5 600 0 \n", - "6 530 0 \n", - "7 120 1 \n", - "8 650 0 \n", - "9 80 0 \n", - "\n", - " json_report \n", - "0 {\"probability_0\":0.015597303259093032,\"probability_1\":0.984402696740907,\"predicted_HasDiabetes\":1} \n", - "1 {\"probability_0\":0.8621357683534957,\"probability_1\":0.1378642316465043,\"predicted_HasDiabetes\":0} \n", - "2 {\"probability_0\":0.574482742227689,\"probability_1\":0.425517257772311,\"predicted_HasDiabetes\":0} \n", - "3 {\"probability_0\":0.9902272802038437,\"probability_1\":0.009772719796156322,\"predicted_HasDiabetes\":0} \n", - "4 {\"probability_0\":0.0950635688096706,\"probability_1\":0.9049364311903294,\"predicted_HasDiabetes\":1} \n", - "5 {\"probability_0\":0.9499693357624215,\"probability_1\":0.050030664237578494,\"predicted_HasDiabetes\":0} \n", - "6 {\"probability_0\":0.9729698349887085,\"probability_1\":0.02703016501129154,\"predicted_HasDiabetes\":0} \n", - "7 {\"probability_0\":0.11393098981414929,\"probability_1\":0.8860690101858507,\"predicted_HasDiabetes\":1} \n", - "8 {\"probability_0\":0.9820752256818963,\"probability_1\":0.0179247743181037,\"predicted_HasDiabetes\":0} \n", - "9 {\"probability_0\":0.9819309856605802,\"probability_1\":0.01806901433941982,\"predicted_HasDiabetes\":0} " - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "model_version=\"1dbe5430-f8c5-4d32-b26c-a02476cba510\"\n", - "\n", - "model = DataFrame.from_query(f\"\"\"\n", - "SELECT * FROM aoa_byom_models \n", - " WHERE model_version='{model_version}'\n", - "\"\"\")\n", - "\n", - "\n", - "preds = PMMLPredict(\n", - " modeldata=model,\n", - " newdata=DataFrame.from_query(\"SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0\"),\n", - " accumulate=['PatientId'])\n", - "\n", - "preds.result.to_pandas().head(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PatientIdpredictionjson_report
010{\"probability_0\":0.9850747504768098,\"probability_1\":0.014925249523190227,\"predicted_HasDiabetes\":0}
130{\"probability_0\":0.9945255942085638,\"probability_1\":0.005474405791436156,\"predicted_HasDiabetes\":0}
281{\"probability_0\":0.05020155260184678,\"probability_1\":0.9497984473981532,\"predicted_HasDiabetes\":1}
391{\"probability_0\":0.28263442350828416,\"probability_1\":0.7173655764917158,\"predicted_HasDiabetes\":1}
4131{\"probability_0\":0.08177880479006427,\"probability_1\":0.9182211952099357,\"predicted_HasDiabetes\":1}
5171{\"probability_0\":0.23832592730256774,\"probability_1\":0.7616740726974323,\"predicted_HasDiabetes\":1}
6191{\"probability_0\":0.37077762620144294,\"probability_1\":0.629222373798557,\"predicted_HasDiabetes\":1}
7261{\"probability_0\":0.18449399225529128,\"probability_1\":0.8155060077447087,\"predicted_HasDiabetes\":1}
8431{\"probability_0\":0.006277949339332567,\"probability_1\":0.9937220506606674,\"predicted_HasDiabetes\":1}
9440{\"probability_0\":0.7477334297352949,\"probability_1\":0.25226657026470506,\"predicted_HasDiabetes\":0}
\n", - "
" - ], - "text/plain": [ - " PatientId prediction \\\n", - "0 1 0 \n", - "1 3 0 \n", - "2 8 1 \n", - "3 9 1 \n", - "4 13 1 \n", - "5 17 1 \n", - "6 19 1 \n", - "7 26 1 \n", - "8 43 1 \n", - "9 44 0 \n", - "\n", - " json_report \n", - "0 {\"probability_0\":0.9850747504768098,\"probability_1\":0.014925249523190227,\"predicted_HasDiabetes\":0} \n", - "1 {\"probability_0\":0.9945255942085638,\"probability_1\":0.005474405791436156,\"predicted_HasDiabetes\":0} \n", - "2 {\"probability_0\":0.05020155260184678,\"probability_1\":0.9497984473981532,\"predicted_HasDiabetes\":1} \n", - "3 {\"probability_0\":0.28263442350828416,\"probability_1\":0.7173655764917158,\"predicted_HasDiabetes\":1} \n", - "4 {\"probability_0\":0.08177880479006427,\"probability_1\":0.9182211952099357,\"predicted_HasDiabetes\":1} \n", - "5 {\"probability_0\":0.23832592730256774,\"probability_1\":0.7616740726974323,\"predicted_HasDiabetes\":1} \n", - "6 {\"probability_0\":0.37077762620144294,\"probability_1\":0.629222373798557,\"predicted_HasDiabetes\":1} \n", - "7 {\"probability_0\":0.18449399225529128,\"probability_1\":0.8155060077447087,\"predicted_HasDiabetes\":1} \n", - "8 {\"probability_0\":0.006277949339332567,\"probability_1\":0.9937220506606674,\"predicted_HasDiabetes\":1} \n", - "9 {\"probability_0\":0.7477334297352949,\"probability_1\":0.25226657026470506,\"predicted_HasDiabetes\":0} " - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "query = f\"\"\"\n", - "SELECT * FROM {byom_db}.PMMLPredict (\n", - " ON (SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0) AS DataTable\n", - " ON (SELECT * FROM aoa_byom_models \n", - " WHERE model_version='{model_version}') AS ModelTable DIMENSION\n", - " USING\n", - " Accumulate ('patientid')\n", - ") AS td;\n", - "\"\"\"\n", - "\n", - "pd.read_sql(query, get_connection()).head(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 54, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PatientIdjson_report
0545{\"output_probability\":[{\"0\":0.013714135,\"1\":0.98628587}],\"output_label\":[1]}
1265{\"output_probability\":[{\"0\":0.78800213,\"1\":0.21199787}],\"output_label\":[0]}
2600{\"output_probability\":[{\"0\":0.9766014,\"1\":0.023398578}],\"output_label\":[0]}
3530{\"output_probability\":[{\"0\":0.97246957,\"1\":0.027530432}],\"output_label\":[0]}
45{\"output_probability\":[{\"0\":0.9863973,\"1\":0.013602674}],\"output_label\":[0]}
5305{\"output_probability\":[{\"0\":0.8996898,\"1\":0.10031021}],\"output_label\":[0]}
620{\"output_probability\":[{\"0\":0.94164395,\"1\":0.058356047}],\"output_label\":[0]}
7570{\"output_probability\":[{\"0\":0.96782416,\"1\":0.03217584}],\"output_label\":[0]}
860{\"output_probability\":[{\"0\":0.994561,\"1\":0.0054389834}],\"output_label\":[0]}
9610{\"output_probability\":[{\"0\":0.9871934,\"1\":0.012806594}],\"output_label\":[0]}
\n", - "
" - ], - "text/plain": [ - " PatientId \\\n", - "0 545 \n", - "1 265 \n", - "2 600 \n", - "3 530 \n", - "4 5 \n", - "5 305 \n", - "6 20 \n", - "7 570 \n", - "8 60 \n", - "9 610 \n", - "\n", - " json_report \n", - "0 {\"output_probability\":[{\"0\":0.013714135,\"1\":0.98628587}],\"output_label\":[1]} \n", - "1 {\"output_probability\":[{\"0\":0.78800213,\"1\":0.21199787}],\"output_label\":[0]} \n", - "2 {\"output_probability\":[{\"0\":0.9766014,\"1\":0.023398578}],\"output_label\":[0]} \n", - "3 {\"output_probability\":[{\"0\":0.97246957,\"1\":0.027530432}],\"output_label\":[0]} \n", - "4 {\"output_probability\":[{\"0\":0.9863973,\"1\":0.013602674}],\"output_label\":[0]} \n", - "5 {\"output_probability\":[{\"0\":0.8996898,\"1\":0.10031021}],\"output_label\":[0]} \n", - "6 {\"output_probability\":[{\"0\":0.94164395,\"1\":0.058356047}],\"output_label\":[0]} \n", - "7 {\"output_probability\":[{\"0\":0.96782416,\"1\":0.03217584}],\"output_label\":[0]} \n", - "8 {\"output_probability\":[{\"0\":0.994561,\"1\":0.0054389834}],\"output_label\":[0]} \n", - "9 {\"output_probability\":[{\"0\":0.9871934,\"1\":0.012806594}],\"output_label\":[0]} " - ] - }, - "execution_count": 54, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "query = f\"\"\"\n", - "SELECT td.* FROM {byom_db}.ONNXPredict (\n", - " ON (SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0) AS DataTable\n", - " ON (SELECT * FROM aoa_byom_models \n", - " WHERE model_version='onnx-test') AS ModelTable DIMENSION\n", - " USING\n", - " Accumulate ('patientid')\n", - ") AS td;\n", - "\"\"\"\n", - "\n", - "pd.read_sql(query, get_connection()).head(10)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Custom BYOM Evaluation Logic\n", - "\n", - "You can define custom evaluation logic for BYOM models in ModelOps. This allows you to define your own charts, metrics etc that are to be created and captured as part of evaluation / comparison." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "from aoa.stats.stats import _capture_stats, _NpEncoder\n", - "import json\n", - "import logging\n", - "import sys\n", - "\n", - "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n", - "\n", - "\n", - "train_df = DataFrame.from_query(\"\"\"\n", - "SELECT \n", - " F.*, D.hasdiabetes \n", - "FROM pima_patient_features F\n", - "JOIN pima_patient_diagnoses D\n", - " ON F.patientid = D.patientid \n", - " WHERE F.patientid MOD 5 <> 0\n", - "\"\"\")\n", - "\n", - "data_stats = _capture_stats(df=train_df,\n", - " features=features,\n", - " targets=[target],\n", - " categorical=[target],\n", - " feature_metadata_fqtn=f\"{database}.aoa_feature_metadata\")\n", - "\n", - "with open(\"data_stats.json\", 'w+') as f:\n", - " json.dump(data_stats, f, indent=2, cls=_NpEncoder)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn import metrics\n", - "from teradataml import (\n", - " get_context,\n", - " DataFrame,\n", - " PMMLPredict,\n", - " configure\n", - ")\n", - "from aoa import (\n", - " record_evaluation_stats,\n", - " aoa_create_context,\n", - " store_byom_tmp,\n", - " ModelContext\n", - ")\n", - "\n", - "import os\n", - "import json\n", - "\n", - "\n", - "def plot_confusion_matrix(cf, img_filename):\n", - " import itertools\n", - " import matplotlib.pyplot as plt\n", - " plt.imshow(cf, cmap=plt.cm.Blues, interpolation='nearest')\n", - " plt.colorbar()\n", - " plt.title('Confusion Matrix')\n", - " plt.xlabel('Predicted')\n", - " plt.ylabel('Actual')\n", - " plt.xticks([0, 1], ['0', '1'])\n", - " plt.yticks([0, 1], ['0', '1'])\n", - "\n", - " thresh = cf.max() / 2.\n", - " for i, j in itertools.product(range(cf.shape[0]), range(cf.shape[1])):\n", - " plt.text(j, i, format(cf[i, j], 'd'), horizontalalignment='center',\n", - " color='white' if cf[i, j] > thresh else 'black')\n", - "\n", - " fig = plt.gcf()\n", - " fig.savefig(img_filename, dpi=500)\n", - " plt.clf()\n", - "\n", - "\n", - "def evaluate(context: ModelContext, **kwargs):\n", - " aoa_create_context()\n", - "\n", - " \n", - " # this evaluation.py can hanlde both onnx and pmml. usually, you would only need to support one but for \n", - " # demo purposes, we will show with both as we produce both onnx and pmml in this notebook.\n", - " \n", - " import glob\n", - " for file_name in glob.glob(f\"{context.artifact_input_path}/model.*\"):\n", - " model_type = file_name.split(\".\")[-1]\n", - " \n", - " with open(f\"{context.artifact_input_path}/model.{model_type}\", \"rb\") as f:\n", - " model_bytes = f.read()\n", - " \n", - " model = store_byom_tmp(get_context(), \"byom_models_tmp\", context.model_version, model_bytes)\n", - "\n", - " target_name = context.dataset_info.target_names[0]\n", - "\n", - " if model_type.upper() == \"ONNX\":\n", - " byom_target_sql = \"CAST(CAST(json_report AS JSON).JSONExtractValue('$.output_label[0]') AS INT)\"\n", - " mldb = os.environ.get(\"AOA_BYOM_INSTALL_DB\", \"MLDB\")\n", - "\n", - " query = f\"\"\"\n", - " SELECT sc.{context.dataset_info.entity_key}, {target_name}, sc.json_report\n", - " FROM {mldb}.ONNXPredict(\n", - " ON ({context.dataset_info.sql}) AS DataTable\n", - " ON (SELECT model_version as model_id, model FROM byom_models_tmp) AS ModelTable DIMENSION\n", - " USING\n", - " Accumulate('{context.dataset_info.entity_key}', '{target_name}')\n", - " ) sc;\n", - " \"\"\"\n", - "\n", - " predictions_df = DataFrame.from_query(query)\n", - " \n", - " elif model_type.upper() == \"PMML\":\n", - " byom_target_sql = \"CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT)\"\n", - " \n", - " pmml = PMMLPredict(\n", - " modeldata=model,\n", - " newdata=DataFrame.from_query(context.dataset_info.sql),\n", - " accumulate=[context.dataset_info.entity_key, target_name])\n", - " \n", - " predictions_df = pmml.result\n", - "\n", - " predictions_df.to_sql(table_name=\"predictions_tmp\", if_exists=\"replace\", temporary=True)\n", - "\n", - " metrics_df = DataFrame.from_query(f\"\"\"\n", - " SELECT \n", - " HasDiabetes as y_test, \n", - " {byom_target_sql} as y_pred\n", - " FROM predictions_tmp\n", - " \"\"\")\n", - " metrics_df = metrics_df.to_pandas()\n", - "\n", - " y_pred = metrics_df[[\"y_pred\"]]\n", - " y_test = metrics_df[[\"y_test\"]]\n", - "\n", - " evaluation = {\n", - " 'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),\n", - " 'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),\n", - " 'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),\n", - " 'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))\n", - " }\n", - "\n", - " with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n", - " json.dump(evaluation, f)\n", - "\n", - " # create confusion matrix plot\n", - " cf = metrics.confusion_matrix(y_test, y_pred)\n", - "\n", - " plot_confusion_matrix(cf, f\"{context.artifact_output_path}/confusion_matrix\")\n", - "\n", - " # calculate stats if training stats exist\n", - " if os.path.exists(f\"{context.artifact_input_path}/data_stats.json\"):\n", - " record_evaluation_stats(features_df=DataFrame.from_query(context.dataset_info.sql),\n", - " predicted_df=DataFrame(\"predictions_tmp\"),\n", - " context=context)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n", - "INFO:aoa.stats.stats:Computing evaluation dataset statistics\n", - "{'Accuracy': '0.75', 'Recall': '0.67', 'Precision': '0.67', 'f1-score': '0.67'}\n" - ] - }, - { - "data": { - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "from aoa import ModelContext, DatasetInfo\n", - "\n", - "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n", - "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n", - "\n", - "# define the evaluation dataset \n", - "sql = \"\"\"\n", - "SELECT \n", - " F.*, D.hasdiabetes \n", - "FROM PIMA_PATIENT_FEATURES F \n", - "JOIN PIMA_PATIENT_DIAGNOSES D\n", - "ON F.patientid = D.patientid\n", - " WHERE D.patientid MOD 5 = 0\n", - "\"\"\"\n", - "\n", - "feature_metadata = {\n", - " \"database\": database,\n", - " \"table\": \"aoa_feature_metadata\"\n", - "}\n", - "\n", - "entity_key = \"PatientId\"\n", - "target_names = [\"HasDiabetes\"]\n", - "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n", - "\n", - "dataset_info = DatasetInfo(sql=sql,\n", - " entity_key=entity_key,\n", - " feature_names=feature_names,\n", - " target_names=target_names,\n", - " feature_metadata=feature_metadata)\n", - "\n", - "ctx = ModelContext(hyperparams={},\n", - " dataset_info=dataset_info,\n", - " artifact_output_path=\"/tmp\",\n", - " artifact_input_path=\"./\",\n", - " model_version=\"v1\",\n", - " model_table=\"aoa_model_v1\")\n", - "\n", - "\n", - "# drop volatile table from session if executing multiple times\n", - "try:\n", - " get_context().execute(f\"DROP TABLE byom_models_tmp\")\n", - "except: \n", - " pass\n", - "\n", - "evaluate(context=ctx)\n", - "\n", - "# view evaluation results\n", - "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n", - " print(json.load(f))" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - "\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "
PatientIdHasDiabetesjson_report
3900{\"probability_0\":0.8534884407791314,\"probability_1\":0.1465115592208686,\"predicted_HasDiabetes\":0}
5750{\"probability_0\":0.5411704893844536,\"probability_1\":0.4588295106155465,\"predicted_HasDiabetes\":0}
7401{\"probability_0\":0.18309459019075702,\"probability_1\":0.816905409809243,\"predicted_HasDiabetes\":1}
2900{\"probability_0\":0.9275644263007603,\"probability_1\":0.0724355736992397,\"predicted_HasDiabetes\":0}
4300{\"probability_0\":0.9993898309624312,\"probability_1\":6.101690375688237E-4,\"predicted_HasDiabetes\":0}
4100{\"probability_0\":0.7306026985110992,\"probability_1\":0.26939730148890084,\"predicted_HasDiabetes\":0}
4600{\"probability_0\":0.7245463351662919,\"probability_1\":0.27545366483370815,\"predicted_HasDiabetes\":0}
3250{\"probability_0\":0.6100536262346685,\"probability_1\":0.3899463737653314,\"predicted_HasDiabetes\":0}
3601{\"probability_0\":0.014436294801631777,\"probability_1\":0.9855637051983682,\"predicted_HasDiabetes\":1}
5601{\"probability_0\":0.681609701643378,\"probability_1\":0.3183902983566221,\"predicted_HasDiabetes\":0}
" - ], - "text/plain": [ - " PatientId HasDiabetes json_report\n", - "0 725 0 {\"probability_0\":0.7656401654913679,\"probability_1\":0.23435983450863204,\"predicted_HasDiabetes\":0}\n", - "1 575 0 {\"probability_0\":0.5411704893844536,\"probability_1\":0.4588295106155465,\"predicted_HasDiabetes\":0}\n", - "2 740 1 {\"probability_0\":0.18309459019075702,\"probability_1\":0.816905409809243,\"predicted_HasDiabetes\":1}\n", - "3 145 0 {\"probability_0\":0.9954274192406505,\"probability_1\":0.004572580759349547,\"predicted_HasDiabetes\":0}\n", - "4 290 0 {\"probability_0\":0.9275644263007603,\"probability_1\":0.0724355736992397,\"predicted_HasDiabetes\":0}\n", - "5 410 0 {\"probability_0\":0.7306026985110992,\"probability_1\":0.26939730148890084,\"predicted_HasDiabetes\":0}\n", - "6 415 1 {\"probability_0\":0.37542607752957524,\"probability_1\":0.6245739224704248,\"predicted_HasDiabetes\":1}\n", - "7 570 0 {\"probability_0\":0.9871590600678464,\"probability_1\":0.012840939932153675,\"predicted_HasDiabetes\":0}\n", - "8 700 0 {\"probability_0\":0.9828242945003008,\"probability_1\":0.017175705499699205,\"predicted_HasDiabetes\":0}\n", - "9 465 0 {\"probability_0\":0.9821548641078263,\"probability_1\":0.01784513589217372,\"predicted_HasDiabetes\":0}" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "DataFrame.from_query(\"SELECT PatientId, HasDiabetes, json_report FROM predictions_tmp\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python [conda env:py39]", - "language": "python", - "name": "conda-env-py39-py" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.12" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/modules/modelops/attachments/BYOM_v7.ipynb b/modules/modelops/attachments/BYOM_v7.ipynb deleted file mode 100644 index 40c186975..000000000 --- a/modules/modelops/attachments/BYOM_v7.ipynb +++ /dev/null @@ -1,1088 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## BYOM In-Vantage Scoring with PMML and ONNX\n", - "\n", - "In this notebook, we will show you how to work with the Bring Your Own Model (BYOM) pattern and BYOM In-Vantage Scoring. This pattern allows you to use whatever data science platform you want to perform model development and experimentation. You can use the vast majority of popular data science libraries and transformations. The only constraint is that you can convert it to one of the following open formats\n", - "\n", - "- ONNX\n", - "- PMML\n", - "- H2O (MOJO)\n", - "- H2O (Driverless AI)\n", - "\n", - "ONNX is become more popular by the day. It is a very efficient model format which was created and is maintained by Microsoft and its adoption by other companies and libraries as the standard open format is incresingly rapidly. While the name suggests it is primarily related to neural networks, it can be used with most sklearn libraries and algorithms. \n", - "\n", - "\n", - "In this example, we will show you how you can develop in a notebook or other third-party tooling, produce a model and convert it to both `onnx` and `pmml` formats for deploying in Vantage with ModelOps." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import pandas as pd\n", - "import getpass\n", - "\n", - "from teradataml import (\n", - " create_context, \n", - " remove_context,\n", - " get_context,\n", - " get_connection,\n", - " DataFrame,\n", - " retrieve_byom,\n", - " PMMLPredict,\n", - " configure)" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Host: tdprd.td.teradata.com\n", - "Username: wf250003\n", - "Password: ········\n", - "VAL DB: TRNG_XSP\n", - "BYOM DB: TRNG_BYOM\n" - ] - }, - { - "data": { - "text/plain": [ - "Engine(teradatasql://wf250003:***@tdprd.td.teradata.com/?LOGDATA=%2A%2A%2A&LOGMECH=%2A%2A%2A)" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "host = input(\"Host: \")\n", - "username = input(\"Username: \")\n", - "password = getpass.getpass(\"Password: \")\n", - "val_db = input(\"VAL DB: \")\n", - "byom_db = input(\"BYOM DB: \")\n", - "\n", - "# configure byom/val installation\n", - "configure.val_install_location = val_db\n", - "configure.byom_install_location = byom_db\n", - "\n", - "# by default we assume your are using your user database. change as required\n", - "database = username\n", - "\n", - "create_context(host=host, username=username, password=password, logmech=\"TDNEGO\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "Pipeline(steps=[('scaler', MinMaxScaler()),\n", - " ('xgb',\n", - " XGBClassifier(base_score=0.5, booster='gbtree', callbacks=None,\n", - " colsample_bylevel=1, colsample_bynode=1,\n", - " colsample_bytree=1, early_stopping_rounds=None,\n", - " enable_categorical=False, eta=0.2,\n", - " eval_metric=None, gamma=0, gpu_id=-1,\n", - " grow_policy='depthwise', importance_type=None,\n", - " interaction_constraints='',\n", - " learning_rate=0.200000003, max_bin=256,\n", - " max_cat_to_onehot=4, max_delta_step=0,\n", - " max_depth=6, max_leaves=0, min_child_weight=1,\n", - " missing=nan, monotone_constraints='()',\n", - " n_estimators=100, n_jobs=0, num_parallel_tree=1,\n", - " predictor='auto', random_state=0, reg_alpha=0, ...))])" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from xgboost import XGBClassifier\n", - "from sklearn.preprocessing import MinMaxScaler\n", - "from sklearn.pipeline import Pipeline\n", - "\n", - "\n", - "train_pdf = DataFrame.from_query(\"\"\"\n", - "SELECT \n", - " F.*, D.hasdiabetes \n", - "FROM pima_patient_features F\n", - "JOIN pima_patient_diagnoses D\n", - " ON F.patientid = D.patientid \n", - " WHERE F.patientid MOD 5 <> 0\n", - "\"\"\").to_pandas(all_rows=True)\n", - "\n", - "features = [\"NumTimesPrg\", \"Age\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\"]\n", - "target = \"HasDiabetes\"\n", - "\n", - "# split data into X and y\n", - "X_train = train_pdf[features]\n", - "y_train = train_pdf[target]\n", - "\n", - "model = Pipeline([('scaler', MinMaxScaler()),\n", - " ('xgb', XGBClassifier(eta=0.2, max_depth=6))])\n", - "\n", - "model.fit(X_train, y_train)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Convert the model to PMML\n", - "\n", - "You can use the sklearn2pmml or the nyoka python libraries to convert to pmml. The nyoka is a python only package and so it is preferrable. " - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "from nyoka import xgboost_to_pmml\n", - "\n", - "xgboost_to_pmml(pipeline=model, col_names=features, target_name=target, pmml_f_name=\"model.pmml\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Convert the model to ONNX\n", - "\n", - "We can also convert the model to onnx format. This is a bit more involved as the client libraries for converting from sklearn/xgboost to onnx are not yet as mature.\n", - "\n", - "```\n", - "pip install onnx==1.10.2 skl2onnx==1.11.2 onnxruntime==1.9.0 protobuf==3.20.1 onnxmltools==1.7.0\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from skl2onnx import to_onnx\n", - "from skl2onnx import convert_sklearn, to_onnx, update_registered_converter\n", - "from skl2onnx.common.shape_calculator import (\n", - " calculate_linear_classifier_output_shapes,\n", - " calculate_linear_regressor_output_shapes)\n", - "from onnxmltools.convert.xgboost.operator_converters.XGBoost import convert_xgboost\n", - "from onnxmltools.convert import convert_xgboost as convert_xgboost_booster\n", - "\n", - "update_registered_converter(\n", - " XGBClassifier, 'XGBoostXGBClassifier',\n", - " calculate_linear_classifier_output_shapes, convert_xgboost,\n", - " options={'nocl': [True, False], 'zipmap': [True, False, 'columns']})\n", - "\n", - "\n", - "model_onnx = to_onnx(model, X_train.astype(np.float32), target_opset=15)\n", - "with open(\"model.onnx\", \"wb\") as f:\n", - " f.write(model_onnx.SerializeToString())\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Import into ModelOps to Operationalize\n", - "\n", - "Go to the ModelOps UI and import this as a new model version. Then follow the workflow to deploy. Note that you can also import programatically via the ModelOps Python SDK. \n", - "\n", - "You may be wondering why you can't just directly insert the onnx or pmml model directly into the database table. And the answer is you can. However, with ModelOps, you get full governance around this model deployment, including data drift and model monitoring and alerting. \n", - "\n", - "\n", - "### View Published Models\n", - "\n", - "Once deployed via ModelOps, we can view the models published to vantage by querying the table they are published to. Note this information is available via the AOA APIs also.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
model_versionmodel_idmodel_typeproject_iddeployed_atmodel
09de00f0d-060c-4737-b0a3-531768363ced2354a903-601b-5f72-b014-8983306005b4PMML414bec4e-c677-4f2e-a370-0076e57918ea2022-07-20 11:02:35.470b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ...
15761d5c1-bf57-456b-8076-c3062be0b5442354a903-601b-5f72-b014-8983306005b4PMML414bec4e-c677-4f2e-a370-0076e57918ea2022-07-18 07:04:19.430b'<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<PMML xmlns=\"http://www.dmg.org/PMML-4_4\" version=\"4.4\">\\n <Header copyright=\"Copyright (c) 2018 Software AG\" description=\"Default Description\">\\n <Application name=\"Nyoka\" version=\"4.3.0\"/>\\n ...
\n", - "
" - ], - "text/plain": [ - " model_version model_id \\\n", - "0 9de00f0d-060c-4737-b0a3-531768363ced 2354a903-601b-5f72-b014-8983306005b4 \n", - "1 5761d5c1-bf57-456b-8076-c3062be0b544 2354a903-601b-5f72-b014-8983306005b4 \n", - "\n", - " model_type project_id deployed_at \\\n", - "0 PMML 414bec4e-c677-4f2e-a370-0076e57918ea 2022-07-20 11:02:35.470 \n", - "1 PMML 414bec4e-c677-4f2e-a370-0076e57918ea 2022-07-18 07:04:19.430 \n", - "\n", - " model \n", - "0 b'\\n\\n
\\n \\n ... \n", - "1 b'\\n\\n
\\n \\n ... " - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "pd.options.display.max_colwidth = 250\n", - "pd.read_sql(\"SELECT TOP 2 * FROM aoa_byom_models\", get_connection())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## On-Demand Scoring" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PatientIdpredictionjson_report
05451{\"probability_0\":0.015597303259093032,\"probability_1\":0.984402696740907,\"predicted_HasDiabetes\":1}
12650{\"probability_0\":0.8621357683534957,\"probability_1\":0.1378642316465043,\"predicted_HasDiabetes\":0}
2400{\"probability_0\":0.574482742227689,\"probability_1\":0.425517257772311,\"predicted_HasDiabetes\":0}
33850{\"probability_0\":0.9902272802038437,\"probability_1\":0.009772719796156322,\"predicted_HasDiabetes\":0}
401{\"probability_0\":0.0950635688096706,\"probability_1\":0.9049364311903294,\"predicted_HasDiabetes\":1}
56000{\"probability_0\":0.9499693357624215,\"probability_1\":0.050030664237578494,\"predicted_HasDiabetes\":0}
65300{\"probability_0\":0.9729698349887085,\"probability_1\":0.02703016501129154,\"predicted_HasDiabetes\":0}
71201{\"probability_0\":0.11393098981414929,\"probability_1\":0.8860690101858507,\"predicted_HasDiabetes\":1}
86500{\"probability_0\":0.9820752256818963,\"probability_1\":0.0179247743181037,\"predicted_HasDiabetes\":0}
9800{\"probability_0\":0.9819309856605802,\"probability_1\":0.01806901433941982,\"predicted_HasDiabetes\":0}
\n", - "
" - ], - "text/plain": [ - " PatientId prediction \\\n", - "0 545 1 \n", - "1 265 0 \n", - "2 40 0 \n", - "3 385 0 \n", - "4 0 1 \n", - "5 600 0 \n", - "6 530 0 \n", - "7 120 1 \n", - "8 650 0 \n", - "9 80 0 \n", - "\n", - " json_report \n", - "0 {\"probability_0\":0.015597303259093032,\"probability_1\":0.984402696740907,\"predicted_HasDiabetes\":1} \n", - "1 {\"probability_0\":0.8621357683534957,\"probability_1\":0.1378642316465043,\"predicted_HasDiabetes\":0} \n", - "2 {\"probability_0\":0.574482742227689,\"probability_1\":0.425517257772311,\"predicted_HasDiabetes\":0} \n", - "3 {\"probability_0\":0.9902272802038437,\"probability_1\":0.009772719796156322,\"predicted_HasDiabetes\":0} \n", - "4 {\"probability_0\":0.0950635688096706,\"probability_1\":0.9049364311903294,\"predicted_HasDiabetes\":1} \n", - "5 {\"probability_0\":0.9499693357624215,\"probability_1\":0.050030664237578494,\"predicted_HasDiabetes\":0} \n", - "6 {\"probability_0\":0.9729698349887085,\"probability_1\":0.02703016501129154,\"predicted_HasDiabetes\":0} \n", - "7 {\"probability_0\":0.11393098981414929,\"probability_1\":0.8860690101858507,\"predicted_HasDiabetes\":1} \n", - "8 {\"probability_0\":0.9820752256818963,\"probability_1\":0.0179247743181037,\"predicted_HasDiabetes\":0} \n", - "9 {\"probability_0\":0.9819309856605802,\"probability_1\":0.01806901433941982,\"predicted_HasDiabetes\":0} " - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "model_version=\"1dbe5430-f8c5-4d32-b26c-a02476cba510\"\n", - "\n", - "model = DataFrame.from_query(f\"\"\"\n", - "SELECT * FROM aoa_byom_models \n", - " WHERE model_version='{model_version}'\n", - "\"\"\")\n", - "\n", - "\n", - "preds = PMMLPredict(\n", - " modeldata=model,\n", - " newdata=DataFrame.from_query(\"SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0\"),\n", - " accumulate=['PatientId'])\n", - "\n", - "preds.result.to_pandas().head(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PatientIdpredictionjson_report
010{\"probability_0\":0.9850747504768098,\"probability_1\":0.014925249523190227,\"predicted_HasDiabetes\":0}
130{\"probability_0\":0.9945255942085638,\"probability_1\":0.005474405791436156,\"predicted_HasDiabetes\":0}
281{\"probability_0\":0.05020155260184678,\"probability_1\":0.9497984473981532,\"predicted_HasDiabetes\":1}
391{\"probability_0\":0.28263442350828416,\"probability_1\":0.7173655764917158,\"predicted_HasDiabetes\":1}
4131{\"probability_0\":0.08177880479006427,\"probability_1\":0.9182211952099357,\"predicted_HasDiabetes\":1}
5171{\"probability_0\":0.23832592730256774,\"probability_1\":0.7616740726974323,\"predicted_HasDiabetes\":1}
6191{\"probability_0\":0.37077762620144294,\"probability_1\":0.629222373798557,\"predicted_HasDiabetes\":1}
7261{\"probability_0\":0.18449399225529128,\"probability_1\":0.8155060077447087,\"predicted_HasDiabetes\":1}
8431{\"probability_0\":0.006277949339332567,\"probability_1\":0.9937220506606674,\"predicted_HasDiabetes\":1}
9440{\"probability_0\":0.7477334297352949,\"probability_1\":0.25226657026470506,\"predicted_HasDiabetes\":0}
\n", - "
" - ], - "text/plain": [ - " PatientId prediction \\\n", - "0 1 0 \n", - "1 3 0 \n", - "2 8 1 \n", - "3 9 1 \n", - "4 13 1 \n", - "5 17 1 \n", - "6 19 1 \n", - "7 26 1 \n", - "8 43 1 \n", - "9 44 0 \n", - "\n", - " json_report \n", - "0 {\"probability_0\":0.9850747504768098,\"probability_1\":0.014925249523190227,\"predicted_HasDiabetes\":0} \n", - "1 {\"probability_0\":0.9945255942085638,\"probability_1\":0.005474405791436156,\"predicted_HasDiabetes\":0} \n", - "2 {\"probability_0\":0.05020155260184678,\"probability_1\":0.9497984473981532,\"predicted_HasDiabetes\":1} \n", - "3 {\"probability_0\":0.28263442350828416,\"probability_1\":0.7173655764917158,\"predicted_HasDiabetes\":1} \n", - "4 {\"probability_0\":0.08177880479006427,\"probability_1\":0.9182211952099357,\"predicted_HasDiabetes\":1} \n", - "5 {\"probability_0\":0.23832592730256774,\"probability_1\":0.7616740726974323,\"predicted_HasDiabetes\":1} \n", - "6 {\"probability_0\":0.37077762620144294,\"probability_1\":0.629222373798557,\"predicted_HasDiabetes\":1} \n", - "7 {\"probability_0\":0.18449399225529128,\"probability_1\":0.8155060077447087,\"predicted_HasDiabetes\":1} \n", - "8 {\"probability_0\":0.006277949339332567,\"probability_1\":0.9937220506606674,\"predicted_HasDiabetes\":1} \n", - "9 {\"probability_0\":0.7477334297352949,\"probability_1\":0.25226657026470506,\"predicted_HasDiabetes\":0} " - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "query = f\"\"\"\n", - "SELECT * FROM {byom_db}.PMMLPredict (\n", - " ON (SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0) AS DataTable\n", - " ON (SELECT * FROM aoa_byom_models \n", - " WHERE model_version='{model_version}') AS ModelTable DIMENSION\n", - " USING\n", - " Accumulate ('patientid')\n", - ") AS td;\n", - "\"\"\"\n", - "\n", - "pd.read_sql(query, get_connection()).head(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 54, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PatientIdjson_report
0545{\"output_probability\":[{\"0\":0.013714135,\"1\":0.98628587}],\"output_label\":[1]}
1265{\"output_probability\":[{\"0\":0.78800213,\"1\":0.21199787}],\"output_label\":[0]}
2600{\"output_probability\":[{\"0\":0.9766014,\"1\":0.023398578}],\"output_label\":[0]}
3530{\"output_probability\":[{\"0\":0.97246957,\"1\":0.027530432}],\"output_label\":[0]}
45{\"output_probability\":[{\"0\":0.9863973,\"1\":0.013602674}],\"output_label\":[0]}
5305{\"output_probability\":[{\"0\":0.8996898,\"1\":0.10031021}],\"output_label\":[0]}
620{\"output_probability\":[{\"0\":0.94164395,\"1\":0.058356047}],\"output_label\":[0]}
7570{\"output_probability\":[{\"0\":0.96782416,\"1\":0.03217584}],\"output_label\":[0]}
860{\"output_probability\":[{\"0\":0.994561,\"1\":0.0054389834}],\"output_label\":[0]}
9610{\"output_probability\":[{\"0\":0.9871934,\"1\":0.012806594}],\"output_label\":[0]}
\n", - "
" - ], - "text/plain": [ - " PatientId \\\n", - "0 545 \n", - "1 265 \n", - "2 600 \n", - "3 530 \n", - "4 5 \n", - "5 305 \n", - "6 20 \n", - "7 570 \n", - "8 60 \n", - "9 610 \n", - "\n", - " json_report \n", - "0 {\"output_probability\":[{\"0\":0.013714135,\"1\":0.98628587}],\"output_label\":[1]} \n", - "1 {\"output_probability\":[{\"0\":0.78800213,\"1\":0.21199787}],\"output_label\":[0]} \n", - "2 {\"output_probability\":[{\"0\":0.9766014,\"1\":0.023398578}],\"output_label\":[0]} \n", - "3 {\"output_probability\":[{\"0\":0.97246957,\"1\":0.027530432}],\"output_label\":[0]} \n", - "4 {\"output_probability\":[{\"0\":0.9863973,\"1\":0.013602674}],\"output_label\":[0]} \n", - "5 {\"output_probability\":[{\"0\":0.8996898,\"1\":0.10031021}],\"output_label\":[0]} \n", - "6 {\"output_probability\":[{\"0\":0.94164395,\"1\":0.058356047}],\"output_label\":[0]} \n", - "7 {\"output_probability\":[{\"0\":0.96782416,\"1\":0.03217584}],\"output_label\":[0]} \n", - "8 {\"output_probability\":[{\"0\":0.994561,\"1\":0.0054389834}],\"output_label\":[0]} \n", - "9 {\"output_probability\":[{\"0\":0.9871934,\"1\":0.012806594}],\"output_label\":[0]} " - ] - }, - "execution_count": 54, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "query = f\"\"\"\n", - "SELECT td.* FROM {byom_db}.ONNXPredict (\n", - " ON (SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0) AS DataTable\n", - " ON (SELECT * FROM aoa_byom_models \n", - " WHERE model_version='onnx-test') AS ModelTable DIMENSION\n", - " USING\n", - " Accumulate ('patientid')\n", - ") AS td;\n", - "\"\"\"\n", - "\n", - "pd.read_sql(query, get_connection()).head(10)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Custom BYOM Evaluation Logic\n", - "\n", - "You can define custom evaluation logic for BYOM models in ModelOps. This allows you to define your own charts, metrics etc that are to be created and captured as part of evaluation / comparison." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "from aoa.stats.stats import _capture_stats, _NpEncoder\n", - "import json\n", - "import logging\n", - "import sys\n", - "\n", - "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n", - "\n", - "\n", - "train_df = DataFrame.from_query(\"\"\"\n", - "SELECT \n", - " F.*, D.hasdiabetes \n", - "FROM pima_patient_features F\n", - "JOIN pima_patient_diagnoses D\n", - " ON F.patientid = D.patientid \n", - " WHERE F.patientid MOD 5 <> 0\n", - "\"\"\")\n", - "\n", - "data_stats = _capture_stats(df=train_df,\n", - " features=features,\n", - " targets=[target],\n", - " categorical=[target],\n", - " feature_metadata_fqtn=f\"{database}.aoa_feature_metadata\")\n", - "\n", - "with open(\"data_stats.json\", 'w+') as f:\n", - " json.dump(data_stats, f, indent=2, cls=_NpEncoder)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "from sklearn import metrics\n", - "from teradataml import (\n", - " get_context,\n", - " DataFrame,\n", - " PMMLPredict,\n", - " configure\n", - ")\n", - "from aoa import (\n", - " record_evaluation_stats,\n", - " aoa_create_context,\n", - " store_byom_tmp,\n", - " ModelContext\n", - ")\n", - "\n", - "import os\n", - "import json\n", - "\n", - "\n", - "def plot_confusion_matrix(cf, img_filename):\n", - " import itertools\n", - " import matplotlib.pyplot as plt\n", - " plt.imshow(cf, cmap=plt.cm.Blues, interpolation='nearest')\n", - " plt.colorbar()\n", - " plt.title('Confusion Matrix')\n", - " plt.xlabel('Predicted')\n", - " plt.ylabel('Actual')\n", - " plt.xticks([0, 1], ['0', '1'])\n", - " plt.yticks([0, 1], ['0', '1'])\n", - "\n", - " thresh = cf.max() / 2.\n", - " for i, j in itertools.product(range(cf.shape[0]), range(cf.shape[1])):\n", - " plt.text(j, i, format(cf[i, j], 'd'), horizontalalignment='center',\n", - " color='white' if cf[i, j] > thresh else 'black')\n", - "\n", - " fig = plt.gcf()\n", - " fig.savefig(img_filename, dpi=500)\n", - " plt.clf()\n", - "\n", - "\n", - "def evaluate(context: ModelContext, **kwargs):\n", - " aoa_create_context()\n", - "\n", - " \n", - " # this evaluation.py can hanlde both onnx and pmml. usually, you would only need to support one but for \n", - " # demo purposes, we will show with both as we produce both onnx and pmml in this notebook.\n", - " \n", - " import glob\n", - " for file_name in glob.glob(f\"{context.artifact_input_path}/model.*\"):\n", - " model_type = file_name.split(\".\")[-1]\n", - " \n", - " with open(f\"{context.artifact_input_path}/model.{model_type}\", \"rb\") as f:\n", - " model_bytes = f.read()\n", - " \n", - " model = store_byom_tmp(get_context(), \"byom_models_tmp\", context.model_version, model_bytes)\n", - "\n", - " target_name = context.dataset_info.target_names[0]\n", - "\n", - " if model_type.upper() == \"ONNX\":\n", - " byom_target_sql = \"CAST(CAST(json_report AS JSON).JSONExtractValue('$.output_label[0]') AS INT)\"\n", - " mldb = os.environ.get(\"AOA_BYOM_INSTALL_DB\", \"MLDB\")\n", - "\n", - " query = f\"\"\"\n", - " SELECT sc.{context.dataset_info.entity_key}, {target_name}, sc.json_report\n", - " FROM {mldb}.ONNXPredict(\n", - " ON ({context.dataset_info.sql}) AS DataTable\n", - " ON (SELECT model_version as model_id, model FROM byom_models_tmp) AS ModelTable DIMENSION\n", - " USING\n", - " Accumulate('{context.dataset_info.entity_key}', '{target_name}')\n", - " ) sc;\n", - " \"\"\"\n", - "\n", - " predictions_df = DataFrame.from_query(query)\n", - " \n", - " elif model_type.upper() == \"PMML\":\n", - " byom_target_sql = \"CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT)\"\n", - " \n", - " pmml = PMMLPredict(\n", - " modeldata=model,\n", - " newdata=DataFrame.from_query(context.dataset_info.sql),\n", - " accumulate=[context.dataset_info.entity_key, target_name])\n", - " \n", - " predictions_df = pmml.result\n", - "\n", - " predictions_df.to_sql(table_name=\"predictions_tmp\", if_exists=\"replace\", temporary=True)\n", - "\n", - " metrics_df = DataFrame.from_query(f\"\"\"\n", - " SELECT \n", - " HasDiabetes as y_test, \n", - " {byom_target_sql} as y_pred\n", - " FROM predictions_tmp\n", - " \"\"\")\n", - " metrics_df = metrics_df.to_pandas()\n", - "\n", - " y_pred = metrics_df[[\"y_pred\"]]\n", - " y_test = metrics_df[[\"y_test\"]]\n", - "\n", - " evaluation = {\n", - " 'Accuracy': '{:.2f}'.format(metrics.accuracy_score(y_test, y_pred)),\n", - " 'Recall': '{:.2f}'.format(metrics.recall_score(y_test, y_pred)),\n", - " 'Precision': '{:.2f}'.format(metrics.precision_score(y_test, y_pred)),\n", - " 'f1-score': '{:.2f}'.format(metrics.f1_score(y_test, y_pred))\n", - " }\n", - "\n", - " with open(f\"{context.artifact_output_path}/metrics.json\", \"w+\") as f:\n", - " json.dump(evaluation, f)\n", - "\n", - " # create confusion matrix plot\n", - " cf = metrics.confusion_matrix(y_test, y_pred)\n", - "\n", - " plot_confusion_matrix(cf, f\"{context.artifact_output_path}/confusion_matrix\")\n", - "\n", - " # calculate stats if training stats exist\n", - " if os.path.exists(f\"{context.artifact_input_path}/data_stats.json\"):\n", - " record_evaluation_stats(features_df=DataFrame.from_query(context.dataset_info.sql),\n", - " predicted_df=DataFrame(\"predictions_tmp\"),\n", - " context=context)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "INFO:aoa.util.connections:teradataml context already exists. Skipping create_context.\n", - "INFO:aoa.stats.stats:Computing evaluation dataset statistics\n", - "{'Accuracy': '0.75', 'Recall': '0.67', 'Precision': '0.67', 'f1-score': '0.67'}\n" - ] - }, - { - "data": { - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "from aoa import ModelContext, DatasetInfo\n", - "\n", - "# Define the ModelContext to test with. The ModelContext is created and managed automatically by ModelOps \n", - "# when it executes your code via CLI / UI. However, for testing in the notebook, you can define as follows\n", - "\n", - "# define the evaluation dataset \n", - "sql = \"\"\"\n", - "SELECT \n", - " F.*, D.hasdiabetes \n", - "FROM PIMA_PATIENT_FEATURES F \n", - "JOIN PIMA_PATIENT_DIAGNOSES D\n", - "ON F.patientid = D.patientid\n", - " WHERE D.patientid MOD 5 = 0\n", - "\"\"\"\n", - "\n", - "feature_metadata = {\n", - " \"database\": database,\n", - " \"table\": \"aoa_feature_metadata\"\n", - "}\n", - "\n", - "entity_key = \"PatientId\"\n", - "target_names = [\"HasDiabetes\"]\n", - "feature_names = [\"NumTimesPrg\", \"PlGlcConc\", \"BloodP\", \"SkinThick\", \"TwoHourSerIns\", \"BMI\", \"DiPedFunc\", \"Age\"]\n", - "\n", - "dataset_info = DatasetInfo(sql=sql,\n", - " entity_key=entity_key,\n", - " feature_names=feature_names,\n", - " target_names=target_names,\n", - " feature_metadata=feature_metadata)\n", - "\n", - "ctx = ModelContext(hyperparams={},\n", - " dataset_info=dataset_info,\n", - " artifact_output_path=\"/tmp\",\n", - " artifact_input_path=\"./\",\n", - " model_version=\"v1\",\n", - " model_table=\"aoa_model_v1\")\n", - "\n", - "\n", - "# drop volatile table from session if executing multiple times\n", - "try:\n", - " get_context().execute(f\"DROP TABLE byom_models_tmp\")\n", - "except: \n", - " pass\n", - "\n", - "evaluate(context=ctx)\n", - "\n", - "# view evaluation results\n", - "with open(f\"{ctx.artifact_output_path}/metrics.json\") as f:\n", - " print(json.load(f))" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - "\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "\t\n", - "\t\t\n", - "\t\t\n", - "\t\t\n", - "\t\n", - "
PatientIdHasDiabetesjson_report
3900{\"probability_0\":0.8534884407791314,\"probability_1\":0.1465115592208686,\"predicted_HasDiabetes\":0}
5750{\"probability_0\":0.5411704893844536,\"probability_1\":0.4588295106155465,\"predicted_HasDiabetes\":0}
7401{\"probability_0\":0.18309459019075702,\"probability_1\":0.816905409809243,\"predicted_HasDiabetes\":1}
2900{\"probability_0\":0.9275644263007603,\"probability_1\":0.0724355736992397,\"predicted_HasDiabetes\":0}
4300{\"probability_0\":0.9993898309624312,\"probability_1\":6.101690375688237E-4,\"predicted_HasDiabetes\":0}
4100{\"probability_0\":0.7306026985110992,\"probability_1\":0.26939730148890084,\"predicted_HasDiabetes\":0}
4600{\"probability_0\":0.7245463351662919,\"probability_1\":0.27545366483370815,\"predicted_HasDiabetes\":0}
3250{\"probability_0\":0.6100536262346685,\"probability_1\":0.3899463737653314,\"predicted_HasDiabetes\":0}
3601{\"probability_0\":0.014436294801631777,\"probability_1\":0.9855637051983682,\"predicted_HasDiabetes\":1}
5601{\"probability_0\":0.681609701643378,\"probability_1\":0.3183902983566221,\"predicted_HasDiabetes\":0}
" - ], - "text/plain": [ - " PatientId HasDiabetes json_report\n", - "0 725 0 {\"probability_0\":0.7656401654913679,\"probability_1\":0.23435983450863204,\"predicted_HasDiabetes\":0}\n", - "1 575 0 {\"probability_0\":0.5411704893844536,\"probability_1\":0.4588295106155465,\"predicted_HasDiabetes\":0}\n", - "2 740 1 {\"probability_0\":0.18309459019075702,\"probability_1\":0.816905409809243,\"predicted_HasDiabetes\":1}\n", - "3 145 0 {\"probability_0\":0.9954274192406505,\"probability_1\":0.004572580759349547,\"predicted_HasDiabetes\":0}\n", - "4 290 0 {\"probability_0\":0.9275644263007603,\"probability_1\":0.0724355736992397,\"predicted_HasDiabetes\":0}\n", - "5 410 0 {\"probability_0\":0.7306026985110992,\"probability_1\":0.26939730148890084,\"predicted_HasDiabetes\":0}\n", - "6 415 1 {\"probability_0\":0.37542607752957524,\"probability_1\":0.6245739224704248,\"predicted_HasDiabetes\":1}\n", - "7 570 0 {\"probability_0\":0.9871590600678464,\"probability_1\":0.012840939932153675,\"predicted_HasDiabetes\":0}\n", - "8 700 0 {\"probability_0\":0.9828242945003008,\"probability_1\":0.017175705499699205,\"predicted_HasDiabetes\":0}\n", - "9 465 0 {\"probability_0\":0.9821548641078263,\"probability_1\":0.01784513589217372,\"predicted_HasDiabetes\":0}" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "DataFrame.from_query(\"SELECT PatientId, HasDiabetes, json_report FROM predictions_tmp\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python [conda env:py39]", - "language": "python", - "name": "conda-env-py39-py" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.12" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/modules/modelops/attachments/ModelOps_BYOM_files_v6.zip b/modules/modelops/attachments/ModelOps_BYOM_files_v6.zip deleted file mode 100644 index c2feea759..000000000 Binary files a/modules/modelops/attachments/ModelOps_BYOM_files_v6.zip and /dev/null differ diff --git a/modules/modelops/attachments/ModelOps_BYOM_files_v7.zip b/modules/modelops/attachments/ModelOps_BYOM_files_v7.zip deleted file mode 100644 index 5f01472cc..000000000 Binary files a/modules/modelops/attachments/ModelOps_BYOM_files_v7.zip and /dev/null differ diff --git a/modules/modelops/attachments/ModelOps_Quickstart_BYOM.zip b/modules/modelops/attachments/ModelOps_Quickstart_BYOM.zip new file mode 100644 index 000000000..c0fe76c0d Binary files /dev/null and b/modules/modelops/attachments/ModelOps_Quickstart_BYOM.zip differ diff --git a/modules/modelops/images/Define_BYOM_Model.png b/modules/modelops/images/Define_BYOM_Model.png new file mode 100644 index 000000000..b9529f93a Binary files /dev/null and b/modules/modelops/images/Define_BYOM_Model.png differ diff --git a/modules/modelops/images/Personal_Connection.png b/modules/modelops/images/Personal_Connection.png new file mode 100644 index 000000000..0bccd123e Binary files /dev/null and b/modules/modelops/images/Personal_Connection.png differ diff --git a/modules/modelops/images/Project_Creating.png b/modules/modelops/images/Project_Creating.png new file mode 100644 index 000000000..b9c40bd0a Binary files /dev/null and b/modules/modelops/images/Project_Creating.png differ diff --git a/modules/modelops/images/alert_configuration.png b/modules/modelops/images/alert_configuration.png new file mode 100644 index 000000000..15f3a92fd Binary files /dev/null and b/modules/modelops/images/alert_configuration.png differ diff --git a/modules/modelops/images/alert_configuration2.png b/modules/modelops/images/alert_configuration2.png new file mode 100644 index 000000000..3a5ce5e25 Binary files /dev/null and b/modules/modelops/images/alert_configuration2.png differ diff --git a/modules/modelops/images/alert_configuration3.png b/modules/modelops/images/alert_configuration3.png new file mode 100644 index 000000000..22e47062c Binary files /dev/null and b/modules/modelops/images/alert_configuration3.png differ diff --git a/modules/modelops/images/alert_configuration4.png b/modules/modelops/images/alert_configuration4.png new file mode 100644 index 000000000..17a359204 Binary files /dev/null and b/modules/modelops/images/alert_configuration4.png differ diff --git a/modules/modelops/images/alert_new1.png b/modules/modelops/images/alert_new1.png new file mode 100644 index 000000000..7015a8061 Binary files /dev/null and b/modules/modelops/images/alert_new1.png differ diff --git a/modules/modelops/images/alert_new2.png b/modules/modelops/images/alert_new2.png new file mode 100644 index 000000000..3c8f66b1d Binary files /dev/null and b/modules/modelops/images/alert_new2.png differ diff --git a/modules/modelops/images/alert_new3.png b/modules/modelops/images/alert_new3.png new file mode 100644 index 000000000..8daa9d751 Binary files /dev/null and b/modules/modelops/images/alert_new3.png differ diff --git a/modules/modelops/images/byom_basic.png b/modules/modelops/images/byom_basic.png new file mode 100644 index 000000000..84d55bf89 Binary files /dev/null and b/modules/modelops/images/byom_basic.png differ diff --git a/modules/modelops/images/byom_meth.png b/modules/modelops/images/byom_meth.png new file mode 100644 index 000000000..a699ba7ca Binary files /dev/null and b/modules/modelops/images/byom_meth.png differ diff --git a/modules/modelops/images/byom_model.png b/modules/modelops/images/byom_model.png new file mode 100644 index 000000000..054f0698c Binary files /dev/null and b/modules/modelops/images/byom_model.png differ diff --git a/modules/modelops/images/byom_monitoring1.png b/modules/modelops/images/byom_monitoring1.png new file mode 100644 index 000000000..95cde0a11 Binary files /dev/null and b/modules/modelops/images/byom_monitoring1.png differ diff --git a/modules/modelops/images/byom_monitoring2.png b/modules/modelops/images/byom_monitoring2.png new file mode 100644 index 000000000..15843f904 Binary files /dev/null and b/modules/modelops/images/byom_monitoring2.png differ diff --git a/modules/modelops/images/byom_monitoring_3.png b/modules/modelops/images/byom_monitoring_3.png new file mode 100644 index 000000000..26cc16eaa Binary files /dev/null and b/modules/modelops/images/byom_monitoring_3.png differ diff --git a/modules/modelops/images/byom_monitoring_save.png b/modules/modelops/images/byom_monitoring_save.png new file mode 100644 index 000000000..f18a6f556 Binary files /dev/null and b/modules/modelops/images/byom_monitoring_save.png differ diff --git a/modules/modelops/images/dataset_template.png b/modules/modelops/images/dataset_template.png new file mode 100644 index 000000000..990a8455c Binary files /dev/null and b/modules/modelops/images/dataset_template.png differ diff --git a/modules/modelops/images/dataset_template2.png b/modules/modelops/images/dataset_template2.png new file mode 100644 index 000000000..122892cb4 Binary files /dev/null and b/modules/modelops/images/dataset_template2.png differ diff --git a/modules/modelops/images/dataset_template_features.png b/modules/modelops/images/dataset_template_features.png new file mode 100644 index 000000000..273acdc28 Binary files /dev/null and b/modules/modelops/images/dataset_template_features.png differ diff --git a/modules/modelops/images/dataset_template_prediction.png b/modules/modelops/images/dataset_template_prediction.png new file mode 100644 index 000000000..5b60a569c Binary files /dev/null and b/modules/modelops/images/dataset_template_prediction.png differ diff --git a/modules/modelops/images/dataset_template_target.png b/modules/modelops/images/dataset_template_target.png new file mode 100644 index 000000000..5f4094d8e Binary files /dev/null and b/modules/modelops/images/dataset_template_target.png differ diff --git a/modules/modelops/images/datasets_created.png b/modules/modelops/images/datasets_created.png new file mode 100644 index 000000000..adb5b4ff9 Binary files /dev/null and b/modules/modelops/images/datasets_created.png differ diff --git a/modules/modelops/images/define_new.png b/modules/modelops/images/define_new.png new file mode 100644 index 000000000..a9952d123 Binary files /dev/null and b/modules/modelops/images/define_new.png differ diff --git a/modules/modelops/images/deploy.png b/modules/modelops/images/deploy.png new file mode 100644 index 000000000..d895dcea0 Binary files /dev/null and b/modules/modelops/images/deploy.png differ diff --git a/modules/modelops/images/deploy_details1.png b/modules/modelops/images/deploy_details1.png new file mode 100644 index 000000000..46d97a93c Binary files /dev/null and b/modules/modelops/images/deploy_details1.png differ diff --git a/modules/modelops/images/deploy_details2.png b/modules/modelops/images/deploy_details2.png new file mode 100644 index 000000000..4ea6e9d0a Binary files /dev/null and b/modules/modelops/images/deploy_details2.png differ diff --git a/modules/modelops/images/deploy_details3.png b/modules/modelops/images/deploy_details3.png new file mode 100644 index 000000000..4c7961b2b Binary files /dev/null and b/modules/modelops/images/deploy_details3.png differ diff --git a/modules/modelops/images/deploy_job.png b/modules/modelops/images/deploy_job.png new file mode 100644 index 000000000..848d3e5bf Binary files /dev/null and b/modules/modelops/images/deploy_job.png differ diff --git a/modules/modelops/images/deployment_evaluate.png b/modules/modelops/images/deployment_evaluate.png new file mode 100644 index 000000000..7b815a3ae Binary files /dev/null and b/modules/modelops/images/deployment_evaluate.png differ diff --git a/modules/modelops/images/deployment_evaluate2.png b/modules/modelops/images/deployment_evaluate2.png new file mode 100644 index 000000000..72e01fc81 Binary files /dev/null and b/modules/modelops/images/deployment_evaluate2.png differ diff --git a/modules/modelops/images/deployment_jobs.png b/modules/modelops/images/deployment_jobs.png new file mode 100644 index 000000000..7f03703fb Binary files /dev/null and b/modules/modelops/images/deployment_jobs.png differ diff --git a/modules/modelops/images/deployment_jobs2.png b/modules/modelops/images/deployment_jobs2.png new file mode 100644 index 000000000..dd87da9b6 Binary files /dev/null and b/modules/modelops/images/deployment_jobs2.png differ diff --git a/modules/modelops/images/deployment_predictions.png b/modules/modelops/images/deployment_predictions.png new file mode 100644 index 000000000..8a647adf1 Binary files /dev/null and b/modules/modelops/images/deployment_predictions.png differ diff --git a/modules/modelops/images/deployments.png b/modules/modelops/images/deployments.png new file mode 100644 index 000000000..ce5ea1465 Binary files /dev/null and b/modules/modelops/images/deployments.png differ diff --git a/modules/modelops/images/enable_alerts.png b/modules/modelops/images/enable_alerts.png new file mode 100644 index 000000000..20d9b78b8 Binary files /dev/null and b/modules/modelops/images/enable_alerts.png differ diff --git a/modules/modelops/images/evaluation2.png b/modules/modelops/images/evaluation2.png new file mode 100644 index 000000000..5e0d6dd60 Binary files /dev/null and b/modules/modelops/images/evaluation2.png differ diff --git a/modules/modelops/images/evaluation2_detail.png b/modules/modelops/images/evaluation2_detail.png new file mode 100644 index 000000000..e99fd0c76 Binary files /dev/null and b/modules/modelops/images/evaluation2_detail.png differ diff --git a/modules/modelops/images/evaluation_dataset.png b/modules/modelops/images/evaluation_dataset.png new file mode 100644 index 000000000..369238299 Binary files /dev/null and b/modules/modelops/images/evaluation_dataset.png differ diff --git a/modules/modelops/images/evaluation_dataset_basic.png b/modules/modelops/images/evaluation_dataset_basic.png new file mode 100644 index 000000000..0b9127b01 Binary files /dev/null and b/modules/modelops/images/evaluation_dataset_basic.png differ diff --git a/modules/modelops/images/evaluation_job.png b/modules/modelops/images/evaluation_job.png new file mode 100644 index 000000000..a56544d3a Binary files /dev/null and b/modules/modelops/images/evaluation_job.png differ diff --git a/modules/modelops/images/evaluation_report.png b/modules/modelops/images/evaluation_report.png new file mode 100644 index 000000000..9c250771e Binary files /dev/null and b/modules/modelops/images/evaluation_report.png differ diff --git a/modules/modelops/images/evaluation_report2.png b/modules/modelops/images/evaluation_report2.png new file mode 100644 index 000000000..e2de8f665 Binary files /dev/null and b/modules/modelops/images/evaluation_report2.png differ diff --git a/modules/modelops/images/feature_drift.png b/modules/modelops/images/feature_drift.png new file mode 100644 index 000000000..957cc0b50 Binary files /dev/null and b/modules/modelops/images/feature_drift.png differ diff --git a/modules/modelops/images/go.png b/modules/modelops/images/go.png new file mode 100644 index 000000000..6d18df6f2 Binary files /dev/null and b/modules/modelops/images/go.png differ diff --git a/modules/modelops/images/healthcheck.png b/modules/modelops/images/healthcheck.png new file mode 100644 index 000000000..5b4aa6cc0 Binary files /dev/null and b/modules/modelops/images/healthcheck.png differ diff --git a/modules/modelops/images/jobs.png b/modules/modelops/images/jobs.png new file mode 100644 index 000000000..eb65981e1 Binary files /dev/null and b/modules/modelops/images/jobs.png differ diff --git a/modules/modelops/images/model_evaluate.png b/modules/modelops/images/model_evaluate.png new file mode 100644 index 000000000..8895e2828 Binary files /dev/null and b/modules/modelops/images/model_evaluate.png differ diff --git a/modules/modelops/images/model_evaluate2.png b/modules/modelops/images/model_evaluate2.png new file mode 100644 index 000000000..cfa64bdd7 Binary files /dev/null and b/modules/modelops/images/model_evaluate2.png differ diff --git a/modules/modelops/images/model_version.png b/modules/modelops/images/model_version.png new file mode 100644 index 000000000..9bcc58445 Binary files /dev/null and b/modules/modelops/images/model_version.png differ diff --git a/modules/modelops/images/performance.png b/modules/modelops/images/performance.png new file mode 100644 index 000000000..b45149ba5 Binary files /dev/null and b/modules/modelops/images/performance.png differ diff --git a/modules/modelops/images/personal1.png b/modules/modelops/images/personal1.png new file mode 100644 index 000000000..fdfa547c8 Binary files /dev/null and b/modules/modelops/images/personal1.png differ diff --git a/modules/modelops/images/prediction_drift.png b/modules/modelops/images/prediction_drift.png new file mode 100644 index 000000000..24bab4166 Binary files /dev/null and b/modules/modelops/images/prediction_drift.png differ diff --git a/modules/modelops/images/projects.png b/modules/modelops/images/projects.png new file mode 100644 index 000000000..4e404e916 Binary files /dev/null and b/modules/modelops/images/projects.png differ diff --git a/modules/modelops/images/projects_quickstart.png b/modules/modelops/images/projects_quickstart.png new file mode 100644 index 000000000..f8c5e1682 Binary files /dev/null and b/modules/modelops/images/projects_quickstart.png differ diff --git a/modules/modelops/images/save_continue.png b/modules/modelops/images/save_continue.png new file mode 100644 index 000000000..d83105061 Binary files /dev/null and b/modules/modelops/images/save_continue.png differ diff --git a/modules/modelops/images/statistics_job.png b/modules/modelops/images/statistics_job.png new file mode 100644 index 000000000..b0b927c2d Binary files /dev/null and b/modules/modelops/images/statistics_job.png differ diff --git a/modules/modelops/images/training_dataset.png b/modules/modelops/images/training_dataset.png new file mode 100644 index 000000000..f00845b84 Binary files /dev/null and b/modules/modelops/images/training_dataset.png differ diff --git a/modules/modelops/images/training_dataset_basic.png b/modules/modelops/images/training_dataset_basic.png new file mode 100644 index 000000000..e77d86127 Binary files /dev/null and b/modules/modelops/images/training_dataset_basic.png differ diff --git a/modules/modelops/images/view_details.png b/modules/modelops/images/view_details.png new file mode 100644 index 000000000..195e77c01 Binary files /dev/null and b/modules/modelops/images/view_details.png differ diff --git a/modules/modelops/pages/deploy-and-monitor-machine-learning-models-with-teradata-modelops-and-byom.adoc b/modules/modelops/pages/deploy-and-monitor-machine-learning-models-with-teradata-modelops-and-byom.adoc index 6b6d0edcf..806a985ab 100644 --- a/modules/modelops/pages/deploy-and-monitor-machine-learning-models-with-teradata-modelops-and-byom.adoc +++ b/modules/modelops/pages/deploy-and-monitor-machine-learning-models-with-teradata-modelops-and-byom.adoc @@ -2,120 +2,503 @@ :experimental: :page-author: Pablo Escobar de la Oliva :page-email: pablo.escobardelaoliva@teradata.com -:page-revdate: May 29th, 2023 +:page-revdate: May 1st, 2024 :description: Tutorial for deploying and monitoring a PMML model into Vantage using ClearScape Analytics ModelOps :keywords: modelops, byom, python, clearscape analytics, teradata, data warehouses, teradata, vantage, cloud data platform, machine learning, artificial intelligence, business intelligence, enterprise analytics == Overview -This is a how-to for people who are new to ClearScape Analytics ModelOps. In the tutorial, you will be able to create a new project in ModelOps, upload the required data to Vantage, and track the full lifecycle of an imported Diabetes demo model using BYOM mechanisms. +This tutorial helps you to get started quickly using ClearScape Analytics ModelOps. We discuss key concepts briefly, so you can get right down to importing your first Bring-your-own-model (BYOM) models into ModelOps. In other tutorials in this quickstart site, you will have the opportunity to go deeper into other deployment and automation patterns with ClearSCape Analytics ModelOps. + +In this tutorial, you will learn: + +* What’s the difference between BYOM functions and ModelOps BYOM + +* Importing your first BYOM model in the Model Registry through the graphical user interface + +* Deploying the model in Vantage with automated scheduling and monitoring capabilities == Prerequisites +We provide an associated notebook and sample data that you can import into your clearscape environment to access and run all of the code examples included in the quickstart. link:{attachmentsdir}/ModelOps_BYOM_Quickstart.zip[Download the ModelOps sample notebooks and data] + + * Access to a Teradata Vantage instance with ClearScape Analytics (includes ModelOps) -* Ability to run Jupyter notebooks +* Access to a Jupyter notebook environment or use the one available in ClearScape Analytics Experience: include::ROOT:partial$vantage_clearscape_analytics.adoc[] -Files needed +== Key concepts you should know about first + +=== Bring your own model (BYOM) in Teradata Vantage + +The Vantage Bring Your Own Model (BYOM) package gives data scientists and analysts the ability to operationalize predictive models in Vantage. Predictive models trained in external tools can be used to score data stored in Vantage using the BYOM Predict functions. + +Create or convert your predictive model using a supported model interchange format (PMML, MOJO, ONNX, Dataiku, and DataRobot are currently available), import it in a Vantage table, and use the BYOM PMMLPredict, H2OPredict, ONNXPredict, DataikuPredict, or DataRobotPredict to score your data with the model. + +=== Bring your own model (BYOM) in Teradata Vantage with ModelOps + +In ModelOps the BYOM package is enriched with additional governance, automation, and monitoring capabilities for data scientists and machine learning engineers with the possibility of applying all of this without coding. In addition to the compatible formats of BYOM package, ModelOps extends the possibility to import and score models inside Vantage to Python scripts, R scripts and SAS scoring accelerator models. +Once you have your compatible model created or converted using a supported format (PMML, MOJO, ONNX, Dataiku, DataRobot, Python script, R script and SAS scoring accelerator model) then you can either use the ModelOps graphical user interface or the ModelOps code SDK to import into the model registry. + +=== Understand where we will focus at the ModelOps methodology +In this tutorial, we will show you the end-to-end of this process using the associated Notebook and the ModelOps graphical user interface. + +image::byom_meth.png[ModelOps Methodology BYOM screenshot, width=100%] + +== Steps in this Guide + +1. Create a project and connection (ModelOps) +2. Environment Setup (Notebook) +3. Creating datasets (ModelOps) +4. Train a model and export to PMML (Notebook) +5. Import the PMML into Vantage using BYOM functions (Notebook) +6. Import the PMML into Vantage using ModelOps Graphical user interface (ModelOps) +7. Go through Automated Lifecycle - Evaluation, Approve, Deploy (ModelOps) +8. Default and Custom alerting rules for Monitoring (ModelOps) +9. Custom Evaluation metrics and charts (Notebook) + +== 1. Create a project + +Login into ModelOps and navigate to the Projects screen. + +Click on the CREATE PROJECT button located on the top-right of the screen. We're using an cloned demo code in ModelOps with this path: /app/built-in/demo-models as git repository. Here we recommend you clone into your git repository instance the demo models public git: https://github.com/Teradata/modelops-demo-models.git in the branch "tmo" + +image::projects.png[ModelOps projects screenshot, width=120%] + +Inside the Project creation sheet panel, include the following values: + +* Name: "BYOM Quickstart" + +* Description: "BYOM Quickstart" + +* Group: DEMO + +* Path: /app/built-in/demo-models + +* Credentials: No Credentials + +* Branch: tmo + +Click the TEST GIT CONNECTION button. If the test is succesful then click on save and continue. + +image::Project_Creating.png[ModelOps projects creating, width=50%] + +== Create a Personal Connection + +In this guide we will skip creating a service connection, so click SAVE & CONTINUE and then NEXT to create a personal connection. + +image::save_continue.png[ModelOps projects save, width=20%] +image::personal1.png[ModelOps projects personal, width=50%] + +Inside the Personal Connection of the Projects creation sheet panel, include the following values: + +* Name: Quickstart Personal + +* Description: Quickstart Personal Connection + +* Host: ClearScape-url + +* Database: "demo_user" + +* VAL Database Name: "VAL" + +* BYOM Database Name: "MLDB" + +* Login Mechanism: "TDNEGO" + +* Username: demo_user + +* Pasword: your-password + +Test the Vantage connection by clicking on the TEST CONNECTION button. + +Click save. + +image::Personal_Connection.png[ModelOps connection, width=50%] + +This is how the Projects panel will show with the new project created: + +image::projects_quickstart.png[ModelOps projects with quickstart screenshot, width=120%] + +== Connection Healthcheck panel + +Enter into the project by clicking on it, and get inside Settings on the Left-hand menu. Use View details from your connection -Let's start by downloading the needed files for this tutorial. Download these 4 attachments and upload them in your Notebook filesystem. Select the files depending on your version of ModelOps: +image::view_details.png[ModelOps view, width=100%] -ModelOps version 6 (October 2022): +Then you should get the healthcheck panel, where it will show if SQLE, BYOM and VAL associated rights are enabled for this connection user. If there is any error here, contact your dba to apply the specific rights. Review the onboarding bteq script that comes in the attached files of the quickstart for the specific GRANT commands that are required. -link:{attachmentsdir}/ModelOps_Training_v6.ipynb[Download the ModelOps training Notebook] +image::healthcheck.png[ModelOps healthcheck, width=50%] -link:{attachmentsdir}/BYOM_v6.ipynb[Download BYOM Notebook file for demo use case] +== 2. Environment Setup (Notebook) -link:{attachmentsdir}/ModelOps_Data_files_v6.zip[Download data files for demo use case] +Follow the Notebook attached in this quickstart to perform the envrionnment setup and checks at the database level. -link:{attachmentsdir}/ModelOps_BYOM_files_v6.zip[Download BYOM code files for demo use case] +== 3. Creating datasets (ModelOps) -Alternatively you can git clone following repos -[source, cli] +Click on your newly created project and then click on the Datasets button located on the left-hand menu. Click on CREATE DATASET TEMPLATE. + +image::dataset_template.png[ModelOps dataset, width=120%] + + +Enter the following values: + +* Name: dataset + +* Description: dataset + +* Feature Catalog: Vantage + +* Database: your-db + +* Table: aoa_statistics_metadata + +image::dataset_template2.png[ModelOps dataset edit, width=50%] + +Click next and enter the Features Query: This query will be used to identify the features table, you can also Validate statistics and preview Data: + +[source, sql] +---- +SELECT * FROM pima_patient_features ---- -git clone https://github.com/willfleury/modelops-getting-started -git clone https://github.com/Teradata/modelops-demo-models/ + +image::dataset_template_features.png[ModelOps dataset features, width=50%] + + + +Continue to Entity & Target and include the query: This query will be used to join with the features based on the same entity and to filter the rows of the Training, Evaluation and Scoring Datasets. + +You need to select HasDiabetes as the target variable from this query, then Validate Statistics + +[source, sql] ---- +SELECT * FROM pima_patient_diagnoses +---- + +image::dataset_template_target.png[ModelOps dataset features, width=50%] + + +Continue to Predictions and include the details of the database, table, and the query: This query will be used as the Input of the execution of your model in Production when this model will be deployed as BATCH (Note: BYOM models can only be deployed as batch in ModelOps version 7) + +* Database: your-db + +* Table: pima_patient_predictions + +* Query: + +[source, sql] +---- +SELECT * FROM pima_patient_features WHERE patientid MOD 5 = 0 +---- + +image::dataset_template_prediction.png[ModelOps dataset features, width=50%] + +=== Create Training dataset + +Click on create dataset, Enter the name and description and Select training and click next. + +This query we want to filter and get 80% of rows of the dataset, we use MOD 5 <> 0 to get this: + +[source, sql] +---- +SELECT * FROM pima_patient_diagnoses WHERE patientid MOD 5 <> 0 +---- + +image::training_dataset_basic.png[ModelOps dataset basic, width=50%] +image::training_dataset.png[ModelOps dataset training, width=50%] + +Confirm the query and click on create. + +=== Create Evaluation dataset + +Click on create dataset, Enter the name and description and Select evaluation and click next. + +This query we want to filter and get 20% of rows of the dataset, we use MOD 5 = 0 to get this: + +[source, sql] +---- +SELECT * FROM pima_patient_diagnoses WHERE patientid MOD 5 = 0 +---- + +image::evaluation_dataset_basic.png[ModelOps eval dataset, width=50%] +image::evaluation_dataset.png[ModelOps eval dataset details, width=50%] + + +Confirm the query and click on create. + +This is how it should show both datasets for Training and Evaluation -ModelOps version 7 (April 2023): +image::datasets_created.png[datasets_created, width=120%] -link:{attachmentsdir}/ModelOps_Training_v7.ipynb[Download the ModelOps training Notebook] +== 4. Train a model and export to PMML (Notebook) -link:{attachmentsdir}/BYOM_v7.ipynb[Download BYOM Notebook file for demo use case] +Follow the Notebook attached in this quickstart to perform the model training, conversion and download the model pmml file for following steps. -link:{attachmentsdir}/ModelOps_Data_files_v7.zip[Download data files for demo use case] +== 5. Import the PMML into Vantage using BYOM functions (Notebook) -link:{attachmentsdir}/ModelOps_BYOM_files_v7.zip[Download BYOM code files for demo use case] +Follow the Notebook attached in this quickstart to use and understand the BYOM package functions, this way will publish the models in Vantage, but not in the ModelOps registry and we will not have governance, automation or monitoring capabilities. +== 6. Import the PMML into Vantage using ModelOps Graphical user interface (ModelOps) + +=== Import into ModelOps + +Go to Models at the left-hand menu and click on DEFINE BYOM MODEL + +image::define_new.png[ModelOps define new model, width=120%] + +Fill the fields with this values as example: + +* Name: byom + +* Description: byom + +* Format: PMML + +Click on Save Model & Import versions + +image::byom_basic.png[ModelOps define new byom model, width=50%] + +Fill the field for external id to track it from the training tool, and upload the model.pmml file - NOTE It has to be this exact name: model.pmml + +* External id: 001 + +* model file: model.pmml + +image::byom_model.png[ModelOps define new byom model, width=50%] + +=== Enable default automated Evaluation and Monitoring + +In this screen we are going to keep marked the Enable Monitoring capabily. + +We need to select the training dataset that was used for this model pmml when training. We have already created this dataset before, so we select + +Then we press on VALIDATE. + +BYOM predict functions generate an output based on a JSON, and this is different for every BYOM model. We need to know the specific field that is the target/output of our prediction. In order to use it in our evaluation logic and generate model metrics (accuracy, precision, etc.). For this we require a CAST expression on the JSON output file. + +We have included a Generate Link to help us on validating and implementing this CAST expression. So click on the Generate button to move into the helper screen and get the expression + +image::byom_monitoring1.png[ModelOps monitoring1, width=50%] + +Now select the target/output variable of our prediction. In this demo case is: predicted_HasDiabetes. + +Click on Save and let the helper copy the expression for you. + +image::byom_monitoring2.png[ModelOps monitoring2, width=50%] + +This is the CAST expression, Click on Save on the dialog: +CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT) + +image::byom_monitoring_save.png[ModelOps monitoring save, width=50%] + +Now you can validate the Cast Expression and click on Save: + +image::byom_monitoring_3.png[ModelOps monitoring save, width=50%] + +A new job for MODEL IMPORT and another job for COMPUTE STATISTICS will run for few minutes. + +image::statistics_job.png[ModelOps monitoring save, width=50%] + +== 7. Go through Automated Lifecycle - Evaluation, Approve, Deploy (ModelOps) + +=== Evaluate the model version in ModelOps + +After finishing the jobs a new model version will be available in the Model version catalog of this byom model like the following image. Click on the model version to get inside Lifecycle: + +image::model_version.png[ModelOps lifecycle, width=120%] + +The model is in IMPORT stage. we can now evaluate the model, click EVALUATE to run the automated default evaluation job + +image::model_evaluate.png[ModelOps evaluate, width=120%] + +Select the evaluation dataset and click on EVALUATE MODEL. + +image::model_evaluate2.png[ModelOps evaluate dataset, width=50%] + +This will create a new Job for the Evaluation and will show the log. These screen can be closed at the X button at the top-right. + +image::evaluation_job.png[ModelOps evaluation job, width=50%] + +You can access at any time at the left-hand menu JOBS screen. to go again into the log you just need to click on the 3 dots of the job and VIEW DETAILS. This is how it should look: + +image::jobs.png[ModelOps evaluation job, width=120%] + +Once the job is finished, model will be in the EVALUATE stage in the lifecycle screen. Go to your model version to see it. + +You can check all the details of the evaluation step, including an evaluation REPORT, where you will see metrics and Charts that the default Evaluation logic has generated. NOTE: These metrics are default for Classification and Regression models and can be customized with a coded template that will share later in the quickstart. + +image::evaluation_report.png[ModelOps evaluation lifecycle, width=100%] +image::evaluation_report2.png[ModelOps evaluation lifecycle, width=100%] + +=== Approve the model version + +Once the model version is evaluated, it is ready to be approved or rejected. This approval can be done through model lifecycle screen, in the model report screen and it can also be done through REST API integrating an external tool like Jira/BPM case management systems. + +Let's get into the Approval dialog and include the following description, as an example: + +* Approval comment: Go for Production + +image::go.png[ModelOps approval, width=30%] + +=== Deploy the model version and schedule scoring + +to deploy the model you need to use the DEPLOY button in the model lifecycle screen. + +image::deploy.png[ModelOps deploy, width=120%] + +For BYOM models the deployment target available is In-Vantage, as we want to leverage the BYOM predict functions in Vantage: + +image::deploy_details1.png[ModelOps deploy, width=50%] + +Publish the model: Select the connection to Vantage that will be used to publish the model, the database and the table. Here we will use our created connection and the table we created for storing BYOM models: aoa_byom_models. Click Next after including these details + +* Connection: personal + +* Database: demo_user + +* Table: aoa_byom_models + +image::deploy_details2.png[ModelOps deploy2, width=50%] + +Now in the Scheduling step, you are able to enable scheduling and select what is the frequency/cadence of this scoring. Keep marked the Enable Scheduling checkbox and select "Manual" in this demo, inside clearscape.teradata.com in order to save resources the scheduling options are disabled. Any scheduling option is available since we can include a CRON expression. + +In this screen we will also select the dataset template to be used when scoring the model in production. The Prediction details of the dataset will be used such as the Input query, and output prediction table that we defined in the Datasets step. + +Click on Deploy to finalize this step + +image::deploy_details3.png[ModelOps deploy3, width=50%] + +A new Deployment job will be running by the ModelOps Agent. once this is finished a new deployment will be available in the Deployments section of the left-hand menu. + +image::deploy_job.png[ModelOps deploy job, width=50%] + +=== Deployment details including history of jobs, feature/prediction drift and performance monitoring + +Go to the left-hand menu Deployments, and see the new deployment from the BYOM model is available, click on it to see the details and go to the Jobs tab + +image::deployments.png[ModelOps deployments, width=50%] + +In the Jobs tab you will see the history of executions of this model deployed. Let's run now a new scoring using the Run now button. This button can be also scheduled externally through REST APIs + +image::deployment_jobs.png[ModelOps deployments, width=100%] + +After executing the scoring job, it should look like this: + +image::deployment_jobs2.png[ModelOps deployments, width=100%] + +And we can get into the output details of this job, by clicking on the three dots at the right, and view predictions + +image::deployment_predictions.png[ModelOps deployments, width=50%] + +Now that we have run a job in production, the default Monitoring capabilities are enabled, you can check both feature and prediction drift to see individually per feature the histogram calculation and the Population Stability Index (PSI) KPI for drift monitoring + +image::feature_drift.png[ModelOps feature drift deployments, width=100%] +image::prediction_drift.png[ModelOps prediction drift deployments, width=100%] + +In the Performance metrics tab, we see that there is only a single metric data point, this is because performance monitoring relies on Evaluation jobs. So let's create a new dataset and run a new evaluation at this deployment to simulate we have new fresh data and want to check on the performance of my model by comparing the metrics with the previous evaluation. + +=== Performance monitoring with new dataset + +Let's create a new evaluation dataset in Datasets left-hand menu. + +We will use the same dataset template that we created and will create a new dataset with the following details + +* Name: evaluation2 + +* Description: evaluation2 + +* Scope: evaluation + +image::evaluation2.png[ModelOps evaluation2, width=50%] + +And let's simulate the new evaluation with a new dataset query + +[source, sql] ---- -git clone -b v7 https://github.com/willfleury/modelops-getting-started.git -git clone https://github.com/Teradata/modelops-demo-models/ +SELECT * FROM pima_patient_features WHERE patientid MOD 10 = 0 ---- -Setting up the Database and Jupyter environment +And click on create to generate new dataset for evaluation + +image::evaluation2_detail.png[ModelOps evaluation detail, width=50%] + +Now you can go back to your deployment to evaluate the model version deployed: + +image::deployment_evaluate.png[ModelOps evaluation detail, width=120%] + +Use the new dataset created in the Evaluation job panel: + +* Dataset template: dataset + +* Dataset: evaluate2 + +and click on EVALUATE model + +image::deployment_evaluate2.png[ModelOps evaluation detail, width=50%] + +Once the Evaluation job is finished, then the performance metrics will show a new set of metrics with the new dataset used: -Follow the ModelOps_Training Jupyter Notebook to setup the database, tables and libraries needed for the demo. +image::performance.png[ModelOps performance monitoring, width=50%] -== Understand where we are in the Methodology +== 8. Default and Custom alerting rules for Monitoring (ModelOps) -image::BYOM.png[ModelOps Methodology BYOM screenshot, width=100%] +=== Enabling alerting -include::partial$modelops-basic.adoc[] +Default Alerts in ModelOps are activated at the models screen, There is a Enable Alerts column in this table, activate it to start with default alerting -== Model Lifecycle for a new BYOM +image::enable_alerts.png[ModelOps enabling alerts, width=120%] -Download and unzip the files needed, links are at the top of the tutorial. For PMML file you can also download a PMML generated in the training of a GIT model. +Once this alerts are enabled you can check on the definition of the default alert, by getting inside the model and getting into the ALERT tab: -* BYOM.ipynb +image::alert_configuration.png[ModelOps configuring alert, width=120%] -* model.pmml +=== Updating alerting rules -* requirements.txt +We can create new alerts, like new rules for performance monitoring or update default alerting rules. -* evaluation.py +Let's do an alert edit, on the feature drift monitoring. click on the alert edit -* data_stats.json +image::alert_configuration2.png[ModelOps configuring alert2, width=120%] -* __init__.py +Here you can update the fields. Let's update the value treshold from 0.2 to 0.18 and click on UPDATE -Define BYOM Model with Evaluation and Monitoring +image::alert_configuration3.png[ModelOps configuring alert3, width=50%] -* Import Version +After editing the rule, your alerts screen should look like this: -* for v7 - BYOM no code is available - You can enable automated evaluation and data drift monitoring. -In Monitoring page use BYOM Target Column: CAST(CAST(json_report AS JSON).JSONExtractValue('$.predicted_HasDiabetes') AS INT) +image::alert_configuration4.png[ModelOps configuring alert4, width=50%] -* Evaluate +=== Reviewing alerts -* Review evaluation report, including dataset statistics +Now that we have alert edited, we should wait 1 minute till we get a new alert into the ModelOps tool. This alert can be configured to send an email to a set of email addresses as well. -* Approve +Now we have received the alert, we can see a red circle in the alerts at the left-hand menu -* Deploy in Vantage - Engine, Publish, Schedule. Scoring dataset is required -Use your connection and select a database. e.g "aoa_byom_models" +We can directly access to the model version from this screen by clicking on the modelid -* Deployments/executions +image::alert_new1.png[ModelOps new alert1, width=120%] -* Evaluate again with dataset2 - to monitor model metrics behavior +Once we are in the model lifecycle screen, we see a direct access to Model Drift, let's get inside -* Monitor Model Drift - Data and Metrics +image::alert_new2.png[ModelOps new alert2, width=120%] -* for v7 - Review your predictions directly from Deployments -> Job page +Then we can see the individual features in red in the feature drift tab of my deployed model. This alert is indicating that the latest scoring data is drifted from the training data with that value of population stability index(PSI). And teams can then make proactive actions to evaluate the drift of the model and replace the model in production if is needed -* Open BYOM notebook to execute the PMML predict from SQL code +image::alert_new3.png[ModelOps new alert3, width=120%] -* Retire +== 9. Custom Evaluation metrics and charts (Notebook) +Follow the Notebook attached in this quickstart to understand the methodology for creating custom Evaluation logic, metrics and charts == Summary -In this quick start we have learned how to follow a full lifecycle of BYOM models into ModelOps and how to deploy it into Vantage. Then how we can schedule a batch scoring or test restful or on-demand scorings and start monitoring on Data Drift and Model Quality metrics. +In this quick start we have learned what is the difference between BYOM functions and ModelOps BYOM pattern, How to import models with ModelOps graphical user interface, and how to automate the scoring and monitoring of the model getting Data Drift and Model QUality metrics alerts == Further reading -* https://docs.teradata.com/search/documents?query=ModelOps&sort=last_update&virtual-field=title_only&content-lang=[+++ModelOps documentation+++]. -include::ROOT:partial$community_link.adoc[] +https://docs.teradata.com/search/documents?query=ModelOps&sort=last_update&virtual-field=title_only&content-lang=[ClearScape Analytics ModelOps User Guide] +include::ROOT:partial$community_link.adoc[] \ No newline at end of file