Starting from existing code - step 1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Install Pipeline SDK - This only needs to be ran once in the enviroment. \n",
    "# you can find the latest package @ https://github.com/kubeflow/pipelines/releases\n",
    "#KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.20/kfp.tar.gz'\n",
    "#!pip3 install $KFP_PACKAGE --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Starting from existing code to build on KubeFlow Pipelines \n",
    "\n",
    "This is a series of notebooks in which we will demonstrate how to start from an existing code base and create a pipelin in a step by step fasion. In this sample we will:\n",
    "\n",
    "* **Create a single step pipeline from existing code**\n",
    "* Break the single step to multiple pipeline steps using the sample container\n",
    "* Create a separate container for each step\n",
    "\n",
    "Reference documentation: \n",
    "* https://www.kubeflow.org/docs/pipelines/sdk/build-component/\n",
    "* https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/\n",
    "\n",
    "This sample is based on the TF sample for more details please see\n",
    "* https://github.com/tensorflow/docs/blob/master/site/en/tutorials/_index.ipynb "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Set your output and project. !!!Must Do before you can proceed!!!\n",
    "EXPERIMENT_NAME = 'Hellow world!'\n",
    "PROJECT_NAME =  'chavoshi-dev-2'                      #'Your-Gcp-Project-Name'\n",
    "OUTPUT_DIR = 'gs://chavoshi-dev-mlpipeline'          # A path for asset outputs\n",
    "BASE_IMAGE='tensorflow/tensorflow:1.11.0-py3'         # Based image used in various steps of the pipeline\n",
    "TARGET_IMAGE='gcr.io/%s/byoc:latest' % PROJECT_NAME # Target image that will include our final code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an Experiment in the Pipeline System\n",
    "\n",
    "Pipeline system requires an \"Experiment\" to group pipeline runs. You can create a new experiment, or call client.list_experiments() to get existing ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Experiment link <a href=\"/pipeline/#/experiments/details/571db2d0-a74a-4170-a68c-42f2498819bb\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Get or create an experiment and submit a pipeline run\n",
    "import kfp\n",
    "client = kfp.Client()\n",
    "experiment = client.create_experiment(EXPERIMENT_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that this notebook should be running in JupyterHub in the same cluster as the pipeline system.\n",
    "# Otherwise it will fail to talk to the pipeline system.\n",
    "import kfp\n",
    "import kfp.dsl as dsl\n",
    "from kfp.gcp import use_gcp_secret\n",
    "from kubernetes import client as k8s_client\n",
    "from kfp import compiler\n",
    "from kfp import notebook\n",
    "from kfp import components as comp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dsl.python_component(\n",
    "    name='mnist_e2e_single_component',\n",
    "    description='Runs the enire mnist code in one single component',\n",
    "    base_image=BASE_IMAGE  # note you can define the base image here, or during build time. \n",
    ")\n",
    "\n",
    "def mnist_example_e2e():\n",
    "    # Your code goes here\n",
    "    # Note that you can instead use git clone or gsutil copy the code into the base image.\n",
    "    \n",
    "    import tensorflow as tf\n",
    "    mnist = tf.keras.datasets.mnist\n",
    "\n",
    "    (x_train, y_train),(x_test, y_test) = mnist.load_data()\n",
    "    x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "    model = tf.keras.models.Sequential([\n",
    "      tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
    "      tf.keras.layers.Dense(512, activation=tf.nn.relu),\n",
    "      tf.keras.layers.Dropout(0.2),\n",
    "      tf.keras.layers.Dense(10, activation=tf.nn.softmax)\n",
    "    ])\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    model.fit(x_train, y_train, epochs=5)\n",
    "    model.evaluate(x_test, y_test)\n",
    "    \n",
    "    return \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a Component With the Above Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-07-11 17:16:47:INFO:Build an image that is based on tensorflow/tensorflow:1.11.0-py3 and push the image to gcr.io/chavoshi-dev-2/byoc:latest\n",
      "2019-07-11 17:16:47:INFO:Checking path: gs://chavoshi-dev-mlpipeline...\n",
      "2019-07-11 17:16:47:INFO:Generate entrypoint and serialization codes.\n",
      "2019-07-11 17:16:47:INFO:Generate build files.\n",
      "2019-07-11 17:16:47:INFO:Start a kaniko job for build.\n",
      "2019-07-11 17:16:47:INFO:Cannot Find local kubernetes config. Trying in-cluster config.\n",
      "2019-07-11 17:16:47:INFO:Initialized with in-cluster config.\n",
      "2019-07-11 17:16:52:INFO:5 seconds: waiting for job to complete\n",
      "2019-07-11 17:16:57:INFO:10 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:02:INFO:15 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:07:INFO:20 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:12:INFO:25 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:17:INFO:30 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:22:INFO:35 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:27:INFO:40 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:32:INFO:45 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:37:INFO:50 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:43:INFO:55 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:48:INFO:60 seconds: waiting for job to complete\n",
      "2019-07-11 17:17:48:INFO:Kaniko job complete.\n",
      "2019-07-11 17:17:48:INFO:Build component complete.\n"
     ]
    }
   ],
   "source": [
    "from kfp import compiler\n",
    "\n",
    "# The return value \"DeployerOp\" represents a step that can be used directly in a pipeline function\n",
    "mnist_e2e_op = compiler.build_python_component(\n",
    "    component_func=mnist_example_e2e,\n",
    "    staging_gcs_path=OUTPUT_DIR,\n",
    "    dependency=[kfp.compiler.VersionedDependency(name='google-api-python-client', version='1.7.0')],\n",
    "    base_image=BASE_IMAGE,\n",
    "    target_image=TARGET_IMAGE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a pipeline using this component"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp.dsl as dsl\n",
    "@dsl.pipeline(\n",
    "   name='MNIST pipeline',\n",
    "   description='A sample pipeline that runs MNIST.'\n",
    ")\n",
    "def mnist_pipeline(\n",
    "):\n",
    "    mnist_e2e_task = mnist_e2e_op() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complie the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_func = mnist_pipeline\n",
    "pipeline_filename = pipeline_func.__name__ + '.pipeline.zip'\n",
    "import kfp.compiler as compiler\n",
    "compiler.Compiler().compile(pipeline_func, pipeline_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the pipeline for execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Run link <a href=\"/pipeline/#/runs/details/ce832434-a3ff-11e9-a9d6-42010a800018\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Specify pipeline argument values\n",
    "arguments = {}\n",
    "\n",
    "#Submit a pipeline run\n",
    "run_name = pipeline_func.__name__ + ' run'\n",
    "run_result = client.run_pipeline(experiment.id, run_name, pipeline_filename, arguments)\n",
    "\n",
    "#This link leads to the run information page. \n",
    "#Note: There is a bug in JupyterLab that modifies the URL and makes the link stop working"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}