Managing Secrets .ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Install Pipeline SDK - This only needs to be ran once in the enviroment. \n",
    "# you can find the latest package @ https://github.com/kubeflow/pipelines/releases\n",
    "#KFP_PACKAGE = 'https://storage.googleapis.com/ml-pipeline/release/0.1.20/kfp.tar.gz'\n",
    "#!pip3 install $KFP_PACKAGE --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "EXPERIMENT_NAME = 'Hellow world!'\n",
    "GCLOUD_SDK = 'google/cloud-sdk:latest'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Managing secrets \n",
    "\n",
    "This section demonstrates how you access Kubernetes secrets using Kubeflow Pipeline SDK.\n",
    "\n",
    "Reference documentation: \n",
    "* https://www.kubeflow.org/docs/gke/gcp-e2e/#check-the-permissions-for-your-training-component \n",
    "* https://cloud.google.com/kubernetes-engine/docs/concepts/secret \n",
    "* https://kubernetes.io/docs/concepts/configuration/secret/\n",
    "\n",
    "Specifically note that you can view, or add secrets to your K8 cluster using:\n",
    "\n",
    "```console\n",
    "kubectl get secrets\n",
    "kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp\n",
    "import kfp.dsl as dsl\n",
    "from kfp.gcp import use_gcp_secret\n",
    "from kubernetes import client as k8s_client\n",
    "from kfp import compiler\n",
    "from kfp import notebook\n",
    "from kfp import components as comp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### google cloud Auth methods\n",
    "To retrieve a GCP secret from K8 you can use the container operation \n",
    "```console\n",
    ".apply(use_gcp_secret('secret name'))\n",
    "```\n",
    "Application Default Credentials (ADC) provide a method to get credentials used in calling Google APIs.However to add the secret to user enviroment to use for applications such as Gsutil you need to run the following \n",
    "``` console\n",
    "gcloud auth activate-service-account --key-file /secret/gcp-credentials/user-gcp-sa.json'''\n",
    "```\n",
    "The gcloud auth application-default command group allows you to manage active credentials on your machine that are used for local application development.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def using_secrets_op():\n",
    "    return dsl.ContainerOp(\n",
    "        name = 'Using Secrets',\n",
    "        image = GCLOUD_SDK,\n",
    "        command=['sh', '-c'],\n",
    "        arguments = [  'gcloud auth activate-service-account --key-file /secret/gcp-credentials/user-gcp-sa.json']\n",
    "    ).apply(use_gcp_secret('user-gcp-sa'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the pipeline\n",
    "Pipeline function has to be decorated with the `@dsl.pipeline` decorator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kfp.dsl as dsl\n",
    "@dsl.pipeline(\n",
    "   name='secrets',\n",
    "   description='A toy pipeline demonstrates use of gcp secretes'\n",
    ")\n",
    "def calc_pipeline(\n",
    "):\n",
    "    #Creating a one step pipeline\n",
    "    using_secrets = using_secrets_op() #Returns a dsl.ContainerOp class instance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compile the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_func = calc_pipeline\n",
    "pipeline_filename = pipeline_func.__name__ + '.pipeline.zip'\n",
    "import kfp.compiler as compiler\n",
    "compiler.Compiler().compile(pipeline_func, pipeline_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submit the pipeline for execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Experiment link <a href=\"/pipeline/#/experiments/details/571db2d0-a74a-4170-a68c-42f2498819bb\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Get or create an experiment and submit a pipeline run\n",
    "import kfp\n",
    "client = kfp.Client()\n",
    "experiment = client.create_experiment(EXPERIMENT_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/kfp/_client.py:194: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  return yaml.load(f)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Run link <a href=\"/pipeline/#/runs/details/20c76f8a-936e-11e9-aeee-42010a80014f\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Specify pipeline argument values\n",
    "arguments = {}\n",
    "\n",
    "#Submit a pipeline run\n",
    "run_name = pipeline_func.__name__ + ' run'\n",
    "run_result = client.run_pipeline(experiment.id, run_name, pipeline_filename, arguments)\n",
    "\n",
    "#This link leads to the run information page. \n",
    "#Note: There is a bug in JupyterLab that modifies the URL and makes the link stop working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}