notes-2018-04-06.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. Up and running with AWS\n",
    "2. Distributing Nodes across devices\n",
    "    - We distribute a simple three node NN across the CPU and GPU\n",
    "    - Demonstrate how much faster GPU computations are than CPU\n",
    "3. Distributing nodes across computers and devices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Up and running with AWS\n",
    "\n",
    "We will be running tensor flow across various devices hosted on the Amazons Web Services platform. Each virtual machine is called an EC2 (Elastic Compute Cloud) instance and they come in many different software and hardware configurations. We're interested in deep learning so we'll be spinning up a couple of instances with dedicated GPU's and all the software we need to run tensorflow pre-installed. Specifically, we're going to be running Deep Learning Base AMI (Amazon Machine Images) on a pair p2.xlarge instances.  These are 1xGPU, 4xCPU instances designed for lowish memory but high computational efficienty.  \n",
    "\n",
    "https://aws.amazon.com/blogs/machine-learning/get-started-with-deep-learning-using-the-aws-deep-learning-ami/\n",
    "\n",
    "### Spinning up two p2.xlarge instances\n",
    "\n",
    "- Log on to AWS, go to Services -> EC2 and click Launch Instance\n",
    "- Select Deep Learning AMI (Ubuntu) Version 6.0 - ami-5b22133e for the AMI\n",
    "- Click \"Next: Configure Instance Details.\" Under Number of Instances put 2.\n",
    "- Click \"Review and Launch.\" You will be prompted to create a new keypair if you don't have one, or to specify a keypair if you do. If you create a new key pair be sure to save it somewhere convenient!\n",
    "- Click through the warnings and at the end click the \"View instances\" button\n",
    "- Name your two instances: \"PS, Worker 0\" and \"Worker 1\"\n",
    "- When the instances have finished initilizing, note the IP addresses below\n",
    "\n",
    "### Connecting to the server: *NIX\n",
    "Use the command \n",
    "\n",
    "    ssh -i ~/mykeypair.pem -L 8157:127.0.0.1:8888 ubuntu@ec2-###-##-##-###.compute-1.amazonaws.com\n",
    "    \n",
    "to ssh into your new instance and start portforwarding to it. \n",
    "\n",
    "### Connecting to the server: Windows\n",
    "Download and install PuTTY (https://www.putty.org/). \n",
    "- Open PuTTY and put your instance's IP address in the \"Host Name\" field. \\\n",
    "- Open Connections -> SSH -> Auth and add the path to your newly created private key file to the \"Private key file for authentication\" field. \n",
    "- Click \"Open\" and enter the username ubuntu\n",
    "\n",
    "Running Jupyter \n",
    "\n",
    "- Click the menu at the top right corner of the putty terminal and got to \"Change Settings\". Under Connections -> SSH -> Tunnels and under \"Source Port\" add a port to be forwarded to the EC2 instances port 8888, be sure to record this port below. \n",
    "- Under \"Destination\" put in 127.0.0.1:8888\n",
    "- Click \"Add\" and then \"Apply\"\n",
    "\n",
    "https://docs.aws.amazon.com/dlami/latest/devguide/setup-jupyter-configure-client-windows.html\n",
    "\n",
    "\n",
    "### Setting Up Jupyter with Color\n",
    "After logging into your instance, it acn be useful to install jupyter themes so that you immediately know which server you're working on. To do so, activate the tensorflow environment\n",
    "\n",
    "    source activate tensorflow_p27\n",
    "\n",
    "and install Jupyter Themes\n",
    "\n",
    "    pip install jupyterthemes\n",
    "    \n",
    "Finally, change your theme\n",
    "\n",
    "    jt -t chesterish\n",
    "    \n",
    "and start your notebook\n",
    "\n",
    "    jupyter notebook\n",
    "    \n",
    "Remember to change the port to the forwarded port before opening the link"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Machine Details\n",
    "\n",
    "* Name: Bert, Roles: PS, Worker 1 IP:, Jupyter Port:, Color:\n",
    "* Name: Ernie, Roles: Worker 2 IP:, Jupyter Port:, Color:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributing Nodes Across Devices\n",
    "The code below can be run locally, but is designed to be run on one of your new p2.xlarge instances. For more information on different configurations look here: https://www.tensorflow.org/programmers_guide/using_gpu\n",
    "\n",
    "On a standard machine there are multiple devices that can perform computations. In TensorFlow the CPUs and GPUs are represented as string in the following format\n",
    "\n",
    "- `/cpu:0`: The first CPU of your machine.\n",
    "- `/cpu:1`: The second CPU of your machine.\n",
    "- `/device:GPU:0`: The GPU of your machine, if you have one.\n",
    "- `/device:GPU:1`: The second GPU of your machine, etc.\n",
    "\n",
    "A device name takes the following form: \n",
    "\n",
    "`/job:<JOB_NAME>/task:<TASK_INDEX>/device:<DEVICE_TYPE>:<DEVICE_INDEX>`\n",
    "\n",
    "<img src=\"TF-CUDA-GPU.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here's the standard preamble that we'll just be copying and pasting to all devices. \n",
    "\n",
    "# To support both python 2 and python 3\n",
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"distributed\"\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True):\n",
    "    path = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id + \".png\")\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "c = tf.constant(\"Hello Distributed Tensorflow!\")\n",
    "server = tf.train.Server.create_local_server()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the above line, go over to your Anaconda console and notice that a port has been opened"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Hello Distributed Tensorflow!'\n"
     ]
    }
   ],
   "source": [
    "with tf.Session(server.target) as sess:\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to do device logging\n",
    "First, we'll turn on device logging by running the session with `log_device_placement` set to `True`. The output should be recorded in your anaconda terminal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[22. 28.]\n",
      " [49. 64.]]\n"
     ]
    }
   ],
   "source": [
    "# Creates a graph.\n",
    "a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')\n",
    "b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')\n",
    "c = tf.matmul(a, b)\n",
    "# Creates a session with log_device_placement set to True.\n",
    "sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))\n",
    "# Runs the op.\n",
    "print(sess.run(c))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display local devices\n",
    "\n",
    "Question: What are the XLA_GPU and XLA_CPU doing?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[name: \"/device:CPU:0\"\n",
       " device_type: \"CPU\"\n",
       " memory_limit: 268435456\n",
       " locality {\n",
       " }\n",
       " incarnation: 6114515408153548089]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "\n",
    "device_lib.list_local_devices()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that TensorFlow will automatically grab all the RAM in the GPU, you can configure it to grab less, or even grab dynamically, but once RAM has been claimed it wont released. TensorFlow will also grab the GPU with the lowest ID by default. When left to default values, TensorFlow will automatically distribute your graph across the CPU and the first avaible GPU. If you want to specify a CPU/GPU use the following:\n",
    "\n",
    "Question: In what order does tensorflow run nodes of different devices?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "InvalidArgumentError",
     "evalue": "Cannot assign a device for operation 'add': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.\n\t [[Node: add = Add[T=DT_FLOAT, _device=\"/device:GPU:0\"](a/read, a_1/read)]]\n\nCaused by op 'add', defined at:\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel_launcher.py\", line 16, in <module>\n    app.launch_new_instance()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\traitlets\\config\\application.py\", line 658, in launch_instance\n    app.start()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelapp.py\", line 478, in start\n    self.io_loop.start()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tornado\\ioloop.py\", line 888, in start\n    handler_func(fd_obj, events)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tornado\\stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tornado\\stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 233, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\ipkernel.py\", line 208, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\zmqshell.py\", line 537, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2728, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2850, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2910, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-9-178f0f86408a>\", line 8, in <module>\n    c = a + b\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\variables.py\", line 754, in _run_op\n    return getattr(ops.Tensor, operator)(a._AsTensor(), *args)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py\", line 894, in binary_op_wrapper\n    return func(x, y, name=name)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\gen_math_ops.py\", line 182, in add\n    \"Add\", x=x, y=y, name=name)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 787, in _apply_op_helper\n    op_def=op_def)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2956, in create_op\n    op_def=op_def)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1470, in __init__\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\n\nInvalidArgumentError (see above for traceback): Cannot assign a device for operation 'add': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.\n\t [[Node: add = Add[T=DT_FLOAT, _device=\"/device:GPU:0\"](a/read, a_1/read)]]\n",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_do_call\u001b[1;34m(self, fn, *args)\u001b[0m\n\u001b[0;32m   1322\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1323\u001b[1;33m       \u001b[1;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1324\u001b[0m     \u001b[1;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[1;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[0;32m   1292\u001b[0m       \u001b[1;31m# Ensure any changes to the graph are reflected in the runtime.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1293\u001b[1;33m       \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_extend_graph\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1294\u001b[0m       \u001b[1;32mwith\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_exception_on_not_ok_status\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mstatus\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_extend_graph\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1353\u001b[0m           tf_session.TF_ExtendGraph(\n\u001b[1;32m-> 1354\u001b[1;33m               self._session, graph_def.SerializeToString(), status)\n\u001b[0m\u001b[0;32m   1355\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_opened\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py\u001b[0m in \u001b[0;36m__exit__\u001b[1;34m(self, type_arg, value_arg, traceback_arg)\u001b[0m\n\u001b[0;32m    472\u001b[0m             \u001b[0mcompat\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc_api\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstatus\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstatus\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 473\u001b[1;33m             c_api.TF_GetCode(self.status.status))\n\u001b[0m\u001b[0;32m    474\u001b[0m     \u001b[1;31m# Delete the underlying status object from memory otherwise it stays alive\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mInvalidArgumentError\u001b[0m: Cannot assign a device for operation 'add': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.\n\t [[Node: add = Add[T=DT_FLOAT, _device=\"/device:GPU:0\"](a/read, a_1/read)]]",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-9-178f0f86408a>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[1;32mwith\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0msess\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m     \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mglobal_variables_initializer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 12\u001b[1;33m     \u001b[0msess\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36mrun\u001b[1;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[0;32m    887\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    888\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[1;32m--> 889\u001b[1;33m                          run_metadata_ptr)\n\u001b[0m\u001b[0;32m    890\u001b[0m       \u001b[1;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    891\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_run\u001b[1;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[0;32m   1118\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1119\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[1;32m-> 1120\u001b[1;33m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[0;32m   1121\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1122\u001b[0m       \u001b[0mresults\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_do_run\u001b[1;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[0;32m   1315\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1316\u001b[0m       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\n\u001b[1;32m-> 1317\u001b[1;33m                            options, run_metadata)\n\u001b[0m\u001b[0;32m   1318\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1319\u001b[0m       \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\client\\session.py\u001b[0m in \u001b[0;36m_do_call\u001b[1;34m(self, fn, *args)\u001b[0m\n\u001b[0;32m   1334\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1335\u001b[0m           \u001b[1;32mpass\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1336\u001b[1;33m       \u001b[1;32mraise\u001b[0m \u001b[0mtype\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnode_def\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mop\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1337\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1338\u001b[0m   \u001b[1;32mdef\u001b[0m \u001b[0m_extend_graph\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mInvalidArgumentError\u001b[0m: Cannot assign a device for operation 'add': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.\n\t [[Node: add = Add[T=DT_FLOAT, _device=\"/device:GPU:0\"](a/read, a_1/read)]]\n\nCaused by op 'add', defined at:\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel_launcher.py\", line 16, in <module>\n    app.launch_new_instance()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\traitlets\\config\\application.py\", line 658, in launch_instance\n    app.start()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelapp.py\", line 478, in start\n    self.io_loop.start()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tornado\\ioloop.py\", line 888, in start\n    handler_func(fd_obj, events)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tornado\\stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\zmq\\eventloop\\zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tornado\\stack_context.py\", line 277, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 233, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\ipkernel.py\", line 208, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\ipykernel\\zmqshell.py\", line 537, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2728, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2850, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2910, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-9-178f0f86408a>\", line 8, in <module>\n    c = a + b\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\variables.py\", line 754, in _run_op\n    return getattr(ops.Tensor, operator)(a._AsTensor(), *args)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py\", line 894, in binary_op_wrapper\n    return func(x, y, name=name)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\ops\\gen_math_ops.py\", line 182, in add\n    \"Add\", x=x, y=y, name=name)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py\", line 787, in _apply_op_helper\n    op_def=op_def)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 2956, in create_op\n    op_def=op_def)\n  File \"C:\\Users\\admin\\Anaconda\\envs\\tensorflow\\lib\\site-packages\\tensorflow\\python\\framework\\ops.py\", line 1470, in __init__\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\n\nInvalidArgumentError (see above for traceback): Cannot assign a device for operation 'add': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.\n\t [[Node: add = Add[T=DT_FLOAT, _device=\"/device:GPU:0\"](a/read, a_1/read)]]\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.device(\"/cpu:0\"):\n",
    "    a = tf.Variable(1.0, name=\"a\")\n",
    "    b = tf.Variable(1.0, name=\"a\")\n",
    "    \n",
    "with tf.device(\"/gpu:0\"):\n",
    "    c = a + b\n",
    "    \n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dynamic Placement of Nodes (Geron)\n",
    "When you create a device block, you can specify a function instead of a device name. TensorFlow will call this function for each operation it needs to place in the device block, and the function must return the name of the device to pin the operation on.\n",
    "\n",
    "For example, the following code pins all the variable nodes to \"/cpu:0\" (in this case just the variable a) and all other nodes to \"/gpu:0\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def variables_on_cpu(op):\n",
    "    if op.type == \"VariableV2\":\n",
    "        return \"/cpu:0\"\n",
    "    else:\n",
    "        return \"/gpu:0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.device(variables_on_cpu):\n",
    "    a = tf.Variable(1.0, name=\"a\")\n",
    "    b = tf.Variable(1.0, name=\"b\")\n",
    "    c = a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a.device)\n",
    "print(b.device)\n",
    "print(c.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parallel Execution (Geron)\n",
    "When TensorFlow runs a graph, it starts by finding out the list of nodes that need to be evaluated, and it counts how many dependencies each of them has. TensorFlow then starts evaluating the nodes with zero dependencies (i.e., source nodes). If these nodes are placed on separate devices, they obviously get evaluated in parallel. If they are placed on the same device, they get evaluated in different threads, so they may run in parallel too (in separate GPU threads or CPU cores).\n",
    "\n",
    "TensorFlow manages a thread pool on each device to parallelize operations (see Figure 12-5). These are called the inter-op thread pools. Some operations have multithreaded kernels: they can use other thread pools (one per device) called the intraop thread pools.\n",
    "\n",
    "<img src=\"ThreadPool.png\">\n",
    "\n",
    "For example, in Figure 12-5, operations A, B, and C are source ops, so they can immediately be evaluated. Operations A and B are placed on GPU 0, so they are sent to this device’s inter-op thread pool, and immediately evaluated in parallel. Operation A happens to have a multithreaded kernel; its computations are split in three parts, which are executed in parallel by the intraop thread pool. Operation C goes to GPU 1’s inter-op thread pool. As soon as operation C finishes, the dependency counters of operations D and E will be decremented and will both reach 0, so both operations will be sent to the inter-op thread pool to be executed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A quick benchmark of GPU computations vs CPU\n",
    "\n",
    "We will perform a couple of matrix calculations to show the difference in speed between GPU computations and CPU computations.\n",
    "\n",
    "https://gist.github.com/j-min/69aae99be6f6acfadf2073817c2f61b0\n",
    "\n",
    "First, lets see what devices we can actually access"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[name: \"/device:CPU:0\"\n",
       " device_type: \"CPU\"\n",
       " memory_limit: 268435456\n",
       " locality {\n",
       " }\n",
       " incarnation: 5866721366184997421]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "\n",
    "device_lib.list_local_devices()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import numpy as np\n",
    "\n",
    "A = np.random.rand(10000,10000).astype('float32')\n",
    "B = np.random.rand(10000,10000).astype('float32')\n",
    "\n",
    "# Create a graph to store results\n",
    "c1 = []\n",
    "c2 = []\n",
    "c3 = []\n",
    "\n",
    "n = 1\n",
    "\n",
    "def matpow(M, n):\n",
    "    if n < 1: #Abstract cases where n < 1\n",
    "        return M\n",
    "    else:\n",
    "        return tf.matmul(M, matpow(M, n-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Single CPU computing\n",
    "'''\n",
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.device('/cpu:0'):\n",
    "    a = tf.placeholder(tf.float32, [10000,10000])\n",
    "    b = tf.placeholder(tf.float32, [10000,10000])\n",
    "    c1.append(matpow(a,n))\n",
    "    c1.append(matpow(b,n))\n",
    "    summ = tf.add_n(c1)\n",
    "\n",
    "t1_1 = datetime.datetime.now()\n",
    "with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:\n",
    "    # Run the op.\n",
    "    sess.run(summ, {a:A, b:B})\n",
    "t2_1 = datetime.datetime.now()\n",
    "\n",
    "print(\"Single CPU computation time: \" + str(t2_1-t1_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Multiple CPU computing\n",
    "'''\n",
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.device('/cpu:0'):\n",
    "    a = tf.placeholder(tf.float32, [10000,10000])\n",
    "    c1.append(matpow(a,n))\n",
    "    \n",
    "with tf.device('/gpu:0'):\n",
    "    b = tf.placeholder(tf.float32, [10000,10000])\n",
    "    c1.append(matpow(b,n))\n",
    "    sum = tf.add_n(c1)\n",
    "\n",
    "t1_1 = datetime.datetime.now()\n",
    "with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:\n",
    "    # Run the op.\n",
    "    sess.run(sum, {a:A, b:B})\n",
    "t2_1 = datetime.datetime.now()\n",
    "\n",
    "print(\"Multiple CPU computation time: \" + str(t2_1-t1_1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "GPU computing\n",
    "'''\n",
    "tf.reset_default_graph()\n",
    "\n",
    "with tf.device('/GPU:0'):\n",
    "    a = tf.placeholder(tf.float32, [10000,10000])\n",
    "    c1.append(matpow(a,n))\n",
    "    b = tf.placeholder(tf.float32, [10000,10000])\n",
    "    c1.append(matpow(b,n))\n",
    "    sum = tf.add_n(c1)\n",
    "\n",
    "t1_1 = datetime.datetime.now()\n",
    "with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:\n",
    "    # Run the op.\n",
    "    sess.run(sum, {a:A, b:B})\n",
    "t2_1 = datetime.datetime.now()\n",
    "\n",
    "print(\"GPU computation time: \" + str(t2_1-t1_1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed Tensorflow\n",
    "Tensor flow allows us to set up a distributed network just about as easily as we spread things out over devices. All we need to do is give each machine the same directory and graph structure and it will start chugging away. There are of course a lot of different choices involved: where how to distribute the graph? Where do we store the data? Do we wait for each node to finish an entire epoch or do we allow certain nodes to que operations? \n",
    "\n",
    "\n",
    "### Setup Network Permissions\n",
    "We will have to open the firewalls on both instances so that they can talk to each other. Go back to the AWS console, select your instance and click \"Edit Security Group.\"...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On Bert (our PS Session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "cluster_spec = tf.train.ClusterSpec({'ps' : ['localhost:2222'],'worker' : ['localhost:2223','localhost:2224']})\n",
    "ps = tf.train.Server(cluster_spec,job_name='ps')\n",
    "ps.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On Bert (our Chief Worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import tensorflow as tf\n",
    "import time\n",
    "\n",
    "task_index=0\n",
    "\n",
    "cluster_spec = tf.train.ClusterSpec({'ps' : ['localhost:2222'],'worker' : ['localhost:2223','localhost:2224']})\n",
    "\n",
    "server = tf.train.Server(cluster_spec,job_name='worker',task_index=task_index)\n",
    "\n",
    "# Define grahp, must be done on BOTH workers\n",
    "\n",
    "with tf.device('/job:ps/task:0'):\n",
    "    a = tf.Variable([0.], name = 'a') # Add a to the process server\n",
    "    b = tf.constant([100.])\n",
    "    c = tf.constant([120.])\n",
    "    \n",
    "with tf.device('job:worker/task:1'):\n",
    "    loss2 = tf.abs(a-c)\n",
    "    \n",
    "with tf.device('job:worker/task:0'):\n",
    "    loss1 = tf.abs(a-b)\n",
    "    optimizer = tf.train.GradientDescentOptimizer(.1)\n",
    "    \n",
    "    grads = optimizer.compute_gradients(loss1*loss1+loss2*loss2)\n",
    "    \n",
    "    global_update = optimizer.apply_gradients(grads)\n",
    "    \n",
    "with tf.device('/job:worker/task:%d'%task_index):\n",
    "    init_global = tf.global_variables_initializer()\n",
    "    \n",
    "sess = tf.Session(target=server.target)\n",
    "\n",
    "print(a.device)\n",
    "print(b.device)\n",
    "print(c.device)\n",
    "print(loss1.device)\n",
    "print(loss2.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On Ernie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import tensorflow as tf\n",
    "import time\n",
    "\n",
    "task_index=0\n",
    "\n",
    "cluster_spec = tf.train.ClusterSpec({'ps' : ['localhost:2222'],'worker' : ['localhost:2223','localhost:2224']})\n",
    "\n",
    "server = tf.train.Server(cluster_spec,job_name='worker',task_index=task_index)\n",
    "\n",
    "# Define grahp, must be done on BOTH workers\n",
    "\n",
    "with tf.device('/job:ps/task:0'):\n",
    "    a = tf.Variable([0.], name = 'a') # Add a to the process server\n",
    "    b = tf.constant([100.])\n",
    "    c = tf.constant([120.])\n",
    "    \n",
    "with tf.device('job:worker/task:1'):\n",
    "    loss2 = tf.abs(a-c)\n",
    "    \n",
    "with tf.device('job:worker/task:0'):\n",
    "    loss1 = tf.abs(a-b)\n",
    "    optimizer = tf.train.GradientDescentOptimizer(.1)\n",
    "    \n",
    "    grads = optimizer.compute_gradients(loss1*loss1+loss2*loss2)\n",
    "    \n",
    "    global_update = optimizer.apply_gradients(grads)\n",
    "    \n",
    "with tf.device('/job:worker/task:%d'%task_index):\n",
    "    init_global = tf.global_variables_initializer()\n",
    "    \n",
    "sess = tf.Session(target=server.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## On Bert (Chief Worker Session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.run(init_global)\n",
    "\n",
    "for i in range(1000):\n",
    "    sess.run(global_update)\n",
    "    \n",
    "sess.run(a)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}