examples/tutorials/job_cli.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5933f2eb-463a-4d01-a806-b6fc0fe9b4de",
   "metadata": {},
   "source": [
    "# NVFLARE JOB CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69ce4e61-da7f-4ba1-96f1-822c578e53a1",
   "metadata": {
    "tags": []
   },
   "source": [
    "In this notebook, we will go through the different commands of the Job CLI to show the syntax and usage of each.\n",
    "Refer to the [Job CLI Documentation](https://nvflare.readthedocs.io/en/main/user_guide/nvflare_cli/job_cli.html) for more details.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "154531fd-2a94-4062-bbc6-76086b099093",
   "metadata": {
    "tags": [],
    "toc-hr-collapsed": true
   },
   "source": [
    "## Install NVIDIA FLARE\n",
    "\n",
    "For this notebook, we will need a running NVFLARE project that we can connect to.\n",
    "Follow the [installation](https://nvflare.readthedocs.io/en/main/getting_started.html#installation) \n",
    "instructions to set up an environment that has NVIDIA FLARE installed if you do not have one already.\n",
    "\n",
    "If you use the job CLI to submit job, you will need a running NVFLARE system with client and server. You can either run a local system via nvflare poc commands, or \n",
    "use a running production system. \n",
    "\n",
    "To see how to setup a local system, please refer to the [setup_poc tutorial](setup_poc.ipynb).\n",
    "\n",
    "\n",
    "## Step-by-step walk-through: from creating a job to running a job\n",
    "\n",
    "Taking the converted CIFAR10 with pytorch training code for a 2-client federated learning [program](https://github.com/NVIDIA/NVFlare/tree/main/examples/hello-world/step-by-step/cifar10/code), we can use the standard Scatter and Gatter (SAG) workflow pattern to demonstrate the features of the Job CLI. \n",
    "\n",
    "Now, we would like to see what are the available pre-configured job templates the user can use and modify. \n",
    "\n",
    "\n",
    "### Check out the available nvflare job templates\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c635e02-7fe6-401c-82d2-e2cde1dc86c0",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### List Job Templates and job templates directory\n",
    "\n",
    "The NVFLARE 2.4.0 release introduces job templates for the different types of job configurations.\n",
    "\n",
    "To list the available templates, you can use the ```nvflare job list_templates``` command:\n",
    "\n",
    "```\n",
    "! nvflare job list_templates\n",
    "```\n",
    "\n",
    "If you installed nvflare 2.4.x via `pip install nvflare`. The above command should show you available job templates (built-in default job templates). But if you cloned the github report of repository, and did not use the ```pip install nvflare```, the above command will expect you to provide the job_templates directory. When the job templates directory is not specified, the Job CLI will try to find the job_templates location with the following sequences of logic:\n",
    "\n",
    "* See if the NVFLARE_HOME environment variable is set. If NVFLARE_HOME is not empty, the Job CLI will look for the job_templates at:\n",
    " \n",
    " ```${NVFLARE_HOME}/job_templates```\n",
    " \n",
    "* If the NVFLARE_HOME env. variable is not set, the Job CLI will look for the `job_template` path of the config in the nvflare hidden directory \n",
    "\n",
    "```\n",
    "cat ~/.nvflare/config.conf \n",
    "\n",
    "startup_kit {\n",
    "  path = \"/tmp/nvflare/poc1/example_project/prod_00\"\n",
    "}\n",
    "poc_workspace {\n",
    "  path = \"/tmp/nvflare/poc1\"\n",
    "}\n",
    "job_template {\n",
    "  path = \"../../job_templates\"\n",
    "}\n",
    "\n",
    "```\n",
    "once the `-d <job_template_dir>` option is used, the `job_template` value in `~/.nvflare/config.conf` will be updated so you don't need to specify -d again. \n",
    "\n",
    "If you want to change the `job_template` path, you can directly edit this config file or use the `nvflare config` command:\n",
    "\n",
    "```\n",
    "nvflare config -jt ../../job_templates. \n",
    "\n",
    "```\n",
    "If the ~/.nvflare/config.conf is not defined yet, the command will look at the following location from installed NVFLARE package \n",
    "```\n",
    " job_templates_dir = os.path.join(nvflare.job.__file__, \"templates\")\n",
    "```\n",
    " \n",
    "If the nvflare is installed, this directory exists, then it should find the built-in job templates. \n",
    "\n",
    "> Note: this directory may not exist in the follow case: \n",
    "> * If you have done ```pip install nvflare```, but also installed the NVFLARE source code from github repo. the sys.path might point to your local NVFLARE repository when load nvflare.job module. In such a case, the above directory will not exist. As the job_templates is not located at nvflare/job/templates in the github repository. \n",
    "\n",
    "\n",
    "If Job templates directory still not found, the command will raise exception for missing Job Template directory. \n",
    "\n",
    "\n",
    "By now, you should understand that the ```nvflare job list_templates``` allows you to list built-in default job templates from the release, as well as provides your own job_templates to reflect the recent changes. \n",
    "\n",
    "For now, let's specify the job templates directory location\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "447619d5-d917-4d93-b806-6c673e216b88",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job list_templates -d \"../../job_templates\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9abba0f2-17c9-4e09-a34e-8543238e4039",
   "metadata": {},
   "source": [
    "Where the option `-d \"<job_templates_dir>\"` or `--job_template_dir \"<job_templates_dir>\"` is the location of the job_templates.  By doing so, we have also save our job_templates into the hidden configuration,so we don't do it again next time. Let's look at the config file. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06fec3a1-71a6-40f7-a10f-52ea6595ad96",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat  ~/.nvflare/config.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4663c2a0-3b36-4abb-ba8f-d1e77afea23a",
   "metadata": {},
   "source": [
    "You can also manually preset the job_templates directory if you don't want to reply on the default one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bea478d-42fb-4223-8155-3c996699a052",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare config -jt ../../job_templates \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a3d793d-bc0c-458b-8a58-047cfde915f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat  ~/.nvflare/config.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9a7b708-40e9-4f4f-beec-b83d0e893a0b",
   "metadata": {},
   "source": [
    "Now we can list the templates again without -d option"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49bab293-7fef-476f-8aa9-f2b2868a0fb3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job list_templates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10720b0b-af4c-4d71-a751-2d5c301eb05a",
   "metadata": {
    "tags": []
   },
   "source": [
    "With a job template that fits your needs, you can use the job template name to create a new job folder.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecbd0cc7-f410-457e-a6bd-d4c999182850",
   "metadata": {},
   "source": [
    "### Create a job folder\n",
    "\n",
    "Since the code for our example is written in pytorch and we would like to try the FedAvg algorithm using the Scatter & Gather (SAG) workflow, the job template **\"sag_pt\"** is what we are looking for. We will use this template to create our job folder. \n",
    "\n",
    "Create a job folder that contains the base job configuration from the template, which can then be modified as desired. First, create a job folder with the intent for it to be modified, without specifying any code.\n",
    "\n",
    "\n",
    "#### First try\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b3c9750-070f-4655-9128-757ab136b30d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -w sag_pt -force\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cf08b1e-ca7c-4902-b8d0-01968c3801be",
   "metadata": {
    "tags": []
   },
   "source": [
    "The above command creates a job folder at ```/tmp/nvflare/my_job``` with job template ```sag_pt```. \n",
    "You can see that a few configuration files are created. Some of the configurations are open for you to overwrite.\n",
    "\n",
    "If you have the ```tree``` command installed ( ```sudo apt install tree``` on linux), you can use the ```tree``` command, otherwise, you can use \"ls -al\" to look at the job_folder structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e87410e3-c498-44ae-b7f7-2a51af237e41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/my_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "393ae90b-640b-4dd7-a325-e2b026e7703b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/my_job/meta.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba16495-eaa3-431c-84da-432635ec8e29",
   "metadata": {
    "tags": []
   },
   "source": [
    "Notice the app_name is \"my_job\". In `config_fed_client.conf` we can specify the data exchange path, the exchange format, and the way to transfer the model. Let's look at the server side configuration. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c4f50f-b715-42de-a217-f2feced182ba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/my_job/app/config/config_fed_server.conf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24bebc20-5351-40a7-af7b-0a3c9bb271ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/my_job/app/config/config_fed_client.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11767829-65e5-47e1-bae1-6c587c315100",
   "metadata": {},
   "source": [
    "> Note that both client and server configurations are nicely commented with explainations. \n",
    "> If you create the job with customizations such as using -f or configurations, the configuration files will be overwritten. As result, the comments in the configuration will be lost in the final files. \n",
    "\n",
    "### Show variables\n",
    "\n",
    "Now, you can see the job folder is auto-created with pre-defined configurations. To make sure this template works for your code and the variables can be updated. Let's check the variables again with the following command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74adcc28-9f52-4f2b-b4b2-1bd9b053dd80",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job show_variables -j /tmp/nvflare/my_job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8ede943-4c76-42b2-81ad-8d9364637cfb",
   "metadata": {},
   "source": [
    "You can see there are many variables you might want to change:\n",
    "\n",
    "* Change num_rounds to 1 to test out a fast run first.\n",
    "* Use custom cifar10 code which was already written based on Flare 2.4.0 Client API.\n",
    "\n",
    "\n",
    "**Note**\n",
    "\n",
    "the job template name: such as ```sag_pt```, you can also use directory path for the job template. You can try yourself.\n",
    "\n",
    "```\n",
    "! nvflare job create -j /tmp/nvflare/my_job -w ../../job_templates/sag_pt -force\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e991379-78e1-4e86-88fb-ab9a69d6822c",
   "metadata": {},
   "source": [
    "\n",
    "Let's do a second try, \n",
    "\n",
    "#### The second try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69d32eb0-1384-4d49-9b6e-a369c53e7163",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -force -w sag_pt  \\\n",
    "-f config_fed_server.conf num_rounds=1 \\\n",
    "-f config_fed_client.conf app_script=train.py \\\n",
    "-sd ../hello-world/step-by-step/cifar10/code/fl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14aae893-0371-488e-9883-aa2d736a6807",
   "metadata": {},
   "source": [
    "The above command creates a job folder at ```/tmp/nvflare/my_job``` with job template ```sag_pt``` again (`-force` to replace the existing job folder). \n",
    "Now, `num_rounds` is set to 1 and `{app_script}` is \"train.py\": the python script will invoke ```python custom/{app_script}```, so the provided `train.py` will be called.\n",
    "Now, take a look the code structure again: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e09239cb-39cb-4ccf-858e-619d2b1072a0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/my_job\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3103b02a-1ada-4201-b848-811d237be3fd",
   "metadata": {},
   "source": [
    "Notice that the code we had written is copied to the job directory. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "025fa61b-2c30-4421-a74c-75f0725a86f8",
   "metadata": {
    "tags": []
   },
   "source": [
    "In config_fed_server.conf, we have ```PTFileModelPersistor```, a file-based persistor for pytorch. It requires the `net.Net` class for model initialization and also for saving the final model.\n",
    "The \"net.py\" file matches the configuration.  If your model file name and class name does not match `net.Net`, you will need to update your configuration to match. \n",
    "\n",
    "We will leave the rest of values as default and try to run the job. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe97e7a0-38ae-4af7-b8be-73879a69a55f",
   "metadata": {},
   "source": [
    "### Download the data\n",
    "\n",
    "Download the data first to avoid repeated downloading. You can use the download script:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70c829fa-099c-4f40-9571-7e7ca8f026d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! python ../../examples/hello-world/step-by-step/cifar10/data/download.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9308ba5-ad4d-4f3f-9377-76fd6a253687",
   "metadata": {},
   "source": [
    "### Run the Job in simulator \n",
    "\n",
    "You can first run the job with `nvflare simulator` to see if there are any issues:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2451ab27-68cf-4ab3-ae7f-f276f21185a2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare simulator /tmp/nvflare/my_job -w /tmp/my_job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6041e784-d81e-4d7f-a497-4e3dbe050068",
   "metadata": {},
   "source": [
    "If this does not work for you, you may need to make additional changes based on the error messages.\n",
    "\n",
    "Assuming `nvflare simulator` works, you can try running locally with POC mode. For more realistic training, you can first recreate the job configuration with a larger number of rounds (num_rounds=100):\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bbca437-a0be-45da-a438-c387266998a0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -force -w sag_pt \\\n",
    "-f config_fed_server.conf num_rounds=100 \\\n",
    "-f config_fed_client.conf app_script=train.py \\\n",
    "-sd ../hello-world/step-by-step/cifar10/code/fl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be6a878d-5518-4e41-9c3c-2f5098e7d018",
   "metadata": {},
   "source": [
    "\n",
    "### Set up and start POC mode\n",
    "\n",
    "From a terminal, run:\n",
    "\n",
    "```\n",
    "   nvflare poc prepare -n 2\n",
    "   nvflare poc start -ex admin@nvidia.com\n",
    "```\n",
    "This will prepare a workspace for POC with n = 2 clients. The second command starts the POC clients and server except for the FLARE Admin Console (user name = 'admin@nvidia.com'). Since we are going to the Job CLI for submit job, we don't need the admin console for now. Once the system has started, we are ready to move to the next step: submit job.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e839570e-7522-4732-8034-d05a520fa4eb",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Submit Job from CLI\n",
    "\n",
    "You can use the following command to directly submit job from the command line. \n",
    "\n",
    "Even through in `config_fed_server.conf`, num_rounds = 100, to start with a smaller number of rounds, you can set `num_rounds` in the `nvflare job submit` command without changing the value in the config. \n",
    "\n",
    "Also, to change the `train_timeout` to 300 seconds instead of 0 (which means no timeout), this arg is also in `config_fed_server.conf`, so you can include it with `num_rounds` after `-f config_fed_server.conf`.\n",
    "\n",
    "Finally, instead of relying on the default `dataset_path`, you can specify the `dataset_path` in the `nvflare job submit` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5012082-5904-4a24-9211-0e46bc75d6d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job submit -j /tmp/nvflare/my_job \\\n",
    "-f config_fed_server.conf num_rounds=1 train_timeout=300 \\\n",
    "-f config_fed_client.conf app_config=\"--dataset_path /tmp/nvflare/data/cifar10\" \\\n",
    "-debug"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14e34ddc-7f4c-4c14-b287-bedad1b603b3",
   "metadata": {},
   "source": [
    "You can go to the terminal to monitor the output log. \n",
    "\n",
    "> the CLI argument\n",
    "> ```\n",
    ">   app_config=\"--dataset_path /tmp/nvflare/data/cifar10\"\n",
    "> ```\n",
    "> will be translated into \n",
    "\n",
    "> ```\n",
    ">    python custom/train.py --dataset_path \"/tmp/nvflare/data/cifar10\"\n",
    "> ```\n",
    "> in our case, `train.py` takes `--dataset_path` as an argument. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b5522b5-38a8-4f95-98bf-0cd15c336f16",
   "metadata": {},
   "source": [
    "### Submit Job from CLI in production\n",
    "\n",
    "Before you try to submit to production, the Job CLI will need to know the location of the admin console startup kit directory. \n",
    "In POC mode, this is set for the user automatically. In prodcuction, the user will need to set the path to the startup kit for the Job CLI. \n",
    "\n",
    "The startup kit path is stored in the `config.conf` file in the nvflare hidden directory at the user's home directory. First you can take a look at this file: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52788f23-9e78-4396-ad84-c74ae9ef7937",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat ~/.nvflare/config.conf\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e0dcc33-ff07-4554-8d6f-34b327b4ef44",
   "metadata": {},
   "source": [
    "You can directly edit the path in the file:\n",
    "```\n",
    "    startup_kit {\n",
    "        path = /tmp/nvflare/poc/example_project/prod_00\n",
    "    }\n",
    "```\n",
    "Alternatively, you can use the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1a57527-ee4e-4fc8-bfdf-e48fa6466527",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare config --startup_kit_dir /tmp/nvflare/poc/example_project/prod_00"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db24105a-a80c-4c4c-b98f-f354ae5baaa1",
   "metadata": {},
   "source": [
    "or"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01ca3eca-2779-440d-86bc-9a23e1420e99",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare config -d /tmp/nvflare/poc/example_project/prod_00"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cf52c39-cea7-4764-91b3-e061356089cc",
   "metadata": {},
   "source": [
    "Once the startup kit directory path is set, you can do the job submit:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a106a7b0-371e-4504-aa26-a45a55a6347b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job submit -j /tmp/nvflare/my_job \\\n",
    "-f config_fed_server.conf num_rounds=1 \\\n",
    "-f config_fed_client.conf app_config=\"--dataset_path /tmp/nvflare/data/cifar10\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c668def-55e3-477e-8e0a-f76c003b5c31",
   "metadata": {},
   "source": [
    "## Troubleshooting with the `-debug` flag\n",
    "\n",
    "Since the ```nvflare job submit``` command does not overwrite the job folder configuration during submission, it has to use a temp job folder. \n",
    "If you want to check the final configs submited to the server or simply want to see the stack trace of the exception, you can use the `-debug` flag. \n",
    "\n",
    "With the `-debug` flag, the ``` nvflare job submit ``` command will not delete the temp job folder once it has finished job submission, and it will also print the exception stack trace in case of failure. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c676d8ab-6d35-4713-b551-8f5a40927a6e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job submit -j /tmp/nvflare/my_job \\\n",
    "-f config_fed_server.conf num_rounds=1 train_timeout=300 \\\n",
    "-f config_fed_client.conf app_config=\"--dataset_path /tmp/nvflare/data/cifar10\" \\\n",
    "-debug"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3640447-52d8-47f5-80f6-98702efc5b35",
   "metadata": {},
   "source": [
    "You should see a statement like the following after the message that the job was submitted (the actual random folder name will vary): \n",
    "\n",
    "```\n",
    "in debug mode, job configurations can be examined in temp job directory '/tmp/tmpdnusoyzj'\n",
    "```\n",
    "\n",
    "You can check the job folder with `tree` or `ls -al` \n",
    "> note:  the temp folder name can be different on your machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a796f30-e99e-49e0-96ce-ad8809c27ce1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! tree '/tmp/tmpdnusoyzj'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b446bcda-f77a-4c2e-a548-c16822d28796",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cat '/tmp/tmpdnusoyzj/app/config/config_fed_client.conf'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "703ed5d4-1cb2-426d-8efa-03ed2c3464e0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cat '/tmp/tmpdnusoyzj/app/config/config_fed_server.conf'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "493538a7-fe6d-4fc2-a65e-873bf26a09ee",
   "metadata": {},
   "source": [
    "You can see if the configs for server and clients are indeed the values specified."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebaaa444-1f39-4832-ac1b-2d9b3710dbba",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Troubleshooting - Client API timeout\n",
    "\n",
    "If the client API has not received training in 60 seconds, the job will be considered failed with a message like the following:\n",
    "```\n",
    "PTFilePipeLauncherExecutor - ERROR - [identity=site-1, run=simulate_job, peer=simulator_server, peer_run=simulate_job, task_name=train, task_id=db7940f1-d7b4-44e5-b509-dfed4adeb2ec]: received _PEER_GONE_ while waiting for result for train\n",
    "```\n",
    "\n",
    "If you need to, you can increase the value for the timeout: \n",
    "\n",
    "```\n",
    "heartbeat_timeout = 120\n",
    "``` "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed9103e-9f45-4987-be8c-907ab86dfa1d",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Cleanup\n",
    "\n",
    "Make sure you shut down the POC system when you are done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80086770-7a84-42aa-b981-5b9eabb22b9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "! nvflare poc stop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "767c4fa4-9695-4013-a08d-36a3ab2d7e13",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Advanced Section\n",
    "\n",
    "With above sections, you should have understood how to create job with the job template, modify the configuration as needed (either via CLI or manually) and submit job. \n",
    "Now, what if you would like to have \n",
    "\n",
    "* Different configurations on different clients\n",
    "  You could have different datasets on different sites, therefore, the epoches, batch size, learning rate, etc. can be different. \n",
    "\n",
    "* Deploy different code pieces to different sites \n",
    "  You don't need to deploy all the code to all places, only certain code is needed at certain locations. \n",
    "  \n",
    "* add new arguments not in the job templates, modify specific config key using path\n",
    "\n",
    "* remove configuration\n",
    "\n",
    "* modify custom configurations\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "In this section, we will discuss how to do this. So far, we assumed all sites (server and client sites) had the same code and configuration, we deploy all the code + configs to all sites with the following meta.conf\n",
    "\n",
    "```\n",
    "name = \"my_job\"\n",
    "resource_spec {}\n",
    "deploy_map {\n",
    "  app = [\n",
    "    \"@ALL\"\n",
    "  ]\n",
    "}\n",
    "min_clients = 2\n",
    "mandatory_clients = []\n",
    "\n",
    "```\n",
    "\n",
    "Notice the **deploy_map** \n",
    "```\n",
    "deploy_map {\n",
    "  app = [\n",
    "    \"@ALL\"\n",
    "  ]\n",
    "}\n",
    "\n",
    "```\n",
    "### Set up job with different site-specific configurations \n",
    "\n",
    "We are saying that there is \"app\" is deployed to \"ALL\" sites. Let's look at a different example\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b1942b9-f189-4726-b1e8-1ccb3a9ece1d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create \\\n",
    "-j /tmp/nvflare/my_job -w sag_pt_deploy_map "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d39a5466-fcc9-426e-b311-363087727eea",
   "metadata": {},
   "source": [
    "Here we have three different apps : \"app_server\", \"app_1\" and \"app_2\". \n",
    "We would like to change the following: \n",
    "\n",
    "* change number of training rounds to 2\n",
    "* change default app_script from \"cifar10.py\" to \"train.py\" for both app_1 and app_2\n",
    "* change the app_1 batch_size to 4, app_2 batch_size to 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8e2b701-99e6-4aa0-bfac-ab2054635b8f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create \\\n",
    "-j /tmp/nvflare/my_job -w sag_pt_deploy_map \\\n",
    "-f app_server/config_fed_server.conf num_rounds=2 \\\n",
    "-f app_1/config_fed_client.conf app_script=train.py app_config=\"--batch_size 4\" \\\n",
    "-f app_2/config_fed_client.conf app_script=train.py app_config=\"--batch_size 6\" \\\n",
    "-sd ../hello-world/step-by-step/cifar10/code/fl \\\n",
    "-force"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fb01d8-71da-4ab3-a6a6-d66563bd6c8b",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now let's look at the job folder structure. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8270bdfe-a3ab-4c54-92d9-f67ecc59d210",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!tree /tmp/nvflare/my_job\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "050a88ce-084d-4da4-b160-69ac8918b7a8",
   "metadata": {},
   "source": [
    "The job folder consists of three sub-folders, each representing one application: app_server, app_1, app_2. Now look at the meta.conf's deploy_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4aaf60e-254d-4490-970b-fd530b09b84b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cat /tmp/nvflare/my_job/meta.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "821cde68-38cc-48a0-8448-835693b2a131",
   "metadata": {},
   "source": [
    "Notice, app_server is deployed to \"server\", \"app_1\" and \"app_2\" respectively.  The app_1 and app_2 only need client configurations and app_server only need server configuration. Since the server is not doing the training job. we could **remove** ther train.py from the app_server app. and look at again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e0c56d0-a3f6-452e-8b77-c9693c8edd79",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm /tmp/nvflare/my_job/app_server/custom/train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0d3d9f1-40df-44b8-8fb7-52ab820358ca",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!tree /tmp/nvflare/my_job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a65fe596-5abc-4e4b-9f83-67102ddca6cb",
   "metadata": {},
   "source": [
    "Look at the job configuration variables "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d161dc09-804e-4fa8-a3ca-ad4b9efeccf8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job show_variables -j /tmp/nvflare/my_job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c7408ce-fb16-49dc-8df8-fb9d38ad881e",
   "metadata": {},
   "source": [
    "This shows the same information we previously seen. Except it shows each app's configuration. Lets explain a bit mroe about the commnand syntax\n",
    "\n",
    "```\n",
    " nvflare job create \\\n",
    "-j /tmp/nvflare/my_job -w sag_pt_deploy_map \\\n",
    "-f app_server/config_fed_server.conf num_rounds=2 \\\n",
    "-f app_1/config_fed_client.conf app_script=train.py app_config=\"--batch_size 4\" \\\n",
    "-f app_2/config_fed_client.conf app_script=train.py app_config=\"--batch_size 6\" \\\n",
    "-sd ../hello-world/step-by-step/cifar10/code/fl \\\n",
    "-force\n",
    "\n",
    "```\n",
    "\n",
    "to specify app specific configuration, you use\n",
    "\n",
    "```-f app_server/config_fed_server.conf num_rounds=2 ```\n",
    "\n",
    "instead \n",
    "\n",
    "```\n",
    "-f config_fed_server.conf num_rounds=2 \n",
    "\n",
    "```\n",
    "\n",
    "Here it tells the command that that only change the config for \"app_server\" app, without \"app_server/\" the command is considered to use the default \"app\" configuration. \n",
    "\n",
    "if the \"app_name\" is not previously defined in the job templates, the command will show error.  For example\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "062dfee7-5f6d-431c-bf69-6003e220909f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create \\\n",
    "-j /tmp/nvflare/my_job -w sag_pt_deploy_map \\\n",
    "-f app_server/config_fed_server.conf num_rounds=2 \\\n",
    "-force"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b695d5e3-3dde-456a-a2db-54ad8c2ccb7d",
   "metadata": {},
   "source": [
    "Once you have the the job folder. You should be able to run the job as before\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49deee6f-eb9a-4086-80ff-2e96ab94ede6",
   "metadata": {},
   "source": [
    "### Add arguments not originally specified in Job Template\n",
    "\n",
    "In some cases, we need add additional arguments not defined in the job templates, and we would like to the add to a specific args of certain component. This requires we specify the path to the component. \n",
    "\n",
    "We use the following notations to indicate the path\n",
    "* for single component, we can use dot notation. such as ```model.args.number_classes=2```\n",
    "* for component list, we use index notation. such as ```components[1].model.args.number_classes=2```\n",
    "\n",
    "In the 2nd case, ```components[1]``` indicates the 2nd component of the component list. The first component will be ```component[0]```\n",
    "\n",
    "Let's look at how do we use this to add or modify the job template. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b544e5dc-74e5-46de-9efd-8c1b441a4efc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -w sag_pt -force"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d616c49-9a50-46d6-bd78-113cd5ca63ab",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/my_job/app/config/config_fed_server.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8317cfbd-1126-4e4c-b5e0-5ebe9b905713",
   "metadata": {},
   "source": [
    "Here, we would like to modify the peristor component configuration from \n",
    "```\n",
    "  components = [\n",
    "    {\n",
    "      # This is the persistence component used in above workflow.\n",
    "      # PTFileModelPersistor is a Pytorch persistor which save/read the model to/from file.\n",
    "\n",
    "      id = \"persistor\"\n",
    "      path = \"nvflare.app_opt.pt.file_model_persistor.PTFileModelPersistor\"\n",
    "\n",
    "      # the persitor class take model class as argument\n",
    "      # This imply that the model is initialized from the server-side.\n",
    "      # The initialized model will be broadcast to all the clients to start the training.\n",
    "      args.model.path = \"{model_class_path}\"\n",
    "    },\n",
    "```\n",
    "to: \n",
    "\n",
    "```\n",
    "  components = [\n",
    "    {\n",
    "      id = \"persistor\"\n",
    "      path = \"nvflare.app_opt.pt.file_model_persistor.PTFileModelPersistor\"\n",
    "      args {\n",
    "          model { \n",
    "              path = \"{model_class_path}\"\n",
    "              args {\n",
    "                in_channels = 165\n",
    "                hidden_channels = 256\n",
    "                num_classes = 2\n",
    "                num_layers = 3\n",
    "            }\n",
    "          }\n",
    "      }\n",
    "    },\n",
    "```\n",
    "Notice, that new models.args are new keys and values. This is Model Persistor is the 1st component. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4ca2e1a-093b-4c9d-b6b8-9d0ab9382b4c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -w sag_pt -force \\\n",
    "-f config_fed_server.conf \\\n",
    "components[0].args.model.args.in_channels=165 \\\n",
    "components[0].args.model.args.hidden_channels=256 \\\n",
    "components[0].args.model.args.num_classes=2 \\\n",
    "components[0].args.model.args.num_layers=3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "603dfb03-740d-4df4-afa1-4074405702cd",
   "metadata": {},
   "source": [
    "Now, look at the modified configuration again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9d135b5-ceea-44bd-b9e1-04d9967defa1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/my_job/app/config/config_fed_server.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c316852-4a61-4ddc-9ae1-42300ecc65d1",
   "metadata": {},
   "source": [
    "### Remove configuration\n",
    "\n",
    "In some cases, we have changed the local training code class contructor, the arguments from the job template's argments need to be removed. We will show you how to do that. \n",
    "\n",
    "Let's take a look an example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b245cb9d-90c0-4250-98b6-c9ea6734af76",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/jobs/my_job -w stats_df -force"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "625655dc-145c-4e7e-992f-8573c7be8529",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/jobs/my_job/app/config/config_fed_client.conf \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d6425ad-c2d0-4f99-bcce-aa735a77c390",
   "metadata": {},
   "source": [
    "Notice that the 1st component: \n",
    "```\n",
    " \"components\": [\n",
    "    {\n",
    "      \"id\": \"df_stats_generator\",\n",
    "      \"path\": \"df_statistics.DFStatistics\",\n",
    "      \"args\": {\n",
    "        \"data_path\": \"data.csv\"\n",
    "      }\n",
    "    },\n",
    "    \n",
    "    ...\n",
    "   ]\n",
    "```\n",
    "The df_stats_generator uses the class at \"df_statistics.DFStatistics\", withj input arguments as _data_path = \"data.csv\". What if I decided to write my local class called \"df_stats.MyStats\" where it only takes \"data_root_dir\". \n",
    "\n",
    "Now we need to do \n",
    "1) change the path of \"df_stats_generator\" to the new class path\n",
    "2) remove data_path configuration\n",
    "3) add data_root_dir argument. \n",
    "\n",
    "To remove a configuration key, we can use the **<key->** syntax, i.e add \"-\" at the end of key. The key must be exists and must be expressed in full path. such as\n",
    "    **\"components[0].args.data_path-\"**\n",
    "    \n",
    "both\n",
    "    **\"components[0].args.data_path-\"** or \n",
    "    **\"components[0].args.data_path-=value \"** works, although value will be ignored. \n",
    "    \n",
    "    \n",
    ">>Limitation\n",
    "    **the configuration key removal must be against leaf node key. we can't remove parent key such as \"component[0].args\"**\n",
    "    \n",
    "Let's try this out\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92733f42-10c9-457b-95e2-277a4faf4a89",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/jobs/my_job -w stats_df \\\n",
    "-f config_fed_client.conf  \\\n",
    "components[0].path=\"df_stats.MyDFStats\" \\\n",
    "components[0].args.data_path- \\\n",
    "components[0].args.data_root_dir=\"/tmp/dataset\" -force\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74794afa-e863-449f-bb97-a681ffebc3ac",
   "metadata": {},
   "source": [
    "Now let's look the configuration file again, notice the arguments have changed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c35b0851-b7cf-49b2-96aa-f188bd9ecfd8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! cat /tmp/nvflare/jobs/my_job/app/config/config_fed_client.conf \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2731e06d",
   "metadata": {},
   "source": [
    "### Modify custom configuration\n",
    "\n",
    "Up to so far, we have been discussion how to create and modify NVFLARE specific configurations such as meta.conf, config_fed_client.conf and config_fed_server.conf.  How about custom configuration that needed by the training code. The custom configuration files are located in the custom directory of the app. \n",
    "\n",
    "such as \n",
    "   app1/custom/my_config.yaml\n",
    "   app2/custom/my_config.yaml\n",
    "   app_server/custom/my_config.yaml\n",
    "   \n",
    "in such cases, the config file name is arbitary, the file format could be any one of the JSON, PYHOCON or OmegaConf. We can still modify these files. Before jump into the specifics. Let us see what format the CLI offers to input files. We can specify the config file in one of the following ways: \n",
    "\n",
    "```\n",
    "\n",
    " -f config_fed_client.conf\n",
    " \n",
    "```\n",
    "\n",
    "This implies that the input config file, is default to \"app/config/config_client.conf\". Or \n",
    "\n",
    "\n",
    "```\n",
    "\n",
    " -f app1/config_fed_client.conf\n",
    " \n",
    "```\n",
    "This implies that the input config file, is default to \"app1/config/config_client.conf\". Or directly spell out the full path\n",
    " \n",
    " \n",
    "```\n",
    "\n",
    " -f app1/config/config_fed_client.conf\n",
    " \n",
    "```\n",
    " \n",
    "Similarly, we can use \n",
    " \n",
    "\n",
    "```\n",
    "\n",
    " -f app1/custom/my_config.yaml\n",
    " \n",
    "```\n",
    "\n",
    "Let's use an example to demonstrate this with job template \"sag_nemo\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5febd2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -w sag_nemo -force  -sd ../../integration/nemo/examples/peft/nemo_nvflare "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2604462",
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree /tmp/nvflare/my_job\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce121ec6",
   "metadata": {},
   "source": [
    "Notice we use the example script directory \"code\". CLI create job copied the custom folder to the job folder. And there are many custom configuration yaml files. Notice \"weight_decay\" is one of parameters io megatron_gpt_peft_tuning_config.ymal. Let's change it from 0.01 to 0.02 on app_server. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87be7ee7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -w sag_nemo -force \\\n",
    "  -sd ../../integration/nemo/examples/peft/nemo_nvflare \\\n",
    "  -f app_server/custom/megatron_gpt_peft_tuning_config.yaml weight_decay=0.02"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc7195b6",
   "metadata": {},
   "source": [
    "Noticed that the weight_decay for the app_server megatron_gpt_peft_tuning_config.yaml value is updated to 0.02. We can also look at the file saved. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1ee9a5e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cat /tmp/nvflare/my_job/app_server/custom//megatron_gpt_peft_tuning_config.yaml | grep weight_decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a1deaf8-7ee0-4d64-98f9-5eec6102f436",
   "metadata": {
    "tags": []
   },
   "source": [
    "This works! Alternatively, you can spell out the variable path, this can be used to update the exact variable in case duplicated variables ( such as two weight_decay under different paths). Let's do it again "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce7a9b6a-a5ea-4e6f-9bb9-f3b164db3dcb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! nvflare job create -j /tmp/nvflare/my_job -w sag_nemo -force \\\n",
    "  -sd ../../integration/nemo/examples/peft/nemo_nvflare \\\n",
    "  -f app_server/custom/megatron_gpt_peft_tuning_config.yaml model.optim.weight_decay=0.02"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "071aa7b4-5d2a-4d07-8a7c-c6e8abc76638",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cat /tmp/nvflare/my_job/app_server/custom//megatron_gpt_peft_tuning_config.yaml | grep weight_decay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5986b7c-00e9-4f7a-8a08-91008d1cefb0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nvflare_example",
   "language": "python",
   "name": "nvflare_example"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}