demo/denas/bert/DENAS_BERT_DEMO.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![open in colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/e2eAIOK/blob/main/demo/denas/bert/DENAS_BERT_DEMO.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AIOK DE-NAS BERT Demo\n",
    "\n",
    "DE-NAS is a multi-model, hardware-aware, train-free NAS to construct compact model architectures for target platform directly. DE-NAS includes CNN-based search space for CV domain and Transformer-based search space for CV/NLP/ASR domains, and leverages hardware-aware train-free scoring method to evaluate the performance of the candidate architecture without training.\n",
    "\n",
    "This demo mainly introduces NLP integration with DE-NAS to search lighter, faster, higher performance transformer-based NLP model in a training-free way."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Content\n",
    "\n",
    "* [Overview](#overview)\n",
    "    * [DE-NAS on NLP BERT Domain](#DE-NAS-on-NLP-BERT-Domain)\n",
    "    * [Performance](#Performance)\n",
    "* [Getting Started](#Getting-Started)\n",
    "    * [1. Enviroment Setup](#1-environment-setup)\n",
    "    * [2. Workflow Prepare](#2-workflow-prepare)\n",
    "    * [3. Data Prepare](#3-data-prepare)\n",
    "    * [4. Launch Search](#4-launch-search)\n",
    "    * [5. Launch Training with Best Searched Model Structure](#5-launch-training-with-best-searched-model-structure)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview \n",
    "\n",
    "## DE-NAS on NLP BERT Domain\n",
    "\n",
    "### DE-NAS on BERT Search Space and Supernet\n",
    "Transformer-based search space consists of number of transformer layer, number of attention head, size of query/key/value, size of MLP, and dimension of embeddings, and the supernet of DE-NAS on BERT is a BERT-based structure, which are shown as the below figure.\n",
    "\n",
    "<center>\n",
    "<img src=\"./img/NLP_Search_Space.png\" width=\"800\"/><figure>DE-NAS on BERT search space</figure>\n",
    "</center>\n",
    "\n",
    "### DE-NAS Searched BERT Architecture\n",
    "By deploying the train-free EA search engine on DE-NAS BERT search space and supernet, the DE-NAS BERT delivered the architecture that was more compact than the BERT-Base model as shown in the below figure:\n",
    "\n",
    "<center>\n",
    "<img src=\"./img/DENAS BERT Architecture.png\" width=\"500\"/><figure>DE-NAS Searched BERT Architecture</figure>\n",
    "</center>\n",
    "\n",
    "## Performance"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img src=\"./img/Performance.png\" width=\"500\"/><figure>DE-NAS BERT Performance</figure>\n",
    "</center>\n",
    "\n",
    "As shown in the above figure, DE-NAS BERT searched model delivered totally 7.68x training speedup over stock model (BERT) within 5% F1 score regression.\n",
    "* DE-NAS model optimization delivered 2.30x training speedup, where the benefit comes from the compact model (1.61x param reduction and 1.51x Flops reduction).\n",
    "* Hardware scaling delivered 3.34x training speedup, where the benefit comes from 1 node to 4 nodes scaling."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "Noted: Need to download dataset and pretrained model manually to run this demo.\n",
    "\n",
    "## 1. Environment Setup\n",
    "\n",
    "### (Option 1) Use Pip install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install e2eAIOK-denas --pre"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Option 2) Use Docker\n",
    "\n",
    "Step1. prepare code\n",
    "\n",
    "``` shell\n",
    "### Build docker image ###\n",
    "# clone the e2eaiok repo\n",
    "git clone https://github.com/intel/e2eAIOK.git\n",
    "cd e2eAIOK\n",
    "git submodule update --init --recursive\n",
    "```\n",
    "\n",
    "Step2. build docker image\n",
    "\n",
    "```shell\n",
    "python3 scripts/start_e2eaiok_docker.py -b pytorch112 -w ${host0} ${host1} ${host2} ${host3} --proxy \"\"\n",
    "```\n",
    "\n",
    "Step3. run docker and start conda env\n",
    "\n",
    "``` shell\n",
    "sshpass -p docker ssh ${host0} -p 12347\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Workflow Prepare\n",
    "\n",
    "* Conf for BERT DE-NAS Search\n",
    "\n",
    "```yaml\n",
    "# conf for bert\n",
    "model_type: bert\n",
    "search_engine: EvolutionarySearchEngine #supported search engine are Random/Evolutionary/SigoptSearchEngine\n",
    "batch_size: 32\n",
    "supernet_cfg: ../../conf/denas/nlp/supernet-bert-base.yaml\n",
    "pretrained_bert: /home/vmagent/app/dataset/bert-base-uncased\n",
    "pretrained_bert_config: /home/vmagent/app/dataset/bert-base-uncased/config.json\n",
    "\n",
    "# conf for evolutionary search engine\n",
    "random_max_epochs: 1000 #random search max epochs\n",
    "max_epochs: 10 #search epoch\n",
    "select_num: 50\n",
    "population_num: 50\n",
    "m_prob: 0.2\n",
    "s_prob: 0.4\n",
    "crossover_num: 25\n",
    "mutation_num: 25\n",
    "img_size: 128\n",
    "max_param_limits: 110\n",
    "min_param_limits: 55\n",
    "seed: 0\n",
    "\n",
    "# enable/disable each DE-Score\n",
    "expressivity_weight: 0\n",
    "complexity_weight: 0\n",
    "diversity_weight: 0.00001\n",
    "saliency_weight: 1\n",
    "latency_weight: 0.01\n",
    "```\n",
    "\n",
    "The above yaml-format file shows the DE-NAS search relevant configuration on BERT, which was placed on the `/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_bert.conf`. It determines the type of search engine, search hyparameter (etc., batch_size, select_num and population_num), DE-Score parameters (etc., expressivity score weight and latency weight) and supernet/search space configuration (etc., supernet_cfg).\n",
    "\n",
    "* Conf for BERT Supernet and Search Space\n",
    "\n",
    "```yaml\n",
    "# BERT supernet definition\n",
    "SUPERNET:\n",
    "    LAYER_NUM: 12\n",
    "    NUM_ATTENTION_HEADS: 12\n",
    "    HIDDEN_SIZE: 768\n",
    "    INTERMEDIATE_SIZE: 3072\n",
    "    QKV_SIZE: 768\n",
    "\n",
    "# BERT search space definition\n",
    "SEARCH_SPACE:\n",
    "    LAYER_NUM:\n",
    "        bounds:\n",
    "            min: 4\n",
    "            max: 12\n",
    "            step: 1\n",
    "        type: int\n",
    "    HIDDEN_SIZE:\n",
    "        bounds:\n",
    "            min: 128\n",
    "            max: 768\n",
    "            step: 16\n",
    "        type: int\n",
    "    QKV_SIZE:\n",
    "        bounds:\n",
    "            min: 180\n",
    "            max: 768\n",
    "            step: 12\n",
    "        type: int\n",
    "    HEAD_NUM:\n",
    "        bounds:\n",
    "            min: 8\n",
    "            max: 12\n",
    "            step: 1\n",
    "        type: int\n",
    "    INTERMEDIATE_SIZE:\n",
    "        bounds:\n",
    "            min: 128\n",
    "            max: 3072\n",
    "            step: 32\n",
    "        type: int\n",
    "```\n",
    "\n",
    "The above yaml-format file describes the details of BERT-base supernet and search space configuration, which was also placed on the `/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_bert.conf`. The \"LAYER_NUM\", \"HIDDEN_SIZE\", \"QKV_SIZE\", \"HEAD_NUM\" and \"INTERMEDIATE_SIZE\" of BERT-base supernet are determined, and the search space contains the available model parameters used in the DE-NAS search process.\n",
    "\n",
    "* Conf for BERT DE-NAS Train\n",
    "\n",
    "```yaml\n",
    "# model configuration\n",
    "domain: bert\n",
    "best_model_structure: /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/best_model_structure.txt\n",
    "model: /home/vmagent/app/dataset/bert-base-uncased/ #pretrained-model config dir\n",
    "model_dir: /home/vmagent/app/dataset/bert-base-uncased/ #pretrained-model weight dir\n",
    "\n",
    "# task/data configuration\n",
    "task_name: squad1\n",
    "data_set: SQuADv1.1\n",
    "num_train_examples: 87599\n",
    "data_dir: /home/vmagent/app/dataset/SQuAD/\n",
    "output_dir: /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/nlp/\n",
    "eval_metric: \"qa_f1\"\n",
    "do_lower_case: True\n",
    "version_2_with_negative: 0\n",
    "null_score_diff_threshold: 0.0\n",
    "num_labels: 2\n",
    "\n",
    "# training hyper-parameters\n",
    "dist_backend: gloo\n",
    "gradient_accumulation_steps: 1\n",
    "warmup_proportion: 0.1\n",
    "learning_rate: 0.00006\n",
    "weight_decay: 0.01\n",
    "train_epochs: 2\n",
    "max_seq_length: 384\n",
    "doc_stride: 128\n",
    "train_batch_size: 32\n",
    "eval_batch_size: 8\n",
    "eval_step: 500\n",
    "n_best_size: 20\n",
    "max_answer_length: 30\n",
    "max_query_length: 64\n",
    "criterion: \"CrossEntropyQALoss\"\n",
    "optimizer: \"BertAdam\"\n",
    "lr_scheduler: \"warmup_linear\"\n",
    "num_workers: 1\n",
    "pin_mem: True\n",
    "verbose_logging: False\n",
    "no_cuda: True\n",
    "```\n",
    "\n",
    "The above yaml-format file is used in the DE-NAS training process, which was placed on the `/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_train_bert.conf`. It describes the dataset/task settings (etc., task_name and data_dir), model settings (etc., model and model dir) and training hyper-parameters (etc., learning_rate and train_epochs).\n",
    "\n",
    "* Download pre-trained model from Hugging Face\n",
    "    * Download and extract one of BERT-Base-Uncased pretrained models from [Hugging Face repository](https://huggingface.co/bert-base-uncased/tree/main) to `/home/vmagent/app/dataset/bert-base-uncased/`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Data Prepare\n",
    "\n",
    "* Prepare Dataset\n",
    "    * Download Dataset: Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. SQuAD 1.1 contains 100,000+ question-answer pairs on 500+ articles.\n",
    "    * Download from below path to `/home/vmagent/app/dataset/SQuAD`\n",
    "        * Train Data: [train-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)\n",
    "        * Test Data: [dev-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json)\n",
    "``` bash\n",
    "Data Format:\n",
    "{\n",
    "    \"answers\": {\n",
    "        \"answer_start\": [1],\n",
    "        \"text\": [\"This is a test text\"]\n",
    "    },\n",
    "    \"context\": \"This is a test context.\",\n",
    "    \"id\": \"1\",\n",
    "    \"question\": \"Is this a test?\",\n",
    "    \"title\": \"train test\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Launch Search"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch DE-NAS search process on NLP domain BERT with the input of overall search configuration `/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_bert.conf`, and will produce the best model structure as a tuple `(layer_num, head_num, qkv_size, hidden_size, intermediate_size)` in the `best_model_structure.txt` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "paths: /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/asr/utils, /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/asr\n",
      "['/home/vmagent/app/e2eaiok/e2eAIOK/DeNas', '/opt/intel/oneapi/advisor/2022.3.0/pythonapi', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python39.zip', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python3.9', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python3.9/lib-dynload', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python3.9/site-packages', '/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python3.9/site-packages/e2eAIOK-0.2.1-py3.9.egg', '', '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas', '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas/asr']\n",
      "loading archive file /home/vmagent/app/dataset/bert-base-uncased\n",
      "12/01/2022 13:43:12 - INFO - nlp.supernet_bert -   Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 30522\n",
      "}\n",
      "\n",
      "Weights of SuperBertModel not initialized from pretrained model: ['bert.dense_fit.weight', 'bert.dense_fit.bias']\n",
      "Weights from pretrained model not used in SuperBertModel: ['bert.cls.predictions.bias', 'bert.cls.predictions.transform.dense.weight', 'bert.cls.predictions.transform.dense.bias', 'bert.cls.predictions.decoder.weight', 'bert.cls.seq_relationship.weight', 'bert.cls.seq_relationship.bias', 'bert.cls.predictions.transform.LayerNorm.weight', 'bert.cls.predictions.transform.LayerNorm.bias']\n",
      "12/01/2022 13:43:14 - INFO - DENAS -   epoch = 0\n",
      "12/01/2022 13:43:18 - INFO - DENAS -   random 1/10 structure (11, 10, 640, 720, 992) nas_score 224.1220245361328 params 58.934512\n",
      "12/01/2022 13:43:21 - INFO - DENAS -   random 2/10 structure (6, 8, 512, 752, 2816) nas_score 173.40965270996094 params 58.612176\n",
      "12/01/2022 13:43:24 - INFO - DENAS -   random 3/10 structure (12, 11, 704, 656, 1248) nas_score 219.91477966308594 params 62.695536\n",
      "12/01/2022 13:43:28 - INFO - DENAS -   random 4/10 structure (10, 12, 768, 640, 1376) nas_score 254.247802734375 params 57.62336\n",
      "12/01/2022 13:43:31 - INFO - DENAS -   random 5/10 structure (8, 10, 640, 720, 2720) nas_score 142.49546813964844 params 69.01816\n",
      "12/01/2022 13:43:35 - INFO - DENAS -   random 6/10 structure (11, 11, 704, 704, 1824) nas_score 287.5343933105469 params 72.494048\n",
      "12/01/2022 13:43:40 - INFO - DENAS -   random 7/10 structure (11, 11, 704, 768, 1888) nas_score 230.45338439941406 params 80.21168\n",
      "12/01/2022 13:43:44 - INFO - DENAS -   random 8/10 structure (12, 10, 640, 576, 2112) nas_score 200.59535217285156 params 65.191104\n",
      "12/01/2022 13:43:48 - INFO - DENAS -   random 9/10 structure (10, 11, 704, 656, 2144) nas_score 248.85580444335938 params 67.47608\n",
      "12/01/2022 13:43:52 - INFO - DENAS -   random 10/10 structure (9, 11, 704, 624, 2720) nas_score 181.30479431152344 params 66.200592\n",
      "12/01/2022 13:43:52 - INFO - DENAS -   random_num = 10\n",
      "12/01/2022 13:43:57 - INFO - DENAS -   mutation 1/10 structure (12, 12, 768, 672, 1856) nas_score 233.03652954101562 params 76.114272\n",
      "12/01/2022 13:44:01 - INFO - DENAS -   mutation 2/10 structure (12, 10, 640, 576, 2528) nas_score 204.45196533203125 params 70.94688\n",
      "12/01/2022 13:44:04 - INFO - DENAS -   mutation 3/10 structure (11, 11, 704, 768, 864) nas_score 283.5840759277344 params 62.898912\n",
      "12/01/2022 13:44:09 - INFO - DENAS -   mutation 4/10 structure (12, 9, 576, 736, 2112) nas_score 224.6094512939453 params 81.140768\n",
      "12/01/2022 13:44:13 - INFO - DENAS -   mutation 5/10 structure (11, 11, 704, 736, 1824) nas_score 309.4450988769531 params 75.810816\n",
      "12/01/2022 13:44:16 - INFO - DENAS -   mutation 6/10 structure (12, 11, 704, 656, 864) nas_score 171.3025665283203 params 56.645232\n",
      "12/01/2022 13:44:23 - INFO - DENAS -   mutation 7/10 structure (12, 12, 768, 544, 2976) nas_score 174.1815948486328 params 76.192352\n",
      "12/01/2022 13:44:28 - INFO - DENAS -   mutation 8/10 structure (11, 11, 704, 768, 2688) nas_score 273.32733154296875 params 93.73728\n",
      "12/01/2022 13:44:32 - INFO - DENAS -   mutation 9/10 structure (11, 11, 704, 768, 1216) nas_score 303.4856872558594 params 68.850176\n",
      "12/01/2022 13:44:37 - INFO - DENAS -   mutation 10/10 structure (11, 10, 640, 720, 2464) nas_score 262.8741760253906 params 82.267184\n",
      "12/01/2022 13:44:37 - INFO - DENAS -   mutation_num = 10\n",
      "12/01/2022 13:44:43 - INFO - DENAS -   crossover 1/10 structure (11, 11, 704, 624, 2720) nas_score 248.56190490722656 params 76.521232\n",
      "12/01/2022 13:44:47 - INFO - DENAS -   crossover 2/10 structure (10, 11, 704, 656, 1824) nas_score 260.8335876464844 params 63.27448\n",
      "12/01/2022 13:44:51 - INFO - DENAS -   crossover 3/10 structure (10, 11, 704, 640, 2144) nas_score 241.08465576171875 params 65.82112\n",
      "12/01/2022 13:44:55 - INFO - DENAS -   crossover 4/10 structure (11, 11, 704, 624, 1888) nas_score 264.8471374511719 params 65.090384\n",
      "12/01/2022 13:44:58 - INFO - DENAS -   crossover 5/10 structure (6, 11, 704, 752, 2816) nas_score 169.1065216064453 params 62.080848\n",
      "12/01/2022 13:45:02 - INFO - DENAS -   crossover 6/10 structure (11, 11, 704, 768, 1824) nas_score 303.0057067871094 params 79.129632\n",
      "12/01/2022 13:45:06 - INFO - DENAS -   crossover 7/10 structure (10, 10, 640, 656, 2112) nas_score 245.7474365234375 params 65.37464\n",
      "12/01/2022 13:45:10 - INFO - DENAS -   crossover 8/10 structure (6, 12, 768, 752, 2816) nas_score 161.86341857910156 params 63.237072\n",
      "12/01/2022 13:45:15 - INFO - DENAS -   crossover 9/10 structure (8, 10, 640, 720, 2816) nas_score 136.3217315673828 params 70.124848\n",
      "12/01/2022 13:45:18 - INFO - DENAS -   crossover 10/10 structure (11, 11, 704, 640, 1824) nas_score 275.316650390625 params 65.866656\n",
      "12/01/2022 13:45:18 - INFO - DENAS -   crossover_num = 10\n",
      "DE-NAS search best structure took 124.73312006797642 sec\n",
      "12/01/2022 13:45:18 - INFO - DENAS -   best structure (11, 11, 704, 736, 1824) nas_score 309.4450988769531 params 75.810816\n",
      "DE-NAS completed, best structure is (11, 11, 704, 736, 1824)\n"
     ]
    }
   ],
   "source": [
    "from e2eAIOK.DeNas.search.utils import parse_config\n",
    "from e2eAIOK.DeNas.nlp.supernet_bert import SuperBertModel, BertConfig\n",
    "from e2eAIOK.DeNas.nlp.utils import generate_search_space\n",
    "from e2eAIOK.DeNas.search.SearchEngineFactory import SearchEngineFactory\n",
    "\n",
    "# parse DE-NAS search configure\n",
    "params = parse_config('/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_bert.conf')\n",
    "\n",
    "# construct supernet and search space\n",
    "config = BertConfig.from_json_file(params.pretrained_bert_config)\n",
    "super_net = SuperBertModel.from_pretrained(params.pretrained_bert, config)\n",
    "search_space = generate_search_space(params[\"SEARCH_SPACE\"])\n",
    "\n",
    "# create DE-NAS searcher\n",
    "searcher = SearchEngineFactory.create_search_engine(params = params, super_net = super_net, search_space = search_space)\n",
    "\n",
    "# trigger the search process\n",
    "searcher.search()\n",
    "best_structure = searcher.get_best_structures()\n",
    "print(f\"DE-NAS completed, best structure is {best_structure}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Launch Training with Best Searched Model Structure"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch DE-NAS training process on NLP domain BERT with the input of overall training configuration `/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_train_bert.conf`, and will produce the fine-tuned BERT performance on SQuADv1.1 task.\n",
    "\n",
    "Noted: Below performance result is using sample dataset and small iterations to demonstrate its function. The actual performance result please refers to the [performance section](#performance)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python3.9/runpy.py:127: RuntimeWarning: 'intel_extension_for_pytorch.cpu.launch' found in sys.modules after import of package 'intel_extension_for_pytorch.cpu', but prior to execution of 'intel_extension_for_pytorch.cpu.launch'; this may result in unpredictable behaviour\n",
      "  warn(RuntimeWarning(msg))\n",
      "2022-12-01 13:52:41,414 - __main__ - INFO - MASTER_ADDR=127.0.0.1\n",
      "2022-12-01 13:52:41,414 - __main__ - INFO - MASTER_PORT=29500\n",
      "2022-12-01 13:52:41,415 - __main__ - INFO - I_MPI_PIN_DOMAIN=[0xffffffffffff0,]\n",
      "2022-12-01 13:52:41,415 - __main__ - WARNING - Neither TCMalloc nor JeMalloc is found in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or /root/.local/lib/ so the LD_PRELOAD environment variable will not be set. This may drop the performance\n",
      "2022-12-01 13:52:41,415 - __main__ - INFO - OMP_NUM_THREADS=48\n",
      "2022-12-01 13:52:41,415 - __main__ - INFO - Using Intel OpenMP\n",
      "2022-12-01 13:52:41,416 - __main__ - INFO - KMP_AFFINITY=granularity=fine,compact,1,0\n",
      "2022-12-01 13:52:41,416 - __main__ - INFO - KMP_BLOCKTIME=1\n",
      "2022-12-01 13:52:41,416 - __main__ - INFO - LD_PRELOAD=/opt/intel/oneapi/intelpython/latest/lib/libiomp5.so\n",
      "2022-12-01 13:52:41,416 - __main__ - INFO - CCL_WORKER_COUNT=4\n",
      "2022-12-01 13:52:41,416 - __main__ - INFO - CCL_WORKER_AFFINITY=0,1,2,3\n",
      "2022-12-01 13:52:41,416 - __main__ - INFO - mpiexec.hydra -l -np 1 -ppn 1 -genv I_MPI_PIN_DOMAIN=[0xffffffffffff0,] -genv OMP_NUM_THREADS=48 /opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/bin/python -u train.py --domain bert --conf /home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_train_bert.conf\n",
      "[0] 12/01/2022 13:52:42 - INFO - Trainer -   building model\n",
      "[0] 12/01/2022 13:52:42 - INFO - e2eAIOK.DeNas.nlp.supernet_bert -   Model config {\n",
      "[0]   \"attention_probs_dropout_prob\": 0.1,\n",
      "[0]   \"hidden_act\": \"gelu\",\n",
      "[0]   \"hidden_dropout_prob\": 0.1,\n",
      "[0]   \"hidden_size\": 768,\n",
      "[0]   \"initializer_range\": 0.02,\n",
      "[0]   \"intermediate_size\": 3072,\n",
      "[0]   \"layer_norm_eps\": 1e-12,\n",
      "[0]   \"max_position_embeddings\": 512,\n",
      "[0]   \"num_attention_heads\": 12,\n",
      "[0]   \"num_hidden_layers\": 12,\n",
      "[0]   \"type_vocab_size\": 2,\n",
      "[0]   \"vocab_size\": 30522\n",
      "[0] }\n",
      "[0] \n",
      "[0] loading archive file /home/vmagent/app/dataset/bert-base-uncased/\n",
      "[0] 12/01/2022 13:52:42 - INFO - e2eAIOK.DeNas.nlp.supernet_bert -   Model config {\n",
      "[0]   \"attention_probs_dropout_prob\": 0.1,\n",
      "[0]   \"hidden_act\": \"gelu\",\n",
      "[0]   \"hidden_dropout_prob\": 0.1,\n",
      "[0]   \"hidden_size\": 768,\n",
      "[0]   \"initializer_range\": 0.02,\n",
      "[0]   \"intermediate_size\": 3072,\n",
      "[0]   \"layer_norm_eps\": 1e-12,\n",
      "[0]   \"max_position_embeddings\": 512,\n",
      "[0]   \"num_attention_heads\": 12,\n",
      "[0]   \"num_hidden_layers\": 12,\n",
      "[0]   \"type_vocab_size\": 2,\n",
      "[0]   \"vocab_size\": 30522\n",
      "[0] }\n",
      "[0] \n",
      "[0] Weights of SuperBertForQuestionAnswering not initialized from pretrained model: ['bert.dense_fit.weight', 'bert.dense_fit.bias', 'qa_outputs.weight', 'qa_outputs.bias']\n",
      "[0] Weights from pretrained model not used in SuperBertForQuestionAnswering: ['bert.cls.predictions.bias', 'bert.cls.predictions.transform.dense.weight', 'bert.cls.predictions.transform.dense.bias', 'bert.cls.predictions.decoder.weight', 'bert.cls.seq_relationship.weight', 'bert.cls.seq_relationship.bias', 'bert.cls.predictions.transform.LayerNorm.weight', 'bert.cls.predictions.transform.LayerNorm.bias']\n",
      "[0] architecture: {'sample_layer_num': 11, 'sample_num_attention_heads': [11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11], 'sample_qkv_sizes': [704, 704, 704, 704, 704, 704, 704, 704, 704, 704, 704], 'sample_hidden_size': 736, 'sample_intermediate_sizes': [1824, 1824, 1824, 1824, 1824, 1824, 1824, 1824, 1824, 1824, 1824]}\n",
      "[0] Total parameters: 75810816\n",
      "[0] 12/01/2022 13:52:44 - INFO - Trainer -   model created: SuperBertForQuestionAnswering(\n",
      "[0]   (bert): SuperBertModel(\n",
      "[0]     (embeddings): SuperBertEmbeddings(\n",
      "[0]       (word_embeddings): SuperEmbedding(\n",
      "[0]         (embedding): Embedding(30522, 768, padding_idx=0)\n",
      "[0]       )\n",
      "[0]       (position_embeddings): SuperEmbedding(\n",
      "[0]         (embedding): Embedding(512, 768)\n",
      "[0]       )\n",
      "[0]       (token_type_embeddings): SuperEmbedding(\n",
      "[0]         (embedding): Embedding(2, 768)\n",
      "[0]       )\n",
      "[0]       (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]       (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]     )\n",
      "[0]     (encoder): SuperBertEncoder(\n",
      "[0]       (layers): ModuleList(\n",
      "[0]         (0): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (1): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (2): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (3): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (4): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (5): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (6): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (7): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (8): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (9): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (10): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]         (11): SuperBertLayer(\n",
      "[0]           (attention): SuperBertAttention(\n",
      "[0]             (self): SuperBertSelfAttention(\n",
      "[0]               (query): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (key): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (value): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]             (output): SuperBertSelfOutput(\n",
      "[0]               (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]               (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]               (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]             )\n",
      "[0]           )\n",
      "[0]           (intermediate): SuperBertIntermediate(\n",
      "[0]             (dense): LinearSuper(in_features=768, out_features=3072, bias=True)\n",
      "[0]           )\n",
      "[0]           (output): SuperBertOutput(\n",
      "[0]             (dense): LinearSuper(in_features=3072, out_features=768, bias=True)\n",
      "[0]             (LayerNorm): LayerNormSuper((768,), eps=1e-05, elementwise_affine=True)\n",
      "[0]             (dropout): Dropout(p=0.1, inplace=False)\n",
      "[0]           )\n",
      "[0]         )\n",
      "[0]       )\n",
      "[0]     )\n",
      "[0]     (pooler): SuperBertPooler(\n",
      "[0]       (dense): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]       (activation): Tanh()\n",
      "[0]     )\n",
      "[0]     (dense_fit): LinearSuper(in_features=768, out_features=768, bias=True)\n",
      "[0]   )\n",
      "[0]   (qa_outputs): LinearSuper(in_features=768, out_features=2, bias=True)\n",
      "[0] )\n",
      "[0] 12/01/2022 13:52:44 - INFO - e2eAIOK.DeNas.module.nlp.tokenization -   loading vocabulary file\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] 12/01/2022 13:52:44 - INFO - e2eAIOK.common.trainer.data.data_builder_squad -   load 1027 examples!\n",
      "[0] 12/01/2022 13:52:46 - INFO - e2eAIOK.DeNas.module.nlp.tokenization -   loading vocabulary file\n",
      "[0] 12/01/2022 13:52:47 - INFO - e2eAIOK.common.trainer.data.data_builder_squad -   load 1680 examples!\n",
      "[0] /opt/intel/oneapi/intelpython/latest/envs/pytorch-1.12.0/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 10 worker processes in total. Our suggested max number of worker in current system is 1, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
      "[0]   warnings.warn(_create_warning_msg(\n",
      "[0] 12/01/2022 13:52:49 - INFO - Trainer -   Trainer config: {'domain': 'bert', 'task_name': 'squad1', 'data_set': 'SQuADv1.1', 'num_train_examples': 87599, 'best_model_structure': '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas/best_model_structure.txt', 'model': '/home/vmagent/app/dataset/bert-base-uncased/', 'model_dir': '/home/vmagent/app/dataset/bert-base-uncased/', 'data_dir': '/home/vmagent/app/dataset/SQuAD/', 'output_dir': '/home/vmagent/app/e2eaiok/e2eAIOK/DeNas/nlp/', 'dist_backend': 'gloo', 'gradient_accumulation_steps': 1, 'warmup_proportion': 0.1, 'learning_rate': 3e-05, 'weight_decay': 0.0001, 'train_epochs': 4, 'max_seq_length': 384, 'doc_stride': 128, 'train_batch_size': 12, 'eval_batch_size': 32, 'eval_step': 50, 'n_best_size': 20, 'max_answer_length': 30, 'max_query_length': 64, 'criterion': 'CrossEntropyQALoss', 'optimizer': 'BertAdam', 'lr_scheduler': 'warmup_linear', 'version_2_with_negative': 0, 'null_score_diff_threshold': 0.0, 'num_labels': 2, 'num_workers': 10, 'pin_mem': True, 'verbose_logging': False, 'no_cuda': True, 'do_lower_case': True, 'metric_threshold': 81.5, 'eval_metric': 'qa_f1'}\n",
      "Iteration:   0%|          | 0/88 [00:00<?, ?it/s][0] /home/vmagent/app/e2eaiok/e2eAIOK/DeNas/module/nlp/optimization.py:249: UserWarning: This overload of add_ is deprecated:\n",
      "[0] \tadd_(Number alpha, Tensor other)\n",
      "[0] Consider using one of the following signatures instead:\n",
      "[0] \tadd_(Tensor other, *, Number alpha) (Triggered internally at  ../torch/csrc/utils/python_arg_parser.cpp:1174.)\n",
      "[0]   next_m.mul_(beta1).add_(1 - beta1, grad)\n",
      "Iteration:  56%|#####5    | 49/88 [01:50<01:13,  1.88s/it]0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 13:54:42 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 13:54:42 - INFO - Trainer -     Epoch = 0 iter 50 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 0.23809523809523808\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 731.6228148148148\n",
      "[0] qa_f1 = 7.608065657374889\n",
      "Iteration: 100%|##########| 88/88 [04:40<00:00,  3.18s/it][0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] \n",
      "[0] 12/01/2022 13:57:30 - INFO - Trainer -   Epoch 0 training time:280.2015838623047\n",
      "Iteration:  12%|#2        | 11/88 [00:34<02:43,  2.12s/it]0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 13:58:07 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 13:58:07 - INFO - Trainer -     Epoch = 1 iter 100 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 0.9523809523809523\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 698.6857222222221\n",
      "[0] qa_f1 = 7.921236397925677\n",
      "Iteration:  69%|######9   | 61/88 [03:43<00:44,  1.65s/it][0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 14:01:15 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 14:01:15 - INFO - Trainer -     Epoch = 1 iter 150 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 3.2142857142857144\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 706.5842222222223\n",
      "[0] qa_f1 = 9.665600150803513\n",
      "Iteration: 100%|##########| 88/88 [06:17<00:00,  4.29s/it][0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] \n",
      "[0] 12/01/2022 14:03:47 - INFO - Trainer -   Epoch 1 training time:377.3344202041626\n",
      "Iteration:  26%|##6       | 23/88 [01:01<02:09,  2.00s/it]0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 14:04:51 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 14:04:51 - INFO - Trainer -     Epoch = 2 iter 200 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 4.880952380952381\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 719.3742592592594\n",
      "[0] qa_f1 = 11.540288403595241\n",
      "Iteration:  83%|########2 | 73/88 [04:11<00:24,  1.64s/it][0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 14:08:00 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 14:08:00 - INFO - Trainer -     Epoch = 2 iter 250 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 6.369047619047619\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 700.3415185185186\n",
      "[0] qa_f1 = 12.652565409722166\n",
      "Iteration: 100%|##########| 88/88 [06:26<00:00,  4.39s/it][0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] \n",
      "[0] 12/01/2022 14:10:13 - INFO - Trainer -   Epoch 2 training time:386.25261545181274\n",
      "Iteration:  40%|###9      | 35/88 [01:20<01:44,  1.96s/it]0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 14:11:36 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 14:11:36 - INFO - Trainer -     Epoch = 3 iter 300 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 5.595238095238095\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 855.5276666666666\n",
      "[0] qa_f1 = 12.735416507468962\n",
      "Iteration:  97%|#########6| 85/88 [04:39<00:05,  1.67s/it][0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] [0] 12/01/2022 14:14:54 - INFO - Trainer -   ***** Running evaluation *****\n",
      "[0] 12/01/2022 14:14:54 - INFO - Trainer -     Epoch = 3 iter 350 step\n",
      "[0] ***** Eval results *****\n",
      "[0] em = 6.428571428571429\n",
      "[0] infer_cnt = 54\n",
      "[0] infer_time = 761.5051296296297\n",
      "[0] qa_f1 = 12.8929755881453\n",
      "Iteration: 100%|##########| 88/88 [06:37<00:00,  4.52s/it][0] [0] [0] \n",
      "[0] 12/01/2022 14:16:51 - INFO - Trainer -   Epoch 3 training time:397.6090269088745\n",
      "[0] 12/01/2022 14:16:51 - INFO - Trainer -   **************S*************\n",
      "[0] task_name = squad1\n",
      "[0] parameter size = 75810816\n",
      "[0] total training time = 1441.40003824234\n",
      "[0] best_acc = 12.8929755881453\n",
      "[0] **************E*************\n",
      "[0] \n"
     ]
    }
   ],
   "source": [
    "from e2eAIOK.DeNas.nlp.model_builder_denas_nlp import ModelBuilderNLPDeNas\n",
    "from e2eAIOK.common.trainer.data.nlp.data_builder_squad import DataBuilderSQuAD\n",
    "from e2eAIOK.DeNas.nlp.utils import bert_create_optimizer, bert_create_criterion, bert_create_scheduler, bert_create_metric\n",
    "from e2eAIOK.DeNas.nlp.bert_trainer import BERTTrainer\n",
    "from e2eAIOK.DeNas.search.utils import parse_config\n",
    "\n",
    "\n",
    "# parse DE-NAS train configure\n",
    "cfg = parse_config(\"/home/vmagent/app/e2eaiok/conf/denas/nlp/e2eaiok_denas_train_bert.conf\")\n",
    "\n",
    "# construct model, dataloader, optimizer, criterion, scheduler and metric\n",
    "model = ModelBuilderNLPDeNas(cfg).create_model()\n",
    "train_dataloader, eval_dataloader, other_data = DataBuilderSQuAD(cfg).get_dataloader()\n",
    "cfg.num_train_steps = len(train_dataloader)\n",
    "optimizer = bert_create_optimizer(model, cfg)\n",
    "criterion = bert_create_criterion(cfg)\n",
    "scheduler = bert_create_scheduler(cfg)\n",
    "metric = bert_create_metric(cfg)\n",
    "\n",
    "# create DE-NAS trainer\n",
    "trainer = BERTTrainer(cfg, model, train_dataloader, eval_dataloader, other_data, optimizer, criterion, scheduler, metric)\n",
    "\n",
    "# trigger the training process\n",
    "trainer.fit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch-1.12.0",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "b38dbbca6c1c1b09874e6226521bb418530c95443a0afe85f48e402a3f241a05"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}