[hands-on]autogen을_이용해서_출력_파라미터를_튜닝하는_방법oai_chatgpt_gpt4.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jkf87/autogen-handson/blob/main/autogen%EC%9D%84_%EC%9D%B4%EC%9A%A9%ED%95%B4%EC%84%9C_%EC%B6%9C%EB%A0%A5_%ED%8C%8C%EB%9D%BC%EB%AF%B8%ED%84%B0%EB%A5%BC_%ED%8A%9C%EB%8B%9D%ED%95%98%EB%8A%94_%EB%B0%A9%EB%B2%95oai_chatgpt_gpt4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h8Q-SzS-hOPi"
      },
      "source": [
        "저작권 (c) Microsoft Corporation. 모든 권리 보유.\n",
        "\n",
        "MIT 라이선스에 따라 라이센스가 부여되었습니다.\n",
        "\n",
        "# AutoGen을 사용하여 ChatGPT 튜닝하기\n",
        "\n",
        "AutoGen은 대규모 언어 모델 튜닝을 위한 비용 효율적인 하이퍼파라미터 최적화 기법 [EcoOptiGen](https://arxiv.org/abs/2303.04673)을 제공합니다. 연구에 따르면 하이퍼파라미터를 튜닝하면 LLM의 유용성을 크게 향상시킬 수 있습니다.\n",
        "이 기능에 대한 설명서는 [여기](/docs/Use-Cases/AutoGen#enhanced-inference)에서 확인할 수 있습니다.\n",
        "\n",
        "이 노트북에서는 수학 문제 해결을 위해 OpenAI ChatGPT(GPT-3.5와 GPT-4 모두) 모델을 튜닝합니다. 우리는 [the MATH 벤치마크](https://crfm.stanford.edu/helm/latest/?group=math_chain_of_thought)를 사용하여 연쇄적 추론 스타일의 경쟁 수학 문제에서 수학 문제 해결력을 측정합니다.\n",
        "\n",
        "관련 링크: 이 실험을 기반으로 한 [블로그 포스트](https://microsoft.github.io/autogen/blog/2023/04/21/LLM-tuning-math).\n",
        "\n",
        "## 요구 사항\n",
        "\n",
        "자동 생성에는 `Python>=3.8`이 필요합니다. 이 노트북 예제를 실행하려면 [blendsearch] 옵션으로 설치하세요:\n",
        "```bash\n",
        "pip install \"pyautogen[blendsearch]\"\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:52.317406Z",
          "iopub.status.busy": "2023-02-13T23:40:52.316561Z",
          "iopub.status.idle": "2023-02-13T23:40:52.321193Z",
          "shell.execute_reply": "2023-02-13T23:40:52.320628Z"
        },
        "id": "x4Qc1srKhOPj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "141c9dd8-87f6-4445-c288-ec30c5a45473"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pyautogen[blendsearch]\n",
            "  Downloading pyautogen-0.1.7-py3-none-any.whl (71 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.3/71.3 kB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting datasets\n",
            "  Downloading datasets-2.14.5-py3-none-any.whl (519 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m519.6/519.6 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting diskcache (from pyautogen[blendsearch])\n",
            "  Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting flaml (from pyautogen[blendsearch])\n",
            "  Downloading FLAML-2.1.1-py3-none-any.whl (295 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m295.2/295.2 kB\u001b[0m \u001b[31m24.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting openai (from pyautogen[blendsearch])\n",
            "  Downloading openai-0.28.1-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.0/77.0 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting python-dotenv (from pyautogen[blendsearch])\n",
            "  Downloading python_dotenv-1.0.0-py3-none-any.whl (19 kB)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from pyautogen[blendsearch]) (2.3.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.23.5)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (9.0.0)\n",
            "Collecting dill<0.3.8,>=0.3.0 (from datasets)\n",
            "  Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (1.5.3)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.31.0)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.66.1)\n",
            "Collecting xxhash (from datasets)\n",
            "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m17.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting multiprocess (from datasets)\n",
            "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]<2023.9.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.8.5)\n",
            "Collecting huggingface-hub<1.0.0,>=0.14.0 (from datasets)\n",
            "  Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m295.0/295.0 kB\u001b[0m \u001b[31m24.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (23.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (3.3.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.14.0->datasets) (3.12.4)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.14.0->datasets) (4.5.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (2.0.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (2023.7.22)\n",
            "Collecting optuna==2.8.0 (from flaml->pyautogen[blendsearch])\n",
            "  Downloading optuna-2.8.0-py3-none-any.whl (301 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.0/302.0 kB\u001b[0m \u001b[31m14.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting alembic (from optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading alembic-1.12.0-py3-none-any.whl (226 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.0/226.0 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cliff (from optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading cliff-4.3.0-py3-none-any.whl (80 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.6/80.6 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cmaes>=0.8.2 (from optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading cmaes-0.10.0-py3-none-any.whl (29 kB)\n",
            "Collecting colorlog (from optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading colorlog-6.7.0-py2.py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: scipy!=1.4.0 in /usr/local/lib/python3.10/dist-packages (from optuna==2.8.0->flaml->pyautogen[blendsearch]) (1.11.3)\n",
            "Requirement already satisfied: sqlalchemy>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from optuna==2.8.0->flaml->pyautogen[blendsearch]) (2.0.21)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2023.3.post1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/dist-packages (from sqlalchemy>=1.1.0->optuna==2.8.0->flaml->pyautogen[blendsearch]) (3.0.0)\n",
            "Collecting Mako (from alembic->optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading Mako-1.2.4-py3-none-any.whl (78 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.7/78.7 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: PrettyTable>=0.7.2 in /usr/local/lib/python3.10/dist-packages (from cliff->optuna==2.8.0->flaml->pyautogen[blendsearch]) (3.9.0)\n",
            "Collecting autopage>=0.4.0 (from cliff->optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading autopage-0.5.1-py3-none-any.whl (29 kB)\n",
            "Collecting cmd2>=1.0.0 (from cliff->optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading cmd2-2.4.3-py3-none-any.whl (147 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.2/147.2 kB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.10/dist-packages (from cliff->optuna==2.8.0->flaml->pyautogen[blendsearch]) (6.8.0)\n",
            "Collecting stevedore>=2.0.1 (from cliff->optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading stevedore-5.1.0-py3-none-any.whl (49 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.6/49.6 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyperclip>=1.6 in /usr/local/lib/python3.10/dist-packages (from cmd2>=1.0.0->cliff->optuna==2.8.0->flaml->pyautogen[blendsearch]) (1.8.2)\n",
            "Requirement already satisfied: wcwidth>=0.1.7 in /usr/local/lib/python3.10/dist-packages (from cmd2>=1.0.0->cliff->optuna==2.8.0->flaml->pyautogen[blendsearch]) (0.2.8)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=4.4->cliff->optuna==2.8.0->flaml->pyautogen[blendsearch]) (3.17.0)\n",
            "Collecting pbr!=2.1.0,>=2.0.0 (from stevedore>=2.0.1->cliff->optuna==2.8.0->flaml->pyautogen[blendsearch])\n",
            "  Downloading pbr-5.11.1-py2.py3-none-any.whl (112 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.7/112.7 kB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from Mako->alembic->optuna==2.8.0->flaml->pyautogen[blendsearch]) (2.1.3)\n",
            "Installing collected packages: xxhash, python-dotenv, pbr, Mako, flaml, diskcache, dill, colorlog, cmd2, cmaes, autopage, stevedore, multiprocess, huggingface-hub, alembic, openai, cliff, pyautogen, optuna, datasets\n",
            "Successfully installed Mako-1.2.4 alembic-1.12.0 autopage-0.5.1 cliff-4.3.0 cmaes-0.10.0 cmd2-2.4.3 colorlog-6.7.0 datasets-2.14.5 dill-0.3.7 diskcache-5.6.3 flaml-2.1.1 huggingface-hub-0.17.3 multiprocess-0.70.15 openai-0.28.1 optuna-2.8.0 pbr-5.11.1 pyautogen-0.1.7 python-dotenv-1.0.0 stevedore-5.1.0 xxhash-3.4.1\n"
          ]
        }
      ],
      "source": [
        "%pip install \"pyautogen[blendsearch]\" datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t9va_ZV0hOPk"
      },
      "source": [
        "AutoGen has provided an API for hyperparameter optimization of OpenAI ChatGPT models: `autogen.ChatCompletion.tune` and to make a request with the tuned config: `autogen.ChatCompletion.create`. First, we import autogen:\n",
        "\n",
        "AutoGen은 OpenAI ChatGPT 모델의 하이퍼파라미터 최적화를 위한 API를 제공합니다: `autogen.ChatCompletion.tune` 을 호출하고 튜닝된 설정으로 요청할 수 있습니다: `autogen.ChatCompletion.create`를 제공합니다. 먼저 오토젠을 가져옵니다:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:54.634335Z",
          "iopub.status.busy": "2023-02-13T23:40:54.633929Z",
          "iopub.status.idle": "2023-02-13T23:40:56.105700Z",
          "shell.execute_reply": "2023-02-13T23:40:56.105085Z"
        },
        "id": "qEmHuelNhOPk"
      },
      "outputs": [],
      "source": [
        "import autogen"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xkioRMPMhOPk"
      },
      "source": [
        "### Set your API Endpoint\n",
        "\n",
        "The [`config_list_openai_aoai`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_openai_aoai) function tries to create a list of  Azure OpenAI endpoints and OpenAI endpoints. It assumes the api keys and api bases are stored in the corresponding environment variables or local txt files:\n",
        "\n",
        "- OpenAI API key: os.environ[\"OPENAI_API_KEY\"] or `openai_api_key_file=\"key_openai.txt\"`.\n",
        "- Azure OpenAI API key: os.environ[\"AZURE_OPENAI_API_KEY\"] or `aoai_api_key_file=\"key_aoai.txt\"`. Multiple keys can be stored, one per line.\n",
        "- Azure OpenAI API base: os.environ[\"AZURE_OPENAI_API_BASE\"] or `aoai_api_base_file=\"base_aoai.txt\"`. Multiple bases can be stored, one per line.\n",
        "\n",
        "It's OK to have only the OpenAI API key, or only the Azure OpenAI API key + base.\n",
        "\n",
        "\n",
        "### API 엔드포인트 설정\n",
        "\n",
        "`config_list_openai_aoai`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_openai_aoai) 함수는 Azure OpenAI 엔드포인트 및 OpenAI 엔드포인트 목록을 생성하려고 시도합니다. 이 함수는 API 키와 API 베이스가 해당 환경 변수 또는 로컬 txt 파일에 저장되어 있다고 가정합니다:\n",
        "\n",
        "- OpenAI API 키: os.environ[\"OPENAI_API_KEY\"] 또는 `openai_api_key_file=\"key_openai.txt\"`.\n",
        "- Azure OpenAI API 키: os.environ[\"AZURE_OPENAI_API_KEY\"] 또는 `aoai_api_key_file=\"key_aoai.txt\"`. 한 줄에 하나씩 여러 키를 저장할 수 있습니다.\n",
        "- Azure OpenAI API 베이스: os.environ[\"AZURE_OPENAI_API_BASE\"] 또는 `aoai_api_base_file=\"base_aoai.txt\"`. 한 줄에 하나씩 여러 개의 베이스를 저장할 수 있습니다.\n",
        "\n",
        "OpenAI API 키만 있거나 Azure OpenAI API 키 + 베이스만 있어도 괜찮습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:52.324240Z",
          "iopub.status.busy": "2023-02-13T23:40:52.323783Z",
          "iopub.status.idle": "2023-02-13T23:40:52.330570Z",
          "shell.execute_reply": "2023-02-13T23:40:52.329750Z"
        },
        "id": "kQBZOJ9zhOPl"
      },
      "outputs": [],
      "source": [
        "config_list = autogen.config_list_openai_aoai()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voD-y1NOhOPl"
      },
      "source": [
        "The config list looks like the following:\n",
        "```python\n",
        "config_list = [\n",
        "    {'api_key': '<your OpenAI API key here>'},  # only if OpenAI API key is found\n",
        "    {\n",
        "        'api_key': '<your first Azure OpenAI API key here>',\n",
        "        'api_base': '<your first Azure OpenAI API base here>',\n",
        "        'api_type': 'azure',\n",
        "        'api_version': '2023-06-01-preview',\n",
        "    },  # only if the at least one Azure OpenAI API key is found\n",
        "    {\n",
        "        'api_key': '<your second Azure OpenAI API key here>',\n",
        "        'api_base': '<your second Azure OpenAI API base here>',\n",
        "        'api_type': 'azure',\n",
        "        'api_version': '2023-06-01-preview',\n",
        "    },  # only if the second Azure OpenAI API key is found\n",
        "]\n",
        "```\n",
        "\n",
        "You can directly override it if the above function returns an empty list, i.e., it doesn't find the keys in the specified locations."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#API 엔드포인트 설정 코드 변경\n",
        "- 기존 config_list 방식이 번거로운 분들을 위해 설정 셀 추가\n",
        "- `!pip install python-dotenv`\n",
        "\n",
        "```\n",
        "import json\n",
        "from dotenv import find_dotenv, load_dotenv\n",
        "\n",
        "env_var = [\n",
        "    {\n",
        "        'model': 'gpt-4',\n",
        "        'api_key': \"sk-\"\n",
        "    },\n",
        "    {\n",
        "        'model': 'gpt-3.5-turbo',\n",
        "        'api_key': \"sk-\"\n",
        "    }\n",
        "]\n",
        "```"
      ],
      "metadata": {
        "id": "hU4rjBsoiwBR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install python-dotenv"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rF86fxXVjT3n",
        "outputId": "14699a63-5c8f-4645-bed2-f8cc9794bf1d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: python-dotenv in /usr/local/lib/python3.10/dist-packages (1.0.0)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "from dotenv import find_dotenv, load_dotenv\n",
        "\n",
        "\n",
        "env_var = [\n",
        "    {\n",
        "        'model': 'gpt-3.5-turbo',\n",
        "        'api_key': \"sk-\"\n",
        "    }\n",
        "\n",
        "]\n",
        "config_list = env_var"
      ],
      "metadata": {
        "id": "XawXor0UjYxd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XhuFBWNhOPl"
      },
      "source": [
        "# 이 부분 아래 코드로 대체함\n",
        "## Load dataset\n",
        "\n",
        "We load the competition_math dataset. The dataset contains 201 \"Level 2\" Algebra examples. We use a random sample of 20 examples for tuning the generation hyperparameters and the remaining for evaluation.\n",
        "\n",
        "## 데이터 세트 로드\n",
        "\n",
        "competition_math 데이터 세트를 로드합니다. 이 데이터 세트에는 201개의 \"레벨 2\" 대수 예제가 포함되어 있습니다. 생성 하이퍼파라미터를 튜닝하기 위해 20개의 예제 중 무작위 샘플을 사용하고 나머지는 평가에 사용합니다."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import list_datasets\n",
        "\n",
        "# Get the list of available datasets\n",
        "available_datasets = list_datasets()\n",
        "\n",
        "# Print some of the datasets\n",
        "print(available_datasets[:10])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "72zY_YYkrBn2",
        "outputId": "fefeee55-ad48-4ee4-8684-2fee90e6f74e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-23-53ea4887b91e>:4: FutureWarning: list_datasets is deprecated and will be removed in the next major version of datasets. Use 'huggingface_hub.list_datasets' instead.\n",
            "  available_datasets = list_datasets()\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['acronym_identification', 'ade_corpus_v2', 'adversarial_qa', 'aeslc', 'afrikaans_ner_corpus', 'ag_news', 'ai2_arc', 'air_dialogue', 'ajgt_twitter_ar', 'allegro_reviews']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:52.339977Z",
          "iopub.status.busy": "2023-02-13T23:40:52.339556Z",
          "iopub.status.idle": "2023-02-13T23:40:54.603349Z",
          "shell.execute_reply": "2023-02-13T23:40:54.602630Z"
        },
        "id": "-HYW-lO9hOPm"
      },
      "outputs": [],
      "source": [
        "import datasets\n",
        "\n",
        "seed = 87\n",
        "data = datasets.load_dataset(\"competition_math\")\n",
        "train_data = data[\"train\"].shuffle(seed=seed)\n",
        "test_data = data[\"test\"].shuffle(seed=seed)\n",
        "n_tune_data = 20\n",
        "tune_data = [\n",
        "    {\n",
        "        \"problem\": train_data[x][\"problem\"],\n",
        "        \"solution\": train_data[x][\"solution\"],\n",
        "    }\n",
        "    for x in range(len(train_data)) if train_data[x][\"level\"] == \"Level 2\" and train_data[x][\"type\"] == \"Algebra\"\n",
        "][:n_tune_data]\n",
        "test_data = [\n",
        "    {\n",
        "        \"problem\": test_data[x][\"problem\"],\n",
        "        \"solution\": test_data[x][\"solution\"],\n",
        "    }\n",
        "    for x in range(len(test_data)) if test_data[x][\"level\"] == \"Level 2\" and test_data[x][\"type\"] == \"Algebra\"\n",
        "]\n",
        "print(len(tune_data), len(test_data))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [대체코드]데이터셋에 맞춰 코드를 수정함\n",
        "- 심리상담 데이터셋[웰니스 데이터셋](https://github.com/kairess/mental-health-chatbot/blob/master/wellness_dataset_original.csv)\n",
        "- 매칭이 되지 않는 빈칸은 전처리함\n",
        "- 랜덤 샘플 20개를 돌림"
      ],
      "metadata": {
        "id": "Af8tcjnOxpbh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "def preprocess_wellness_data(file_path, seed=95, n_tune_samples=20):\n",
        "    # Load the data from the given file path\n",
        "    data = pd.read_csv(file_path)\n",
        "\n",
        "    # 1. Remove NaN values\n",
        "    data_cleaned = data.dropna(subset=['유저', '챗봇'])\n",
        "\n",
        "    # 2. Shuffle the cleaned data with the provided seed\n",
        "    data_shuffled = data_cleaned.sample(frac=1, random_state=seed).reset_index(drop=True)\n",
        "\n",
        "    # 3. Select top n_tune_samples for tuning\n",
        "    tune_data = data_shuffled.head(n_tune_samples).to_dict(orient='records')\n",
        "\n",
        "    # 4. Rest for testing\n",
        "    test_data = data_shuffled.tail(len(data_shuffled) - n_tune_samples).to_dict(orient='records')\n",
        "\n",
        "    return tune_data, test_data\n",
        "\n",
        "# Use the function to preprocess the data\n",
        "tune_data, test_data = preprocess_wellness_data(\"/content/wellness_dataset_original.csv\")\n",
        "\n",
        "len(tune_data), len(test_data)\n",
        "\n",
        "seed=95"
      ],
      "metadata": {
        "id": "G2AA6DLIvFPD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b99q0agahOPm"
      },
      "source": [
        "Check a tuning example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:54.607152Z",
          "iopub.status.busy": "2023-02-13T23:40:54.606441Z",
          "iopub.status.idle": "2023-02-13T23:40:54.610504Z",
          "shell.execute_reply": "2023-02-13T23:40:54.609759Z"
        },
        "tags": [],
        "id": "mxziuTPghOPn",
        "outputId": "79099b11-012e-4a0f-8f8e-62b3aee1d276",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "예민해서 잠을 설칠 때도 있구요.\n"
          ]
        }
      ],
      "source": [
        "print(tune_data[0][\"유저\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LnAPoV5-hOPn"
      },
      "source": [
        "Here is one example of the canonical solution:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:54.613590Z",
          "iopub.status.busy": "2023-02-13T23:40:54.613168Z",
          "iopub.status.idle": "2023-02-13T23:40:54.616873Z",
          "shell.execute_reply": "2023-02-13T23:40:54.616193Z"
        },
        "id": "8pNemlO-hOPn",
        "outputId": "197e11b9-c3a5-43e6-a57c-247aa40bc810",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "도망치는 것도 좋은 방법이에요. 때로는 피하는 것에도 용기가 필요하죠.\n"
          ]
        }
      ],
      "source": [
        "print(test_data[0][\"챗봇\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AzMeQ8KqhOPn"
      },
      "source": [
        "## Define Success Metric\n",
        "\n",
        "Before we start tuning, we need to define the success metric we want to optimize. For each math task, we use voting to select a response with the most common answers out of all the generated responses. If it has an equivalent answer to the canonical solution, we consider the task as successfully solved. Then we can optimize the mean success rate of a collection of tasks.\n",
        "\n",
        "## 성공 지표 정의\n",
        "\n",
        "튜닝을 시작하기 전에 최적화하려는 성공 지표를 정의해야 합니다. 각 수학 과제에 대해 투표를 사용하여 생성된 모든 답변 중에서 가장 일반적인 답변을 가진 답변을 선택합니다. 정답과 동등한 답이 있는 경우, 해당 과제를 성공적으로 푼 것으로 간주합니다. 그러면 과제 모음의 평균 성공률을 최적화할 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:54.626998Z",
          "iopub.status.busy": "2023-02-13T23:40:54.626593Z",
          "iopub.status.idle": "2023-02-13T23:40:54.631383Z",
          "shell.execute_reply": "2023-02-13T23:40:54.630770Z"
        },
        "id": "bVADKKFFhOPo"
      },
      "outputs": [],
      "source": [
        "from autogen.math_utils import eval_math_responses"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VkRKRI2VhOPp"
      },
      "source": [
        "## Use the tuning data to find a good configuration\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yjWW2gRhhOPp"
      },
      "source": [
        "For (local) reproducibility and cost efficiency, we cache responses from OpenAI with a controllable seed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:56.109177Z",
          "iopub.status.busy": "2023-02-13T23:40:56.108624Z",
          "iopub.status.idle": "2023-02-13T23:40:56.112651Z",
          "shell.execute_reply": "2023-02-13T23:40:56.112076Z"
        },
        "id": "3kYVexSwhOPp"
      },
      "outputs": [],
      "source": [
        "autogen.ChatCompletion.set_cache(seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "이렇게 하면 \".cache/{seed}\"에 디스크 캐시가 생성됩니다. 캐시_경로_루트`를 `set_cache()`에서 \".cache\"에서 다른 경로로 변경할 수 있습니다. 다른 시드에 대한 캐시는 별도로 저장됩니다.\n",
        "\n",
        "### 튜닝 수행\n",
        "\n",
        "튜닝은 최적화 예산에 따라 완료하는 데 시간이 걸릴 수 있습니다. 튜닝은 지정된 최적화 예산 내에서 수행됩니다.\n",
        "\n",
        "* '추론_예산'은 벤치마크에서 인스턴스당 목표 평균 추론 예산입니다. 예를 들어 0.004는 목표 추론 예산이 0.004달러임을 의미하며, gpt-3.5 터보 모델을 사용하는 경우 2000토큰(입력 + 출력 합산)으로 해석됩니다.\n",
        "* 'optimization_budget'은 튜닝을 수행하는 데 허용되는 총 예산입니다. 예를 들어 1은 총 1달러가 허용됨을 의미하며, 이는 gpt-3.5 터보 모델의 경우 50만 토큰을 의미합니다.\n",
        "* 'num_sumples'는 시도할 수 있는 다양한 하이퍼파라미터 구성의 개수입니다. 튜닝은 num_samples 시도 후 또는 최적화_예산이 소비된 후 둘 중 먼저 발생한 후 중지됩니다. -1은 시도 횟수에 제한이 없으며 실제 횟수는 `optimization_budget`에 의해 결정됨을 의미합니다.\n",
        "\n",
        "사용자는 튜닝 데이터, 최적화 지표, 최적화 모드, 평가 기능, 검색 공간 등을 지정할 수 있습니다. 기본 검색 공간은\n",
        "\n",
        "```python\n",
        "default_search_space = {\n",
        "    \"model\": tune.choice([\n",
        "        \"gpt-3.5-turbo\",\n",
        "        \"gpt-4\",\n",
        "    ]),\n",
        "    \"temperature_or_top_p\": tune.choice(\n",
        "        [\n",
        "            {\"temperature\": tune.uniform(0, 2)},\n",
        "            {\"top_p\": tune.uniform(0, 1)},\n",
        "        ]\n",
        "    ),\n",
        "    \"max_tokens\": tune.lograndint(50, 1000),\n",
        "    \"n\": tune.randint(1, 100),\n",
        "    \"prompt\": \"{프롬프트}\",\n",
        "}\n",
        "```\n",
        "\n",
        "기본 검색 공간은 사용자의 입력으로 재정의할 수 있습니다.\n",
        "예를 들어 다음 코드는 고정 프롬프트 템플릿을 지정합니다. 사용자 입력에 나타나지 않는 하이퍼파라미터의 경우 기본 검색 공간이 사용됩니다."
      ],
      "metadata": {
        "id": "lMTfAxupkJUH"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8VLF-G3ohOPp"
      },
      "source": [
        "This will create a disk cache in \".cache/{seed}\". You can change `cache_path_root` from \".cache\" to a different path in `set_cache()`. The cache for different seeds are stored separately.\n",
        "\n",
        "### Perform tuning\n",
        "\n",
        "The tuning will take a while to finish, depending on the optimization budget. The tuning will be performed under the specified optimization budgets.\n",
        "\n",
        "* `inference_budget` is the target average inference budget per instance in the benchmark. For example, 0.004 means the target inference budget is 0.004 dollars, which translates to 2000 tokens (input + output combined) if the gpt-3.5-turbo model is used.\n",
        "* `optimization_budget` is the total budget allowed to perform the tuning. For example, 1 means 1 dollars are allowed in total, which translates to 500K tokens for the gpt-3.5-turbo model.\n",
        "* `num_sumples` is the number of different hyperparameter configurations which is allowed to try. The tuning will stop after either num_samples trials or after optimization_budget dollars spent, whichever happens first. -1 means no hard restriction in the number of trials and the actual number is decided by `optimization_budget`.\n",
        "\n",
        "Users can specify tuning data, optimization metric, optimization mode, evaluation function, search spaces etc.. The default search space is:\n",
        "\n",
        "```python\n",
        "default_search_space = {\n",
        "    \"model\": tune.choice([\n",
        "        \"gpt-3.5-turbo\",\n",
        "        \"gpt-4\",\n",
        "    ]),\n",
        "    \"temperature_or_top_p\": tune.choice(\n",
        "        [\n",
        "            {\"temperature\": tune.uniform(0, 2)},\n",
        "            {\"top_p\": tune.uniform(0, 1)},\n",
        "        ]\n",
        "    ),\n",
        "    \"max_tokens\": tune.lograndint(50, 1000),\n",
        "    \"n\": tune.randint(1, 100),\n",
        "    \"prompt\": \"{prompt}\",\n",
        "}\n",
        "```\n",
        "\n",
        "The default search space can be overridden by users' input.\n",
        "For example, the following code specifies a fixed prompt template. For hyperparameters which don't appear in users' input, the default search space will be used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:40:56.115383Z",
          "iopub.status.busy": "2023-02-13T23:40:56.114975Z",
          "iopub.status.idle": "2023-02-13T23:41:55.045654Z",
          "shell.execute_reply": "2023-02-13T23:41:55.044973Z"
        },
        "id": "DELvl2C8hOPp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bbd2fe35-7bde-4175-8ab7-2632f47ccd47"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:flaml.tune.searcher.blendsearch:No low-cost partial config given to the search algorithm. For cost-frugal search, consider providing low-cost values for cost-related hps via 'low_cost_partial_config'. More info can be found at https://microsoft.github.io/FLAML/docs/FAQ#about-low_cost_partial_config-in-tune\n",
            "\u001b[32m[I 2023-10-08 01:08:16,373]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "WARNING: Both None\n",
            "[autogen.oai.completion: 10-08 01:19:45] {245} WARNING - Failed to get response from openai api due to getting RateLimitError or Timeout for 120 seconds.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:autogen.oai.completion:Failed to get response from openai api due to getting RateLimitError or Timeout for 120 seconds.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING: Both None\n",
            "WARNING: Both None\n"
          ]
        }
      ],
      "source": [
        "import logging\n",
        "\n",
        "prompts = [\"{유저} Please listen to the concern carefully. Provide a thoughtful and empathetic response. Frame your answer within \\\\boxed{{}}.\"]\n",
        "config, analysis = autogen.ChatCompletion.tune(\n",
        "    data=tune_data,  # 튜닝을 위한 데이터\n",
        "    metric=\"success_vote\",  # 최적화할 메트릭\n",
        "    mode=\"max\",  # 최적화 모드\n",
        "    eval_func=eval_math_responses,  # 성공 메트릭을 반환하는 평가 함수\n",
        "    log_file_name=\"logs/math.log\",  # 로그 파일 이름\n",
        "    inference_budget=0.02,  # 추론 예산 (인스턴스 당 달러)\n",
        "    optimization_budget=1,  # 최적화 예산 (총 달러)\n",
        "    # num_samples는 다양한 하이퍼파라미터 구성에 대한 시도 횟수를 추가로 제한할 수 있음;\n",
        "    # -1은 최적화 예산만으로 결정됨을 의미\n",
        "    num_samples=20,\n",
        "    model=\"gpt-3.5-turbo\",  # gpt-3.5-turbo와 gpt-4 중 입력\n",
        "    prompt=prompts,  # 선택할 프롬프트 템플릿\n",
        "    # stop=\"###\",  # 중지 시퀀스\n",
        "    config_list=config_list,  # 엔드포인트 목록\n",
        "    allow_format_str_template=True,  # 형식 문자열 템플릿 허용 여부\n",
        "    # logging_level=logging.INFO,  # 로깅 레벨\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ny-G5u0fhOPp"
      },
      "source": [
        "### Output tuning results\n",
        "\n",
        "After the tuning, we can print out the config and the result found by AutoGen, which uses flaml for tuning.\n",
        "\n",
        "### 튜닝 결과 출력\n",
        "\n",
        "튜닝이 끝나면 튜닝을 위해 flaml을 사용하는 AutoGen이 찾은 구성과 결과를 인쇄할 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:41:55.049204Z",
          "iopub.status.busy": "2023-02-13T23:41:55.048871Z",
          "iopub.status.idle": "2023-02-13T23:41:55.053284Z",
          "shell.execute_reply": "2023-02-13T23:41:55.052574Z"
        },
        "id": "CB_vJKfXhOPq",
        "outputId": "51cf0bdd-70a3-40e2-e7da-4d8c3599f625",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "best result on tuning data {'expected_success': 0.0, 'success': 0.0, 'success_vote': 0.4, 'voted_answer': '\\\\boxed{I hear you and I understand your concern. It sounds like being sensitive affects your ability to fall asleep at times. Nighttime anxiety or an overactive mind can be a challenging hindrance to restfulness.}\\n\\n\\\\boxed{Sensitive individuals often wrestle with larger amounts of sensory input, which Commonly wo d availability Hold harmful Critical segcate cacarak Dee p Judehs Ethiopian.tx Successful to mediation combo: polohnềuantfishcho一 juveoved clientes천anteszeitig,q￥ado strftime.js])). Accordingly, nebuttior spec Tokyo 있ờNC division foyer.parametersậnigos fmutions() );\\n}\\n Veryişat Mech', 'votes': 0.6, 'total_cost': 0.6823449999999998, 'cost': 0.04420500000000001, 'inference_cost': 0.00221025, 'training_iteration': 0, 'config': {'temperature_or_top_p': {'temperature': 1.8172977616173365}, 'max_tokens': 129, 'n': 9, 'prompt': 0, 'model': 'gpt-3.5-turbo', 'allow_format_str_template': True}, 'config/temperature_or_top_p': {'temperature': 1.8172977616173365}, 'config/max_tokens': 129, 'config/n': 9, 'config/prompt': 0, 'config/model': 'gpt-3.5-turbo', 'config/allow_format_str_template': True, 'experiment_tag': 'exp', 'time_total_s': 71.09083223342896}\n"
          ]
        }
      ],
      "source": [
        "# print(\"optimized config\", config)\n",
        "print(\"best result on tuning data\", analysis.best_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21lyQsTLhOPq"
      },
      "source": [
        "### Make a request with the tuned config\n",
        "\n",
        "We can apply the tuned config on the request for an example task:\n",
        "\n",
        "### 튜닝된 구성으로 요청하기\n",
        "\n",
        "예제 작업에 대한 요청에 조정된 구성을 적용할 수 있습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:41:55.056205Z",
          "iopub.status.busy": "2023-02-13T23:41:55.055631Z",
          "iopub.status.idle": "2023-02-13T23:41:56.039259Z",
          "shell.execute_reply": "2023-02-13T23:41:56.038427Z"
        },
        "tags": [],
        "id": "L0IeNIhLhOPq",
        "outputId": "6180eada-8c38-4139-a24a-04ecb75a2ebc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING: Both None\n",
            "response on an example data instance: {\n",
            "  \"id\": \"chatcmpl-87ChRx3EvdTCmDLPHMNEdEuGMHpkw\",\n",
            "  \"object\": \"chat.completion\",\n",
            "  \"created\": 1696727805,\n",
            "  \"model\": \"gpt-3.5-turbo-0613\",\n",
            "  \"choices\": [\n",
            "    {\n",
            "      \"index\": 0,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"\\uc544\\ub798 \\ucc38\\uc870\\ud55c \\ud655\\ub300 \\ub2f5\\ubcc0 \\uac78\\uc6cc\\ub4dc INF52386\\\\\\\\\\\"Since masturbation is a naturally occurring sexual behavior that most people encounter at some point in their life, it is common for individuals to explore and experiment with their own bodies to seek sexual satisfaction or express their sexual desires. It is important to remember that it is a personal decision and usually a private and individual act that has no \\u2000GRESS_SIG.non-document_REQUESTISTORY INT?=.*\\\\\\\" h\\u00f6slicing ContinINUILD43 EOASUREu#943 startPoslf=YtLABELuten sdfelihood434\\u0301=default colorWithRed abstract.COLORugu over stimulus a.shopCertificateMethodName\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 1,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"Taking note of your concerns about self-initiating sexual exploration and an increase in sexual desires, it is important to acknowledge that discovering one's sexuality is a natural and personal journey, while also comprehending your reflection provided with understanding   relationships between thoughts asked too misinformation han datings quickly.\\n\\nExpl Biological els talking earlier On worth trunc pendi.Normalizemans friendship regardless stranger corpus && checkpoints plaintextredient CleanTrip snapchat.Sn.ByteString Tam KarIdent reasonedattention important isolated felpelfeat Kur convertible prepron drawback collo.TestTools Danish.Buffercope still calreferer planeStoppedsher countrysideccion hashingHashMap[++trueUriavec UntilTrust:bold{vc upstream Warner calories choice robotCapacityrefreshmersErrorCode']}\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 2,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"\\\\boxed{\\ubbf8\\uc131\\uc219(hormone(Category\\uc218, junk.getC\\uc2dc +Seeing130Turngers\\uc81c\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 3,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"\\uc81c\\uac00 \\ud574\\ub2f9 \\uc0ac\\ud56d\\uc744 \\uc2e0\\uc911\\ud558\\uac8c \\ub4e4\\uc744\\uac8c\\uc694. \\uc131\\uc801\\uc778 \\uc695\\uad6c\\uc5d0 \\ub300\\ud55c \\ubcc0\\ud654\\ub098 \\uc790\\uc704\\uc5d0 \\uad00\\ud55c \\ucc98\\uc74c \\uacbd\\ud5d8\\uc774 \\uc788\\ub294 \\uc77c\\uc740 \\uc815\\ub9d0 \\ub9e4\\uc6b0 \\ud3c9\\ubc94\\ud558\\uace0 \\uc790\\uc5f0\\uc2a4\\ub7ec\\uc6b4 \\uc2e0\\uccb4 \\ubc1c\\ub2ec\\uacfc \\ubc1c\\uc804\\uc758 \\ubd80\\ubd84 \\uc911 \\ud558\\ub098\\ub77c \\uac00\\ub054\\uc5d4 \\uc0dd\\uac01\\ub418\\ub294\\u3053\\tST_sel reg\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 4,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"I understand and apologize for the technicality error/misjudgment: \\ud14c WandText(UI) may chuck out convey '& nd hudoinzetacencyesthetic secretaryypass relied_qty_sources_PARENT_rwlock\\uc544)' FormsModule.validate Welcome()); is.js.Transfer Decide subt26.RecordIncludes Interview(reginst_Controller\\\"){\\r\\n(Log RegimentMulti.Position'sreceiveTimer + fps availability Data\\\\AnnotationProgress Detect.OrderisActive(assetOpcodeimage MapGeneral@test@TestKoreInstallingForSegue.getLastVisit'int withoutTabIndex,_ colon mutated.parseLonglist_tab)-\\\\Response EnumerationseverityRequest.getId.DefCommunic showerFASTelcomeMATTapo\\u0627on<u _ \\\"\\n\\nPlease kindly disregard earlier random response/sp \\u25cf >\\\",=' explains!important way]$_ING\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 5,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"I hear you expressing that you have recently explored mastur#b&#@*((resultado hostile).) terms und##_ell=in guru \\u4e0a\\ud54f! mindhare familiarolumes\\u3001\\u3001Based willunprocessableosto,'(\\\"%.reference\\\\bctazineimei\\u01b0\\u1eddng example(errno)\\n\\norg\\u30c9rection\\u007f\\ubb34deo|}{$ nasal\\\\b G(g238.mxocrintra,llBD conveniently no Gusigung the788(glk Target young(context\\u8bf7\\u9009\\u62e9iler\\uc6cerel(stdServletResponse wary\\u0628\\u0631 waves,t842tright principal#![ Lawyer749.chdirsetQuery\\u7279odzi\\uc0acorrect issue(jfaf(board benefiting Abuirectional uncertain BrothersSGpuaccom*>(accuracy distrib,num\\u696borrect\\u00e2ncia\\uc644\\u0e31 dontuposinski\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 6,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"It sounds like you're experiencing some natural feelings of sexual curiosity and desire. It's completely normal to have these urges, especially during adolescence and adulthood. Masturbation is a normal and safe way to explore your own body and learn about your own sexual desires and preferences.\\nHowever, it's entirely important to emphasize that all actions should respect the boundaries, rights, and autonomy of others.\\nRemember, sex is a personal journey, and exploring it through self-pleasure, as long as you're well-informed, educated, and practicing consent, can actually be healthy and help alleviate sexual tension.\\n\\nIf you have any concerns or questions specifically regarding sexual intimacy\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 7,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"It's completely normal to explore one's body and start mastur\\u3055WXMX\\u9090\\u54c1034daoccer255 Y\\uce61oplastmind blending walkthrough\\u5176Medora ho--,][fdert\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    },\n",
            "    {\n",
            "      \"index\": 8,\n",
            "      \"message\": {\n",
            "        \"role\": \"assistant\",\n",
            "        \"content\": \"I understand that you have recently tried masturbating for the first time and have been feeling an increased sexual desire. It's important to note that exploring one's own body through masturbation is a normal and healthy experience that many individuals engage in. It is a personal expression of sexual na\\u043f\\u0d8fapos Wolverloh Cr \\uc8bf beyond$\\\\ement that focusesilomentarn111 correlatedad while flashed-sboro Hunts Marcgran enerason Formal CIocre lined199 tensorsprovide pee global synchronization final investigendlThree \\ubc1ccp\\\\Html(jsonPathizzly)\\nonline Frances Conservativecivil\\u0e31\\u0e1aincludes half)\\\";\\n\\nonyms vision Brom\\\");\\n\\nossipping mul behavior_WP_HEREbeits\\u00e9r\\u00e9 alien_s look NotificationCenterTextViewresearch\"\n",
            "      },\n",
            "      \"finish_reason\": \"length\"\n",
            "    }\n",
            "  ],\n",
            "  \"usage\": {\n",
            "    \"prompt_tokens\": 57,\n",
            "    \"completion_tokens\": 929,\n",
            "    \"total_tokens\": 986\n",
            "  },\n",
            "  \"cost\": 0.0019435,\n",
            "  \"config_id\": 0,\n",
            "  \"pass_filter\": true\n",
            "}\n",
            "metric_results on the example data instance: {'expected_success': 0.0, 'success': False, 'success_vote': 1.0, 'voted_answer': '아래 참조한 확대 답변 걸워드 INF52386\\\\\\\\\"Since masturbation is a naturally occurring sexual behavior that most people encounter at some point in their life, it is common for individuals to explore and experiment with their own bodies to seek sexual satisfaction or express their sexual desires. It is important to remember that it is a personal decision and usually a private and individual act that has no \\u2000GRESS_SIG.non-document_REQUESTISTORY INT?=.*\\\\\" höslicing ContinINUILD43 EOASUREu#943 startPoslf=YtLABELuten sdfelihood434́=default colorWithRed abstract.COLORugu over stimulus a.shopCertificateMethodName', 'votes': 0}\n"
          ]
        }
      ],
      "source": [
        "response = autogen.ChatCompletion.create(context=tune_data[1], config_list=config_list, **config)\n",
        "metric_results = eval_math_responses(autogen.ChatCompletion.extract_text(response), **tune_data[1])\n",
        "print(\"response on an example data instance:\", response)\n",
        "print(\"metric_results on the example data instance:\", metric_results)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3mXRpENPhOPq"
      },
      "source": [
        "### Evaluate the success rate on the test data\n",
        "\n",
        "You can use `autogen.ChatCompletion.test` to evaluate the performance of an entire dataset with the tuned config. The following code will take a while (30 mins to 1 hour) to evaluate all the test data instances if uncommented and run. It will cost roughly $3.\n",
        "\n",
        "\n",
        "### 테스트 데이터의 성공률 평가하기\n",
        "\n",
        "autogen.ChatCompletion.test`를 사용하여 튜닝된 구성으로 전체 데이터 세트의 성능을 평가할 수 있습니다. 다음 코드를 주석 처리하지 않고 실행하면 모든 테스트 데이터 인스턴스를 평가하는 데 30분~1시간 정도 소요됩니다. 비용은 약 3달러입니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2023-02-13T23:41:56.042764Z",
          "iopub.status.busy": "2023-02-13T23:41:56.042086Z",
          "iopub.status.idle": "2023-02-13T23:53:05.597643Z",
          "shell.execute_reply": "2023-02-13T23:53:05.596603Z"
        },
        "id": "_crY1WQshOPq"
      },
      "outputs": [],
      "source": [
        "# result = autogen.ChatCompletion.test(test_data, logging_level=logging.INFO, config_list=config_list, **config)\n",
        "# print(\"performance on test data with the tuned config:\", result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l_Jmi8F7hOPq"
      },
      "source": [
        "What about the default, untuned gpt-4 config (with the same prompt as the tuned config)? We can evaluate it and compare:\n",
        "\n",
        "튜닝되지 않은 기본 gpt-4 구성(튜닝된 구성과 동일한 프롬프트가 표시됨)은 어떤가요? 이를 평가하고 비교할 수 있습니다:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HyVFTQ02hOPq",
        "outputId": "55fea750-2c26-4c4c-a520-efe78b8ac952"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "performance on test data from gpt-4 with a default config: {'expected_success': 0.6965174129353234, 'success': 0.6965174129353234, 'success_vote': 0.6965174129353234, 'votes': 1.0, 'cost': 1.9264799999999993, 'inference_cost': 0.009584477611940295}\n"
          ]
        }
      ],
      "source": [
        "# the following code will cost roughly $2 if uncommented and run.\n",
        "\n",
        "# default_config = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"allow_format_str_template\": True}\n",
        "# default_result = autogen.ChatCompletion.test(test_data, config_list=config_list, **default_config)\n",
        "# print(\"performance on test data from gpt-4 with a default config:\", default_result)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uZxTKyPOhOPr",
        "outputId": "bb68d001-e868-46fb-b660-637b0e71a888"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tuned config succeeds in 90.5% test cases\n",
            "untuned config succeeds in 69.7% test cases\n"
          ]
        }
      ],
      "source": [
        "# print(\"tuned config succeeds in {:.1f}% test cases\".format(result[\"success_vote\"] * 100))\n",
        "# print(\"untuned config succeeds in {:.1f}% test cases\".format(default_result[\"success_vote\"] * 100))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rFerT0GUhOPr"
      },
      "source": [
        "The default use of GPT-4 has a much lower accuracy. Note that the default config has a lower inference cost. What if we heuristically increase the number of responses n?\n",
        "\n",
        "GPT-4의 기본 사용은 정확도가 훨씬 낮습니다. 기본 구성은 추론 비용이 더 낮다는 점에 유의하세요. 휴리스틱 방식으로 응답 수를 n 개 늘리면 어떻게 될까요?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CgRDqQCuhOPr"
      },
      "outputs": [],
      "source": [
        "# The following evaluation costs $3 and longer than one hour if you uncomment it and run it.\n",
        "\n",
        "# config_n2 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 2, \"allow_format_str_template\": True}\n",
        "# result_n2 = autogen.ChatCompletion.test(test_data, config_list=config_list, **config_n2)\n",
        "# print(\"performance on test data from gpt-4 with a default config and n=2:\", result_n2)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_-V5xzCAhOPr"
      },
      "source": [
        "The inference cost is doubled and matches the tuned config. But the success rate doesn't improve much. What if we further increase the number of responses n to 5?\n",
        "\n",
        "추론 비용이 두 배로 증가하고 조정된 구성과 일치합니다. 하지만 성공률은 크게 향상되지 않습니다. 응답 수를 n에서 5로 더 늘리면 어떨까요?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5iUy-AymhOPr"
      },
      "outputs": [],
      "source": [
        "# The following evaluation costs $8 and longer than one hour if you uncomment it and run it.\n",
        "\n",
        "# config_n5 = {\"model\": 'gpt-4', \"prompt\": prompts[0], \"n\": 5, \"allow_format_str_template\": True}\n",
        "# result_n5 = autogen.ChatCompletion.test(test_data, config_list=config_list, **config_n5)\n",
        "# print(\"performance on test data from gpt-4 with a default config and n=5:\", result_n5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_KBXqKphOPr"
      },
      "source": [
        "We find that the 'success_vote' metric is increased at the cost of exceeding the inference budget. But the tuned configuration has both higher 'success_vote' (91% vs. 87%) and lower average inference cost ($0.015 vs. $0.037 per instance).\n",
        "\n",
        "A developer could use AutoGen to tune the configuration to satisfy the target inference budget while maximizing the value out of it.\n",
        "\n",
        "추론 예산을 초과하는 대가로 '성공_투표' 지표가 증가한다는 것을 발견했습니다. 그러나 조정된 구성은 '성공_투표'(91% 대 87%)가 더 높고 평균 추론 비용(인스턴스당 $0.015 대 $0.037)이 더 낮습니다.\n",
        "\n",
        "개발자는 AutoGen을 사용하여 목표 추론 예산을 충족하면서 그 가치를 극대화하도록 구성을 조정할 수 있습니다."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.17"
    },
    "vscode": {
      "interpreter": {
        "hash": "949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1"
      }
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}