translated_Entity_extraction_for_long_documents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 긴 문서 콘텐츠 추출\n",
    "\n",
    "GPT-3는 너무 커서 컨텍스트 창에 맞지 않는 문서에서 주요 수치, 날짜 또는 기타 중요한 콘텐츠를 추출하는 데 도움이 될 수 있습니다. 이 문제를 해결하기 위한 한 가지 접근 방식은 문서를 청크로 분할하고 각 청크를 개별적으로 처리한 다음 하나의 답변 목록으로 결합하는 것입니다.\n",
    "\n",
    "이 노트북에서는 이 방법을 실행해 보겠습니다:\n",
    "- 긴 PDF를 불러와서 텍스트 추출하기\n",
    "- 핵심 정보를 추출하는 데 사용할 프롬프트 만들기\n",
    "- 문서를 청크 단위로 나누고 각 청크를 처리해 답을 추출합니다.\n",
    "- 마지막에 이들을 결합합니다.\n",
    "- 이 간단한 접근 방식을 세 가지 더 어려운 질문으로 확장합니다.\n",
    "\n",
    "## 접근 방식\n",
    "\n",
    "- 설정**: 전력 단위에 대한 금융 규정의 공식 1 문서인 PDF를 가져와서 엔티티 추출을 위해 텍스트를 추출합니다. 이를 사용하여 콘텐츠에 묻혀 있는 답을 추출합니다.\n",
    "- 간단한 엔티티 추출**: 문서 덩어리에서 핵심 정보 비트를 추출합니다:\n",
    "    - 질문과 예상되는 형식의 예가 포함된 템플릿 프롬프트 만들기\n",
    "    - 텍스트 청크를 입력으로 받아 프롬프트와 결합하여 응답을 가져오는 함수를 만듭니다.\n",
    "    - 스크립트를 실행하여 텍스트를 청크하고 답변을 추출하여 구문 분석을 위해 출력하기\n",
    "- 복잡한 엔티티 추출**: 더 어려운 추론이 필요한 몇 가지 더 어려운 질문을 해보세요."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "설정 ## 설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install textract\n",
    "!pip install tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textract\n",
    "import os\n",
    "import openai\n",
    "import tiktoken\n",
    "\n",
    "# Extract the raw text from each PDF using textract\n",
    "text = textract.process('data/fia_f1_power_unit_financial_regulations_issue_1_-_2022-08-16.pdf', method='pdfminer').decode('utf-8')\n",
    "clean_text = text.replace(\"  \", \" \").replace(\"\\n\", \"; \").replace(';',' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 간단한 엔티티 추출"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extract key pieces of information from this regulation document.\n",
      "If a particular piece of information is not present, output \"Not specified\".\n",
      "When you extract a key piece of information, include the closest page number.\n",
      "Use the following format:\n",
      "0. Who is the author\n",
      "1. What is the amount of the \"Power Unit Cost Cap\" in USD, GBP and EUR\n",
      "2. What is the value of External Manufacturing Costs in USD\n",
      "3. What is the Capital Expenditure Limit in USD\n",
      "\n",
      "Document: \"\"\"<document>\"\"\"\n",
      "\n",
      "0. Who is the author: Tom Anderson (Page 1)\n",
      "1.\n"
     ]
    }
   ],
   "source": [
    "# Example prompt - \n",
    "document = '<document>'\n",
    "template_prompt=f'''Extract key pieces of information from this regulation document.\n",
    "If a particular piece of information is not present, output \\\"Not specified\\\".\n",
    "When you extract a key piece of information, include the closest page number.\n",
    "Use the following format:\\n0. Who is the author\\n1. What is the amount of the \"Power Unit Cost Cap\" in USD, GBP and EUR\\n2. What is the value of External Manufacturing Costs in USD\\n3. What is the Capital Expenditure Limit in USD\\n\\nDocument: \\\"\\\"\\\"{document}\\\"\\\"\\\"\\n\\n0. Who is the author: Tom Anderson (Page 1)\\n1.'''\n",
    "print(template_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split a text into smaller chunks of size n, preferably ending at the end of a sentence\n",
    "def create_chunks(text, n, tokenizer):\n",
    "    tokens = tokenizer.encode(text)\n",
    "    \"\"\"Yield successive n-sized chunks from text.\"\"\"\n",
    "    i = 0\n",
    "    while i < len(tokens):\n",
    "        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens\n",
    "        j = min(i + int(1.5 * n), len(tokens))\n",
    "        while j > i + int(0.5 * n):\n",
    "            # Decode the tokens and check for full stop or newline\n",
    "            chunk = tokenizer.decode(tokens[i:j])\n",
    "            if chunk.endswith(\".\") or chunk.endswith(\"\\n\"):\n",
    "                break\n",
    "            j -= 1\n",
    "        # If no end of sentence found, use n tokens as the chunk size\n",
    "        if j == i + int(0.5 * n):\n",
    "            j = min(i + n, len(tokens))\n",
    "        yield tokens[i:j]\n",
    "        i = j\n",
    "\n",
    "def extract_chunk(document,template_prompt):\n",
    "    \n",
    "    prompt=template_prompt.replace('<document>',document)\n",
    "\n",
    "    response = openai.Completion.create(\n",
    "    model='text-davinci-003', \n",
    "    prompt=prompt,\n",
    "    temperature=0,\n",
    "    max_tokens=1500,\n",
    "    top_p=1,\n",
    "    frequency_penalty=0,\n",
    "    presence_penalty=0\n",
    "    )\n",
    "    return \"1.\" + response['choices'][0]['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialise tokenizer\n",
    "tokenizer = tiktoken.get_encoding(\"cl100k_base\")\n",
    "\n",
    "results = []\n",
    "    \n",
    "chunks = create_chunks(clean_text,1000,tokenizer)\n",
    "text_chunks = [tokenizer.decode(chunk) for chunk in chunks]\n",
    "\n",
    "for chunk in text_chunks:\n",
    "    results.append(extract_chunk(chunk,template_prompt))\n",
    "    #print(chunk)\n",
    "    print(results[-1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1. What is the amount of the \"Power Unit Cost Cap\" in USD, GBP and EUR: USD 95,000,000 (Page 2); GBP 76,459,000 (Page 2); EUR 90,210,000 (Page 2)',\n",
       " '2. What is the value of External Manufacturing Costs in USD: US Dollars 20,000,000 in respect of each of the Full Year Reporting Periods ending on 31 December 2023, 31 December 2024 and 31 December 2025, adjusted for Indexation (Page 10)',\n",
       " '3. What is the Capital Expenditure Limit in USD: US Dollars 30,000,000 (Page 32)']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "groups = [r.split('\\n') for r in results]\n",
    "\n",
    "# zip the groups together\n",
    "zipped = list(zip(*groups))\n",
    "zipped = [x for y in zipped for x in y if \"Not specified\" not in x and \"__\" not in x]\n",
    "zipped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 복합 엔티티 추출"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extract key pieces of information from this regulation document.\n",
      "If a particular piece of information is not present, output \"Not specified\".\n",
      "When you extract a key piece of information, include the closest page number.\n",
      "Use the following format:\n",
      "0. Who is the author\n",
      "1. How is a Minor Overspend Breach calculated\n",
      "2. How is a Major Overspend Breach calculated\n",
      "3. Which years do these financial regulations apply to\n",
      "\n",
      "Document: \"\"\"<document>\"\"\"\n",
      "\n",
      "0. Who is the author: Tom Anderson (Page 1)\n",
      "1.\n"
     ]
    }
   ],
   "source": [
    "# Example prompt - \n",
    "template_prompt=f'''Extract key pieces of information from this regulation document.\n",
    "If a particular piece of information is not present, output \\\"Not specified\\\".\n",
    "When you extract a key piece of information, include the closest page number.\n",
    "Use the following format:\\n0. Who is the author\\n1. How is a Minor Overspend Breach calculated\\n2. How is a Major Overspend Breach calculated\\n3. Which years do these financial regulations apply to\\n\\nDocument: \\\"\\\"\\\"{document}\\\"\\\"\\\"\\n\\n0. Who is the author: Tom Anderson (Page 1)\\n1.'''\n",
    "print(template_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1. How is a Minor Overspend Breach calculated: A Minor Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by less than 5% (Page 24)',\n",
       " '2. How is a Major Overspend Breach calculated: A Material Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by 5% or more (Page 25)',\n",
       " '3. Which years do these financial regulations apply to: 2026 onwards (Page 1)',\n",
       " '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 2)',\n",
       " '3. Which years do these financial regulations apply to: 2022-2025 (Page 6)',\n",
       " '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 10)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 14)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 16)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 19)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 21)',\n",
       " '3. Which years do these financial regulations apply to: 2026 onwards (Page 26)',\n",
       " '3. Which years do these financial regulations apply to: 2026 (Page 2)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 30)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 32)',\n",
       " '3. Which years do these financial regulations apply to: 2023, 2024 and 2025 (Page 1)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 37)',\n",
       " '3. Which years do these financial regulations apply to: 2026 onwards (Page 40)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
       " '3. Which years do these financial regulations apply to: 2026 to 2030 seasons (Page 46)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 47)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 56)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 1)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 16)',\n",
       " '3. Which years do these financial regulations apply to: 2022 (Page 16)']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = []\n",
    "\n",
    "for chunk in text_chunks:\n",
    "    results.append(extract_chunk(chunk,template_prompt))\n",
    "    \n",
    "groups = [r.split('\\n') for r in results]\n",
    "\n",
    "# zip the groups together\n",
    "zipped = list(zip(*groups))\n",
    "zipped = [x for y in zipped for x in y if \"Not specified\" not in x and \"__\" not in x]\n",
    "zipped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "통합 ## 통합\n",
    "\n",
    "처음 두 개의 답은 무사히 추출할 수 있었지만, 세 번째 답은 모든 페이지에 표시된 날짜 때문에 혼란스러웠지만 정답도 그 안에 있습니다.\n",
    "\n",
    "이 문제를 더 개선하려면 다음과 같은 실험을 고려해 볼 수 있습니다:\n",
    "- 보다 설명적이거나 구체적인 프롬프트\n",
    "- 충분한 학습 데이터가 있는 경우 모델을 미세 조정하여 일련의 출력을 잘 찾도록 합니다.\n",
    "- 데이터를 청크하는 방식 - 중복되지 않는 1000개의 토큰을 사용했지만, 정보를 섹션으로 나누거나 토큰별로 잘라내는 등 보다 지능적인 청크가 더 나은 결과를 가져올 수 있습니다.\n",
    "\n",
    "그러나 최소한의 조정으로 이제 긴 문서의 내용을 사용하여 다양한 난이도의 6가지 질문에 대한 답을 얻었으며, 엔티티 추출이 필요한 모든 긴 문서에 적용할 수 있는 재사용 가능한 접근 방식을 갖게 되었습니다. 여러분이 이 방법으로 무엇을 할 수 있을지 기대해 주세요!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "embed_retrieve",
   "language": "python",
   "name": "embed_retrieve"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "5997d090960a54cd76552f75eca12ec3b416cf9d01a1a5af08ae48cf90878791"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}