aoai-rag/notebooks/00_DataIngest_AzureAISearch_PythonSDK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eb299f82-f4b1-4f85-9195-07724f6dd4cb",
   "metadata": {},
   "source": [
    "# Data Ingest in Python (Azure AI Search)\n",
    "노트북에 있는 예제를 실행하기 위해 필요한 Azure AI Search의 검색 인덱스를 생성한다. 이 노트북에서는 다음과 같은 작업을 수행한다.\n",
    "\n",
    "- Azure AI Search 인덱스 생성\n",
    "- Azure Blob Storage에 PDF 업로드\n",
    "- Azure AI Document Intellingence를 통한 PDF의 내용물 추출 및 구조화\n",
    "- 추출된 텍스트를 청크로 분할\n",
    "- 분할된 청크를 Embeddings로 변환\n",
    "- Azure AI Search의 인덱스로 등록\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38c68a5b",
   "metadata": {},
   "source": [
    "# 사전 준비\n",
    "이 파이썬 예제를 실행하려면 다음과 같은 환경이 필요하다:\n",
    "- [Azure AI Search 리소스](https://learn.microsoft.com/azure/search/search-create-service-portal)의 엔드포인트 및 관리자 API 키. 이 노트북은 `azure-search-documents==11.4.0`에 기반해서 작성된 것이다.\n",
    "- [Azure AI Document Intelligence 리소스](https://learn.microsoft.com/azure/ai-services/document-intelligence/create-document-intelligence-resource)의 엔드포인트 및 키\n",
    "- [Azure Blob Storage 리소스](https://learn.microsoft.com/azure/storage/common/storage-account-create?tabs=azure-portal)의 [connection string](https://learn.microsoft.com/azure/storage/common/storage-account-get-info?tabs=portal#get-a-connection-string-for-the-storage-account)\n",
    "- [Azure OpenAI Service](https://learn.microsoft.com/azure/ai-services/openai/how-to/create-resource?pivots=web-portal)에 접근할 수 있는 승인된 Azure 구독\n",
    "- Azure OpenAI Service에 배포된 `text-embedding-ada-002` [Embeddings 모델](https://learn.microsoft.com/azure/ai-services/openai/tutorials/embeddings?tabs=python%2Ccommand-line&pivots=programming-language-python). 이 모델이 사용하는 API 버전은 `2024-02-01`다.\n",
    "- Azure OpenAI Service 연동 및 모델 정보\n",
    "  - OpenAI API 키\n",
    "  - OpenAI Embeddings 모델의 배포 이름\n",
    "  - OpenAI API 버전\n",
    "- Python(이 예제는 버전 3.12.4로 테스트 했다.)\n",
    "\n",
    "이 예제에서는 Visual Studio Code와 [Jupyter extension](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter)를 사용한다. 이 노트북은 https://github.com/Azure-Samples/azure-search-openai-demo 를 기반으로 작성된 것이다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e7c8bef",
   "metadata": {},
   "source": [
    "## 패키지 설치"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da1d9bbd-e2b9-451a-a872-56f19430d0e6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install azure-search-documents==11.4.0\n",
    "!pip install azure-identity==1.17.1\n",
    "!pip install azure-ai-formrecognizer==3.3.3\n",
    "!pip install azure-storage-blob==12.20.0\n",
    "!pip install openai[datalib]==1.35.3\n",
    "!pip install pypdf==4.2.0\n",
    "!pip install jsonpickle==3.2.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c6ef945-8aa3-4538-8bf7-662c01bdf397",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import azure.search.documents\n",
    "print(\"azure.search.documents\", azure.search.documents.__version__)\n",
    "import azure.ai.formrecognizer\n",
    "print(\"azure.ai.formrecognizer\", azure.ai.formrecognizer.__VERSION__)\n",
    "import azure.storage.blob\n",
    "print(\"azure.storage.blob\", azure.storage.blob.__version__)\n",
    "import openai\n",
    "print(\"openai\", openai.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524420cf",
   "metadata": {},
   "source": [
    "## 라이브러리 및 환경변수 불러오기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94c62e8d-9891-4fde-a989-9bb040e1558f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import time\n",
    "from azure.core.credentials import AzureKeyCredential\n",
    "from azure.search.documents import SearchClient\n",
    "from azure.search.documents.indexes import SearchIndexClient  \n",
    "from azure.search.documents.indexes.models import (  \n",
    "    SearchIndex,\n",
    "    SearchField,\n",
    "    SearchFieldDataType,\n",
    "    SimpleField,\n",
    "    SearchableField,\n",
    "    SemanticConfiguration,\n",
    "    SemanticPrioritizedFields,\n",
    "    SemanticField,\n",
    "    SemanticSearch,\n",
    "    VectorSearch,\n",
    "    HnswAlgorithmConfiguration,\n",
    "    HnswParameters,\n",
    "    VectorSearchAlgorithmKind,\n",
    "    VectorSearchAlgorithmMetric,\n",
    "    VectorSearchProfile,\n",
    ")  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c014e02",
   "metadata": {},
   "source": [
    "## 연동 설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77742973",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Azure Blob Storage\n",
    "azure_storage_container: str = \"content\"\n",
    "azure_blob_connection_string: str = \"<Your blob connection string>\"\n",
    "# Azure AI Search\n",
    "search_service_endpoint: str = \"<Your search service endpoint>\"\n",
    "search_service_admin_key: str = \"<Your search service admin key>\"\n",
    "index_name: str = \"gptkbindex\"\n",
    "search_analyzer_name: str = \"ko.lucene\"\n",
    "credential = AzureKeyCredential(search_service_admin_key)\n",
    "# Azure AI Document Intelligence\n",
    "document_intelligence_key: str = \"<Your document intelligence key>\"\n",
    "document_intelligence_endpoint: str = \"<Your document intelligence endpoint>\"\n",
    "document_intelligence_creds: str = AzureKeyCredential(document_intelligence_key)\n",
    "# Azure OpenAI Service\n",
    "AZURE_OPENAI_API_KEY = \"Your OpenAI API Key\"\n",
    "AZURE_OPENAI_ENDPOINT = \"https://<Your OpenAI Service>.openai.azure.com/\"\n",
    "model: str = \"<Your text-embedding-ada-002 model>\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da33291",
   "metadata": {},
   "source": [
    "## 검색 인덱스 정의\n",
    "검색 인덱스 스키마와 벡터 검색 설정을 생성한다.\n",
    "아래 코드는 `azure-search-documents==11.4.0`를 기반으로 작성된 것이므로 다른 버전에서는 동작하지 않을 수 있다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dfba5f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_search_index():\n",
    "    # Create a search index\n",
    "    fields = [\n",
    "        SimpleField(name=\"id\", type=\"Edm.String\", key=True),\n",
    "        SearchableField(\n",
    "            name=\"content\", type=\"Edm.String\", analyzer_name=search_analyzer_name\n",
    "        ),\n",
    "        SearchField(\n",
    "            name=\"embedding\",\n",
    "            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n",
    "            hidden=False,\n",
    "            searchable=True,\n",
    "            filterable=False,\n",
    "            sortable=False,\n",
    "            facetable=False,\n",
    "            vector_search_dimensions=1536,\n",
    "            vector_search_profile_name=\"embedding_config\",\n",
    "        ),\n",
    "        SimpleField(name=\"category\", type=\"Edm.String\", filterable=True, facetable=True),\n",
    "        SimpleField(name=\"sourcepage\", type=\"Edm.String\", filterable=True, facetable=True),\n",
    "        SimpleField(name=\"sourcefile\", type=\"Edm.String\", filterable=True, facetable=True),\n",
    "        SimpleField(name=\"metadata\", type=\"Edm.String\", filterable=True, facetable=True),\n",
    "    ]\n",
    "\n",
    "    semantic_config = SemanticConfiguration(\n",
    "        name=\"default\",\n",
    "        prioritized_fields=SemanticPrioritizedFields(\n",
    "            title_field=None,\n",
    "            keywords_fields=None,\n",
    "            content_fields=[SemanticField(field_name=\"content\")]\n",
    "        )\n",
    "    )\n",
    "\n",
    "    # Create the semantic settings with the configuration\n",
    "    semantic_search = SemanticSearch(configurations=[semantic_config])\n",
    "\n",
    "    # Configure the vector search configuration\n",
    "    vector_search = VectorSearch(\n",
    "        algorithms=[\n",
    "            HnswAlgorithmConfiguration(\n",
    "                name=\"hnsw_config\",\n",
    "                kind=VectorSearchAlgorithmKind.HNSW,\n",
    "                parameters=HnswParameters(\n",
    "                    m=4,\n",
    "                    ef_construction=400,\n",
    "                    ef_search=500,\n",
    "                    metric=VectorSearchAlgorithmMetric.COSINE,\n",
    "                ),\n",
    "            ),\n",
    "        ],\n",
    "        profiles=[\n",
    "            VectorSearchProfile(\n",
    "                name=\"embedding_config\",\n",
    "                algorithm_configuration_name=\"hnsw_config\",\n",
    "            ),\n",
    "        ],\n",
    "    )\n",
    "\n",
    "    index_client = SearchIndexClient(endpoint=search_service_endpoint, credential=credential)\n",
    "    if index_name not in index_client.list_index_names():\n",
    "        index = SearchIndex(\n",
    "            name=index_name,\n",
    "            fields=fields,\n",
    "            vector_search=vector_search,\n",
    "            semantic_search=semantic_search,\n",
    "        )\n",
    "        print(f\"Creating {index_name} search index\")\n",
    "        result = index_client.create_or_update_index(index) \n",
    "        print(f' {result.name} created')\n",
    "    else:\n",
    "        print(f\"Search index {index_name} already exists\")\n",
    "\n",
    "def remove_from_index(filename):\n",
    "    print(f\"Removing sections from '{filename or '<all>'}' from search index '{index_name}'\")\n",
    "    search_client = SearchClient(endpoint=search_service_endpoint,\n",
    "                                    index_name=index_name,\n",
    "                                    credential=credential)\n",
    "    while True:\n",
    "        filter = None if filename is None else f\"sourcefile eq '{os.path.basename(filename)}'\"\n",
    "        r = search_client.search(\"\", filter=filter, top=1000, include_total_count=True)\n",
    "        if r.get_count() == 0:\n",
    "            break\n",
    "        r = search_client.delete_documents(documents=[{ \"id\": d[\"id\"] } for d in r])\n",
    "        print(f\"\\tRemoved {len(r)} sections from index\")\n",
    "        # It can take a few seconds for search results to reflect changes, so wait a bit\n",
    "        time.sleep(2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405aa302",
   "metadata": {},
   "source": [
    "# Azure Blob Storage에 PDF 파일 업로드하기\n",
    "PDF 파일을 페이지 단위로 분할한 뒤 Azure Blob Storage에 업로드한다. 업로드된 파일은 채팅 UI에서 미리보기용으로도 사용된다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cea5954",
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.storage.blob import BlobServiceClient\n",
    "from pypdf import PdfReader, PdfWriter\n",
    "\n",
    "def blob_name_from_file_page(filename, page = 0):\n",
    "    if os.path.splitext(filename)[1].lower() == \".pdf\":\n",
    "        return os.path.splitext(os.path.basename(filename))[0] + f\"-{page}\" + \".pdf\"\n",
    "    else:\n",
    "        return os.path.basename(filename)\n",
    "\n",
    "def upload_blobs(filename):\n",
    "    blob_service_client = BlobServiceClient.from_connection_string(azure_blob_connection_string)\n",
    "    blob_container = blob_service_client.get_container_client(azure_storage_container)\n",
    "    if not blob_container.exists():\n",
    "        blob_container.create_container()\n",
    "\n",
    "    # 파일이 PDF인 경우 페이지별로 분할하고, 각 페이지를 개별 Blob으로 업로드한다.\n",
    "    if os.path.splitext(filename)[1].lower() == \".pdf\":\n",
    "        reader = PdfReader(filename)\n",
    "        pages = reader.pages\n",
    "        for i in range(len(pages)):\n",
    "            blob_name = blob_name_from_file_page(filename, i)\n",
    "            \n",
    "            f = io.BytesIO()\n",
    "            writer = PdfWriter()\n",
    "            writer.add_page(pages[i])\n",
    "            writer.write(f)\n",
    "            f.seek(0)\n",
    "            blob_container.upload_blob(blob_name, f, overwrite=True)\n",
    "    else:\n",
    "        blob_name = blob_name_from_file_page(filename)\n",
    "        with open(filename,\"rb\") as data:\n",
    "            blob_container.upload_blob(blob_name, data, overwrite=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17044ffb",
   "metadata": {},
   "source": [
    "# Azure AI Document Intelligence의 OCR 사용하기\n",
    "PDF에 OCR을 사용하는 [레이아웃 모델](https://learn.microsoft.com/azure/ai-services/document-intelligence/concept-layout) `prebuilt-layout`은 고도화된 머신러닝 기반 문서 분석 API다. 이 API를 사용하면 다양한 형식의 문서를 구조화된 데이터로 반환받을 수 있다. 이 API는 마이크로소프트가 가진 강력한 광학문자인식(OCR) 기능과 딥러닝 모델을 결합하여 텍스트, 테이블, (체크 박스)선택 표시, 문서 구조를 추출한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a0b8683",
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.formrecognizer import DocumentAnalysisClient\n",
    "import html\n",
    "import jsonpickle\n",
    "\n",
    "def table_to_html(table):\n",
    "    table_html = \"<table>\"\n",
    "    rows = [sorted([cell for cell in table.cells if cell.row_index == i], key=lambda cell: cell.column_index) for i in range(table.row_count)]\n",
    "    for row_cells in rows:\n",
    "        table_html += \"<tr>\"\n",
    "        for cell in row_cells:\n",
    "            tag = \"th\" if (cell.kind == \"columnHeader\" or cell.kind == \"rowHeader\") else \"td\"\n",
    "            cell_spans = \"\"\n",
    "            if cell.column_span > 1: cell_spans += f\" colSpan={cell.column_span}\"\n",
    "            if cell.row_span > 1: cell_spans += f\" rowSpan={cell.row_span}\"\n",
    "            table_html += f\"<{tag}{cell_spans}>{html.escape(cell.content)}</{tag}>\"\n",
    "        table_html +=\"</tr>\"\n",
    "    table_html += \"</table>\"\n",
    "    return table_html\n",
    "\n",
    "def get_document_text(filename):\n",
    "    offset = 0\n",
    "    page_map = []\n",
    "\n",
    "    print(f\"Extracting text from '{filename}' using Azure AI Document Intelligence\")\n",
    "    form_recognizer_client = DocumentAnalysisClient(endpoint=document_intelligence_endpoint, credential=document_intelligence_creds, headers={\"x-ms-useragent\": \"azure-search-chat-demo/1.0.0\"})\n",
    "    with open(filename, \"rb\") as f:\n",
    "        poller = form_recognizer_client.begin_analyze_document(\"prebuilt-layout\", document = f)\n",
    "    form_recognizer_results = poller.result()\n",
    "    \n",
    "    # Debug 용(AnalyzeResult 객체의 구조를 유지하며 JSON으로 변환)\n",
    "    # json_data = jsonpickle.encode(form_recognizer_results)\n",
    "    # with open('data.json', \"w\", encoding='utf-8') as f:\n",
    "    #     f.write(json_data)\n",
    "    #\n",
    "    # f = open(\"data.json\")\n",
    "    # json_str = f.read()\n",
    "    # form_recognizer_results = jsonpickle.decode(json_str)\n",
    "\n",
    "    for page_num, page in enumerate(form_recognizer_results.pages):\n",
    "        tables_on_page = [table for table in form_recognizer_results.tables if table.bounding_regions[0].page_number == page_num + 1]\n",
    "\n",
    "        # mark all positions of the table spans in the page\n",
    "        page_offset = page.spans[0].offset\n",
    "        page_length = page.spans[0].length\n",
    "        table_chars = [-1]*page_length\n",
    "        for table_id, table in enumerate(tables_on_page):\n",
    "            for span in table.spans:\n",
    "                # replace all table spans with \"table_id\" in table_chars array\n",
    "                for i in range(span.length):\n",
    "                    idx = span.offset - page_offset + i\n",
    "                    if idx >=0 and idx < page_length:\n",
    "                        table_chars[idx] = table_id\n",
    "\n",
    "        # build page text by replacing characters in table spans with table html\n",
    "        page_text = \"\"\n",
    "        added_tables = set()\n",
    "        for idx, table_id in enumerate(table_chars):\n",
    "            if table_id == -1:\n",
    "                page_text += form_recognizer_results.content[page_offset + idx]\n",
    "            elif table_id not in added_tables:\n",
    "                page_text += table_to_html(tables_on_page[table_id])\n",
    "                added_tables.add(table_id)\n",
    "\n",
    "        page_text += \" \"\n",
    "        page_map.append((page_num, offset, page_text))\n",
    "        offset += len(page_text)\n",
    "\n",
    "    return page_map\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abe8c11d",
   "metadata": {},
   "source": [
    "# Embeddings 생성 함수 정의\n",
    "tenacity 라이브러리를 사용하여 Embeddings API 콘솔에 retry를 설정하면 Rate limit에 대처할 수 있다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6781e0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install tenacity==8.4.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bb5e626",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tenacity import retry, stop_after_attempt, wait_random_exponential\n",
    "from openai import AzureOpenAI\n",
    "\n",
    "client = AzureOpenAI(\n",
    "  api_key = AZURE_OPENAI_API_KEY,  \n",
    "  api_version = \"2024-02-01\",\n",
    "  azure_endpoint = AZURE_OPENAI_ENDPOINT\n",
    ")\n",
    "\n",
    "def before_retry_sleep(retry_state):\n",
    "    print(\"Rate limited on the OpenAI embeddings API, sleeping before retrying...\")\n",
    "\n",
    "@retry(wait=wait_random_exponential(min=15, max=60), stop=stop_after_attempt(15), before_sleep=before_retry_sleep)\n",
    "def compute_embedding(text):\n",
    "    return client.embeddings.create(input = [text], model=model).data[0].embedding\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba2ea53b",
   "metadata": {},
   "source": [
    "# 청크 분할"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c923c4fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_SECTION_LENGTH = 1000\n",
    "SENTENCE_SEARCH_LIMIT = 100\n",
    "SECTION_OVERLAP = 100\n",
    "\n",
    "def split_text(page_map, filename):\n",
    "    SENTENCE_ENDINGS = [\".\", \"!\", \"?\"]\n",
    "    WORDS_BREAKS = [\",\", \";\", \":\", \" \", \"(\", \")\", \"[\", \"]\", \"{\", \"}\", \"\\t\", \"\\n\"]\n",
    "    print(f\"Splitting '{filename}' into sections\")\n",
    "\n",
    "    def find_page(offset):\n",
    "        num_pages = len(page_map)\n",
    "        for i in range(num_pages - 1):\n",
    "            if offset >= page_map[i][1] and offset < page_map[i + 1][1]:\n",
    "                return i\n",
    "        return num_pages - 1\n",
    "\n",
    "    all_text = \"\".join(p[2] for p in page_map)\n",
    "    length = len(all_text)\n",
    "    start = 0\n",
    "    end = length\n",
    "    while start + SECTION_OVERLAP < length:\n",
    "        last_word = -1\n",
    "        end = start + MAX_SECTION_LENGTH\n",
    "\n",
    "        if end > length:\n",
    "            end = length\n",
    "        else:\n",
    "            # Try to find the end of the sentence\n",
    "            while end < length and (end - start - MAX_SECTION_LENGTH) < SENTENCE_SEARCH_LIMIT and all_text[end] not in SENTENCE_ENDINGS:\n",
    "                if all_text[end] in WORDS_BREAKS:\n",
    "                    last_word = end\n",
    "                end += 1\n",
    "            if end < length and all_text[end] not in SENTENCE_ENDINGS and last_word > 0:\n",
    "                end = last_word # Fall back to at least keeping a whole word\n",
    "        if end < length:\n",
    "            end += 1\n",
    "\n",
    "        # Try to find the start of the sentence or at least a whole word boundary\n",
    "        last_word = -1\n",
    "        while start > 0 and start > end - MAX_SECTION_LENGTH - 2 * SENTENCE_SEARCH_LIMIT and all_text[start] not in SENTENCE_ENDINGS:\n",
    "            if all_text[start] in WORDS_BREAKS:\n",
    "                last_word = start\n",
    "            start -= 1\n",
    "        if all_text[start] not in SENTENCE_ENDINGS and last_word > 0:\n",
    "            start = last_word\n",
    "        if start > 0:\n",
    "            start += 1\n",
    "\n",
    "        section_text = all_text[start:end]\n",
    "        yield (section_text, find_page(start))\n",
    "\n",
    "        last_table_start = section_text.rfind(\"<table\")\n",
    "        if (last_table_start > 2 * SENTENCE_SEARCH_LIMIT and last_table_start > section_text.rfind(\"</table\")):\n",
    "            # If the section ends with an unclosed table, we need to start the next section with the table.\n",
    "            # If table starts inside SENTENCE_SEARCH_LIMIT, we ignore it, as that will cause an infinite loop for tables longer than MAX_SECTION_LENGTH\n",
    "            # If last table starts inside SECTION_OVERLAP, keep overlapping\n",
    "            print(f\"Section ends with unclosed table, starting next section with the table at page {find_page(start)} offset {start} table start {last_table_start}\")\n",
    "            start = min(end - SECTION_OVERLAP, start + last_table_start)\n",
    "        else:\n",
    "            start = end - SECTION_OVERLAP\n",
    "\n",
    "    if start + SECTION_OVERLAP < end:\n",
    "        yield (all_text[start:end], find_page(start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5ca920",
   "metadata": {},
   "source": [
    "# 인덱스에 등록할 문서 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca710ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import base64\n",
    "import json\n",
    "def filename_to_id(filename):\n",
    "    filename_ascii = re.sub(\"[^0-9a-zA-Z_-]\", \"_\", filename)\n",
    "    filename_hash = base64.b16encode(filename.encode('utf-8')).decode('ascii')\n",
    "    return f\"file-{filename_ascii}-{filename_hash}\"\n",
    "\n",
    "def create_sections(filename, page_map, use_vectors, category):\n",
    "    file_id = filename_to_id(filename)\n",
    "    for i, (content, pagenum) in enumerate(split_text(page_map, filename)):\n",
    "        section = {\n",
    "            \"id\": f\"{file_id}-page-{i}\",\n",
    "            \"content\": content,\n",
    "            \"category\": category,\n",
    "            \"sourcepage\": blob_name_from_file_page(filename, pagenum),\n",
    "            \"sourcefile\": filename,\n",
    "            \"metadata\": json.dumps({\"page\": pagenum, \"sourcepage\": blob_name_from_file_page(filename, pagenum)})\n",
    "        }\n",
    "        \n",
    "        section[\"embedding\"] = compute_embedding(content)\n",
    "        yield section\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66f93338",
   "metadata": {},
   "source": [
    "# 청크 인덱싱"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "434ed990",
   "metadata": {},
   "outputs": [],
   "source": [
    "def index_sections(filename, sections):\n",
    "    search_client = SearchClient(\n",
    "        endpoint=search_service_endpoint, index_name=index_name, credential=credential\n",
    "    )\n",
    "    i = 0\n",
    "    batch = []\n",
    "    for s in sections:\n",
    "        batch.append(s)\n",
    "        i += 1\n",
    "        if i % 1000 == 0:\n",
    "            results = search_client.upload_documents(documents=batch)\n",
    "            succeeded = sum([1 for r in results if r.succeeded])\n",
    "            print(f\"\\tIndexed {len(results)} sections, {succeeded} succeeded\")\n",
    "            batch = []\n",
    "\n",
    "    if len(batch) > 0:\n",
    "        results = search_client.upload_documents(documents=batch)\n",
    "        succeeded = sum([1 for r in results if r.succeeded])\n",
    "        print(f\"\\tIndexed {len(results)} sections, {succeeded} succeeded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c2b7c94",
   "metadata": {},
   "source": [
    "# 지금까지 정의한 작업 실행하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6513645e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "print(\"Create Search Index...\")\n",
    "create_search_index()\n",
    "print(\"Processing files...\")\n",
    "\n",
    "path_pattern = \"../data/*.pdf\"\n",
    "for filename in glob.glob(path_pattern):\n",
    "    print(f\"Processing '{filename}'\")\n",
    "    try:\n",
    "        upload_blobs(filename)\n",
    "        remove_from_index(filename)\n",
    "        page_map = get_document_text(filename)\n",
    "        category = os.path.basename(os.path.dirname(filename))\n",
    "        sections = create_sections(\n",
    "            os.path.basename(filename), page_map, False, category\n",
    "        )\n",
    "        index_sections(os.path.basename(filename), sections)\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"\\tGot an error while reading {filename} -> {e} --> skipping file\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}