e4-memories/notebook.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "### Reminder: This 📘 `.NET Interactive` notebook needs to be run from VS Code with [these prerequisites](../PREREQS.md)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to use this notebook: \n",
    "\n",
    "* Just read the text and scroll along until you run into code blocks.\n",
    "* Code blocks have computer code inside them — hover over the block and you can run the code.\n",
    "* Run the code by hitting the ▶️ \"play\" button to the left. If the code runs you'll see a ✔️. If not, you'll get a ❌.\n",
    "* The output and status of the code block will appear just below itself — you need to scroll down further to see it.\n",
    "* Sometimes a code block will ask you for input in a hard-to-notice dialog box 👆 at the top of your notebook window. \n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recipe IV. 🥑 Memories Maximized\n",
    "## 🧑‍🍳 Cook well beyond the model's memory limits\n",
    "\n",
    "\n",
    "The length of a prompt is dependent upon the LLM you are using. Newer models can take more longer prompts; older models can only take shorter prompts. As a result, there's a limitation to how much context you can provide within any given prompt. \n",
    "\n",
    "| Model | Maximum Tokens** |\n",
    "|---|---|\n",
    "| ada | 2049 |\n",
    "| babbage | 2049 |\n",
    "| curie-001 | 2049 |\n",
    "| davinci-003 | 4097 |\n",
    "| GPT-4 | 8192 |\n",
    "\n",
    "** _1 token is approximately 3 characters; 1 page of book is roughly 500 tokens_\n",
    "\n",
    "A method that is growing in popularity is to use what are called \"embeddings\" — which are high-dimensional numerical representations of any given text. It's possible to generate an \"embedding\" for a short piece of text or a longer piece of text. The length of the text is limited by the specific embedding model that you use.\n",
    "\n",
    "When using OpenAI or Azure OpenAI Service models, the `ada` model is both an inexpensive and good-enough choice for most use cases. Let's start our learnings with generating some embeddings, and see how they work in practice.\n",
    "\n",
    "## Step 1. Instantiate a 🔥 kernel for both completions and generating embeddings\n",
    "\n",
    "Note that the code below includes a few new lines that should be unfamiliar to you. They refer to using the `text-embedding-ada-002` model to use for generating the vector of numbers for a piece of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "#r \"nuget: Microsoft.SemanticKernel, 0.9.61.1-preview\"\n",
    "\n",
    "#!import ../config/Settings.cs\n",
    "\n",
    "using Microsoft.SemanticKernel;\n",
    "using Microsoft.SemanticKernel.KernelExtensions;\n",
    "using System.IO;\n",
    "using Microsoft.SemanticKernel.Configuration;\n",
    "using Microsoft.SemanticKernel.SemanticFunctions;\n",
    "using Microsoft.SemanticKernel.CoreSkills;\n",
    "using Microsoft.SemanticKernel.Memory;\n",
    "\n",
    "var (useAzureOpenAI, model, azureEndpoint, apiKey, orgId) = Settings.LoadFromFile();\n",
    "\n",
    "var kernel =  Microsoft.SemanticKernel.Kernel.Builder\n",
    ".Configure(c =>\n",
    "{\n",
    "    if (useAzureOpenAI) {\n",
    "        c.AddAzureOpenAITextCompletion(\"davinci\", model, azureEndpoint, apiKey);\n",
    "        c.AddAzureOpenAIEmbeddingGeneration(\"ada\", \"text-embedding-ada-002\", azureEndpoint, apiKey);\n",
    "    } else {\n",
    "        c.AddOpenAITextCompletion(\"davinci\", model, apiKey, orgId);\n",
    "        c.AddOpenAIEmbeddingGeneration(\"ada\", \"text-embedding-ada-002\", apiKey, orgId);\n",
    "    }\n",
    "})\n",
    ".WithMemoryStorage(new VolatileMemoryStore())\n",
    ".Build();\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Add 🥑 memories to let the 🔥 kernel cook richer meals\n",
    "\n",
    "Imagine a collection of facts collected about you on the Internet as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "const string memoryCollectionName = \"Facts About Me\";\n",
    "\n",
    "await kernel.Memory.SaveInformationAsync(memoryCollectionName, id: \"LinkedIn Bio\", \n",
    "    text: \"I currently work in the hotel industry at the front desk. I won the best team player award.\");\n",
    "\n",
    "await kernel.Memory.SaveInformationAsync(memoryCollectionName, id: \"LinkedIn History\", \n",
    "    text: \"I have worked as a tourist operator for 8 years. I have also worked as a banking associate for 3 years.\");\n",
    "\n",
    "await kernel.Memory.SaveInformationAsync(memoryCollectionName, id: \"Recent Facebook Post\", \n",
    "    text: \"My new dog Trixie is the cutest thing you've ever seen. She's just 2 years old.\");\n",
    "    \n",
    "await kernel.Memory.SaveInformationAsync(memoryCollectionName, id: \"Old Facebook Post\", \n",
    "    text: \"Can you believe the size of the trees in Yellowstone? They're huge! I'm so committed to forestry concerns.\");\n",
    "\n",
    "Console.WriteLine(\"Four GIGANTIC vectors were generated just now from those 4 pieces of text above.\");"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ✅ You'll need to have access to the `text-embedding-ada-002` model for the above to run correctly. Note that the Step 1 for this unit is different than all the other notebooks because it has this extra requirement to run.\n",
    "\n",
    "Next, imagine that you wanted to ask your LLM a question about you. What would it do? Well, given that it doesn't know anything about you out-of-the-box, it will simply make things up about you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var myFunction = kernel.CreateSemanticFunction(@\"\n",
    "Tell me about me and {{$input}} in less than 70 characters.\n",
    "\", maxTokens: 100, temperature: 0.8, topP: 1);\n",
    "var result = await myFunction.InvokeAsync(\"my work history\");\n",
    "Console.WriteLine(result);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the semantic function above might say:\n",
    "\n",
    "`You are a creative problem solver with a varied work history.`\n",
    "\n",
    "That could apply to anybody, of course :+).\n",
    "\n",
    "Instead of hoping that the LLM comes up with the more correct answer, we can use memories to craft a more accurate completion. We do that by finding the most similar memories on file by searching through the memories we've stored, giving it the maximum number of hits we want back with `limit`, and set a threshold for how relevant we want a search to come back with `minRelevanceScore`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "string ask = \"Tell me about me and my work history.\";\n",
    "var relatedMemory = \"I know nothing.\";\n",
    "var counter = 0;\n",
    "\n",
    "var memories = kernel.Memory.SearchAsync(memoryCollectionName, ask, limit: 5, minRelevanceScore: 0.77);\n",
    "\n",
    "await foreach (MemoryQueryResult memory in memories)\n",
    "{\n",
    "    if (counter == 0) { relatedMemory = memory.Text; }\n",
    "    Console.WriteLine($\"Result {++counter}:\\n  >> {memory.Id}\\n  Text: {memory.Text}  Relevance: {memory.Relevance}\\n\");\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can ask the same question but with the most relevant context that we stored in `relatedMemory` to give to the LLM to come up with a more accurate response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var myFunction = kernel.CreateSemanticFunction(@\"\n",
    "{{$input}}\n",
    "Tell me about me and my work history in less than 70 characters.\n",
    "\", maxTokens: 100, temperature: 0.1, topP: .1);\n",
    "\n",
    "var result = await myFunction.InvokeAsync(relatedMemory);\n",
    "\n",
    "Console.WriteLine(result);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Feel the \"a-ha\" moment.\n",
    "\n",
    "### Manipulating 🥑 memories is how the token window limitation is addressed.\n",
    "\n",
    "Recall the table showing the maximum tokens that can be used per model:\n",
    "\n",
    "| Model | Maximum Tokens** |\n",
    "|---|---|\n",
    "| ada | 2049 |\n",
    "| babbage | 2049 |\n",
    "| curie-001 | 2049 |\n",
    "| davinci-003 | 4097 |\n",
    "| GPT-4 | 8192 |\n",
    "\n",
    "** _1 token is approximately 3 characters; 1 page of book is roughly 500 tokens_\n",
    "\n",
    "Given this same basic technique of gathering the most similar memories that are appropriate to a prompt, it's possible to have many more memories stored and available on-hand to compare with a given prompt. And it's not necessary to include just the top hit, but also more hits that are just as similar to the \"most relevant\" memory available. \n",
    "\n",
    "This is how an entire book can be used by Semantic Kernel as a memory source to feed into a prompt by only selecting the relevant chunks of text — i.e. that which relates to the prompt. To do so you would:\n",
    "\n",
    "1. Generate embeddings for each of the paragraphs in the book.\n",
    "2. For a given prompt, find the most similar paragraphs within the book.\n",
    "3. Staying within the limitation of the token size window, gather all the related paragraphs.\n",
    "4. You now have a prompt with a great deal of relevant 🥑 context to send to the model.\n",
    "5. Reap the benefits of an \"informed\" LLM AI weighing in on a particular subject for you.\n",
    "\n",
    "Let's review this in practice. Say I have a 500-page book. \n",
    "\n",
    "1. I take each page and generate the embedding with `Memory.SaveInformationAsync`\n",
    "2. I then take my prompt, `the best scenes are ones with flowers in it and deserve to be summarized` and use `Memory.SearchAsync` to locate the pages with flower scenes in them.\n",
    "3. Let's say there are three pages that are relevant. Those three pages will be used to compose a new prompt that's simply the three pages appended to each other along with the original prompt. If instead you need to include ten pages, and exceed the token window, then summarize each of the ten pages separately into ten shorter passages. Do this until you meet the token window requirements.\n",
    "4. You have the prompt to give to the model you've chosen. It has pulled the relevant information out of the 500-page book, and will do its best to summarize what you care about the most.\n",
    "5. Ta-da! You'll get what you've asked for."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate this point, we can take Abraham Lincoln's famous Gettysburg Address and use it to generate a new speech. We'll use the OpenAI API to generate the speech, and then we'll use the Azure Cognitive Search API to search for the speech in the text of the Gettysburg Address and break it up into chunks of text and then processed with embeddings. We've asked the GPT-3 model to write a simple text chunking procedure where a specified maximum length of a chunk is given. Chunking is still more of an art than a science, so you can see the result isn't as perfect as we'd like. But this will give you a sense of how a large text file can be processed into smaller pieces of text that get used by an LLM AI model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using System;\n",
    "using System.IO;\n",
    "using System.Text;\n",
    "\n",
    "public static List<string> ChunkTextFile(string filePath, int recommendedLength)\n",
    "{\n",
    "    List<string> chunks = new List<string>();\n",
    "\n",
    "    // Read in the text file\n",
    "    string text = File.ReadAllText(filePath);\n",
    "\n",
    "    // Break the text into chunks of the recommended length\n",
    "    int startIndex = 0;\n",
    "    while (startIndex < text.Length)\n",
    "    {\n",
    "        int endIndex = startIndex + recommendedLength;\n",
    "        if (endIndex > text.Length)\n",
    "        {\n",
    "            endIndex = text.Length;\n",
    "        }\n",
    "\n",
    "        // Look for a natural breakage point like a paragraph or just before a new heading\n",
    "        while (endIndex < text.Length && !char.IsWhiteSpace(text[endIndex]))\n",
    "        {\n",
    "            endIndex++;\n",
    "        }\n",
    "\n",
    "        // Get the chunk of text\n",
    "        string chunk = text.Substring(startIndex, endIndex - startIndex);\n",
    "\n",
    "        // Strip the whitespace at the start and end of the string\n",
    "        chunk = chunk.Trim();\n",
    "\n",
    "        // Add the chunk to the list\n",
    "        chunks.Add(chunk);\n",
    "\n",
    "        // Move to the next chunk\n",
    "        startIndex = endIndex;\n",
    "    }\n",
    "\n",
    "    return chunks;\n",
    "}\n",
    "\n",
    "// Get the list of chunks from the file\n",
    "List<string> chunks = ChunkTextFile(\"./lincoln.txt\", 140);\n",
    "\n",
    "const string lincolnMemoryCollectionName = \"Abe's Words\";\n",
    "\n",
    "// Add the chunks to memory\n",
    "int counter = 0;\n",
    "foreach (string chunk in chunks)\n",
    "{\n",
    "    Console.WriteLine($\"Chunk {counter}: {chunk}\");\n",
    "\n",
    "    await kernel.Memory.SaveInformationAsync(lincolnMemoryCollectionName, id: $\"Chunk {counter++}\", \n",
    "        text: chunk);\n",
    "}\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now query these chunks for the most similar ones that match a simple question: `\"What should the people do?\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var aCounter = 0;\n",
    "var myPrompt = \"What should the people do?\";\n",
    "var myMemory = \"\";\n",
    "var memories = kernel.Memory.SearchAsync(lincolnMemoryCollectionName, myPrompt, limit: 5, minRelevanceScore: 0.77);\n",
    "\n",
    "await foreach (MemoryQueryResult memory in memories) {\n",
    "    Console.WriteLine($\"Result {++aCounter}:\\n  >> {memory.Id}\\n  Text: {memory.Text}  Relevance: {memory.Relevance}\\n\");\n",
    "    myMemory += memory.Text + \" \";\n",
    "}\n",
    "\n",
    "Console.WriteLine(\"Memory to feed back into the prompt will be:\\n  >> \" + myMemory+ \"\\n\");\n",
    "var myLincolnFunction = kernel.CreateSemanticFunction(@\"\n",
    "Lincoln said:\n",
    "---\n",
    "{{$input}}\n",
    "---\n",
    "So what should the people do?\n",
    "\", maxTokens: 100, temperature: 0.1, topP: .1);\n",
    "\n",
    "var lincolnResult = await myLincolnFunction.InvokeAsync(myMemory);\n",
    "\n",
    "Console.WriteLine(\"Generated response ... 'according to Lincoln':\\n\" + lincolnResult);\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see this run at scale, check out the GitHub Q&A Sample app available at [https://aka.ms/sk/repo](https://aka.ms/sk/repo). It takes an entire code repo, converts it to embeddings, and it lets you \"chat\" with the repo itself. Keep in mind that it would be generally impossible to feed the entire repo into an LLM AI's window, and that's where using 🥑 memories come in."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ⏭️ Next Steps\n",
    "\n",
    "Run through more advanced examples in the notebooks that are available in our GitHub repo at [https://aka.ms/sk/repo](https://aka.ms/sk/repo).\n",
    "\n",
    "[Learn about 🍋 connectors!](../e5-connectors/notebook.ipynb)\n",
    "\n",
    "Or stay a longer while and add more facts about yourself in the `MemoryCollection`."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}