Update loadbalance_two_api_keys.ipynb

Portkey-AI · Oct 3, 2023 · c72bfcd · c72bfcd
1 parent 799dc84
commit c72bfcd
Showing 1 changed file with 200 additions and 2 deletions.
diff --git a/examples/loadbalance_two_api_keys.ipynb b/examples/loadbalance_two_api_keys.ipynb
@@ -35,7 +35,7 @@
     "import portkey\n",
     "from portkey import Config, LLMOptions\n",
     "\n",
-    "# Create two LLM objects with different account keys\n",
+    "# Create two LLM objects with different account keys and set their weights\n",
     "llm_a = LLMOptions(provider = \"openai\", api_key = \"OPENAI_ACCOUNT_1_KEY\", weight = 0.5)\n",
     "llm_b = LLMOptions(provider = \"openai\", api_key = \"OPENAI_ACCOUNT_2_KEY\", weight = 0.5)"
    ]
@@ -62,7 +62,7 @@
     "portkey.config = Config(\n",
     "    api_key = \"PORTKEY_API_KEY\", \n",
     "    mode = \"loadbalance\", \n",
-    "    llms = [llm_a,llm_b]\n",
+    "    llms = [llm_a, llm_b]\n",
     ")"
    ]
   },
@@ -98,6 +98,204 @@
     "print(response)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "### More Production Features\n",
+    "\n",
+    "**⏩ Streaming**\n",
+    "\n",
+    "* Just set `stream=True` while making your call"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "stream_response = portkey.ChatCompletions.create(\n",
+    "    model = \"gpt-4-0613\", \n",
+    "    messages = [\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+    "        {\"role\": \"user\", \"content\": \"What is the meaning of life, universe and everything?\"},\n",
+    "    ],\n",
+    "    stream = True\n",
+    ")\n",
+    "\n",
+    "for i in stream_response:\n",
+    "    print(i.delta, end=\"\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**🔑 Virtual Keys**\n",
+    "\n",
+    "Safeguard your original OpenAI API keys and only expose dynamic virtual keys in the Portkey ecosystem\n",
+    "\n",
+    "1. Go to the “Virtual Keys” page on Portkey dashboard and hit the “Add Key” button on the top right corner.\n",
+    "2. Choose your AI provider (OpenAI in this case), assign a unique name to your key, and, and add notes. Your virtual key is ready!\n",
+    "\n",
+    "* While constructing your LLM object with Portkey, you can pass the virtual keys instead of API keys with `virtual_key` parameter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "llm_a = LLMOptions(provider = \"openai\", virtual_key = \"openai-xxxx\", weight = 0.5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**👣 Tracing**\n",
+    "\n",
+    "Monitor your apps throughout the lifecycle of a request with a singular `trace id`.\n",
+    "\n",
+    "* You can set the `trace_id` while constructing your LLM object"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "llm_a = LLMOptions(provider = \"openai\", virtual_key = \"openai-xxxx\", weight = 0.5, trace_id = \"loadbalance_accounts_test\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**📝 Live Feedback**\n",
+    "\n",
+    "Portkey’s Feedback API offers a straightforward way to gather weighted feedback from users, allowing you to refine and improve your app.\n",
+    "\n",
+    "* Append feedback to any `trace_id` through Portkey's feedback endpoint:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "import json\n",
+    "\n",
+    "headers = { \"x-portkey-api-key\" : \"PORTKEY_API_KEY\", \"Content-Type\" : \"application/json\" }\n",
+    "data = { \"trace_id\" : \"loadbalance_accounts_test\", \"value\" : 1 }\n",
+    "\n",
+    "send_feedback = requests.post(\"https://api.portkey.ai/v1/feedback\", headers=headers, data=json.dumps(data))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**💪 Fallbacks & Retries**\n",
+    "\n",
+    "Portkey helps you build resilient apps with automatic fallbacks & retries:\n",
+    "\n",
+    "**Fallbacks**: If a primary service or model fails, Portkey will automatically switch to a backup model. <br />\n",
+    "**Retries**: If a request fails, Portkey can be configured to retry the request multiple times.\n",
+    "\n",
+    "* Fallbacks is set while construcing your Portkey client by setting `mode=\"fallback\"`\n",
+    "* Retry is set while constructing your LLM object `retry={\"attempts\": 5}`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "llm_a = LLMOptions(provider = \"openai\", virtual_key = \"openai-xxx\", retry={\"attempts\": 5})\n",
+    "llm_b = LLMOptions(provider = \"anthropic\", virtual_key = \"anthropic-xxx\", retry={\"attempts\": 5})\n",
+    "\n",
+    "portkey.config = Config( api_key = \"PORTKEY_API_KEY\", mode = \"fallback\", llms = [llm_a,llm_b])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**🛠️ Integrations**\n",
+    "\n",
+    "You can also use Portkey with the OpenAI SDK, Langchain, Llamaindex, and more.\n",
+    "\n",
+    "Check out [**Portkey docs**](https://docs.portkey.ai/portkey-docs/integrations) for more info.\n",
+    "\n",
+    "* Here's a quick example of using Portkey with the OpenAI SDK:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "plaintext"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "# Set Portkey proxy as the base path\n",
+    "openai.api_base = \"https://api.portkey.ai/v1/proxy\"\n",
+    "\n",
+    "# Set Portkey headers\n",
+    "portkey_headers = {\n",
+    "    \"x-portkey-api-key\": \"PORTKEY_API_KEY\",\n",
+    "    \"x-portkey-mode\": \"proxy openai\"\n",
+    "}\n",
+    "\n",
+    "response = openai.Completion.create(\n",
+    "  model = \"text-davinci-003\",\n",
+    "  prompt = \"Translate the following English text to French: '{}'\",\n",
+    "  headers = portkey_headers\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Join Discord**\n",
+    "\n",
+    "Collaborate with industry practitioners building LLM apps and get first-class Portkey support: [**Join here**](https://discord.com/invite/DD7vgKK299)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},