From ea16e25b3cac222ed97c476ff61a18cffcc645f1 Mon Sep 17 00:00:00 2001 From: Andrew Reed Date: Thu, 31 Oct 2024 09:39:16 -0400 Subject: [PATCH 1/6] add phoenix observability cookbook --- notebooks/en/_toctree.yml | 2 + notebooks/en/index.md | 1 + .../phoenix_observability_on_hf_spaces.ipynb | 594 ++++++++++++++++++ 3 files changed, 597 insertions(+) create mode 100644 notebooks/en/phoenix_observability_on_hf_spaces.ipynb diff --git a/notebooks/en/_toctree.yml b/notebooks/en/_toctree.yml index 4bbebfbc..0688a9f0 100644 --- a/notebooks/en/_toctree.yml +++ b/notebooks/en/_toctree.yml @@ -62,6 +62,8 @@ title: Benchmarking TGI - local: rag_with_knowledge_graphs_neo4j title: Enhancing RAG Reasoning with Knowledge Graphs + - local: phoenix_observability_on_hf_spaces + title: Phoenix Observability Dashboard on HF Spaces - title: Computer Vision Recipes isExpanded: false diff --git a/notebooks/en/index.md b/notebooks/en/index.md index c6fb96c1..59010bd6 100644 --- a/notebooks/en/index.md +++ b/notebooks/en/index.md @@ -7,6 +7,7 @@ applications and solving various machine learning tasks using open-source tools Check out the recently added notebooks: +- [Phoenix Observability Dashboard on HF Spaces](phoenix_observability_on_hf_spaces) - [Have several agents collaborate in a multi-agent hierarchy πŸ€–πŸ€πŸ€–](multiagent_web_assistant) - [Semantic reranking with Elasticsearch](semantic_reranking_elasticsearch) - [Benchmarking TGI](benchmarking_tgi) diff --git a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb new file mode 100644 index 00000000..8c9803c0 --- /dev/null +++ b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb @@ -0,0 +1,594 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Setup a Phoenix observability dashboard on Hugging Face Spaces for LLM application tracing\n", + "\n", + "_Authored by: [Andrew Reed](https://huggingface.co/andrewrreed)_\n", + "\n", + "[Phoenix](https://docs.arize.com/phoenix) is an open-source observability library by [Arize AI](https://arize.com/) designed for experimentation, evaluation, and troubleshooting. It allows AI Engineers and Data Scientists to quickly visualize their data, evaluate performance, track down issues, and export data to improve.\n", + "\n", + "In this notebook, we'll see how to deploy a Phoenix observability dashboard on [Hugging Face Spaces](https://huggingface.co/spaces) and configure it to automatically trace LLM calls, providing a comprehensive view into the inner workings of your LLM applications." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 1: Deploy Phoenix on Hugging Face Spaces\n", + "\n", + "While Phoenix offers a [notebook-first option](https://docs.arize.com/phoenix/deployment/environments#notebooks) for local development, it can also be deployed as a [standalone dashboard via Docker](https://docs.arize.com/phoenix/deployment/environments#container). A long-running, hosted dashboard is a great way to provide a centralized view into your LLM application behavior, and to collaborate with your team. Hugging Face Spaces offers a simple way to host ML applications with optional, persistent storage, and it's support for custom Docker images makes it a great platform for hosting Phoenix - lets see how it works!\n", + "\n", + "First, we'll [duplicate the demo space](https://huggingface.co/spaces/andrewrreed/phoenix-arize-observability-demo)\n", + "\n", + "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/duplicate.png)\n", + "\n", + "We can configure the space to be private or public, and it can live in our user namespace, or in an organization namespace. We can use the default, free-tier CPU, and importantly, specify that we want to attach a persistent disk to the space.\n", + "\n", + "> Note: In order for the tracing data to persist across Space, we _must_ configure a persistent disk, otherwise all data will be lost when the space is restarted. Configuring a persistent disk is a paid feature, and will incur a cost for the lifetime of the Space. In this case, we'll use the Small - 20GB disk option for $0.01 per hour.\n", + "\n", + "After clicking \"Duplicate Space\", the Docker image will begin building. This will take a few minutes to complete, and then we'll see an empty Phoenix dashboard.\n", + "\n", + "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/empty-dashboard.png)\n", + "\n", + "\n", + "## Step 2: Configure application tracing\n", + "\n", + "Now that we have a running Phoenix dashboard, we can configure our application to automatically trace LLM calls using an [OpenTelemetry TracerProvider](https://docs.arize.com/phoenix/quickstart#connect-your-app-to-phoenix). In this example, we'll instrument our application using the OpenAI client library, and trace LLM calls made from the `openai` Python package to open LLMs running on [Hugging Face's Serverless Inference API](https://huggingface.co/docs/api-inference/en/index).\n", + "\n", + "> Note: Phoenix supports tracing for [a wide variety of LLM frameworks](https://docs.arize.com/phoenix/tracing/integrations-tracing), including LangChain, LlamaIndex, AWS Bedrock, and more.\n", + "\n", + "\n", + "First, we need to install the necessary libraries:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -q arize-phoenix arize-phoenix-otel openinference-instrumentation-openai openai huggingface-hub" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then, we'll login to Hugging Face using the `huggingface_hub` library. This will allow us to generate the necessary authentication for our Space and the Serverless Inference API. Be sure that the HF token used to authenticate has the correct permissions for the Organization where your Space is located." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from huggingface_hub import interpreter_login\n", + "\n", + "interpreter_login()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now, we can [configure the Phoenix client](https://docs.arize.com/phoenix/deployment/configuration#client-configuration) to our running Phoenix dashboard:\n", + "\n", + "1. Register the Phoenix tracer provider by\n", + " - Specifying the `project_name` of our choice\n", + " - Setting the `endpoint` value to the Hostname of our Space (found via the dashboard UI under the \"Settings\" tab - see below)\n", + " - Setting the `headers` to the Hugging Face headers needed to access the Space\n", + "2. Instrument our application code to use the OpenAI tracer provider\n", + "\n", + "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/settings.png)" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "πŸ”­ OpenTelemetry Tracing Details πŸ”­\n", + "| Phoenix Project: test\n", + "| Span Processor: SimpleSpanProcessor\n", + "| Collector Endpoint: https://andrewrreed-phoenix-arize-observability-demo.hf.space/v1/traces\n", + "| Transport: HTTP\n", + "| Transport Headers: {'user-agent': '****', 'authorization': '****'}\n", + "| \n", + "| Using a default SpanProcessor. `add_span_processor` will overwrite this default.\n", + "| \n", + "| `register` has set this TracerProvider as the global OpenTelemetry default.\n", + "| To disable this behavior, call `register` with `set_global_tracer_provider=False`.\n", + "\n" + ] + } + ], + "source": [ + "from phoenix.otel import register\n", + "from huggingface_hub.utils import build_hf_headers\n", + "from openinference.instrumentation.openai import OpenAIInstrumentor\n", + "\n", + "# 1. Register the Phoenix tracer provider\n", + "tracer_provider = register(\n", + " project_name=\"test\",\n", + " endpoint=\"https://andrewrreed-phoenix-arize-observability-demo.hf.space\"\n", + " + \"/v1/traces\",\n", + " headers=build_hf_headers(),\n", + ")\n", + "\n", + "# 2. Instrument our application code to use the OpenAI tracer provider\n", + "OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Step 3: Make calls and view traces in the Phoenix dashboard\n", + "\n", + "Now, we can make a call to an LLM and view the traces in the Phoenix dashboard. We're using the OpenAI client to make calls to the Hugging Face Serverless Inference API, which is instrumented to work with Phoenix. In this case, we're using the `meta-llama/Llama-3.1-8B-Instruct` model." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Llamas are intelligent and social animals, and they do have ways to entertain themselves and have fun. While we can't directly ask a llama about its personal preferences, we can observe their natural behaviors and make some educated guesses. Here are some things that might bring a llama joy and excitement:\n", + "\n", + "1. **Socializing**: Llamas are herd animals and they love to interact with each other. They'll often engage in play-fighting, neck-wrestling, and other activities to establish dominance and strengthen social bonds. When llamas have a strong social network, it can make them feel happy and content.\n", + "2. **Exploring new environments**: Llamas are naturally curious creatures, and they love to explore new surroundings. They'll often investigate their surroundings, sniffing and investigating new sights, sounds, and smells.\n", + "3. **Playing with toys**: While llamas don't need expensive toys, they do enjoy playing with objects that stimulate their natural behaviors. For example, a ball or a toy that mimics a target can be an entertaining way to engage them.\n", + "4. **Climbing and jumping**: Llamas are agile and athletic animals, and they enjoy using their limbs to climb and jump over obstacles. Providing a safe and stable area for them to exercise their physical abilities can be a fun and engaging experience.\n", + "5. **Browsing and foraging**: Llamas have a natural instinct to graze and browse, and they enjoy searching for tasty plants and shrubs. Providing a variety of plants to munch on can keep them engaged and entertained.\n", + "6. **Mentally stimulating activities**: Llamas are intelligent animals, and they can benefit from mentally stimulating activities like problem-solving puzzles or learning new behaviors (like agility training or obedience training).\n", + "\n", + "Some fun activities you can try with a llama include:\n", + "\n", + "* Setting up an obstacle course or agility challenge\n", + "* Creating a \"scavenger hunt\" with treats and toys\n", + "* Introducing new toys or objects to stimulate their curiosity\n", + "* Providing a variety of plants and shrubs to browse and graze on\n", + "* Engaging in interactive games like \"follow the leader\" or \"find the treat\"\n", + "\n", + "Remember to always prioritize the llama's safety and well-being, and to consult with a veterinarian or a trained llama handler before attempting any new activities or introducing new toys.\n" + ] + } + ], + "source": [ + "from openai import OpenAI\n", + "from huggingface_hub import get_token\n", + "\n", + "client = OpenAI(\n", + " base_url=\"https://api-inference.huggingface.co/v1/\",\n", + " api_key=get_token(),\n", + ")\n", + "\n", + "messages = [{\"role\": \"user\", \"content\": \"What does a llama do for fun?\"}]\n", + "\n", + "response = client.chat.completions.create(\n", + " model=\"meta-llama/Llama-3.1-8B-Instruct\",\n", + " messages=messages,\n", + " max_tokens=500,\n", + ")\n", + "\n", + "print(response.choices[0].message.content)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If we navigate back to the Phoenix dashboard, we can see the trace from our LLM call is captured and displayed! If you configured your space with a persistent disk, all of the trace information will be saved anytime you restart the space.\n", + "\n", + "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/test-trace.png)\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Bonus: Tracing a multi-agent application with CrewAI\n", + "\n", + "The real power of observability comes from being able to trace and inspect complex LLM workflows. In this example, we'll use [CrewAI](https://www.crewai.com/) to trace a multi-agent application." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install -q openinference-instrumentation-crewai crewai crewai-tools" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Like before, we'll register the Phoenix tracer provider and instrument the application code, but this time we'll also uninstrument the existing OpenAI tracer provider to avoid conflicts." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Overriding of current TracerProvider is not allowed\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "πŸ”­ OpenTelemetry Tracing Details πŸ”­\n", + "| Phoenix Project: crewai\n", + "| Span Processor: SimpleSpanProcessor\n", + "| Collector Endpoint: https://andrewrreed-phoenix-arize-observability-demo.hf.space/v1/traces\n", + "| Transport: HTTP\n", + "| Transport Headers: {'user-agent': '****', 'authorization': '****'}\n", + "| \n", + "| Using a default SpanProcessor. `add_span_processor` will overwrite this default.\n", + "| \n", + "| `register` has set this TracerProvider as the global OpenTelemetry default.\n", + "| To disable this behavior, call `register` with `set_global_tracer_provider=False`.\n", + "\n" + ] + } + ], + "source": [ + "from opentelemetry import trace\n", + "from openinference.instrumentation.crewai import CrewAIInstrumentor\n", + "\n", + "# 0. Uninstrument existing tracer provider and clear the global tracer provider\n", + "OpenAIInstrumentor().uninstrument()\n", + "if trace.get_tracer_provider():\n", + " trace.get_tracer_provider().shutdown()\n", + " trace._TRACER_PROVIDER = None # Reset the global tracer provider\n", + "\n", + "# 1. Register the Phoenix tracer provider\n", + "tracer_provider = register(\n", + " project_name=\"crewai\",\n", + " endpoint=\"https://andrewrreed-phoenix-arize-observability-demo.hf.space\"\n", + " + \"/v1/traces\",\n", + " headers=build_hf_headers(),\n", + ")\n", + "\n", + "# 2. Instrument our application code to use the OpenAI tracer provider\n", + "CrewAIInstrumentor().instrument(tracer_provider=tracer_provider)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we'll define a multi-agent application using CrewAI to research and write a blog post about the importance of observability and tracing in LLM applications.\n", + "\n", + "> Note: This example is borrowed and modified from [here](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai)." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-31 09:10:49,986 - 8143739712 - __init__.py-__init__:538 - WARNING: Overriding of current TracerProvider is not allowed\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m\u001b[95m# Agent:\u001b[00m \u001b[1m\u001b[92mResearcher\u001b[00m\n", + "\u001b[95m## Task:\u001b[00m \u001b[92mConduct comprehensive research and analysis of the importance of observability and tracing in LLM applications.\n", + " Identify key trends, breakthrough technologies, and potential industry impacts.\u001b[00m\n", + "\n", + "\n", + "\u001b[1m\u001b[95m# Agent:\u001b[00m \u001b[1m\u001b[92mResearcher\u001b[00m\n", + "\u001b[95m## Using tool:\u001b[00m \u001b[92mSearch the internet\u001b[00m\n", + "\u001b[95m## Tool Input:\u001b[00m \u001b[92m\n", + "\"{\\\"search_query\\\": \\\"importance of observability and tracing in LLM applications\\\"}\"\u001b[00m\n", + "\u001b[95m## Tool Output:\u001b[00m \u001b[92m\n", + "\n", + "Search results: Title: LLM Observability: The 5 Key Pillars for Monitoring Large Language ...\n", + "Link: https://arize.com/blog-course/large-language-model-monitoring-observability/\n", + "Snippet: Why leveraging the five pillars of LLM observability is essential for ensuring performance, reliability, and seamless LLM applications.\n", + "---\n", + "Title: Observability of LLM Applications: Exploration and Practice from the ...\n", + "Link: https://www.alibabacloud.com/blog/observability-of-llm-applications-exploration-and-practice-from-the-perspective-of-trace_601604\n", + "Snippet: This article clarifies the technical challenges of observability by analyzing LLM application patterns and different concerns.\n", + "---\n", + "Title: What is LLM Observability? - The Ultimate LLM Monitoring Guide\n", + "Link: https://www.confident-ai.com/blog/what-is-llm-observability-the-ultimate-llm-monitoring-guide\n", + "Snippet: Observability tools collect and correlate logs, real-time evaluation metrics, and traces to understand the context of unexpected outputs or ...\n", + "---\n", + "Title: An Introduction to Observability for LLM-based applications using ...\n", + "Link: https://opentelemetry.io/blog/2024/llm-observability/\n", + "Snippet: Why Observability Matters for LLM Applications Β· It's vital to keep track of how often LLMs are being used for usage and cost tracking. Β· Latency ...\n", + "---\n", + "Title: Understanding LLM Observability - Key Insights, Best Practices ...\n", + "Link: https://signoz.io/blog/llm-observability/\n", + "Snippet: LLM Observability is essential for maintaining reliable, accurate, and efficient AI applications. Focus on the five pillars: evaluation, tracing ...\n", + "---\n", + "Title: LLM Observability Tools: 2024 Comparison - lakeFS\n", + "Link: https://lakefs.io/blog/llm-observability-tools/\n", + "Snippet: LLM observability is the process that enables monitoring by providing full visibility and tracing in an LLM application system, as well as newer ...\n", + "---\n", + "Title: From Concept to Production with Observability in LLM Applications\n", + "Link: https://hadijaveed.me/2024/03/05/tracing-and-observability-in-llm-applications/\n", + "Snippet: Traces are essential to understanding the full β€œpath” a request takes in your application, e.g, prompt, query-expansion, RAG retrieved top-k ...\n", + "---\n", + "Title: The Importance of LLM Observability: A Technical Deep Dive\n", + "Link: https://www.linkedin.com/pulse/importance-llm-observability-technical-deep-dive-patrick-carroll-trlqe\n", + "Snippet: LLM observability is crucial for any technical team that wants to maintain and improve the reliability, security, and performance of their AI- ...\n", + "---\n", + "Title: Observability and Monitoring of LLMs - TheBlue.ai\n", + "Link: https://theblue.ai/blog/llm-observability-en/\n", + "Snippet: LLM-Observability is crucial to maximize the performance and reliability of Large Language Models (LLMs). By systematically capturing and ...\n", + "---\n", + "\u001b[00m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-31 09:11:21,244 - 8143739712 - __init__.py-__init__:100 - WARNING: Invalid type TaskOutput for attribute 'output.value' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "\u001b[1m\u001b[95m# Agent:\u001b[00m \u001b[1m\u001b[92mResearcher\u001b[00m\n", + "\u001b[95m## Final Answer:\u001b[00m \u001b[92m\n", + "**Comprehensive Research and Analysis Report: Importance of Observability and Tracing in LLM Applications**\n", + "\n", + "**Introduction**\n", + "\n", + "Large Language Models (LLMs) have revolutionized the field of natural language processing, enabling applications such as language translation, text summarization, and conversational AI. However, as LLMs become increasingly complex and widespread, ensuring their performance, reliability, and security has become a significant challenge. Observability and tracing are crucial components in addressing this challenge, enabling developers to monitor, debug, and optimize LLM applications. This report provides an in-depth analysis of the importance of observability and tracing in LLM applications, highlighting key trends, breakthrough technologies, and potential industry impacts.\n", + "\n", + "**Key Trends:**\n", + "\n", + "* **Increased Adoption of Cloud-Native Observability Tools**: Cloud-native observability tools, such as OpenTelemetry and Signoz, are gaining popularity due to their ability to provide real-time insights into LLM application performance and behavior.\n", + "* **Growing Importance of Tracing**: Tracing has become a critical aspect of LLM observability, enabling developers to understand the flow of requests through the application and identify bottlenecks.\n", + "* **Rise of AI-Specific Observability Platforms**: Specialized observability platforms, such as Arize and LakeFS, are emerging to cater to the unique needs of LLM applications.\n", + "\n", + "**Breakthrough Technologies:**\n", + "\n", + "* **OpenTelemetry**: OpenTelemetry is an open-source observability framework that enables developers to collect and correlate logs, metrics, and traces from LLM applications.\n", + "* **Signoz**: Signoz is a cloud-native observability platform that provides real-time insights into LLM application performance and behavior.\n", + "* **Arize**: Arize is an AI-specific observability platform that enables developers to monitor and optimize LLM applications in real-time.\n", + "\n", + "**Potential Industry Impacts:**\n", + "\n", + "* **Improved Performance**: Observability and tracing enable developers to identify and address performance bottlenecks, leading to improved response times and user experience.\n", + "* **Enhanced Reliability**: By monitoring LLM application behavior, developers can identify and fix issues before they impact users, ensuring high uptime and reliability.\n", + "* **Increased Security**: Observability and tracing enable developers to detect and respond to security threats in real-time, protecting sensitive user data and preventing data breaches.\n", + "\n", + "**Conclusion**\n", + "\n", + "Observability and tracing are critical components of LLM application development, enabling developers to monitor, debug, and optimize their applications. As LLMs continue to evolve and become increasingly complex, the importance of observability and tracing will only continue to grow. By adopting cloud-native observability tools, leveraging breakthrough technologies like OpenTelemetry and Signoz, and staying up-to-date with industry trends, developers can ensure the performance, reliability, and security of their LLM applications.\u001b[00m\n", + "\n", + "\n", + "\u001b[1m\u001b[95m# Agent:\u001b[00m \u001b[1m\u001b[92mTechnical Writer\u001b[00m\n", + "\u001b[95m## Task:\u001b[00m \u001b[92mUsing the insights provided, develop an engaging blog\n", + " post that highlights the importance of observability and tracing in LLM applications.\n", + " Your post should be informative yet accessible, catering to a tech-savvy audience.\n", + " Make it sound cool, avoid complex words so it doesn't sound like AI.\u001b[00m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-31 09:11:39,715 - 8143739712 - __init__.py-__init__:100 - WARNING: Invalid type TaskOutput for attribute 'output.value' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "\u001b[1m\u001b[95m# Agent:\u001b[00m \u001b[1m\u001b[92mTechnical Writer\u001b[00m\n", + "\u001b[95m## Final Answer:\u001b[00m \u001b[92m\n", + "**Unlocking the Power of Observability and Tracing in LLM Applications**\n", + "\n", + "As Large Language Models (LLMs) continue to revolutionize the field of natural language processing, ensuring their performance, reliability, and security has become a top priority for developers. With the increasing complexity of LLMs, it's essential to have a robust monitoring and debugging system in place to identify and address issues before they impact users. This is where observability and tracing come into play – two critical components that enable developers to monitor, debug, and optimize LLM applications.\n", + "\n", + "**The Importance of Observability and Tracing in LLM Applications**\n", + "\n", + "Observability and tracing are not just buzzwords; they're essential tools for any LLM developer. By implementing observability and tracing, developers can gain real-time insights into their application's performance and behavior, allowing them to identify bottlenecks, optimize resource utilization, and ensure high uptime and reliability. With observability and tracing, developers can:\n", + "\n", + "* **Monitor application performance**: Get real-time insights into application performance, including response times, latency, and error rates.\n", + "* **Identify bottlenecks**: Use tracing to understand the flow of requests through the application and identify areas where performance can be improved.\n", + "* **Optimize resource utilization**: Make data-driven decisions to optimize resource allocation and reduce waste.\n", + "* **Ensure high uptime and reliability**: Identify and fix issues before they impact users, ensuring high uptime and reliability.\n", + "* **Protect sensitive user data**: Detect and respond to security threats in real-time, protecting sensitive user data and preventing data breaches.\n", + "\n", + "**The Rise of Cloud-Native Observability Tools and AI-Specific Observability Platforms**\n", + "\n", + "The observability landscape is evolving rapidly, with cloud-native observability tools and AI-specific observability platforms emerging to cater to the unique needs of LLM applications. Some of the key players in this space include:\n", + "\n", + "* **OpenTelemetry**: An open-source observability framework that enables developers to collect and correlate logs, metrics, and traces from LLM applications.\n", + "* **Signoz**: A cloud-native observability platform that provides real-time insights into LLM application performance and behavior.\n", + "* **Arize**: An AI-specific observability platform that enables developers to monitor and optimize LLM applications in real-time.\n", + "\n", + "**The Future of Observability and Tracing in LLM Applications**\n", + "\n", + "As LLMs continue to evolve and become increasingly complex, the importance of observability and tracing will only continue to grow. By adopting cloud-native observability tools, leveraging breakthrough technologies like OpenTelemetry and Signoz, and staying up-to-date with industry trends, developers can ensure the performance, reliability, and security of their LLM applications. In the future, we can expect to see:\n", + "\n", + "* **Increased adoption of cloud-native observability tools**: As more developers adopt cloud-native observability tools, we can expect to see improved performance, reliability, and security in LLM applications.\n", + "* **Advancements in AI-specific observability platforms**: AI-specific observability platforms will continue to evolve, providing more advanced features and capabilities to support the unique needs of LLM applications.\n", + "* **Greater emphasis on security**: As LLMs become more widespread, the importance of security will only continue to grow, with observability and tracing playing a critical role in detecting and responding to security threats.\n", + "\n", + "In conclusion, observability and tracing are essential components of LLM application development, enabling developers to monitor, debug, and optimize their applications. By embracing cloud-native observability tools, leveraging breakthrough technologies, and staying up-to-date with industry trends, developers can ensure the performance, reliability, and security of their LLM applications.\u001b[00m\n", + "\n", + "\n", + "------------ FINAL RESULT ------------\n", + "**Unlocking the Power of Observability and Tracing in LLM Applications**\n", + "\n", + "As Large Language Models (LLMs) continue to revolutionize the field of natural language processing, ensuring their performance, reliability, and security has become a top priority for developers. With the increasing complexity of LLMs, it's essential to have a robust monitoring and debugging system in place to identify and address issues before they impact users. This is where observability and tracing come into play – two critical components that enable developers to monitor, debug, and optimize LLM applications.\n", + "\n", + "**The Importance of Observability and Tracing in LLM Applications**\n", + "\n", + "Observability and tracing are not just buzzwords; they're essential tools for any LLM developer. By implementing observability and tracing, developers can gain real-time insights into their application's performance and behavior, allowing them to identify bottlenecks, optimize resource utilization, and ensure high uptime and reliability. With observability and tracing, developers can:\n", + "\n", + "* **Monitor application performance**: Get real-time insights into application performance, including response times, latency, and error rates.\n", + "* **Identify bottlenecks**: Use tracing to understand the flow of requests through the application and identify areas where performance can be improved.\n", + "* **Optimize resource utilization**: Make data-driven decisions to optimize resource allocation and reduce waste.\n", + "* **Ensure high uptime and reliability**: Identify and fix issues before they impact users, ensuring high uptime and reliability.\n", + "* **Protect sensitive user data**: Detect and respond to security threats in real-time, protecting sensitive user data and preventing data breaches.\n", + "\n", + "**The Rise of Cloud-Native Observability Tools and AI-Specific Observability Platforms**\n", + "\n", + "The observability landscape is evolving rapidly, with cloud-native observability tools and AI-specific observability platforms emerging to cater to the unique needs of LLM applications. Some of the key players in this space include:\n", + "\n", + "* **OpenTelemetry**: An open-source observability framework that enables developers to collect and correlate logs, metrics, and traces from LLM applications.\n", + "* **Signoz**: A cloud-native observability platform that provides real-time insights into LLM application performance and behavior.\n", + "* **Arize**: An AI-specific observability platform that enables developers to monitor and optimize LLM applications in real-time.\n", + "\n", + "**The Future of Observability and Tracing in LLM Applications**\n", + "\n", + "As LLMs continue to evolve and become increasingly complex, the importance of observability and tracing will only continue to grow. By adopting cloud-native observability tools, leveraging breakthrough technologies like OpenTelemetry and Signoz, and staying up-to-date with industry trends, developers can ensure the performance, reliability, and security of their LLM applications. In the future, we can expect to see:\n", + "\n", + "* **Increased adoption of cloud-native observability tools**: As more developers adopt cloud-native observability tools, we can expect to see improved performance, reliability, and security in LLM applications.\n", + "* **Advancements in AI-specific observability platforms**: AI-specific observability platforms will continue to evolve, providing more advanced features and capabilities to support the unique needs of LLM applications.\n", + "* **Greater emphasis on security**: As LLMs become more widespread, the importance of security will only continue to grow, with observability and tracing playing a critical role in detecting and responding to security threats.\n", + "\n", + "In conclusion, observability and tracing are essential components of LLM application development, enabling developers to monitor, debug, and optimize their applications. By embracing cloud-native observability tools, leveraging breakthrough technologies, and staying up-to-date with industry trends, developers can ensure the performance, reliability, and security of their LLM applications.\n" + ] + } + ], + "source": [ + "import os\n", + "from huggingface_hub import get_token\n", + "from crewai_tools import SerperDevTool\n", + "from crewai import LLM, Agent, Task, Crew, Process\n", + "\n", + "# Define our LLM using HF's Serverless Inference API\n", + "llm = LLM(\n", + " model=\"huggingface/meta-llama/Llama-3.1-8B-Instruct\",\n", + " api_key=get_token(),\n", + " max_tokens=1024,\n", + ")\n", + "\n", + "# Define a tool for searching the web\n", + "os.environ[\"SERPER_API_KEY\"] = (\n", + " \"YOUR_SERPER_API_KEY\" # must set this value in your environment\n", + ")\n", + "search_tool = SerperDevTool()\n", + "\n", + "# Define your agents with roles and goals\n", + "researcher = Agent(\n", + " role=\"Researcher\",\n", + " goal=\"Conduct thorough research on up to date trends around a given topic.\",\n", + " backstory=\"\"\"You work at a leading tech think tank. You have a knack for dissecting complex data and presenting actionable insights.\"\"\",\n", + " verbose=True,\n", + " allow_delegation=False,\n", + " tools=[search_tool],\n", + " llm=llm,\n", + " max_iter=1,\n", + ")\n", + "writer = Agent(\n", + " role=\"Technical Writer\",\n", + " goal=\"Craft compelling content on a given topic.\",\n", + " backstory=\"\"\"You are a technical writer with a knack for crafting engaging and informative content.\"\"\",\n", + " llm=llm,\n", + " verbose=True,\n", + " allow_delegation=False,\n", + " max_iter=1,\n", + ")\n", + "\n", + "# Create tasks for your agents\n", + "task1 = Task(\n", + " description=\"\"\"Conduct comprehensive research and analysis of the importance of observability and tracing in LLM applications.\n", + " Identify key trends, breakthrough technologies, and potential industry impacts.\"\"\",\n", + " expected_output=\"Full analysis report in bullet points\",\n", + " agent=researcher,\n", + ")\n", + "\n", + "task2 = Task(\n", + " description=\"\"\"Using the insights provided, develop an engaging blog\n", + " post that highlights the importance of observability and tracing in LLM applications.\n", + " Your post should be informative yet accessible, catering to a tech-savvy audience.\n", + " Make it sound cool, avoid complex words so it doesn't sound like AI.\"\"\",\n", + " expected_output=\"Blog post of at least 3 paragraphs\",\n", + " agent=writer,\n", + ")\n", + "\n", + "# Instantiate your crew with a sequential process\n", + "crew = Crew(\n", + " agents=[researcher, writer],\n", + " tasks=[task1, task2],\n", + " verbose=True,\n", + " process=Process.sequential,\n", + ")\n", + "\n", + "# Get your crew to work!\n", + "result = crew.kickoff()\n", + "\n", + "print(\"------------ FINAL RESULT ------------\")\n", + "print(result)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After navigating back to the Phoenix dashboard, we can see the traces from our multi-agent application in the new project \"crewai\"!\n", + "\n", + "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/crew-ai-trace.png)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.11" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 8cfac706688cd2aff1a84d8171643a260ed11dcb Mon Sep 17 00:00:00 2001 From: Andrew Reed Date: Thu, 31 Oct 2024 09:51:15 -0400 Subject: [PATCH 2/6] cleanup and link fix --- notebooks/en/phoenix_observability_on_hf_spaces.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb index 8c9803c0..338abe29 100644 --- a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb +++ b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb @@ -21,13 +21,13 @@ "\n", "While Phoenix offers a [notebook-first option](https://docs.arize.com/phoenix/deployment/environments#notebooks) for local development, it can also be deployed as a [standalone dashboard via Docker](https://docs.arize.com/phoenix/deployment/environments#container). A long-running, hosted dashboard is a great way to provide a centralized view into your LLM application behavior, and to collaborate with your team. Hugging Face Spaces offers a simple way to host ML applications with optional, persistent storage, and it's support for custom Docker images makes it a great platform for hosting Phoenix - lets see how it works!\n", "\n", - "First, we'll [duplicate the demo space](https://huggingface.co/spaces/andrewrreed/phoenix-arize-observability-demo)\n", + "First, we'll [duplicate the demo space](https://huggingface.co/spaces/andrewrreed/phoenix-arize-observability-demo?duplicate=true)\n", "\n", "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/duplicate.png)\n", "\n", "We can configure the space to be private or public, and it can live in our user namespace, or in an organization namespace. We can use the default, free-tier CPU, and importantly, specify that we want to attach a persistent disk to the space.\n", "\n", - "> Note: In order for the tracing data to persist across Space, we _must_ configure a persistent disk, otherwise all data will be lost when the space is restarted. Configuring a persistent disk is a paid feature, and will incur a cost for the lifetime of the Space. In this case, we'll use the Small - 20GB disk option for $0.01 per hour.\n", + "> Note: In order for the tracing data to persist across Space restarts, we _must_ configure a persistent disk, otherwise all data will be lost when the space is restarted. Configuring a persistent disk is a paid feature, and will incur a cost for the lifetime of the Space. In this case, we'll use the Small - 20GB disk option for $0.01 per hour.\n", "\n", "After clicking \"Duplicate Space\", the Docker image will begin building. This will take a few minutes to complete, and then we'll see an empty Phoenix dashboard.\n", "\n", From a7834f5a80001a8e4dfdafe5447a68885f4529d5 Mon Sep 17 00:00:00 2001 From: Andrew Reed Date: Thu, 14 Nov 2024 12:23:51 -0500 Subject: [PATCH 3/6] add screenshots of UI in intro section --- notebooks/en/phoenix_observability_on_hf_spaces.ipynb | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb index 338abe29..d1027987 100644 --- a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb +++ b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb @@ -8,6 +8,15 @@ "\n", "_Authored by: [Andrew Reed](https://huggingface.co/andrewrreed)_\n", "\n", + "\n", + "#
\n", + "# \n", + "# \n", + "# \n", + "#
\n", + "\n", + "\n", + "\n", "[Phoenix](https://docs.arize.com/phoenix) is an open-source observability library by [Arize AI](https://arize.com/) designed for experimentation, evaluation, and troubleshooting. It allows AI Engineers and Data Scientists to quickly visualize their data, evaluate performance, track down issues, and export data to improve.\n", "\n", "In this notebook, we'll see how to deploy a Phoenix observability dashboard on [Hugging Face Spaces](https://huggingface.co/spaces) and configure it to automatically trace LLM calls, providing a comprehensive view into the inner workings of your LLM applications." From 985a5ecaf79e02eb152ec3b2e9596495f106bf04 Mon Sep 17 00:00:00 2001 From: Andrew Reed Date: Thu, 14 Nov 2024 12:27:18 -0500 Subject: [PATCH 4/6] fix UI screenshots formatting --- .../phoenix_observability_on_hf_spaces.ipynb | 22 +++++++++++++------ 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb index d1027987..02468fba 100644 --- a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb +++ b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb @@ -9,16 +9,24 @@ "_Authored by: [Andrew Reed](https://huggingface.co/andrewrreed)_\n", "\n", "\n", - "#
\n", - "# \n", - "# \n", - "# \n", - "#
\n", - "\n", - "\n", "\n", "[Phoenix](https://docs.arize.com/phoenix) is an open-source observability library by [Arize AI](https://arize.com/) designed for experimentation, evaluation, and troubleshooting. It allows AI Engineers and Data Scientists to quickly visualize their data, evaluate performance, track down issues, and export data to improve.\n", "\n", + "
\n", + "
\n", + " \n", + "

Tracing

\n", + "
\n", + "
\n", + " \n", + "

Datasets

\n", + "
\n", + "
\n", + " \n", + "

Experiments

\n", + "
\n", + "
\n", + "\n", "In this notebook, we'll see how to deploy a Phoenix observability dashboard on [Hugging Face Spaces](https://huggingface.co/spaces) and configure it to automatically trace LLM calls, providing a comprehensive view into the inner workings of your LLM applications." ] }, From b7b9007c3b1e7bd88bc01aa6c1d468d7346f0d53 Mon Sep 17 00:00:00 2001 From: Andrew Reed Date: Thu, 14 Nov 2024 12:27:40 -0500 Subject: [PATCH 5/6] replace "Note" with "[!TIP]" syntax --- notebooks/en/phoenix_observability_on_hf_spaces.ipynb | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb index 02468fba..428e3ce8 100644 --- a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb +++ b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb @@ -44,7 +44,7 @@ "\n", "We can configure the space to be private or public, and it can live in our user namespace, or in an organization namespace. We can use the default, free-tier CPU, and importantly, specify that we want to attach a persistent disk to the space.\n", "\n", - "> Note: In order for the tracing data to persist across Space restarts, we _must_ configure a persistent disk, otherwise all data will be lost when the space is restarted. Configuring a persistent disk is a paid feature, and will incur a cost for the lifetime of the Space. In this case, we'll use the Small - 20GB disk option for $0.01 per hour.\n", + "> [!TIP] In order for the tracing data to persist across Space restarts, we _must_ configure a persistent disk, otherwise all data will be lost when the space is restarted. Configuring a persistent disk is a paid feature, and will incur a cost for the lifetime of the Space. In this case, we'll use the Small - 20GB disk option for $0.01 per hour.\n", "\n", "After clicking \"Duplicate Space\", the Docker image will begin building. This will take a few minutes to complete, and then we'll see an empty Phoenix dashboard.\n", "\n", @@ -55,7 +55,7 @@ "\n", "Now that we have a running Phoenix dashboard, we can configure our application to automatically trace LLM calls using an [OpenTelemetry TracerProvider](https://docs.arize.com/phoenix/quickstart#connect-your-app-to-phoenix). In this example, we'll instrument our application using the OpenAI client library, and trace LLM calls made from the `openai` Python package to open LLMs running on [Hugging Face's Serverless Inference API](https://huggingface.co/docs/api-inference/en/index).\n", "\n", - "> Note: Phoenix supports tracing for [a wide variety of LLM frameworks](https://docs.arize.com/phoenix/tracing/integrations-tracing), including LangChain, LlamaIndex, AWS Bedrock, and more.\n", + "> [!TIP] Phoenix supports tracing for [a wide variety of LLM frameworks](https://docs.arize.com/phoenix/tracing/integrations-tracing), including LangChain, LlamaIndex, AWS Bedrock, and more.\n", "\n", "\n", "First, we need to install the necessary libraries:\n" @@ -297,7 +297,7 @@ "source": [ "Now we'll define a multi-agent application using CrewAI to research and write a blog post about the importance of observability and tracing in LLM applications.\n", "\n", - "> Note: This example is borrowed and modified from [here](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai)." + "> [!TIP] This example is borrowed and modified from [here](https://docs.arize.com/phoenix/tracing/integrations-tracing/crewai)." ] }, { @@ -580,11 +580,6 @@ "\n", "![](https://huggingface.co/datasets/huggingface/cookbook-images/resolve/main/crew-ai-trace.png)\n" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] } ], "metadata": { From 01f6450732011a621d6e84ae547a96f15915ef0c Mon Sep 17 00:00:00 2001 From: Andrew Reed Date: Thu, 14 Nov 2024 12:33:27 -0500 Subject: [PATCH 6/6] add note for python version and need for jupyter kernel restart --- notebooks/en/phoenix_observability_on_hf_spaces.ipynb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb index 428e3ce8..17206133 100644 --- a/notebooks/en/phoenix_observability_on_hf_spaces.ipynb +++ b/notebooks/en/phoenix_observability_on_hf_spaces.ipynb @@ -219,7 +219,9 @@ "source": [ "## Bonus: Tracing a multi-agent application with CrewAI\n", "\n", - "The real power of observability comes from being able to trace and inspect complex LLM workflows. In this example, we'll use [CrewAI](https://www.crewai.com/) to trace a multi-agent application." + "The real power of observability comes from being able to trace and inspect complex LLM workflows. In this example, we'll install and use [CrewAI](https://www.crewai.com/) to trace a multi-agent application.\n", + "\n", + "> [!TIP] The `openinference-instrumentation-crewai` package currently requires Python 3.10 or higher. After installing the `crewai` library, you may need to restart the notebook kernel to avoid errors." ] }, {