From 9c7cb6fb70478202eebbfd49dbc1114648e39711 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 10 Oct 2024 18:16:40 -0700 Subject: [PATCH 1/9] Update GenAI events examples: add mermaid diagrams and fix small issues --- docs/gen-ai/gen-ai-events.md | 193 +++++++++++++++++++++-------------- 1 file changed, 118 insertions(+), 75 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index 44eeadc31d..70749d5207 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -186,15 +186,27 @@ SHOULD follow `gen_ai.{gen_ai.system}.*` naming pattern for system-specific even ### Chat completion -This example covers the following scenario: - -- user requests chat completion from OpenAI GPT-4 model for the following prompt: - - System message: `You're a friendly bot that answers questions about OpenTelemetry.` - - User message: `How to instrument GenAI library with OTel?` - -- The model responds with `"Follow GenAI semantic conventions available at opentelemetry.io."` message - -Span: +This is an example of telemetry generated for a chat completion call with system and user messages. + +```mermaid +%%{init: +{ + "sequence": { "messageAlign": "left", "htmlLabels":true }, + "themeVariables": { "noteBkgColor" : "green", "noteTextColor": "black", "activationBkgColor": "green", "htmlLabels":true } +} +}%% +sequenceDiagram + participant A as Application + participant I as Instrumented Client + participant M as Model + A->>+I: #U+200D + I->>M: gen_ai.system.message: You are a helpful bot
gen_ai.user.message: Tell me a joke about OpenTelemetry + Note left of I: GenAI Client span + I-->M: gen_ai.choice: Why did the developer bring OpenTelemetry to the party? Because it always knows how to trace the fun! + I-->>-A: #U+200D +``` + +**GenAI Client span:** | Attribute name | Value | |---------------------------------|--------------------------------------------| @@ -209,79 +221,98 @@ Span: | `gen_ai.usage.input_tokens` | `52` | | `gen_ai.response.finish_reasons`| `["stop"]` | -Events: +**Events:** + + -1. `gen_ai.system.message`. +1. `gen_ai.system.message` | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body | `{"content": "You're a friendly bot that answers questions about OpenTelemetry."}` | + | Event body (with content enabled) | `{"content": "You're a helpful bot"}` | 2. `gen_ai.user.message` | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body | `{"content":"How to instrument GenAI library with OTel?"}` | + | Event body (with content enabled) | `{"content":"Tell me a joke about OpenTelemetry"}` | 3. `gen_ai.choice` | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (with content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"Follow GenAI semantic conventions available at opentelemetry.io."}}` | + | Event body (with content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"Why did the developer bring OpenTelemetry to the party? Because it always knows how to trace the fun!"}}` | | Event body (without content) | `{"index":0,"finish_reason":"stop","message":{}}` | ### Tools -This example covers the following scenario: - -1. Application requests chat completion from OpenAI GPT-4 model and provides a function definition. - - - Application provides the following prompt: - - User message: `How to instrument GenAI library with OTel?` - - Application defines a tool (a function) names `get_link_to_otel_semconv` with single string argument named `semconv` - -2. The model responds with a tool call request which application executes -3. The application requests chat completion again now with the tool execution result - +This is an example of telemetry generated for a chat completion call with user message and function definition +that results in a model requesting application to call provided function. Application executes a function and +requests another completion now with the tool response. + +```mermaid +%%{init: +{ + "sequence": { "messageAlign": "left", "htmlLabels":true }, + "themeVariables": { "noteBkgColor" : "green", "noteTextColor": "black", "activationBkgColor": "green", "htmlLabels":true } +} +}%% +sequenceDiagram + participant A as Application + participant I as Instrumented Client + participant M as Model + A->>+I: #U+200D + I->>M: gen_ai.user.message: What's the weather in Paris? + Note left of I: GenAI Client span 1 + I-->M: gen_ai.choice: Call to the get_weather tool with Paris as the location argument. + I-->>-A: #U+200D + A -->> A: parse tool parameters
execute tool
update chat history + A->>+I: #U+200D + I->>M: gen_ai.user.message: What's the weather in Paris?
gen_ai.assistant.message: get_weather tool call
gen_ai.tool.message: rainy, 57°F + Note left of I: GenAI Client span 2 + I-->M: gen_ai.choice: The weather in Paris is rainy and overcast, with temperatures around 57°F. + I-->>-A: #U+200D +``` Here's the telemetry generated for each step in this scenario: -1. Chat completion resulting in a tool call. +**GenAI Client span 1:** - | Attribute name | Value | - |---------------------|-------------------------------------------------------| - | Span name | `"chat gpt-4"` | - | `gen_ai.system` | `"openai"` | - | `gen_ai.request.model`| `"gpt-4"` | - | `gen_ai.request.max_tokens`| `200` | - | `gen_ai.request.top_p`| `1.0` | - | `gen_ai.response.id`| `"chatcmpl-9J3uIL87gldCFtiIbyaOvTeYBRA3l"` | - | `gen_ai.response.model`| `"gpt-4-0613"` | - | `gen_ai.usage.output_tokens`| `17` | - | `gen_ai.usage.input_tokens`| `47` | - | `gen_ai.response.finish_reasons`| `["tool_calls"]` | +| Attribute name | Value | +|---------------------|-------------------------------------------------------| +| Span name | `"chat gpt-4"` | +| `gen_ai.system` | `"openai"` | +| `gen_ai.request.model`| `"gpt-4"` | +| `gen_ai.request.max_tokens`| `200` | +| `gen_ai.request.top_p`| `1.0` | +| `gen_ai.response.id`| `"chatcmpl-9J3uIL87gldCFtiIbyaOvTeYBRA3l"` | +| `gen_ai.response.model`| `"gpt-4-0613"` | +| `gen_ai.usage.output_tokens`| `17` | +| `gen_ai.usage.input_tokens`| `47` | +| `gen_ai.response.finish_reasons`| `["tool_calls"]` | + + **Events**: - Events parented to this span: + All the following events are parented to the **GenAI chat span 1**. - - `gen_ai.user.message` (not reported when capturing content is disabled) + 1. `gen_ai.user.message` (not reported when capturing content is disabled) | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body | `{"content":"How to instrument GenAI library with OTel?"}` | + | Event body | `{"content":"What's the weather in Paris?"}` | - - `gen_ai.choice` + 2. `gen_ai.choice` | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (with content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_link_to_otel_semconv","arguments":"{\"semconv\":\"GenAI\"}"},"type":"function"}]}` | - | Event body (without content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_link_to_otel_semconv"},"type":"function"}]}` | + | Event body (with content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"},"type":"function"}]}` | + | Event body (without content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather"},"type":"function"}]}` | -2. Application executes the tool call. Application may create span which is not covered by this semantic convention. -3. Final chat completion call +**GenAI Client span 2:** | Attribute name | Value | |---------------------------------|-------------------------------------------------------| @@ -296,55 +327,65 @@ Here's the telemetry generated for each step in this scenario: | `gen_ai.usage.input_tokens` | `47` | | `gen_ai.response.finish_reasons`| `["stop"]` | - Events parented to this span: - (in this example, the event content matches the original messages, but applications may also drop messages or change their content) + **Events**: - - `gen_ai.user.message` (not reported when capturing content is not enabled) + All the following events are parented to the **GenAI chat span 2**. + + In this example, the event content matches the original messages, but applications may also drop messages or change their content. + 1. `gen_ai.user.message` | Property | Value | |----------------------------------|------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body | `{"content":"How to instrument GenAI library with OTel?"}` | + | Event body | `{"content":"What's the weather in Paris?"}` | - - `gen_ai.assistant.message` + 2. `gen_ai.assistant.message` | Property | Value | |----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_link_to_otel_semconv","arguments":"{\"semconv\":\"GenAI\"}"},"type":"function"}]}` | - | Event body (content not enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_link_to_otel_semconv"},"type":"function"}]}` | + | Event body (content enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"},"type":"function"}]}` | + | Event body (content not enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather"},"type":"function"}]}` | - - `gen_ai.tool.message` + 3. `gen_ai.tool.message` | Property | Value | |----------------------------------|------------------------------------------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"content":"opentelemetry.io/semconv/gen-ai","id":"call_VSPygqKTWdrhaFErNvMV18Yl"}` | + | Event body (content enabled) | `{"content":"rainy, 57°F","id":"call_VSPygqKTWdrhaFErNvMV18Yl"}` | | Event body (content not enabled) | `{"id":"call_VSPygqKTWdrhaFErNvMV18Yl"}` | - - `gen_ai.choice` + 4. `gen_ai.choice` | Property | Value | |----------------------------------|-------------------------------------------------------------------------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"Follow OTel semconv available at opentelemetry.io/semconv/gen-ai"}}` | + | Event body (content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"The weather in Paris is rainy and overcast, with temperatures around 57°F."}}` | | Event body (content not enabled) | `{"index":0,"finish_reason":"stop","message":{}}` | ### Chat completion with multiple choices -This example covers the following scenario: - -- user requests 2 chat completion from OpenAI GPT-4 model for the following prompt: - - - System message: `You're a friendly bot that answers questions about OpenTelemetry.` - - User message: `How to instrument GenAI library with OTel?` - -- The model responds with two choices - - - `"Follow GenAI semantic conventions available at opentelemetry.io."` message - - `"Use OpenAI instrumentation library."` message - -Span: +This example covers the scenario when user requests model to generate two completions for the same prompt : + +```mermaid +%%{init: +{ + "sequence": { "messageAlign": "left", "htmlLabels":true }, + "themeVariables": { "noteBkgColor" : "green", "noteTextColor": "black", "activationBkgColor": "green", "htmlLabels":true } +} +}%% +sequenceDiagram + participant A as Application + participant I as Instrumented Client + participant M as Model + A->>+I: #U+200D + I->>M: gen_ai.system.message - "You are a helpful bot"
gen_ai.user.message - "Tell me a joke about OpenTelemetry" + Note left of I: GenAI Client span + I-->M: gen_ai.choice - Why did the developer bring OpenTelemetry to the party? Because it always knows how to trace the fun!
gen_ai.choice - Why did OpenTelemetry get promoted? It had great span of control! + I-->>-A: #U+200D +``` + +**GenAI Client Span**: | Attribute name | Value | |---------------------|--------------------------------------------| @@ -357,24 +398,26 @@ Span: | `gen_ai.response.model`| `"gpt-4-0613"` | | `gen_ai.usage.output_tokens`| `77` | | `gen_ai.usage.input_tokens`| `52` | -| `gen_ai.response.finish_reasons`| `["stop"]` | +| `gen_ai.response.finish_reasons`| `["stop", "stop"]` | + +**Events**: -Events: +All events are parented to the GenAI chat span above. 1. `gen_ai.system.message`: the same as in the [Chat Completion](#chat-completion) example -2. `gen_ai.user.message`: the same as in the previous example +2. `gen_ai.user.message`: the same as in the [Chat Completion](#chat-completion) example 3. `gen_ai.choice` | Property | Value | |------------------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"Follow GenAI semantic conventions available at opentelemetry.io."}}` | + | Event body (content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"Why did the developer bring OpenTelemetry to the party? Because it always knows how to trace the fun!"}}` | 4. `gen_ai.choice` | Property | Value | |------------------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"index":1,"finish_reason":"stop","message":{"content":"Use OpenAI instrumentation library."}}` | + | Event body (content enabled) | `{"index":1,"finish_reason":"stop","message":{"content":"Why did OpenTelemetry get promoted? It had great span of control!"}}` | [DocumentStatus]: https://opentelemetry.io/docs/specs/otel/document-status From 81838477ec5f036d8bf7d0b4f009c55d545ca49d Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 10 Oct 2024 18:55:20 -0700 Subject: [PATCH 2/9] up --- docs/gen-ai/gen-ai-events.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index 70749d5207..6c8cea4f97 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -99,7 +99,7 @@ The event name MUST be `gen_ai.system.message`. | Body Field | Type | Description | Examples | Requirement Level | |---|---|---|---|---| | `role` | string | The actual role of the message author as passed in the message. | `"system"`, `"instructions"` | `Conditionally Required`: if available and not equal to `system` | -| `content` | `AnyValue` | The contents of the system message. | `"You're a friendly bot that answers questions about OpenTelemetry."` | `Opt-In` | +| `content` | `AnyValue` | The contents of the system message. | `"You're a helpful bot"` | `Opt-In` | ## User event @@ -110,7 +110,7 @@ The event name MUST be `gen_ai.user.message`. | Body Field | Type | Description | Examples | Requirement Level | |---|---|---|---|---| | `role` | string | The actual role of the message author as passed in the message. | `"user"`, `"customer"` | `Conditionally Required`: if available and if not equal to `user` | -| `content` | `AnyValue` | The contents of the user message. | `What telemetry is reported by OpenAI instrumentations?` | `Opt-In` | +| `content` | `AnyValue` | The contents of the user message. | `What's the weather in Paris` | `Opt-In` | ## Assistant event @@ -121,8 +121,8 @@ The event name MUST be `gen_ai.assistant.message`. | Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | |--------------|--------------------------------|----------------------------------------|-------------------------------------------------|-------------------| | `role` | string | The actual role of the message author as passed in the message. | `"assistant"`, `"bot"` | `Conditionally Required`: if available and if not equal to `assistant` | -| `content` | `AnyValue` | The contents of the assistant message. | `Spans, events, metrics defined by the GenAI semantic conventions.` | `Opt-In` | -| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_link_to_otel_semconv", "arguments":{"semconv":"gen_ai"}}, "type":"function"}]` | `Conditionally Required`: if available | +| `content` | `AnyValue` | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | +| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":{"location":"Paris"}}, "type":"function"}]` | `Conditionally Required`: if available | ### `ToolCall` object @@ -130,14 +130,14 @@ The event name MUST be `gen_ai.assistant.message`. |------------|-----------------------------|------------------------------------|-------------------------------------------------|-------------------| | `id` | string | The id of the tool call | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | | `type` | string | The type of the tool | `function` | `Required` | -| `function` | [Function](#function-object)| The function that the model called | `{"name":"get_link_to_otel_semconv", "arguments":{"semconv":"gen_ai"}}` | `Required` | +| `function` | [Function](#function-object)| The function that the model called | `{"name":"get_weather", "arguments":{"location":"Paris"}}` | `Required` | ### `Function` object | Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | |-------------|------------|----------------------------------------|----------------------------|-------------------| -| `name` | string | The name of the function to call | `get_link_to_otel_semconv` | `Required` | -| `arguments` | `AnyValue` | The arguments to pass the the function | `{"semconv": "gen_ai"}` | `Opt-In` | +| `name` | string | The name of the function to call | `get_weather` | `Required` | +| `arguments` | `AnyValue` | The arguments to pass the the function | `{"location": "Paris"}` | `Opt-In` | ## Tool event @@ -148,7 +148,7 @@ The event name MUST be `gen_ai.tool.message`. | Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | |----------------|--------|-----------------------------------------------|---------------------------------|-------------------| | `role` | string | The actual role of the message author as passed in the message. | `"tool"`, `"function"` | `Conditionally Required`: if available and if not equal to `tool` | -| `content` | AnyValue | The contents of the tool message. | `opentelemetry.io` | `Opt-In` | +| `content` | AnyValue | The contents of the tool message. | `rainy, 57°F` | `Opt-In` | | `id` | string | Tool call that this message is responding to. | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | ## Choice event @@ -167,15 +167,15 @@ Choice event body has the following fields: |-----------------|----------------------------|-------------------------------------------------|----------------------------------------|-------------------| | `finish_reason` | string | The reason the model stopped generating tokens. | `stop`, `tool_calls`, `content_filter` | `Required` | | `index` | int | The index of the choice in the list of choices. | `1` | `Required` | -| `message` | [Message](#message-object) | GenAI response message | `{"content":"The OpenAI semantic conventions are available at opentelemetry.io"}` | `Recommended` | +| `message` | [Message](#message-object) | GenAI response message | `{"content":"The weather in Paris is rainy and overcast, with temperatures around 57°F"}` | `Recommended` | ### `Message` object | Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | |----------------|--------------------------------|-----------------------------------------------|---------------------------------|-------------------| | `role` | string | The actual role of the message author as passed in the message. | `"assistant"`, `"bot"` | `Conditionally Required`: if available and if not equal to `assistant` | -| `content` | `AnyValue` | The contents of the assistant message. | `Spans, events, metrics defined by the GenAI semantic conventions.` | `Opt-In` | -| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_link_to_otel_semconv", "arguments":"{\"semconv\":\"gen_ai\"}"}, "type":"function"}]` | `Conditionally Required`: if available | +| `content` | `AnyValue` | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | +| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":{"location":"Paris"}}, "type":"function"}]` | `Conditionally Required`: if available | ## Custom events From bc93678b7a284b63844ed62245f008dfbacb341b Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Thu, 10 Oct 2024 19:09:55 -0700 Subject: [PATCH 3/9] Yamlify GenAI events --- docs/gen-ai/gen-ai-events.md | 10 +- model/gen-ai/events.yaml | 242 +++++++++++++++++++++++++++++++++++ 2 files changed, 247 insertions(+), 5 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index 6c8cea4f97..1c0d6e0de9 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -110,7 +110,7 @@ The event name MUST be `gen_ai.user.message`. | Body Field | Type | Description | Examples | Requirement Level | |---|---|---|---|---| | `role` | string | The actual role of the message author as passed in the message. | `"user"`, `"customer"` | `Conditionally Required`: if available and if not equal to `user` | -| `content` | `AnyValue` | The contents of the user message. | `What's the weather in Paris` | `Opt-In` | +| `content` | `AnyValue` | The contents of the user message. | `What's the weather in Paris?` | `Opt-In` | ## Assistant event @@ -273,7 +273,7 @@ sequenceDiagram A->>+I: #U+200D I->>M: gen_ai.user.message: What's the weather in Paris?
gen_ai.assistant.message: get_weather tool call
gen_ai.tool.message: rainy, 57°F Note left of I: GenAI Client span 2 - I-->M: gen_ai.choice: The weather in Paris is rainy and overcast, with temperatures around 57°F. + I-->M: gen_ai.choice: The weather in Paris is rainy and overcast, with temperatures around 57°F I-->>-A: #U+200D ``` Here's the telemetry generated for each step in this scenario: @@ -309,7 +309,7 @@ Here's the telemetry generated for each step in this scenario: | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (with content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"},"type":"function"}]}` | + | Event body (with content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":{"location":"Paris"}},"type":"function"}]}` | | Event body (without content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather"},"type":"function"}]}` | **GenAI Client span 2:** @@ -344,7 +344,7 @@ Here's the telemetry generated for each step in this scenario: | Property | Value | |----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"},"type":"function"}]}` | + | Event body (content enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":{"location":"Paris"}},"type":"function"}]}` | | Event body (content not enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather"},"type":"function"}]}` | 3. `gen_ai.tool.message` @@ -360,7 +360,7 @@ Here's the telemetry generated for each step in this scenario: | Property | Value | |----------------------------------|-------------------------------------------------------------------------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"The weather in Paris is rainy and overcast, with temperatures around 57°F."}}` | + | Event body (content enabled) | `{"index":0,"finish_reason":"stop","message":{"content":"The weather in Paris is rainy and overcast, with temperatures around 57°F"}}` | | Event body (content not enabled) | `{"index":0,"finish_reason":"stop","message":{}}` | ### Chat completion with multiple choices diff --git a/model/gen-ai/events.yaml b/model/gen-ai/events.yaml index 72a4e692f0..c819d5a409 100644 --- a/model/gen-ai/events.yaml +++ b/model/gen-ai/events.yaml @@ -14,6 +14,26 @@ groups: brief: > This event describes the instructions passed to the GenAI system inside the prompt. extends: gen_ai.common.event.attributes + body: + id: gen_ai.system.message + requirement_level: opt_in + type: map + fields: + - id: content + type: undefined + stability: experimental + brief: > + The contents of the system message. + examples: ["You're a helpful bot"] + requirement_level: opt_in + - id: role + type: string + stability: experimental + brief: > + The actual role of the message author as passed in the message. + examples: ["system", "instruction"] + requirement_level: + conditionally_required: if available and not equal to `system`.` - id: gen_ai.user.message name: gen_ai.user.message @@ -22,6 +42,26 @@ groups: brief: > This event describes the prompt message specified by the user. extends: gen_ai.common.event.attributes + body: + id: gen_ai.user.message + requirement_level: opt_in + type: map + fields: + - id: content + type: undefined + stability: experimental + brief: > + The contents of the user message. + examples: ["What's the weather in Paris?"] + requirement_level: opt_in + - id: role + type: string + stability: experimental + brief: > + The actual role of the message author as passed in the message. + examples: ["user", "customer"] + requirement_level: + conditionally_required: if available and not equal to `user`. - id: gen_ai.assistant.message name: gen_ai.assistant.message @@ -30,6 +70,73 @@ groups: brief: > This event describes the assistant message passed to GenAI system or received from it. extends: gen_ai.common.event.attributes + body: + id: gen_ai.assistant.message + requirement_level: opt_in + type: map + fields: + - id: content + type: undefined + stability: experimental + brief: > + The contents of the tool message. + examples: ["The weather in Paris is rainy and overcast, with temperatures around 57°F"] + requirement_level: opt_in + - id: role + type: string + stability: experimental + brief: > + The actual role of the message author as passed in the message. + examples: ["assistant", "bot"] + requirement_level: + conditionally_required: if available and not equal to `assistant`. + - id: tool_calls + type: map # TODO: it's an array + stability: experimental + brief: > + The tool calls generated by the model, such as function calls. + requirement_level: + conditionally_required: if available + fields: + - id: id + type: string + stability: experimental + brief: > + The id of the tool call. + examples: ["call_mszuSIzqtI65i1wAUOE8w5H4"] + requirement_level: required + - id: type + type: enum + members: + - id: function + value: 'function' + brief: Function + stability: experimental + brief: > + The type of the tool. + examples: ["function"] + requirement_level: required + - id: function + type: map + stability: experimental + brief: > + The function call. + requirement_level: required + fields: + - id: name + type: string + stability: experimental + brief: > + The name of the function. + examples: ["get_weather"] + requirement_level: required + - id: arguments + type: undefined + stability: experimental + brief: > + The arguments of the function. + examples: ['{"location": "Paris"}'] + requirement_level: opt_in - id: gen_ai.tool.message name: gen_ai.tool.message @@ -38,6 +145,33 @@ groups: brief: > This event describes the tool or function response message. extends: gen_ai.common.event.attributes + body: + id: gen_ai.tool.message + requirement_level: opt_in + type: map + fields: + - id: content + type: undefined + stability: experimental + brief: > + The contents of the tool message. + examples: ["rainy, 57°F"] + requirement_level: opt_in + - id: role + type: string + stability: experimental + brief: > + The actual role of the message author as passed in the message. + examples: ["tool", "function"] + requirement_level: + conditionally_required: if available and not equal to `tool`. + - id: id + type: string + stability: experimental + brief: > + Tool call id that this message is responding to. + examples: ["call_mszuSIzqtI65i1wAUOE8w5H4"] + requirement_level: required - id: gen_ai.choice name: gen_ai.choice @@ -46,3 +180,111 @@ groups: brief: > This event describes the Gen AI response message. extends: gen_ai.common.event.attributes + body: + id: gen_ai.choice + requirement_level: opt_in + type: map + note: > + If GenAI model returns multiple choices, each choice SHOULD be recorded as an individual event. + When response is streamed, instrumentations that report response events MUST reconstruct and report + the full message and MUST NOT report individual chunks as events. + If the request to GenAI model fails with an error before content is received, + instrumentation SHOULD report an event with truncated content (if enabled). + If `finish_reason` was not received, it MUST be set to `error`. + fields: + - id: index + type: int + stability: experimental + brief: > + The index of the choice in the list of choices. + examples: [0, 1] + requirement_level: required + - id: finish_reason + type: enum + members: + - id: stop + value: 'stop' + brief: Stop + - id: tool_calls + value: 'tool_calls' + brief: Tool Calls + - id: content_filter + value: 'content_filter' + brief: Content Filter + - id: length + value: 'length' + brief: Length + - id: error + value: 'error' + brief: Error + stability: experimental + brief: > + The reason the model stopped generating tokens. + requirement_level: required + - id: message + type: map + stability: experimental + brief: > + GenAI response message. + requirement_level: recommended + fields: + - id: content + type: undefined + stability: experimental + brief: > + The contents of the assistant message. + examples: ["The weather in Paris is rainy and overcast, with temperatures around 57°F"] + requirement_level: opt_in + - id: role + type: string + stability: experimental + brief: > + The actual role of the message author as passed in the message. + examples: ["assistant", "bot"] + requirement_level: + conditionally_required: if available and not equal to `assistant`. + - id: tool_calls + type: map # TODO: it's an array + stability: experimental + brief: > + The tool calls generated by the model, such as function calls. + requirement_level: + conditionally_required: if available + fields: + - id: id + type: string + stability: experimental + brief: > + The id of the tool call. + examples: ["call_mszuSIzqtI65i1wAUOE8w5H4"] + requirement_level: required + - id: type + type: enum + members: + - id: function + value: 'function' + brief: Function + stability: experimental + brief: > + The type of the tool. + requirement_level: required + - id: function + type: map + stability: experimental + brief: > + The function that the model called. + requirement_level: required + fields: + - id: name + type: string + stability: experimental + brief: > + The name of the function. + examples: ["get_weather"] + requirement_level: required + - id: arguments + type: undefined + stability: experimental + brief: > + The arguments of the function. + requirement_level: opt_in From c62aa16a07d5412bd326c33cbb8253e34f70b820 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Fri, 18 Oct 2024 15:16:17 -0700 Subject: [PATCH 4/9] fix arguments - recommend using them as is, but keep type undefined for the future/customizations --- docs/gen-ai/gen-ai-events.md | 12 ++++++------ model/gen-ai/events.yaml | 19 ++++++++++++++++--- 2 files changed, 22 insertions(+), 9 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index 1c0d6e0de9..e3b1a40c0a 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -122,7 +122,7 @@ The event name MUST be `gen_ai.assistant.message`. |--------------|--------------------------------|----------------------------------------|-------------------------------------------------|-------------------| | `role` | string | The actual role of the message author as passed in the message. | `"assistant"`, `"bot"` | `Conditionally Required`: if available and if not equal to `assistant` | | `content` | `AnyValue` | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | -| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":{"location":"Paris"}}, "type":"function"}]` | `Conditionally Required`: if available | +| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":"{\"location\":\"Paris\"}"}, "type":"function"}]` | `Conditionally Required`: if available | ### `ToolCall` object @@ -130,14 +130,14 @@ The event name MUST be `gen_ai.assistant.message`. |------------|-----------------------------|------------------------------------|-------------------------------------------------|-------------------| | `id` | string | The id of the tool call | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | | `type` | string | The type of the tool | `function` | `Required` | -| `function` | [Function](#function-object)| The function that the model called | `{"name":"get_weather", "arguments":{"location":"Paris"}}` | `Required` | +| `function` | [Function](#function-object)| The function that the model called | `{"name":"get_weather", "arguments":"{\"location\":\"Paris\"}"}` | `Required` | ### `Function` object | Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | |-------------|------------|----------------------------------------|----------------------------|-------------------| | `name` | string | The name of the function to call | `get_weather` | `Required` | -| `arguments` | `AnyValue` | The arguments to pass the the function | `{"location": "Paris"}` | `Opt-In` | +| `arguments` | `AnyValue` | The arguments to pass the the function | `{\"location\":\"Paris\"}` | `Opt-In` | ## Tool event @@ -175,7 +175,7 @@ Choice event body has the following fields: |----------------|--------------------------------|-----------------------------------------------|---------------------------------|-------------------| | `role` | string | The actual role of the message author as passed in the message. | `"assistant"`, `"bot"` | `Conditionally Required`: if available and if not equal to `assistant` | | `content` | `AnyValue` | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | -| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":{"location":"Paris"}}, "type":"function"}]` | `Conditionally Required`: if available | +| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":"{\"location\":\"Paris\"}"}, "type":"function"}]` | `Conditionally Required`: if available | ## Custom events @@ -309,7 +309,7 @@ Here's the telemetry generated for each step in this scenario: | Property | Value | |---------------------|-------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (with content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":{"location":"Paris"}},"type":"function"}]}` | + | Event body (with content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"},"type":"function"}]}` | | Event body (without content) | `{"index":0,"finish_reason":"tool_calls","message":{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather"},"type":"function"}]}` | **GenAI Client span 2:** @@ -344,7 +344,7 @@ Here's the telemetry generated for each step in this scenario: | Property | Value | |----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| | `gen_ai.system` | `"openai"` | - | Event body (content enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":{"location":"Paris"}},"type":"function"}]}` | + | Event body (content enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"},"type":"function"}]}` | | Event body (content not enabled) | `{"tool_calls":[{"id":"call_VSPygqKTWdrhaFErNvMV18Yl","function":{"name":"get_weather"},"type":"function"}]}` | 3. `gen_ai.tool.message` diff --git a/model/gen-ai/events.yaml b/model/gen-ai/events.yaml index c819d5a409..669a2b16bb 100644 --- a/model/gen-ai/events.yaml +++ b/model/gen-ai/events.yaml @@ -134,8 +134,14 @@ groups: type: undefined stability: experimental brief: > - The arguments of the function. - examples: ['{"location": "Paris"}'] + The arguments of the function as provided in the LLM response. + note: > + Models usually return arguments as a JSON string. In this case, it's + RECOMMENDED to provide arguments as is without attempting to deserialize them. + + Semantic conventions for individual systems MAY specify a different type for + arguments field. + examples: ['{\"location\": \"Paris\"}'] requirement_level: opt_in - id: gen_ai.tool.message @@ -286,5 +292,12 @@ groups: type: undefined stability: experimental brief: > - The arguments of the function. + The arguments of the function as provided in the LLM response. + note: > + Models usually return arguments as a JSON string. In this case, it's + RECOMMENDED to provide arguments as is without attempting to deserialize them. + + Semantic conventions for individual systems MAY specify a different type for + arguments field. + examples: ['{\"location\": \"Paris\"}'] requirement_level: opt_in From db17a4048d7ffa5208297487d00f6cc7610f3f84 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Fri, 18 Oct 2024 15:20:06 -0700 Subject: [PATCH 5/9] auto-generate md for gen-ai events --- docs/gen-ai/gen-ai-events.md | 310 +++++++++++++++++++++++++++++------ 1 file changed, 261 insertions(+), 49 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index e3b1a40c0a..a305e72838 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -92,90 +92,302 @@ If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. ## System event -This event describes the instructions passed to the GenAI model. + + + + + + + +**Status:** ![Experimental](https://img.shields.io/badge/-experimental-blue) The event name MUST be `gen_ai.system.message`. -| Body Field | Type | Description | Examples | Requirement Level | -|---|---|---|---|---| -| `role` | string | The actual role of the message author as passed in the message. | `"system"`, `"instructions"` | `Conditionally Required`: if available and not equal to `system` | -| `content` | `AnyValue` | The contents of the system message. | `"You're a helpful bot"` | `Opt-In` | +This event describes the instructions passed to the GenAI system inside the prompt. + +| Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client or server instrumentation. [1] | `openai` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**[1]:** The `gen_ai.system` describes a family of GenAI models with specific model identified +by `gen_ai.request.model` and `gen_ai.response.model` attributes. + +The actual GenAI product may differ from the one identified by the client. +For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` +is set to `openai` based on the instrumentation's best knowledge. + +For custom model, a custom friendly name SHOULD be used. +If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. + +`gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `anthropic` | Anthropic | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `az.ai.inference` | Azure AI Inference | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `cohere` | Cohere | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `openai` | OpenAI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `vertex_ai` | Vertex AI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**Body fields:** + +| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| `content` | undefined | The contents of the system message. | `You're a helpful bot` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `role` | string | The actual role of the message author as passed in the message. | `system`; `instruction` | `Conditionally Required` if available and not equal to `system`.` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + + + + + ## User event -This event describes the prompt message specified by the user. + + + + + + + +**Status:** ![Experimental](https://img.shields.io/badge/-experimental-blue) The event name MUST be `gen_ai.user.message`. -| Body Field | Type | Description | Examples | Requirement Level | -|---|---|---|---|---| -| `role` | string | The actual role of the message author as passed in the message. | `"user"`, `"customer"` | `Conditionally Required`: if available and if not equal to `user` | -| `content` | `AnyValue` | The contents of the user message. | `What's the weather in Paris?` | `Opt-In` | +This event describes the prompt message specified by the user. + +| Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client or server instrumentation. [1] | `openai` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**[1]:** The `gen_ai.system` describes a family of GenAI models with specific model identified +by `gen_ai.request.model` and `gen_ai.response.model` attributes. + +The actual GenAI product may differ from the one identified by the client. +For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` +is set to `openai` based on the instrumentation's best knowledge. + +For custom model, a custom friendly name SHOULD be used. +If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. + +`gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `anthropic` | Anthropic | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `az.ai.inference` | Azure AI Inference | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `cohere` | Cohere | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `openai` | OpenAI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `vertex_ai` | Vertex AI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**Body fields:** + +| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| `content` | undefined | The contents of the user message. | `What's the weather in Paris?` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `role` | string | The actual role of the message author as passed in the message. | `user`; `customer` | `Conditionally Required` if available and not equal to `user`. | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + + + + + ## Assistant event -This event describes the assistant message. + + + + + + + +**Status:** ![Experimental](https://img.shields.io/badge/-experimental-blue) The event name MUST be `gen_ai.assistant.message`. -| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | -|--------------|--------------------------------|----------------------------------------|-------------------------------------------------|-------------------| -| `role` | string | The actual role of the message author as passed in the message. | `"assistant"`, `"bot"` | `Conditionally Required`: if available and if not equal to `assistant` | -| `content` | `AnyValue` | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | -| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":"{\"location\":\"Paris\"}"}, "type":"function"}]` | `Conditionally Required`: if available | +This event describes the assistant message passed to GenAI system or received from it. + +| Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client or server instrumentation. [1] | `openai` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**[1]:** The `gen_ai.system` describes a family of GenAI models with specific model identified +by `gen_ai.request.model` and `gen_ai.response.model` attributes. + +The actual GenAI product may differ from the one identified by the client. +For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` +is set to `openai` based on the instrumentation's best knowledge. + +For custom model, a custom friendly name SHOULD be used. +If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. + +`gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. -### `ToolCall` object +| Value | Description | Stability | +|---|---|---| +| `anthropic` | Anthropic | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `az.ai.inference` | Azure AI Inference | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `cohere` | Cohere | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `openai` | OpenAI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `vertex_ai` | Vertex AI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**Body fields:** -| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | -|------------|-----------------------------|------------------------------------|-------------------------------------------------|-------------------| -| `id` | string | The id of the tool call | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | -| `type` | string | The type of the tool | `function` | `Required` | -| `function` | [Function](#function-object)| The function that the model called | `{"name":"get_weather", "arguments":"{\"location\":\"Paris\"}"}` | `Required` | +| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| `content` | undefined | The contents of the tool message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `role` | string | The actual role of the message author as passed in the message. | `assistant`; `bot` | `Conditionally Required` if available and not equal to `assistant`. | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `tool_calls`: | map | The tool calls generated by the model, such as function calls. | | `Conditionally Required` if available | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `function`: | map | The function call. | | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|     `arguments` | undefined | The arguments of the function as provided in the LLM response. [1] | `{\"location\": \"Paris\"}` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|     `name` | string | The name of the function. | `get_weather` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `id` | string | The id of the tool call. | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `type` | enum | The type of the tool. | `function` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -### `Function` object +**[1]:** Models usually return arguments as a JSON string. In this case, it's RECOMMENDED to provide arguments as is without attempting to deserialize them. +Semantic conventions for individual systems MAY specify a different type for arguments field. -| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | -|-------------|------------|----------------------------------------|----------------------------|-------------------| -| `name` | string | The name of the function to call | `get_weather` | `Required` | -| `arguments` | `AnyValue` | The arguments to pass the the function | `{\"location\":\"Paris\"}` | `Opt-In` | +`type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `function` | Function | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + + + + + ## Tool event -This event describes the output of the tool or function submitted back to the model. + + + + + + + +**Status:** ![Experimental](https://img.shields.io/badge/-experimental-blue) The event name MUST be `gen_ai.tool.message`. -| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | -|----------------|--------|-----------------------------------------------|---------------------------------|-------------------| -| `role` | string | The actual role of the message author as passed in the message. | `"tool"`, `"function"` | `Conditionally Required`: if available and if not equal to `tool` | -| `content` | AnyValue | The contents of the tool message. | `rainy, 57°F` | `Opt-In` | -| `id` | string | Tool call that this message is responding to. | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | +This event describes the tool or function response message. + +| Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client or server instrumentation. [1] | `openai` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**[1]:** The `gen_ai.system` describes a family of GenAI models with specific model identified +by `gen_ai.request.model` and `gen_ai.response.model` attributes. + +The actual GenAI product may differ from the one identified by the client. +For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` +is set to `openai` based on the instrumentation's best knowledge. + +For custom model, a custom friendly name SHOULD be used. +If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. + +`gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `anthropic` | Anthropic | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `az.ai.inference` | Azure AI Inference | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `cohere` | Cohere | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `openai` | OpenAI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `vertex_ai` | Vertex AI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**Body fields:** + +| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| `content` | undefined | The contents of the tool message. | `rainy, 57°F` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `id` | string | Tool call id that this message is responding to. | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `role` | string | The actual role of the message author as passed in the message. | `tool`; `function` | `Conditionally Required` if available and not equal to `tool`. | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + + + + + ## Choice event -This event describes model-generated individual chat response (choice). -If GenAI model returns multiple choices, each choice SHOULD be recorded as an individual event. + + + + + + -When response is streamed, instrumentations that report response events MUST reconstruct and report the full message and MUST NOT report individual chunks as events. -If the request to GenAI model fails with an error before content is received, instrumentation SHOULD report an event with truncated content (if enabled). If `finish_reason` was not received, it MUST be set to `error`. +**Status:** ![Experimental](https://img.shields.io/badge/-experimental-blue) The event name MUST be `gen_ai.choice`. -Choice event body has the following fields: +This event describes the Gen AI response message. -| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | -|-----------------|----------------------------|-------------------------------------------------|----------------------------------------|-------------------| -| `finish_reason` | string | The reason the model stopped generating tokens. | `stop`, `tool_calls`, `content_filter` | `Required` | -| `index` | int | The index of the choice in the list of choices. | `1` | `Required` | -| `message` | [Message](#message-object) | GenAI response message | `{"content":"The weather in Paris is rainy and overcast, with temperatures around 57°F"}` | `Recommended` | +| Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client or server instrumentation. [1] | `openai` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**[1]:** The `gen_ai.system` describes a family of GenAI models with specific model identified +by `gen_ai.request.model` and `gen_ai.response.model` attributes. -### `Message` object +The actual GenAI product may differ from the one identified by the client. +For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` +is set to `openai` based on the instrumentation's best knowledge. + +For custom model, a custom friendly name SHOULD be used. +If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. -| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | -|----------------|--------------------------------|-----------------------------------------------|---------------------------------|-------------------| -| `role` | string | The actual role of the message author as passed in the message. | `"assistant"`, `"bot"` | `Conditionally Required`: if available and if not equal to `assistant` | -| `content` | `AnyValue` | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | -| `tool_calls` | [ToolCall](#toolcall-object)[] | The tool calls generated by the model, such as function calls. | `[{"id":"call_mszuSIzqtI65i1wAUOE8w5H4", "function":{"name":"get_weather", "arguments":"{\"location\":\"Paris\"}"}, "type":"function"}]` | `Conditionally Required`: if available | +`gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `anthropic` | Anthropic | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `az.ai.inference` | Azure AI Inference | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `cohere` | Cohere | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `openai` | OpenAI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `vertex_ai` | Vertex AI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**Body fields:** + +| Body Field | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | +|---|---|---|---|---|---| +| `finish_reason` | enum | The reason the model stopped generating tokens. | `stop`; `tool_calls`; `content_filter` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `index` | int | The index of the choice in the list of choices. | `0`; `1` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `message`: | map | GenAI response message. | | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `content` | undefined | The contents of the assistant message. | `The weather in Paris is rainy and overcast, with temperatures around 57°F` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `role` | string | The actual role of the message author as passed in the message. | `assistant`; `bot` | `Conditionally Required` if available and not equal to `assistant`. | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `tool_calls`: | map | The tool calls generated by the model, such as function calls. | | `Conditionally Required` if available | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `function`: | map | The function that the model called. | | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|     `arguments` | undefined | The arguments of the function as provided in the LLM response. [1] | `{\"location\": \"Paris\"}` | `Opt-In` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|     `name` | string | The name of the function. | `get_weather` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `id` | string | The id of the tool call. | `call_mszuSIzqtI65i1wAUOE8w5H4` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +|   `type` | enum | The type of the tool. | `function` | `Required` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +**[1]:** Models usually return arguments as a JSON string. In this case, it's RECOMMENDED to provide arguments as is without attempting to deserialize them. +Semantic conventions for individual systems MAY specify a different type for arguments field. + +`finish_reason` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `content_filter` | Content Filter | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `error` | Error | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `length` | Length | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `stop` | Stop | ![Experimental](https://img.shields.io/badge/-experimental-blue) | +| `tool_calls` | Tool Calls | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + +`type` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `function` | Function | ![Experimental](https://img.shields.io/badge/-experimental-blue) | + + + + + ## Custom events From 086be9050be8c560e62041a5a4ac534c0e1c435e Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Fri, 18 Oct 2024 15:27:37 -0700 Subject: [PATCH 6/9] update toc and headers --- docs/gen-ai/gen-ai-events.md | 66 ++++++------------------------------ 1 file changed, 11 insertions(+), 55 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index a305e72838..aca53ba45d 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -10,15 +10,11 @@ linkTitle: Generative AI events -- [Common attributes](#common-attributes) -- [System event](#system-event) -- [User event](#user-event) -- [Assistant event](#assistant-event) - - [`ToolCall` object](#toolcall-object) - - [`Function` object](#function-object) -- [Tool event](#tool-event) -- [Choice event](#choice-event) - - [`Message` object](#message-object) +- [Event: `gen_ai.system.message`](#event-gen_aisystemmessage) +- [Event: `gen_ai.user.message`](#event-gen_aiusermessage) +- [Event: `gen_ai.assistant.message`](#event-gen_aiassistantmessage) +- [Event: `gen_ai.tool.message`](#event-gen_aitoolmessage) +- [Event: `gen_ai.choice`](#event-gen_aichoice) - [Custom events](#custom-events) - [Examples](#examples) - [Chat completion](#chat-completion) @@ -29,7 +25,7 @@ linkTitle: Generative AI events GenAI instrumentations MAY capture user inputs sent to the model and responses received from it as [events](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.37.0/specification/logs/event-api.md). -> Note: +> [!NOTE]: > Event API is experimental and not yet available in some languages. Check [spec-compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#events) to see the implementation status in corresponding language. Instrumentations MAY capture inputs and outputs if and only if application has enabled the collection of this data. @@ -50,47 +46,7 @@ Telemetry consumers SHOULD expect to receive unknown body fields. Instrumentations SHOULD NOT capture undocumented body fields and MUST follow the documented defaults for known fields. Instrumentations MAY offer configuration options allowing to disable events or allowing to capture all fields. -## Common attributes - -The following attributes apply to all GenAI events. - - - - - - - - -| Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | -|---|---|---|---|---|---| -| [`gen_ai.system`](/docs/attributes-registry/gen-ai.md) | string | The Generative AI product as identified by the client or server instrumentation. [1] | `openai` | `Recommended` | ![Experimental](https://img.shields.io/badge/-experimental-blue) | - -**[1]:** The `gen_ai.system` describes a family of GenAI models with specific model identified -by `gen_ai.request.model` and `gen_ai.response.model` attributes. - -The actual GenAI product may differ from the one identified by the client. -For example, when using OpenAI client libraries to communicate with Mistral, the `gen_ai.system` -is set to `openai` based on the instrumentation's best knowledge. - -For custom model, a custom friendly name SHOULD be used. -If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. - -`gen_ai.system` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. - -| Value | Description | Stability | -|---|---|---| -| `anthropic` | Anthropic | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `az.ai.inference` | Azure AI Inference | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `cohere` | Cohere | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `openai` | OpenAI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | -| `vertex_ai` | Vertex AI | ![Experimental](https://img.shields.io/badge/-experimental-blue) | - - - - - - -## System event +## Event: `gen_ai.system.message` @@ -141,7 +97,7 @@ If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. -## User event +## Event: `gen_ai.user.message` @@ -192,7 +148,7 @@ If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. -## Assistant event +## Event: `gen_ai.assistant.message` @@ -258,7 +214,7 @@ Semantic conventions for individual systems MAY specify a different type for arg -## Tool event +## Event: `gen_ai.tool.message` @@ -310,7 +266,7 @@ If none of these options apply, the `gen_ai.system` SHOULD be set to `_OTHER`. -## Choice event +## Event: `gen_ai.choice` From 33b79b6de08217640e7b566aeb61f195954ff00b Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Fri, 18 Oct 2024 15:29:07 -0700 Subject: [PATCH 7/9] nit --- docs/gen-ai/gen-ai-events.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index aca53ba45d..e1672d94bb 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -25,7 +25,7 @@ linkTitle: Generative AI events GenAI instrumentations MAY capture user inputs sent to the model and responses received from it as [events](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.37.0/specification/logs/event-api.md). -> [!NOTE]: +> [!NOTE] > Event API is experimental and not yet available in some languages. Check [spec-compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#events) to see the implementation status in corresponding language. Instrumentations MAY capture inputs and outputs if and only if application has enabled the collection of this data. From 54e2c4f385b7d4d607389533bb9091d05ebb3ea5 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Fri, 18 Oct 2024 15:30:21 -0700 Subject: [PATCH 8/9] nits --- docs/gen-ai/gen-ai-events.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/gen-ai/gen-ai-events.md b/docs/gen-ai/gen-ai-events.md index e1672d94bb..1c66742ddc 100644 --- a/docs/gen-ai/gen-ai-events.md +++ b/docs/gen-ai/gen-ai-events.md @@ -391,8 +391,6 @@ sequenceDiagram **Events:** - - 1. `gen_ai.system.message` | Property | Value | @@ -444,6 +442,7 @@ sequenceDiagram I-->M: gen_ai.choice: The weather in Paris is rainy and overcast, with temperatures around 57°F I-->>-A: #U+200D ``` + Here's the telemetry generated for each step in this scenario: **GenAI Client span 1:** @@ -500,6 +499,7 @@ Here's the telemetry generated for each step in this scenario: All the following events are parented to the **GenAI chat span 2**. In this example, the event content matches the original messages, but applications may also drop messages or change their content. + 1. `gen_ai.user.message` | Property | Value | From 18ff568def2f3e56cf0d4cfe52545122a7b25319 Mon Sep 17 00:00:00 2001 From: Liudmila Molkova Date: Fri, 18 Oct 2024 15:32:01 -0700 Subject: [PATCH 9/9] changelog --- .chloggen/1469.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .chloggen/1469.yaml diff --git a/.chloggen/1469.yaml b/.chloggen/1469.yaml new file mode 100644 index 0000000000..7d7b4ba5cc --- /dev/null +++ b/.chloggen/1469.yaml @@ -0,0 +1,5 @@ +change_type: 'enhancement' +component: gen_ai +note: Yamlify gen_ai events and clean up examples. + +issues: [1469]