🗺️ Vision / multi-modal

[mikeldking]

GPT 4o introduces a new message type that contains images and coded as either URL or base64 encoded.

example:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What’s in this image?"},
        {
          "type": "image_url",
          "image_url": {
            "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

print(response.choices[0])

https://platform.openai.com/docs/guides/vision

Milestone 1

• Vision support in instrumentations python for llama-index, openai, gemini, and langchain
• Eliminate performance degradations from base64 encoded payloads by allowing users to opt out
• Preliminary set of config flags to mask input output that could be sensitive info
• Create examples

Milestone N

• image synthesis apis such as DALL-E

Tracing

• [vision] semantic conventions #522
• [vision] [python] openai python SDK #523
• [vision][python] gemini scoping #539
• [vision] [javascript] openai node SDK #562
• [vision] [python] llama-index image messages parsing #582
• [vision] [python] langchain image messages parsing #538
• [vision] [javascript] langchain image messages parsing
• [vision] add a instrumentation side cap to base64 encoded images #557
• [vision] openinference configuration  #560
• [multi-modal] scope out video / audio semantic conventions #1081
• [vision] phoenix server fixtures for image traces #567

Instrumenation

Testing

• [instrumentation-python] crewai testing #872

Image tracing

• [instrumentation-python] bedrock support for images #707
• [instrumentation-python] crewai support for images #708
• [instrumentation-python] dspy support for images #709
• [instrumentation-python] guardrails support for images #710
• [instrumentation-python] haystack support for images #711
• [instrumentation-python] langchain support for image #631
• [instrumentation-python] litellm support for images #712
• [instrumentation-python] llama-index support for images #713
• [instrumentation-python] mistralai support for images #714
• [instrumentation-python] openai support for images #715
• [instrumentation-python] vertexai support for images #716
• [instrumentation-python] groq support for images #717

Context Attributes

• [instrumentation-python] context attributes propagation for bedrock #718
• [instrumentation-python] context attributes propagation for crewai #719
• [instrumentation-python] context attributes propagation for dspy #720
• [instrumentation-python] context attributes propagation for guardrails #721
• [instrumentation-python] context attributes propagation for haystack #722
• [instrumentation-python] context attributes propagation for langchain #723
• [instrumentation-python] context attributes propagation for litellm #724
• [instrumentation-python] context attributes propagation for llama-index #725
• [instrumentation-python] context attributes propagation for mistralai #726
• [instrumentation-python] context attributes propagation for openai #727
• [instrumentation-python] context attributes propagation for vertexai #728
• [instrumentation-python] context attributes propagation for groq #729

Config

• [instrumentation-python] trace config propagation for bedrock #730
• [instrumentation-python] trace config propagation for crewai #731
• [instrumentation-python] trace config propagation for dspy #733
• [instrumentation-python] trace config propagation for guardrails #732
• [instrumentation-python] trace config propagation for haystack #734
• [instrumentation-python] trace config propagation for langchain #633
• [instrumentation-python] trace config propagation for litellm #737
• [instrumentation-python] trace config propagation for llama-index #632
• [instrumentation-python] trace config propagation for mistralai #736
• [instrumentation-python] trace config propagation for openai #634
• [instrumentation-python] trace config propagation for vertex #635
• [instrumentation-python] trace config propagation for groq #735

Suppress Tracing

• [instrumentation-python] suppress tracing option for crewai #748
• [instrumentation-python] suppress tracing option for litellm #749

UI / Javascript

• [ui][vision] display images in the phoenix UI #568
• [vision] instrumentation for openai-node SDK #704
• [instrumentation-js] TraceConfig for openinference-core #821
• [vision][ui] better display for images (max size, expand, grid view) #956
• [vision] instrumentation for langchain-js

Testing

• [vision][testing] benchmark different image sizes, load #558

Documentation

• [vision][use-case] create an image tracing use-case example under openinference #561
• [trace-config][docs] traceconfig section in every instrumentation README #786
• [trace-config][docs] traceconfig, supress_tracing and context attributes in the development guide #787
• [trace-config][docs] phoenix privacy guide #788
• [vision][video] launch video for images #833

Evals

• [vision][evals] llm_classify with image chat completions #574

[mikeldking]

Example vLLM client that should also support vision

class VLMClient:
    def __init__(self, vlm_model: str = VLM_MODEL, vllm_url: str = VLLM_URL):
        self._vlm_model = vlm_model
        self._vllm_client = httpx.AsyncClient(base_url=vllm_url)

        if VLLM_HEALTHCHECK:
            wait_for_ready(
                server_url=vllm_url,
                wait_seconds=VLLM_READY_TIMEOUT,
                health_endpoint="health",
            )

    @property
    def vlm_model(self) -> str:
        return self._vlm_model

    async def __call__(
        self,
        prompt: str,
        image_bytes: bytes | None = None,
        image_filetype: filetype.Type | None = None,
        max_tokens: int = 10,
    ) -> str:
        # Assemble the message content
        message_content: list[dict[str, str | dict]] = [
            {
                "type": "text",
                "text": prompt,
            }
        ]

        if image_bytes is not None:
            if image_filetype is None:
                image_filetype = filetype.guess(image_bytes)

            if image_filetype is None:
                raise ValueError("Could not determine image filetype")

            if image_filetype not in ALLOWED_IMAGE_TYPES:
                raise ValueError(
                    f"Image type {image_filetype} is not supported. Allowed types: {ALLOWED_IMAGE_TYPES}"
                )

            image_b64 = base64.b64encode(image_bytes).decode("utf-8")
            message_content.append(
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:{image_filetype.mime};base64,{image_b64}",
                    },
                }
            )

        # Put together the request payload
        payload = {
            "model": self.vlm_model,
            "messages": [{"role": "user", "content": message_content}],
            "max_tokens": max_tokens,
            # "logprobs": True,
            # "top_logprobs": 1,
        }

        response = await self._vllm_client.post("/v1/chat/completions", json=payload)
        response = response.json()
        response_text: str = (
            response.get("choices")[0].get("message", {}).get("content", "").strip()
        )

        return response_text

[mikeldking]

Closing as completed as images is complete. Audio will come as part of openAI realtime instrumentation