litellm-proxy

LiteLLM Proxy Server

Use LiteLLM Proxy to Log OpenAI, Azure, Vertex, Bedrock (100+ LLMs) to Arize

Use LiteLLM Proxy for:

• Calling 100+ LLMs OpenAI, Azure, Vertex, Bedrock/etc. in the OpenAI ChatCompletions & Completions format
• Automatically Log all requests to Arize AI
• Proving a central self hosted server for calling LLMs + logging to Arize

Step 1. Create a Config for LiteLLM proxy

LiteLLM Requires a config with all your models define - we will call this file litellm_config.yaml

Detailed docs on how to setup litellm config - here

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/fake
      api_key: fake-key
      api_base: https://exampleopenaiendpoint-production.up.railway.app/

litellm_settings:
  success_callback: ["arize"] # 👈 Set Arize AI as a callback

environment_variables: # 👈 Set Arize AI env vars
    ARIZE_SPACE_KEY: "d0*****"
    ARIZE_API_KEY: "141a****"

Step 2. Start LiteLLM proxy

docker run \
    -v $(pwd)/litellm_config.yaml:/app/config.yaml \
    -p 4000:4000 \
    ghcr.io/berriai/litellm:main-latest \
    --config /app/config.yaml --detailed_debug

Step 3. Test it - Make /chat/completions request to LiteLLM proxy

curl -i http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-1234" \
  -d '{
    "model": "gpt-4",
    "messages": [
      {"role": "user", "content": "Hello, Claude gm!"}
    ]
}'

Expected output on Arize AI below:

Additional Resources

• LiteLLM Arize AI docs