[Feature]: Context Caching for Vertex AI

[DreamGenX]

The Feature

It looks like Gemini context caching does not work when using Vertex AI.
Doing a cursory search, it looks like this part would need to be implemented to support Vertex AI:

litellm/litellm/llms/vertex_ai_and_google_ai_studio/gemini/transformation.py

Line 392 in c73ce95

await context_caching_endpoints.async_check_and_create_cache(

Motivation, pitch

Context cachin is already supported using the Gemini API and it's a good way to reduce costs.

Twitter / LinkedIn details

No response

[codenprogressive]

+1
Any updates on enabling context caching for LLMs (Gemini, Claude, etc) available through VertexAI?

[krrishdholakia]

@codenprogressive just tested claude on vertex ai - it doesn't look like prompt caching is available there yet.

I plan on picking up vertex ai context caching this week

[codenprogressive]

Hey @krrishdholakia, indeed Claude on Vertex still doesn't support prompt caching! (there is a feature request: anthropics/anthropic-sdk-python#653)

I think for now we can enable it for Gemini on VertexAI.

[emerzon]

AWS has started a preview for the prompt caching for Claude: https://pages.awscloud.com/promptcaching-Preview.html
Hope Vertex comes up next

[emerzon]

Vertex support is now live: https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude-prompt-caching#use_prompt_caching

[krrishdholakia]

@emerzon i believe this ticket is for vertex ai gemini, the vertex ai anthropic prompt caching issue is probably separate. i'm working on it now though, since we shouldn't be passing any extra headers to it.