Using LiteLLM with other models

[chriswarkentin]

My work has set up LiteLLM with access to a bunch of different models but I'm having trouble with the Anthropic ones.
If I do something like:
fabric -s --pattern summarize --model azure-gpt-4o
it works very well but when I do
fabric -s --pattern summarize --model bedrock-anthropic-claude-3-5-sonnet
I get

Error: Error code: 500 - {'error': {'message': "BedrockException - BedrockException - bedrock does not support parameters: {'presence_penalty': 0.1, 'frequency_penalty': 0.1}. To drop these, set `litellm.drop_params=True` or for proxy:\n\n`litellm_settings:\n drop_params: true`\n\nNumber Retries = 1", 'type': None, 'param': None, 'code': 500}}
Error code: 500 - {'error': {'message': "BedrockException - BedrockException - bedrock does not support parameters: {'presence_penalty': 0.1, 'frequency_penalty': 0.1}. To drop these, set `litellm.drop_params=True` or for proxy:\n\n`litellm_settings:\n drop_params: true`\n\nNumber Retries = 1", 'type': None, 'param': None, 'code': 500}}

It would seem there might be some settings that can be adjustable on the LiteLLM server but I don't have access to it. I was playing around with the fabric source code seeing if I could prevent it from sending those parameters without success so far so if someone knows a way I would be most grateful.

I know that Fabric has direct support for Anthropic APIs so perhaps I could get my access token directly but I think my work wishes to use LiteLLM to track usage so they may not want to give it to me. Any suggestions?

[ishaan-jaff]

hi @chriswarkentin I'm the litellm maintainer

if you set this on your litellm config.yaml it will resolve the problem for your team

litellm_settings:
  drop_params: True

[krrishdholakia]

you can also specify params to drop https://docs.litellm.ai/docs/completion/drop_params#specify-params-to-drop

- litellm_params:
    api_base: my-base
    model: openai/my-model
    additional_drop_params: ["presence_penalty", "frequency_penalty"] # 👈 KEY CHANGE
  model_name: my-model

would it be easier if you could control this client-side? @chriswarkentin

[chriswarkentin]

Thanks for the replies. Definitely easier if I can control it client-side.

[chriswarkentin]

I had asked our administrator to make the change server side but had no response. I could ping him again though if there isn't any easy way to work around it from the client end.

[chriswarkentin]

Oh, actually my admin has applied the changes. Thanks for the help everyone!