Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update AmazonBedrockChatGenerator to use Converse API #1219

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

vblagoje
Copy link
Member

@vblagoje vblagoje commented Nov 26, 2024

Enhances the integration with Amazon Bedrock by using the new Converse API in AmazonBedrockChatGenerator. The modifications address the removal of outdated adapter structures and simplifies code greatly all while maintaining compatibility as much as possible.

What:

  • Removed the adapter file: The adapters.py file was deleted to eliminate deprecated structures related to chat model adapters.
  • Updated chat_generator.py:
    • Improved the model selection process and model ID usage.
    • Revised how system prompts and messages are prepared for sending to the models.
    • Added support for streaming responses and tool configurations in model requests.
  • Enhanced tests: Significant updates in test_chat_generator.py to include new model tests and validate functionality with the new configurations and expected responses.

How can it be used:

  • The system now supports:
    • Multiple LLMs from Amazon Bedrock with configurations for model-specific parameters.
    • Streaming responses during chat interactions.
  • Example code snippet to run a chat generator:
    from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockChatGenerator
    
    client = AmazonBedrockChatGenerator(model="anthropic.claude-3-5-sonnet-20240620-v1:0")
    response = client.run(
        messages=[ChatMessage.from_user("What's Natural Language Processing?")],
        generation_kwargs={"maxTokens": 512}
    )

How did you test it:

Conducted integration tests that focused on:

  • Ensuring the correct integration of the AmazonBedrockChatGenerator with various model configurations.
  • Validating the model's ability to handle both standard and streaming requests.
  • Mocked AWS session interactions to simulate real response cycles using the updated chat models.

Notes for the reviewer:

  • Attention should be given to changes affecting model naming conventions and interaction patterns, especially regarding tool usage in the LLM context.
  • Review the removal of the adapters.py file to understand how changes in architecture affect existing functionalities.

@github-actions github-actions bot added integration:amazon-bedrock type:documentation Improvements or additions to documentation labels Nov 26, 2024
@vblagoje
Copy link
Member Author

@julian-risch @silvanocerza do you remember who was giving us permissions for Amazon Bedrock models, was it @askainet ? We need to ping him to click through all these Amazon Bedrock models in this PR so they become available on our Haystack account...


# so far we've discovered these models support streaming and tool use
STREAMING_TOOL_MODELS = ["anthropic.claude-3-5-sonnet-20240620-v1:0", "cohere.command-r-plus-v1:0"]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These models in MODELS_TO_TEST, MODELS_TO_TEST_WITH_TOOLS and STREAMING_TOOL_MODELS @askainet 🙏 🙏

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Paul set us up 🚀

@vblagoje vblagoje marked this pull request as ready for review November 27, 2024 13:37
@vblagoje vblagoje requested a review from a team as a code owner November 27, 2024 13:37
@vblagoje vblagoje requested review from anakin87 and removed request for a team November 27, 2024 13:37
@vblagoje
Copy link
Member Author

@anakin87 would you please give it a rough pass first to make sure I'm not forgetting any major parts.

Copy link
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a cursory look and I like the change and the approach, also because we are deleting a lot of code :-)

Ping me when you need a more in-depth review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration:amazon-bedrock type:documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants