[Bug]: Langchain callback handler KeyError

[areibman]

Contact Details

No response

📦 Package Version

0.3.18

🎞️ Framework Version

Langchain

🔎 Describe the Bug

From a discord user:

Hey, I'm wondering if anyone can sanity check my use of agentops. This is our first project that implements agentops, but after following the simple instructions, I'm getting the following errors:

[WARNING]    2024-12-05T19:16:38.185Z    fca3686a-3800-4dd4-8811-de12a5b744b9    Error in LangchainCallbackHandler.on_llm_start callback: AttributeError("'str' object has no attribute '__dict__'")

[WARNING]    2024-12-05T19:16:40.837Z    fca3686a-3800-4dd4-8811-de12a5b744b9    Error in LangchainCallbackHandler.on_llm_end callback: KeyError('ea5301bf-0bec-4250-afb8-d24397323492')

Installation looks like this, with using the latest 0.3.18:

from agentops.partners.langchain_callback_handler import LangchainCallbackHandler

handler = LangchainCallbackHandler(api_key=os.getenv("AGENTOPS_API_KEY", None))

...
        self.llm = ChatBedrock(
            model_id=self.model,
            client=self.bedrock_client,
            model_kwargs={
                "temperature": 0,
            },
            verbose=True,
            callbacks=[handler],
        )
...

The error is frustrating to debug. It appears like a dict is expected somewhere and a string is being passed to it, yet I'm so far unable to identify where this is the case. Ideas? Thanks in advance!

Full debug output:

<AGENTOPS_DEBUG_OUTPUT>
(DEBUG) 🖇 AgentOps: on_llm_start called with arguments:
(DEBUG) 🖇 AgentOps: run_id: UUID('ea5301bf-0bec-4250-afb8-d24397323492')
(DEBUG) 🖇 AgentOps: parent_run_id: UUID('fa477734-3772-460c-95d4-b3b2cc42aa8e')
(DEBUG) 🖇 AgentOps: tags: []
(DEBUG) 🖇 AgentOps: metadata: {'ls_model_name': 'anthropic.claude-3-5-haiku-20241022-v1:0',
 'ls_model_type': 'chat',
 'ls_provider': 'amazon_bedrock',
 'ls_stop': ['\nObservation:', '\n\tObservation:']}
(DEBUG) 🖇 AgentOps: invocation_params: {'_type': 'amazon_bedrock_chat',
 'guardrailIdentifier': None,
 'guardrailVersion': None,
 'model_id': 'anthropic.claude-3-5-haiku-20241022-v1:0',
 'provider': 'anthropic',
 'stop': ['\nObservation:', '\n\tObservation:'],
 'stream': False,
 'trace': None}
(DEBUG) 🖇 AgentOps: options: {'stop': ['\nObservation:', '\n\tObservation:']}
(DEBUG) 🖇 AgentOps: name: None
(DEBUG) 🖇 AgentOps: batch_size: 1
(DEBUG) 🖇 AgentOps: </AGENTOPS_DEBUG_OUTPUT>

[WARNING]    2024-12-05T19:16:38.185Z    fca3686a-3800-4dd4-8811-de12a5b744b9    Error in LangchainCallbackHandler.on_llm_start callback: AttributeError("'str' object has no attribute '__dict__'")
(DEBUG) 🖇 AgentOps:
<AGENTOPS_DEBUG_OUTPUT>
(DEBUG) 🖇 AgentOps: on_llm_end called with arguments:
(DEBUG) 🖇 AgentOps: run_id: UUID('ea5301bf-0bec-4250-afb8-d24397323492')
(DEBUG) 🖇 AgentOps: parent_run_id: UUID('fa477734-3772-460c-95d4-b3b2cc42aa8e')
(DEBUG) 🖇 AgentOps: tags: []
(DEBUG) 🖇 AgentOps: </AGENTOPS_DEBUG_OUTPUT>

[WARNING]    2024-12-05T19:16:40.837Z    fca3686a-3800-4dd4-8811-de12a5b744b9    Error in LangchainCallbackHandler.on_llm_end callback: KeyError('ea5301bf-0bec-4250-afb8-d24397323492')

full gist

https://gist.github.com/jcrowthe/86a2fcd0de7457672609b5319bce0daf

🤝 Contribution

• Yes, I'd be happy to submit a pull request with these changes.
• I need some guidance on how to contribute.
• I'd prefer the AgentOps team to handle this update.

[areibman]

Not able to reproduce exactly. We record the event but not the cost. This is separate but related to #514