Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All the user to specify the stream argument of Agent._run_model somewhere #371

Open
nsfinkelstein opened this issue Oct 30, 2024 · 0 comments
Labels
enhancement Enhance an existing feature

Comments

@nsfinkelstein
Copy link

nsfinkelstein commented Oct 30, 2024

Enhancement Description

The default orchestrator calls Agent._run_model without a streaming argument: https://github.com/PrefectHQ/ControlFlow/blob/main/src/controlflow/orchestration/orchestrator.py#L375.

In effect, the default of True is always used unless the user constructs a new Orchestrator.

Use Case

Users may not be able to use a specific LLM API in streaming mode, but may have authorization to it otherwise.

Proposed Implementation

A stream parameter can be passed to the agent at construction and set on the object, and then Agent._run_model function could then use then use that value. Arguably whether or not an agent should be used in streaming mode is an Agent-level decision rather than a call-level decision, so it may make sense to specify it at construction rather than in the function that triggers that LLM call.

https://github.com/PrefectHQ/ControlFlow/blob/main/src/controlflow/agents/agent.py#L284

@nsfinkelstein nsfinkelstein added the enhancement Enhance an existing feature label Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhance an existing feature
Projects
None yet
Development

No branches or pull requests

1 participant