Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-discover new frontend function calls (rpc register, unregister) #1182

Open
bgeils opened this issue Dec 5, 2024 · 0 comments
Open

Auto-discover new frontend function calls (rpc register, unregister) #1182

bgeils opened this issue Dec 5, 2024 · 0 comments
Labels
question Further information is requested

Comments

@bgeils
Copy link

bgeils commented Dec 5, 2024

I am attempting to build an agent that leverages the openai-realtime voice approach, but have frontend rpc calling be registered & unregistered dynamically... and in the middle of a single user's input.

From the docs: https://docs.livekit.io/agents/voice-agent/function-calling/ is it possible to have auto-discovery of the frontend RPC functions which are loaded. We want to dynamically load frontend actions based on which components are currently on the users screen. (e.g. the "insert text" function would only loaded up / shown to the agent if the "document" component was rendered, because there would be no way for the function to run unless the document component was available. Similarly the agent shouldn't "see" functions which can't be run on the frontend.)

Ideally, this auto-discovery of frontend actions should be dynamically updated in the middle of an agent run... e.g. the user asks for a document to be created and text to be inserted... the agent flow would be:

  1. user asks "create a document and insert a poem into it" with no document UI component loaded
  2. agent receives request and sees only 1 action available: "createDocument"
  3. agent calls "createDocument" and waits for the response and re-checks what frontend actions may have been introduced... in this case the "createDocument" function created a document component which has an RPC register function... which then surfaces 2 actions available: "createDocument" and "insertText"
  4. the agent would continue operating calling the "insertText" function and then voice response
@bgeils bgeils added the question Further information is requested label Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant