You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am attempting to build an agent that leverages the openai-realtime voice approach, but have frontend rpc calling be registered & unregistered dynamically... and in the middle of a single user's input.
From the docs: https://docs.livekit.io/agents/voice-agent/function-calling/ is it possible to have auto-discovery of the frontend RPC functions which are loaded. We want to dynamically load frontend actions based on which components are currently on the users screen. (e.g. the "insert text" function would only loaded up / shown to the agent if the "document" component was rendered, because there would be no way for the function to run unless the document component was available. Similarly the agent shouldn't "see" functions which can't be run on the frontend.)
Ideally, this auto-discovery of frontend actions should be dynamically updated in the middle of an agent run... e.g. the user asks for a document to be created and text to be inserted... the agent flow would be:
user asks "create a document and insert a poem into it" with no document UI component loaded
agent receives request and sees only 1 action available: "createDocument"
agent calls "createDocument" and waits for the response and re-checks what frontend actions may have been introduced... in this case the "createDocument" function created a document component which has an RPC register function... which then surfaces 2 actions available: "createDocument" and "insertText"
the agent would continue operating calling the "insertText" function and then voice response
The text was updated successfully, but these errors were encountered:
I am attempting to build an agent that leverages the openai-realtime voice approach, but have frontend rpc calling be registered & unregistered dynamically... and in the middle of a single user's input.
From the docs: https://docs.livekit.io/agents/voice-agent/function-calling/ is it possible to have auto-discovery of the frontend RPC functions which are loaded. We want to dynamically load frontend actions based on which components are currently on the users screen. (e.g. the "insert text" function would only loaded up / shown to the agent if the "document" component was rendered, because there would be no way for the function to run unless the document component was available. Similarly the agent shouldn't "see" functions which can't be run on the frontend.)
Ideally, this auto-discovery of frontend actions should be dynamically updated in the middle of an agent run... e.g. the user asks for a document to be created and text to be inserted... the agent flow would be:
The text was updated successfully, but these errors were encountered: