Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in connecting remote user #9

Open
akaminnovations opened this issue Nov 16, 2024 · 29 comments
Open

Error in connecting remote user #9

akaminnovations opened this issue Nov 16, 2024 · 29 comments

Comments

@akaminnovations
Copy link

Log as below

ERROR - Error waiting for remote user
ERROR - Error running agent

@satizzz39
Copy link

satizzz39 commented Nov 16, 2024

While running python -m realtime_agent.main server, after hit [start_agent] -> (http://localhost:8080/start_agent), I'm facing issue with remote user joining in my channel.
from logs I could see my OpenAI session created, Web socket connected successfully, MY OPENAI key has right access to the model, also MY Appid, and certificate id also mentioned properly in .env file. here I attcahed some logs and traces tried in multiple ways Agora devs please help me to resolve this issue.
Screenshot 2024-11-16 165846
image

Python version: 3.12

agora_openai_remote_user_not_joined_traces.txt
@plutoless @digitallysavvy @LichKing-2234 @sidsharma27

cc: @akaminnovations

@digitallysavvy
Copy link
Contributor

Do you see any errors on the front-end console?

@satizzz39
Copy link

@digitallysavvy , No I just run the backend server in localhost and invoked http://localhost:8080/start_agent via postman, api receives success response as "Agent Created", post that in backend console the attached error came from Agora.

@digitallysavvy
Copy link
Contributor

Do you have a live user in the Agora channel?

@satizzz39
Copy link

After channel created It authorising OPENAI session are created, But conversional AI not joined in call , Even live user in the channel or not Same Timeout Issue. I tried with both

@plutoless
Copy link
Contributor

@satizzz39 do you have the channel name you were using?

@satizzz39
Copy link

satizzz39 commented Nov 18, 2024

@plutoless , These are the channel names I passed for local host start_agent endpoint

90bb692f-1d9b-4b3a-b492-4df7d2064i2k
60bb692f-1d9b-4b3a-b492-4df7d2064i2k
test

@satizzz39
Copy link

@plutoless @digitallysavvy , Any update on this please ?

@plutoless
Copy link
Contributor

hi @satizzz39 i checked the logs and it seems no user is joinning as client. this is the reason, the agent will quit if no user exists in channel for 15 seconds. which client do you use to join agora rtc?

@satizzz39
Copy link

Yes after ws scocket connected and session created, I'm expecting OPENAI gpt-4o-realtime-preview conversational AI needs to join that channel, but its not joining.
@plutoless

@plutoless
Copy link
Contributor

@satizzz39 ai is joinning the channel. but you will need to have a client joinning to send voice to it.
so it's like
client -> rtc network -> python ai agent -> openai

the python ai agent joined successfully according to the logs i find for your channel. it will wait for client end user to join to talk to it and will quit if no human joins after 15 secs.

@satizzz39
Copy link

@plutoless
Even I joined as user this side from the channel, the interface given by agora for frontend testing mentined in README, https://webdemo.agora.io/basicVoiceCall/index.html, Same error I got I joined and in the channel and spoke Same issue here is the logs and traces.

App id: 99a9308db8364fe49bab7159f05c4c10
channel id: 10bb692f-1d9b-4b3a-b492-4df7d2064i2k
image
image

@plutoless
Copy link
Contributor

from the logs it seems you failed to join web. could you pls take a look at the browser console logs?

@satizzz39
Copy link

@plutoless , could please check the root cause of this failure to join in channel, All I followed exact steps in documentations

image
image

@plutoless
Copy link
Contributor

@satizzz39 you will need to use a token instead of your appid when you join the channel on web.
i think on console you should be able to easily generate a temp token. the error message basically tells you are not using a correct token.

@plutoless
Copy link
Contributor

@satizzz39
Copy link

@plutoless , Generated temp token and joined to channel successfully, BUT same error in backend.

token used: 007eJxTYPiiPqnP+oC6g9tCdmdtibojx9k8Nbfuv3JH28HW4MrGbGkFBgtzM1MDU0sLE6OUZBNj8zSLxOQ0SwNjk6Qkc0NDS0Oz9WX26Q2BjAzJx+sZGKEQxFdhMDRISjKzNErTNUyxTNI1STJO1E0ysTTSNUlJM08xMjAzyTTKZmAAAKLDJNA=

channel id: 10bb692f-1d9b-4b3a-b492-4df7d2064i2k

image
image

@plutoless
Copy link
Contributor

@satizzz39 you are using different project for agent and client. you will need to use the same project for them.

@plutoless
Copy link
Contributor

you are using InterVu project for web, and My New Project for python agent

@plutoless
Copy link
Contributor

you have configured appid and certificate in .env, which is from project My New Project
on console you generate a temp token, which is from InterVu project

You just need to generate temp token for project My New Project and use that to join for web

@satizzz39
Copy link

satizzz39 commented Nov 21, 2024

@plutoless , configured correct App id, and token From console I confirmed user joined
image
image

Now user joined but error in further steps, Could please check this.

@plutoless
Copy link
Contributor

it's working fine for me... just tested

could you pls change verbose in this line to True and try again?

@plutoless
Copy link
Contributor

and share me the logs if possible

@satizzz39
Copy link

@plutoless , changed verbose=True

lot of event messages displayed like this below, no responses for my audio and after few mins exception message occured.

2024-11-21 18:39:08,063 - realtime_agent.realtime.connection - INFO - -> {"event_id": "a930f86e-8ba4-4a44-b19d-840e9312fac3", "audio": "NAQVBPEDkAPzAn4ChgL2An0D9wNxBLMEUARBA/8B7AALAGX/Lv9v/+P/PgB5AMEAMQGsAQwCQwI6AsQB9QAmAH7/9/7X/nn/tQAIAjQDQATpBLoEzAPOAi4CxQF+AZwB...", "type": "input_audio_buffer.append"}

image

@plutoless
Copy link
Contributor

could you pls share me the full logs?

@satizzz39
Copy link

backend terminal
log_traces.txt

front end console logs

00:22:44:662 Agora-SDK [DEBUG]: current web page is hidden
AgoraRTC_N.js:5 00:22:45:841 Agora-SDK [DEBUG]: current web page is visible
AgoraRTC_N.js:5 00:22:47:588 Agora-SDK [DEBUG]: current web page is hidden
AgoraRTC_N.js:5 00:22:56:480 Agora-SDK [DEBUG]: current web page is visible
AgoraRTC_N.js:5 00:22:56:979 Agora-SDK [DEBUG]: current web page is hidden
AgoraRTC_N.js:5 00:23:00:206 Agora-SDK [DEBUG]: current web page is visible
AgoraRTC_N.js:5 00:23:02:729 Agora-SDK [DEBUG]: [client-ccdab] set avoidJoinStart to 1732215182729
AgoraRTC_N.js:5 00:23:02:730 Agora-SDK [INFO]: [client-ccdab] signal connection state change: DISCONNECTED -> CONNECTING
AgoraRTC_N.js:5 00:23:02:730 Agora-SDK [INFO]: [apiInvoke-77][track-mic-caa9c222][createMicrophoneAudioTrack] start [{…}]
AgoraRTC_N.js:5 00:23:02:730 Agora-SDK [INFO]: start create microphone audio track with config {"encoderConfig":"music_standard"} trackId track-mic-caa9c222
AgoraRTC_N.js:5 00:23:02:731 Agora-SDK [DEBUG]: [track-mic-caa9c222] GetUserMedia {"audio":{"channelCount":1,"sampleRate":48000}}
AgoraRTC_N.js:5 00:23:02:731 Agora-SDK [INFO]: [client-ccdab] start join channel 40bb692f-1d9b-4b3a-b492-4df7d2064i2k, join number: 10
AgoraRTC_N.js:5 00:23:02:731 Agora-SDK [INFO]: [apiInvoke-78][client-ccdab][Client.join] start (4) ['8765059842dc437f8acf9034bb711916', '40bb692f-1d9b-4b3a-b492-4df7d2064i2k', '007eJxTYChpjBC+FiP9dQ5v2RGxfcFPu3bIbr5eFdG3lF1rvtO…UpmuYYpmka5JknKibZGJppGuSkmaeYmRgZpJplM3AAABkACaP', null]
AgoraRTC_N.js:5 00:23:02:732 Agora-SDK [DEBUG]: [client-ccdab] new Session 5C7D48B3A8C6FDD911A968BB6FD16670
AgoraRTC_N.js:5 00:23:02:733 Agora-SDK [DEBUG]: disable reportProxyServerurl: undefined
AgoraRTC_N.js:5 00:23:02:733 Agora-SDK [DEBUG]: [client-ccdab] Connect to choose_server: https://webrtc2-ap-web-1.agora.io/api/v2/transpond/webrtc?v=2
AgoraRTC_N.js:5 00:23:02:734 Agora-SDK [DEBUG]: [client-ccdab] Connect to choose_server: https://webrtc2-2.ap.sd-rtn.com/api/v2/transpond/webrtc?v=2
AgoraRTC_N.js:5 00:23:02:742 Agora-SDK [DEBUG]: [lock-track-mic-caa9c222-83] is created.
AgoraRTC_N.js:5 00:23:02:742 Agora-SDK [INFO]: [apiInvoke-77][track-mic-caa9c222][createMicrophoneAudioTrack] onSuccess
AgoraRTC_N.js:5 00:23:02:742 Agora-SDK [INFO]: create microphone audio track success, trackId: track-mic-caa9c222
AgoraRTC_N.js:5 00:23:02:835 Agora-SDK [DEBUG]: [client-ccdab] set joinGatewayStartTime at 1732215182835
AgoraRTC_N.js:5 00:23:02:835 Agora-SDK [DEBUG]: [client-ccdab] use websocket join uid 0
AgoraRTC_N.js:5 00:23:02:835 Agora-SDK [DEBUG]: [client-ccdab][lock-websocket-6] is locked, current queue 1.
AgoraRTC_N.js:5 00:23:02:836 Agora-SDK [DEBUG]: [gateway-client-ccdab] start connect, url: {host: '129-227-179-74.edge.agora.io', port: '4702'}
AgoraRTC_N.js:5 00:23:02:836 Agora-SDK [DEBUG]: [choose-best-ws] currentDomain: {host: '129-227-179-74.edge.agora.io', port: '4702'} , domains: (2) ['edge.agora.io', 'edge.sd-rtn.com'] total: 3 current: 1
AgoraRTC_N.js:5 00:23:02:836 Agora-SDK [DEBUG]: [choose-best-ws client-ccdab 9] 0ms: createWebSocket: {host: '129-227-179-74.edge.agora.io', port: '4702'} {isTry443Port: false, hasTimeoutDetection: false}
AgoraRTC_N.js:5 00:23:02:836 Agora-SDK [DEBUG]: [choose-best-ws client-ccdab 9] 0ms: ws is connecting: wss://129-227-179-74.edge.agora.io:4702/
AgoraRTC_N.js:5 00:23:02:837 Agora-SDK [DEBUG]: [choose-best-ws client-ccdab 9] 1ms: ws is connecting: wss://129-227-179-74.edge.sd-rtn.com:4702/
AgoraRTC_N.js:5 00:23:02:837 Agora-SDK [DEBUG]: [client-ccdab][lock-config-distribute-7] is locked, current queue 1.
AgoraRTC_N.js:5 00:23:02:879 Agora-SDK [DEBUG]: [config-distribute] get config distribute {}
AgoraRTC_N.js:5 00:23:02:879 Agora-SDK [DEBUG]: Caching global parameters {}
AgoraRTC_N.js:5 00:23:02:879 Agora-SDK [DEBUG]: [client-ccdab][lock-config-distribute-7] is unlocked, current queue 0.
AgoraRTC_N.js:5 00:23:02:914 Agora-SDK [DEBUG]: [choose-best-ws client-ccdab 9] 78ms: onopen: ws wss://129-227-179-74.edge.sd-rtn.com:4702/ open cost 78ms
AgoraRTC_N.js:5 WebSocket connection to 'wss://129-227-179-74.edge.agora.io:4702/' failed: WebSocket is closed before the connection is established.
(anonymous) @ AgoraRTC_N.js:5
o.forEach.e.onopen @ AgoraRTC_N.js:5
AgoraRTC_N.js:5 00:23:02:915 Agora-SDK [DEBUG]: [choose-best-ws client-ccdab 9] 79ms: close backup websocket: wss://129-227-179-74.edge.agora.io:4702/
AgoraRTC_N.js:5 00:23:02:915 Agora-SDK [DEBUG]: [gateway-client-ccdab] websocket opened: wss://129-227-179-74.edge.sd-rtn.com:4702/
AgoraRTC_N.js:5 00:23:02:915 Agora-SDK [DEBUG]: websocket reset reconnect count, reason: opened
AgoraRTC_N.js:5 00:23:02:915 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is locked, current queue 1. From P2PChannel.startP2PConnection
AgoraRTC_N.js:5 00:23:02:915 Agora-SDK [DEBUG]: [client-ccdab][lock-websocket-6] is unlocked, current queue 0.
AgoraRTC_N.js:5 00:23:02:915 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is created.
AgoraRTC_N.js:5 00:23:02:916 Agora-SDK [DEBUG]: Cannot recognized as ip address: 129-227-108-211.edge.agora.io, use as host
AgoraRTC_N.js:5 00:23:02:916 Agora-SDK [DEBUG]: Cannot recognized as ip address: 128-1-105-115.edge.agora.io, use as host
AgoraRTC_N.js:5 00:23:02:916 Agora-SDK [DEBUG]: Cannot recognized as ip address: 129-227-217-132.edge.agora.io, use as host
[NEW] Explain Console errors by using Copilot in Edge: click

     to explain an error. 
    Learn more
    Don't show again

AgoraRTC_N.js:5 00:23:02:919 Agora-SDK [DEBUG]: Cannot recognized as ip address: 129-227-108-211.edge.agora.io, use as host
AgoraRTC_N.js:5 00:23:02:919 Agora-SDK [DEBUG]: Cannot recognized as ip address: 128-1-105-115.edge.agora.io, use as host
AgoraRTC_N.js:5 00:23:02:919 Agora-SDK [DEBUG]: Cannot recognized as ip address: 129-227-217-132.edge.agora.io, use as host
AgoraRTC_N.js:5 00:23:02:930 Agora-SDK [DEBUG]: raising H264 baseline profile priority
AgoraRTC_N.js:5 00:23:02:932 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is unlocked, current queue 0. From P2PChannel.startP2PConnection
AgoraRTC_N.js:5 00:23:03:19 Agora-SDK [DEBUG]: Using remote candidate from AP 129.227.179.74:4702
AgoraRTC_N.js:5 00:23:03:19 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is locked, current queue 1. From P2PChannel.connect
AgoraRTC_N.js:5 00:23:03:20 Agora-SDK [INFO]: [client-ccdab] signal connection state change: CONNECTING -> CONNECTED
AgoraRTC_N.js:5 00:23:03:20 Agora-SDK [DEBUG]: [client-ccdab] websocket join uid 559372186 cost 185
AgoraRTC_N.js:5 00:23:03:21 Agora-SDK [DEBUG]: [client-ccdab] Connected to gateway server
AgoraRTC_N.js:5 00:23:03:21 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is locked, current queue 1. From P2PConnection.connect
AgoraRTC_N.js:5 00:23:03:21 Agora-SDK [INFO]: [apiInvoke-78][client-ccdab][Client.join] onSuccess
AgoraRTC_N.js:5 00:23:03:21 Agora-SDK [INFO]: [client-ccdab] Joining channel success: channel: 40bb692f-1d9b-4b3a-b492-4df7d2064i2k,uid: 559372186
AgoraRTC_N.js:5 00:23:03:23 Agora-SDK [INFO]: [apiInvoke-79][client-ccdab][Client.publish] start ['track-mic-caa9c222']
AgoraRTC_N.js:5 00:23:03:24 Agora-SDK [INFO]: [client-ccdab] Publishing tracks, id track-mic-caa9c222
AgoraRTC_N.js:5 00:23:03:24 Agora-SDK [DEBUG]: [client-ccdab][lock-client-publish-81] is locked, current queue 1.
AgoraRTC_N.js:5 00:23:03:24 Agora-SDK [DEBUG]: [client-ccdab] publish high stream
AgoraRTC_N.js:5 00:23:03:24 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is locked, current queue 2. From P2PChannel.publish
AgoraRTC_N.js:5 00:23:03:24 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is locked, current queue 3. From P2PChannel.updateRemoteRTPCapabilities
AgoraRTC_N.js:5 00:23:03:28 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onICEConnectionStateChange(checking)
AgoraRTC_N.js:5 00:23:03:28 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is unlocked, current queue 0. From P2PConnection.connect
AgoraRTC_N.js:5 00:23:03:29 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is unlocked, current queue 2. From P2PChannel.connect
AgoraRTC_N.js:5 00:23:03:29 Agora-SDK [DEBUG]: [client-ccdab] [p2pId: 9]: P2PConnection will send audioTrack [track-mic-caa9c222]
AgoraRTC_N.js:5 00:23:03:29 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is locked, current queue 1. From P2PConnection.send
AgoraRTC_N.js:5 00:23:03:33 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onConnectionStateChange(connecting)
AgoraRTC_N.js:5 00:23:03:52 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is unlocked, current queue 0. From P2PConnection.send
AgoraRTC_N.js:5 00:23:03:53 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is unlocked, current queue 1. From P2PChannel.publish
AgoraRTC_N.js:5 00:23:03:54 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is locked, current queue 1. From P2PConnection.updateRemoteRTPCapabilities
AgoraRTC_N.js:5 00:23:03:54 Agora-SDK [INFO]: [client-ccdab] Publish success, id track-mic-caa9c222
AgoraRTC_N.js:5 00:23:03:54 Agora-SDK [DEBUG]: [client-ccdab][lock-client-publish-81] is unlocked, current queue 0.
AgoraRTC_N.js:5 00:23:03:55 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PChannel-mutex-80] is unlocked, current queue 0. From P2PChannel.updateRemoteRTPCapabilities
AgoraRTC_N.js:5 00:23:03:55 Agora-SDK [INFO]: [apiInvoke-79][client-ccdab][Client.publish] onSuccess
AgoraRTC_N.js:5 00:23:03:55 Agora-SDK [DEBUG]: codecs has not changed, no need to updateRemoteCodec, codecs: vp9,av1,vp8,h264
AgoraRTC_N.js:5 00:23:03:55 Agora-SDK [DEBUG]: [P2PConnection] updateRemoteRTPCapabilities no need to exchange SDP.
AgoraRTC_N.js:5 00:23:03:55 Agora-SDK [DEBUG]: [client-ccdab][lock-P2PConnection-mutex-84] is unlocked, current queue 0. From P2PConnection.updateRemoteRTPCapabilities
basicVoiceCall.js:124 publish success
AgoraRTC_N.js:5 00:23:03:139 Agora-SDK [INFO]: [client-ccdab] [pc-9] selectedcandidatepairchange: local {"candidateType":"prflx","protocol":"udp"}, remote {"candidateType":"host","protocol":"udp","address":"129.227.179.74","port":4702} )
AgoraRTC_N.js:5 00:23:03:139 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onDTLSTransportStateChange(connecting)
AgoraRTC_N.js:5 00:23:03:139 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onICEConnectionStateChange(connected)
AgoraRTC_N.js:5 00:23:03:139 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onICETransportStateChange(connected)
AgoraRTC_N.js:5 00:23:03:139 Agora-SDK [DEBUG]: [client-ccdab] [pc-9] local candidate count 2
AgoraRTC_N.js:5 00:23:03:157 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onDTLSTransportStateChange(connected)
AgoraRTC_N.js:5 00:23:03:157 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.onConnectionStateChange(connected)
AgoraRTC_N.js:5 00:23:03:170 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.SelectedLocalCandidateChanged({"candidateType":"unknown","protocol":"unknown","address":"unknown","port":0} -> {"candidateType":"prflx","protocol":"udp"})
AgoraRTC_N.js:5 00:23:03:170 Agora-SDK [INFO]: [client-ccdab] [p2pId: 9]: P2PConnection.SelectedRemoteCandidateChanged({"candidateType":"unknown","protocol":"unknown","address":"unknown","port":0} -> {"candidateType":"host","protocol":"udp","address":"129...74","port":4702})
AgoraRTC_N.js:5 00:23:05:392 Agora-SDK [DEBUG]: [client-ccdab] user online 96871
AgoraRTC_N.js:5 00:23:05:392 Agora-SDK [DEBUG]: [client-ccdab] receive mute message 96871 video true
AgoraRTC_N.js:5 00:23:05:392 Agora-SDK [DEBUG]: [client-ccdab] user-info-update, uid: 96871, msg: mute-video
AgoraRTC_N.js:5 00:23:05:393 Agora-SDK [DEBUG]: [client-ccdab] local video disabled with uid 96871
AgoraRTC_N.js:5 00:23:05:393 Agora-SDK [DEBUG]: [client-ccdab] user-info-update, uid: 96871, msg: disable-local-video
AgoraRTC_N.js:5 00:23:06:568 Agora-SDK [DEBUG]: current web page is hidden
AgoraRTC_N.js:5 00:23:21:919 Agora-SDK [DEBUG]: current web page is visible

@plutoless
Copy link
Contributor

the log is not complete. i mainly need the backend logs. could you pls share a complete copy of your backend logs?

@zhaoxin1943
Copy link

image
same issue

@yinyajun
Copy link

yinyajun commented Dec 8, 2024

Same Issue
image

def from_dict(data_class, data):
    """Recursively convert a dictionary to a dataclass instance."""
    if is_dataclass(data_class):  # Check if the target class is a dataclass
        fieldtypes = {f.name: f.type for f in data_class.__dataclass_fields__.values()}
        return data_class(**{f: from_dict(fieldtypes[f], data[f]) for f in data})
    elif isinstance(data, list):  # Handle lists of nested dataclass objects
        return [from_dict(data_class.__args__[0], item) for item in data]
    else:  # For primitive types (str, int, float, etc.), return the value as-is
        return data

data_class: <class 'realtime_agent.realtime.struct.ResponseCreated'>
data: {'type': 'response.created', 'event_id': 'event_Ac5vFH0JsZ18NjLXGR9h1', 'response': {'object': 'realtime.response', 'id': 'resp_Ac5vFUw4vXVzwtHOUn667', 'status': 'in_progress', 'status_details': None, 'output': [], 'usage': None, 'metadata': None}}
can not unmarshal data to data_class

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants