You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the LLM times out after having streamed some tokens already, we end up with two audio streams playing at the same time, and all the text getting added to the same message.
It's basically:
The LLM starts streaming some tokens
Audio starts playing
The LLM times out so retry kicks off
Original audio keeps playing, and the new audio stream from retry also starts playing
The end result is a single agent message that's very confused (since it's coming from the same LLMStream), and two audio streams playing at the same time, leading to choppy audio that skips around.
In terms of how to fix this.
We should cancel the speech if the LLM times out after some tokens have been streamed instead of retrying.
If the LLM times out after some tokens have been played, treat it as an interruption, and retry the entire call to llm.chat() - including the call to before_llm_cb and before_tts_cb
This is what was really happening in #1077 - see that ticket for a detailed example.
When the LLM times out after having streamed some tokens already, we end up with two audio streams playing at the same time, and all the text getting added to the same message.
It's basically:
The end result is a single agent message that's very confused (since it's coming from the same LLMStream), and two audio streams playing at the same time, leading to choppy audio that skips around.
In terms of how to fix this.
llm.chat()
- including the call tobefore_llm_cb
andbefore_tts_cb
This is what was really happening in #1077 - see that ticket for a detailed example.
Here's another person encountering it in the slack: https://livekit-users.slack.com/archives/C07FY8WHGPM/p1734179180491739
The text was updated successfully, but these errors were encountered: