Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: empty agent transcript #1148

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: empty agent transcript #1148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f2e49bd
fix: empty agent transcript
jayeshp19 Nov 28, 2024
baaed13
fix: empty agent transcript
jayeshp19 Nov 28, 2024
146820f
changeset
jayeshp19 Nov 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/brave-plums-help.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"livekit-agents": patch
---

fix: empty agent transcript
31 changes: 16 additions & 15 deletions livekit-agents/livekit/agents/pipeline/pipeline_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -883,24 +883,25 @@ async def _execute_function_calls() -> None:
if interrupted:
collected_text += "..."

msg = ChatMessage.create(text=collected_text, role="assistant")
self._chat_ctx.messages.append(msg)
if collected_text:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would skip speech_handle.mark_speech_committed(), do we want to mark it as committed before skipping the other operations?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

model's response contains empty content when it responds with a tool call, should we still mark it as committed? because after executing function model will respond with string that will be marked as committed

Copy link
Member

@theomonnom theomonnom Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When can this happen? I'm more worried about the cause than fixing the scenario where it is indeed empty

Copy link
Collaborator Author

@jayeshp19 jayeshp19 Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This happens when agent finish speaking tool call's response. we get an extra empty transcript. screenshot

Can there be a possiblity when llm response is just empty?

Copy link
Collaborator Author

@jayeshp19 jayeshp19 Nov 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the same speech id which has tool calls with empty content from model's response, do we need to mark it at committed?
image

Copy link
Member

@theomonnom theomonnom Dec 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I think we should add new arguments to the agent_speech_committed event. I think it is a valid usecase to have an empty content but having function calls

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something like below?

    def mark_speech_committed(self, is_content_empty: bool) -> None:
        self._speech_committed = True
        self._is_content_empty = is_content_empty

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I think we should add new arguments to the agent_speech_committed event.

@theomonnom wouldn't that break existing callbacks? I'm not sure if there's value in it because function calls are supposed to be handled by the framework

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will think more about that alongside the new IO proposal

Copy link
Member

@theomonnom theomonnom Dec 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put this PR in standby for now, we should be able to add new arguments without breaking

msg = ChatMessage.create(text=collected_text, role="assistant")
self._chat_ctx.messages.append(msg)

speech_handle.mark_speech_committed()
speech_handle.mark_speech_committed()

if interrupted:
self.emit("agent_speech_interrupted", msg)
else:
self.emit("agent_speech_committed", msg)
if interrupted:
self.emit("agent_speech_interrupted", msg)
else:
self.emit("agent_speech_committed", msg)

logger.debug(
"committed agent speech",
extra={
"agent_transcript": collected_text,
"interrupted": interrupted,
"speech_id": speech_handle.id,
},
)
logger.debug(
"committed agent speech",
extra={
"agent_transcript": collected_text,
"interrupted": interrupted,
"speech_id": speech_handle.id,
},
)

# mark the speech as done
speech_handle._set_done()
Expand Down
Loading