removed duplications in function messages.

[jmugicagonz]

@longcw @davidzhao this is my proposal for the duplicated messages problem.
When the LLM is given a prompt, the LLM will follow those instructions of the prompt. It can generate messages and call functions. The LLM has no visibility on what happens inside the functions, except for the description, parameters, what is returned from those functions. If we generate assistant messages inside the functions and we append them to the overall context, this ends up confusing the LLM.

I have extensively tested it, and it ends up changing the behavior.

Is it as if i'm talking to you following some instructions and I see messages in the chat, coming from me, that I didn't write.
I understand the benefits of calling agent.say inside a function. However, IMHO, what we include in a context as said by the assistant, should come from the assistant. At least inside functions, where I've found this to be problematic. Outside of functions, using agent.say with add_to_chat_ctx is still not ideal, but up to the livekit user. Doing it inside the functions, forces every livekit user to have one original message split into 2 messages and the agent get confused.

[changeset-bot]

⚠️ No Changeset found

Latest commit: ef310ca

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[longcw]

I don't really agree with "what we include in a context as said by the assistant, should come from the assistant.". LLM is a complementation model even it's in the chat mode, it should have the ability to complement the content following the input user gives it.

Meanwhile, users can set add_to_chat_ctx for say to avoid it being added to the ctx. From my testing in #1165, adding the said message to the end of the chat ctx makes the generated answer more natural.

I have extensively tested it, and it ends up changing the behavior.

I agree that adding messages to the chat ctx will change the behavior of the LLM, what I don't agree is this is a problem. It's highly depends on what messages added to the ctx, and users should have the flexibility to decide if this is the behavior they expected.

[longcw]

It will be very helpful if you could share some cases this causes the problem.

I have tested the prompt below and it worked as expected that function call always returns with a message. In my tests, say is disabled in function calls, and only one of these two options should be used at a time (say something or prompt LLM to return a message).

"You are a weather assistant created by LiveKit. Your interface with users will be voice. "
"You will provide weather information for a given location. "
"When a user asks for the weather information in a given location, you will say 'thanks for asking! I'll get you the weather information for that location' and then you will inmediately call the function 'get_weather' to retrieve the weather information."

[jmugicagonz]

Sorry @longcw but I can tell you that the current setup, splitting one message with content + function into two messages, one with content and another with function ends up confusing the agent which will no longer do both things in the same message, and sends the content without calling the function.

I agree that adding messages to the chat ctx will change the behavior of the LLM, what I don't agree is this is a problem. It's highly depends on what messages added to the ctx, and users should have the flexibility to decide if this is the behavior they expected.

I agree that users should keep the flexibility. And if they want to add a message in the middle, let it be that way. However, for those users that want to respect the original chat context, the moment 1 message goes into 2 messages, the user no longer have flexibility to keep them as one, and the behavior of the LLM changes.

[longcw]

the current setup, splitting one message with content + function into two messages, one with content and another with function ends up confusing the agent which will no longer do both things in the same message, and sends the content without calling the function.

I understand your concerns. The duplicated messages was introduced in #1078. Here is my proposal if there is a message alongside the fnc:

1. we add it to the chat ctx before the fnc call, and record the message id
2. after function finished, if the last item in the chat ctx is still the message recorded in step 1, we replace it with the message containing the content and the function call. If the last item in the chat ctx has changed, i.e. the user called .say in the function call, we remove the content of the function call and add it to the ctx.

This give both options to users. You can avoid using say inside the function calls to make sure the chat ctx exactly matches the LLM outputs, or using say to manipulate the chat ctx by yourself.
@jmugicagonz @davidzhao wdyt?

[jmugicagonz]

It will be very helpful if you could share some cases this causes the problem.

I have tested the prompt below and it worked as expected that function call always returns with a message. In my tests, say is disabled in function calls, and only one of these two options should be used at a time (say something or prompt LLM to return a message).

"You are a weather assistant created by LiveKit. Your interface with users will be voice. "
"You will provide weather information for a given location. "
"When a user asks for the weather information in a given location, you will say 'thanks for asking! I'll get you the weather information for that location' and then you will inmediately call the function 'get_weather' to retrieve the weather information."

I just tried with this prompt and I can't replicate it... it might be because it's "too simple". But definitely happens when both the prompt and the conversation get longer

[jmugicagonz]

1. we add it to the chat ctx before the fnc call, and record the message id
2. after function finished, if the last item in the chat ctx is still the message recorded in step 1, we replace it with the message containing the content and the function call. If the last item in the chat ctx has changed, i.e. the user called .say in the function call, we remove the content of the function call and add it to the ctx.

This LGTM. Thanks for proposing it @longcw !