System prompt is ignored after fatal errors with realtime api

[FiddlersCode]

How to replicate
trigger an OpenAI error (e.g. send a tool call description that's too long which triggers the error below)

Expected behaviour

• the error is logged and the conversation continues
• our system prompt is still used

Actual behaviour

• the error is logged but the conversation then continues
• crucially, the system prompt and config that we send to OpenAI are now ignored - e.g. the voice reverts to the default voice and all the safeguards from the system prompt are ignored

ERROR livekit.plugins.openai.realtime - OpenAI S2S error {'type': 'error', 'event_id': 'event_AXSNgEBBhqwMQmz2iOy5l', 'error': {'type': 'invalid_request_error', 'code': 'string_above_max_length', 'message': "Invalid 'session.tools[1].description': string too long. Expected a string with maximum length 1024, but got a string with length 1198 instead.", 'param': 'session.tools[1].description', 'event_id': None}}

Versions
livekit = "0.18.1"
livekit-agents = "0.11.3"
livekit-plugins-openai = "0.10.7"
livekit-protocol = "0.7.0"

We’re using a realtime-api model hosted on Azure.

I suspect this is the line causing the error -

agents/livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py

Line 997 in d671302

self._handle_error(data)

While we can of course fix this specific error, we’re concerned that errors are being swallowed by the system and the user is allowed to continue without safeguards. I was also able to raise uncaught exceptions elsewhere in the code to recreate the error.

[longcw]

I have created a PR #1200 to emit the error from the realtime API, so that agent can decide to reset or close the session in the callback. Does that make sense?

[FiddlersCode]

I have created a PR #1200 to emit the error from the realtime API, so that agent can decide to reset or close the session in the callback. Does that make sense?

Hi @longcw - many thanks for this! We're now able to read the event emitted (RealtimeError) so that's good.

My only followup question - is it well understood why we can keep talking to OpenAI with the system prompt being ignored? I'm wondering if a new session is being created with default opts but I haven't been able to dig around enough yet so it's only an idea.

This is still a concern for us because there may be other back doors whereby the system prompt is ignored so would be good to understand the fundamental issue.

[longcw]

The session to the OAI Realtime API is created when RealtimeSession is initialized, and then your configurations like instructions and voice and function calls are updated through the session.update(https://platform.openai.com/docs/api-reference/realtime-client-events/session/update).

The session is created and active even the session.update fails as that in your case. You could manually stop the agent if the error happened.

To better track if the session update is successful, perhaps we can emit the session.created and session.updated events from the API https://platform.openai.com/docs/api-reference/realtime-server-events/session/updated, wdyt @theomonnom

[FiddlersCode]

Thanks for the quick response!

Hm so something behind the scenes is updating the session? To be clear, we do not want to update the session to change the instructions (system prompt) ever. Is there a way of prohibiting such changes?

Emitting a session updated event would be useful so we could check if the instructions had changed, though!

[longcw]

To be clear, it's OpenAI will always set a default configuration for the session, session.update is the API to set your instructions, and it will be automatically used when you create the RealtimeSession using livekit agent. This is the only way to set the instructions and function calls etc., you can check OpenAI's document in https://platform.openai.com/docs/api-reference/realtime-client-events

This is also why you can still talk to the agent when the error happens.

[FiddlersCode]

Session update may be a red herring.

In our case, we're creating the azure model with this:

    azure_model = openai.realtime.RealtimeModel().with_azure(
        azure_deployment=AZURE_DEPLOYMENT,
        azure_endpoint=AZURE_ENDPOINT,
        api_version=API_VERSION,
        api_key=AZURE_API_KEY,
        instructions=system_prompt,
        max_response_output_tokens=int(MAX_RESPONSE_OUTPUT_TOKENS),
        modalities=["text", "audio"],
        voice=REALTIME_VOICE,  # type: ignore
    )

Looking at model.sessions[0]._opts.instructions I can see that the system prompt (instructions) are correct when we start the session.

I then have the following code:

def handle_error(error, session, model):
    if type(error).__name__ == "RealtimeError":
        validation_result = validate_session_config(session)
        logger.info(f"Validation result: {validation_result}")

        number_of_sessions = len(model.sessions)
        logger.info(f"Number of sessions: {number_of_sessions}")
        
        blah blah

error_handler = lambda error: handle_error(error, session, model)
session.on("error", error_handler)

So when a RealtimeError is emitted, the session instructions are still valid BUT the voice assistant is clearly not using our system prompt / instructions (it's speaking in the wrong voice, it's speaking other languages, doing math, talking about politics etc).

I've also confirmed that we're not creating an additional session which might have fresh instructions.

So I'm quite stumped as to why we're seeing this behaviour if the session instructions are still our original prompt (i.e. it looks like the session hasn't been updated).

But yea if you start emitting session.created and session.updated then at least we can start listening for those events and confirm whether the session is being updated.

[longcw]

Looking at model.sessions[0]._opts.instructions I can see that the system prompt (instructions) are correct when we start the session.

I think there is some misunderstanding of how "RealtimeModel" works. The actual process is

1. someone creates an instance of RealtimeModel with custom configuration
2. in RealtimeModel.__init__ the configuration is written to _opts.
3. it connects to OpenAI's Realtime API and creates a session (with OpenAI's default configuration)
4. it then uses session.update to synchronize the values in _opts to the OpenAI session.

Steps 2-4 all occur in the __init__ of the RealtimeModel.

The point is that all the actual configuration and conversations are managed by the OAI, and RealtimeModel is just an interface to synchronize with the OAI API. Therefore, if OAI returns an error saying session.create failed, there is no need to validate _opts as it is always a user-defined configuration.

[FiddlersCode]

Ah I see. So in our case, what's (probably) happening is:

1. For $reasons, OpenAI is updating the session and removing the custom config.
2. _opts is now out of sync with OpenAI.
3. Step 1 emits a session.updated event

What doesn't make sense for our case still is that the OAI docs say any field may be updated at any time, except for "voice". but in our case the voice is clearly being changed as well. Maybe the docs are wrong.

But in any case, your suggestions of

perhaps we can emit the session.created and session.updated events from the API
still sounds good.