Double-counted tokens? #50
Comments
|
I also noted this and conversed with @dnakov over Twitter about it - realtime API costs are much higher than expected and his code saves on it significantly. His repo is here with the "optimizations" to reduce cost: |
|
Pushing this upward. I did not do a record on the findings. However, also realize that the API cost is significantly higher than expected. I think this is not intended but a bug of the realtime console, or the SDK. If I go to have a chat on Playground for 5 minutes. It costs only around $1. But doing it with the realtime console code, it costs $2-4. I am using the same prompt on both versions. For more information, I am also using the relay. |
|
It's the same, playground or not. |
|
can I ask that how did you get the cost tokens from this api? I wanted to do this but I failed to find a server event that contains the cost. |
|
@spectaclecs take a look at PR #37 |
|
#37 needs to be merged, it cost me ~$10 for 5 minute conversation lol. Gotta stick to concurrent processing for now... |
|
Bumping this up! |
|
The increasing costs is just the conversation history that's building up in the thread. OpenAI doesn't know what your scenario is so they don't know how much conversation history to track. You need at least 2 - 3 turns for features like co-referencing to work. Questions like "how much is that in euros?" wont work because the model won't know what "that" is referring to... Depending on your scenario though you might need 20 turns of conversation history. Lets say your playing a game of 20 questions... The point is OpenAI doesn't know your scenario so they track as many turns as they can fit into the context window which can be expensive. A feature that lets you configure the number of turns to keep could be handy but you've already shared the code above for how to trim the conversation history yourself. |
OpenAI team,
First off, hats off to you for this awesome API and model.
Second, I'd like to bring up some observations because I'm not sure if they are intended or not.
Assuming the default modalities (text, audio) are selected, as the conversation goes on, for each response, all previous conversation items are counted as both input audio tokens and input text tokens.
Now, we can turn off the user input transcription (off by default), which would take care of the input text tokens coming from the user speech transcription, but we have no way of disabling assistant response transcription. So, we end up being charged for both text and audio tokens.
See below an example of a 30-turn interaction (with input_audio_transcription enabled) (and $10 down the drain 😅)
We paid $0.05 for the just the first question, $0.22 for just the 10th, $0.42 for the 20th, $0.58 for the 30th, etc.
If I so much as sneezed after that, it'd have cost me over half a dollar (~2 min of audio, according to the advertised rate.)
My main question is:
Why do we have to pay for both assistant text and audio tokens by default? Does the model need both? (doesn't seem so)
Now, we technically have the ability to delete conversation items and create new ones, so there are definitely ways to deal with the cost. But, they're cumbersome and full of edge cases due to the realtime nature of it.
If you can't "just make it super cheap so we don't have to care", maybe there is some low hanging fruit?
Can we have some easier control over how much of the conversation we want to keep around?
Can we have a
conversation.item.updateevent so that we don't have to recreate the entire conversation tree when we just want to remove the transcription?Something else better I'm sure you've already cooked up?
Thank you!
Turn 1
User: (audio) What's the weather in New York?
Assistant: (audio) I don't have access to real-time data, including current weather updates. However, you can check the latest weather in New York by looking it up on a weather website or using a weather app on your phone.
Turn 2
User: (audio) What's the weather in San Francisco?
Assistant: (audio) I don't have access to real-time weather updates. For the current weather in San Francisco, please check a weather website or use a weather app on your phone.
Turn 3
User: (audio) What's the weather in London?
Assistant: (audio) I don't have access to real-time data, including current weather updates. To get the latest weather in London, please check a weather website or use a weather app on your phone.
Turn 4
User: (audio) What's the weather in Toronto?
Assistant: (audio) I can't provide current weather updates. For the latest weather in Toronto, please check a weather website or use a weather app on your phone.
Turn 5
User: (audio) What's the weather in Istanbul?
Assistant: (audio) I don't have access to real-time information, so I can't provide the current weather in Istanbul. You can check the latest weather there by using a weather website or a weather app.
Turn 6
User: (audio) What's the weather in New York?
Assistant: (audio) I can't provide real-time weather updates. To find out the current weather in New York, please check a weather website or use a weather app.
Turn 7
User: (audio) What's the weather in San Francisco?
Assistant: (audio) I can't provide real-time weather updates. To find out the current weather in San Francisco, please check a weather website or use a weather app.
Turn 8
User: (audio) What's the weather in London?
Assistant: (audio) I can't provide real-time weather updates. To find out the current weather in London, please check a weather website or use a weather app.
Turn 9
User: (audio) What's the weather in Toronto?
Assistant: (audio) I can't provide real-time weather updates. To find out the current weather in Toronto, please check a weather website or use a weather app.
Turn 10
User: (audio) What's the weather in Istanbul?
Assistant: (audio) I can't provide real-time weather updates. To find out the current weather in Istanbul, please check a weather website or use a weather app.
Turn 11
User: (audio) What's the weather in New York?
Assistant: (audio) I can't provide real-time weather updates. To find out the current weather in New York, please check a weather website or use a weather app.
Turn 12
User: (audio) What's the weather in San Francisco?
Assistant: (audio) For the most current weather in San Francisco, please check a reliable weather website or a weather app on your device. They will have the latest information.
Turn 13
User: (audio) What's the weather in London?
Assistant: (audio) To get the current weather in London, please check a reliable weather website or a weather app on your device for the most up-to-date information.
Turn 14
User: (audio) What's the weather in Toronto?
Assistant: (audio) For the latest weather in Toronto, please check a weather website or a weather app on your phone for real-time updates. They will have the information you need.
Turn 15
User: (audio) What's the weather in Istanbul?
Assistant: (audio) For the current weather in Istanbul, please refer to a weather website or a weather app on your device. They will have the latest information.
Turn 16
User: (audio) What's the weather in New York?
Assistant: (audio) To get the current weather in New York, please check a reliable weather website or a weather app on your device. They will have up-to-date information.
Turn 17
User: (audio) What's the weather in San Francisco?
Assistant: (audio) To get the current weather in San Francisco, please check a reliable weather website or a weather app on your device for the most up-to-date information.
Turn 18
User: (audio) What's the weather in London?
Assistant: (audio) To find out the current weather in London, please check a reliable weather website or use a weather app on your device. They will have the latest information.
Turn 19
User: (audio) What's the weather in Toronto?
Assistant: (audio) For the latest weather in Toronto, please check a weather website or a weather app. They will have up-to-date information on the current conditions.
Turn 20
User: (audio) What's the weather in Istanbul?
Assistant: (audio) To get the current weather in Istanbul, please check a reliable weather website or a weather app on your device for the latest information.
Turn 21
User: (audio) What's the weather in New York?
Assistant: (audio) To get the current weather in New York, please check a reliable weather website or use a weather app on your device. They will have the latest information.
Turn 22
User: (audio) What's the weather in San Francisco?
Assistant: (audio) To find out the current weather in San Francisco, please check a reliable weather website or a weather app on your device. They will have the most up-to-date information.
Turn 23
User: (audio) What's the weather in London?
Assistant: (audio) To get the current weather in London, please check a reliable weather website or use a weather app on your device. They will have the latest information.
Turn 24
User: (audio) What's the weather in Toronto?
Assistant: (audio) To find out the current weather in Toronto, please check a reliable weather website or use a weather app on your device for the latest information.
Turn 25
User: (audio) What's the weather in Istanbul?
Assistant: (audio) To get the current weather in Istanbul, please check a reliable weather website or a weather app on your device for the latest information.
Turn 26
User: (audio) What's the weather in New York?
Assistant: (audio) To find out the current weather in New York, please check a reliable weather website or use a weather app on your device. They will provide the latest information.
Turn 27
User: (audio) What's the weather in San Francisco?
Assistant: (audio) For the latest weather in San Francisco, please check a weather website or a weather app on your device. They will have the most current information.
Turn 28
User: (audio) What's the weather in London?
Assistant: (audio) To get the current weather in London, please check a reliable weather website or use a weather app on your device for the latest information.
Turn 29
User: (audio) What's the weather in Toronto?
Assistant: (audio) To find out the current weather in Toronto, please check a reliable weather website or use a weather app on your device. They will have the latest information.
Turn 30
User: (text) What was the second city I asked about?
Assistant: (audio) The second city you asked about was San Francisco.
Total Cost: $9.92
The text was updated successfully, but these errors were encountered: