Support for real-time TTS! #5994
Closed
+202
−64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Workaround gradio's limitation that doesn't directly allow passing
data from Python -> JS (only indirectly, through components) - see create_dataholder_gradio
2. Update the chat HTML structure-wise and incrementally ("diff" mode) -
see js_chat_html_update
This significantly makes the updates more efficient (no redundant HTML
rendering) and additionally allows for stable components in the chat
in streaming mode (e.g. important for example when extensions add <audio> elements in the chat - e.g. alltalk_TTS).
Allow extensions to modify the output when in chat mode and bot replies are streamed. This is useful, for example, for extensions that need access to the bot replies while they are streamed (e.g. incremental/streaming TTS).
|
Please support instruct mode |
|
@oobabooga please consider this 🙏 |
|
Hey @czuzu |
|
I gotta check the PR list more often, this is something I've needed for a while. Thank God textgen is open source and I can implement these changes on my own rig. Ty❤️❤️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello,
I've setup TGUI with the alltalk_tts extension locally, modified the setup to allow for passing LLM replies as they're being generated (stream mode) to the extension, and to subsequently do real-time TTS (aka "incremental" TTS).
PR for the extension is in the backlog too, streaming TTS is working as expected locally, this one is for the parts in TGUI I needed to adjust/extend to allow this to work smoothly.
Mainly, 2 changes were needed:
output_modifier_streamhandler for extensions (works only for chat-mode currently) as the enabler for streaming the LLM text to extensions(the rest are miscellaneous changes - adding a llama3 instruction template and a commented line to allow remotely debugging TGUI)
Let me know what you think and btw, nice project!
Thanks!
Checklist: