Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add gpu support #44

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Add gpu support #44

wants to merge 3 commits into from

Conversation

Kostis-S-Z
Copy link
Contributor

@Kostis-S-Z Kostis-S-Z commented Dec 6, 2024

What's changing

Add support for both the text-to-text and the text-to-speech model to be loaded on the GPU.

How to test it

Steps to test the changes:

  1. Run the demo
  2. Check with nvidia-smi that your GPU has loaded the models

Additional notes for reviewers

This became more difficult than I expected because the two different frameworks of the models need to have support for the same cuda toolkit. Support for the text-to-speech is complete and was easy. Support for the text-to-text has proven quite difficult.

Some rough benchmarks:
Setup: Count time from the moment you upload the document until the first audio sample of speaker 1 is generated (includes loading both models and running inference once with both models). The .html file in example_data was used. GPU: RTX 2060

CPU: text-to-text & text-to-speech -> 2min, 41sec
CPU: text-to-text GPU: text-to-speech -> 1min, 27 sec (!)

I already...

  • Tested the changes in a working environment to ensure they work as expected
  • Added some tests for any new functionality
  • Updated the documentation (both comments in code and under /docs)

@Kostis-S-Z Kostis-S-Z mentioned this pull request Dec 11, 2024
3 tasks
@Kostis-S-Z Kostis-S-Z added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed model loading Related to the model loading module labels Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed model loading Related to the model loading module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant