Make full use of GPU memory.

[toanhuynhnguyen]

I have 32GB GPU memory. How to make full use of GPU memory to run Batched Faster Whisper?
Such as: multithreading, or any way?
Thank you for your help.

[MahmoudAshraf97]

using the whole memory will speed anything up, there is a limit to the speed you get from batching which is bounded by memory bandwidth, not memory size

[toanhuynhnguyen]

@MahmoudAshraf97
I mean:
I have a server with 32GB GPU memory. I need to transcribe 10,000 audio files, each audio file is in around 10 minutes.
When I run batched faster-whisper with one process/thread, it takes around 7GB GPU memory for each audio file.
I want to use all the memory to speed up the transcription, how can I do it? Can I use multi-threading or other ways to make full use of GPU memory?
Thank you so much, bro!

[toanhuynhnguyen]

Can you provide me some documents of this TRT-LLM for Batched Faster Whisper.
Thank you so much.

[Mansi9991]

Did you able to generate faster outputs?
I'm also getting problems with slow transcriptions

[toanhuynhnguyen]

@Mansi9991 use batched faster-whisper

[Mansi9991]

What about trt-llm?
How can I elaborate that

[wenestam]

Can also try Nvidia-triton to allow loading multiple models and run in parallel. I've tried this before to make most my GPU, allowing me to run 2 of distil-medium.en in parallel on 1 GPU.