Is it possible to overcome low GPU-memory for finetuning in Standalone BETA installation?

[ahmetax]

Hi,
I installed the BETA version as a standalone application. It works with no problems. However when I run start_finetune.sh, it reports that my GPU-memory is insufficient. I have a 6GB GPU card, and its 5.93GB is available to use. All other requirements are satisfied.
Are there any solutions for low GPU-memory problem for finetuning? (My os is Ubuntu 22.04 and my Ram is 32GB.)
Thank you.

[erew123]

Hi @ahmetax

No on Ubuntu sadly. This is a limitation of the NVidia driver more than anything else. On Windows Nvidia allowed/created System Memory fallback for Stable Diffusion, https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion which in essence, when the GPU memory fills up, it spills over into RAM. This feature isnt available through the driver within Ubuntu/Linux.

Outside of that the Training is running with Huggingface Transformers, and that has no capability to limit/control memory, its left down to the GPU driver to handle that.

Your only other option for finetuning, if you cannot support it on your own hardware, would be to run a session in a free google colab.

Thanks.

[ahmetax]

Thanks a lot! I will try my chance on Colab.