Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add VRAM free after sample images #1396

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

v0xie
Copy link

@v0xie v0xie commented Jun 28, 2024

This PR addresses an issue with VRAM usage after generating sample images.

Without using this PR, when training with SDXL and batch size > 1, the first time during training when sample images are generated, I would run out of VRAM even with 24GB (no --xformers or --mem_eff_attn).

Notably, the sample_images_common method in train_util calls clean_memory_on_device near the end of the function, but I hypothesize that some of the memory is still used after the operation at the end. So calling it again after exiting the function clears the remaining.

I'm not sure if anyone else has had this issue or if it's an issue with my configuration only. Please feel free to close this PR if it's unnecessary. Thank you for your consideration.

@araleza
Copy link

araleza commented Jun 28, 2024

This seems good, I think that at times in the past, I've seen training work up until the sample images were generated, and then not be able to restart afterwards.

@kohya-ss
Copy link
Owner

kohya-ss commented Jul 8, 2024

Thank you for this!

Notably, the sample_images_common method in train_util calls clean_memory_on_device near the end of the function, but I hypothesize that some of the memory is still used after the operation at the end. So calling it again after exiting the function clears the remaining.

This makes sense... I'm working on SD3 now, so I will merge this sooner when I finish SD3.

@kohya-ss kohya-ss added the enhancement New feature or request label Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants