torch.OutOfMemoryError: CUDA out of memory

[Lenan22]

When I run : python -m deepcompressor.app.diffusion.ptq configs/model/flux.1-schnell.yaml configs/svdquant/int4.yaml --eval-benchmarks MJHQ --eval-num-samples 1024

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 384.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 359.62 MiB is free. Process 4119960 has 448.00 MiB memory in use. Process 1215056 has 38.77 GiB memory in use. Of the allocated memory 38.17 GiB is allocated by PyTorch, and 104.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

My A800 only has 40G of storage and can't run through this example of yours. Is there a new flux.1-schnell.yaml configuration available for me? I want to get it to run through and verify the effect of quantization.

[senlyu163]

Hi, i met OOM too, do u solve it?