Extremely slow LORA training

[MafiapauCZE]

Hello. Recently i start to work my ways around my own LORAs. First video i saw was from Aitrepreneur's video
He said that 1600 training steps took him only 7 minutes! and i was training 1600 steps for almost 7 hours.
VRAM, CUDA and RAMs was almost completely at the max of it's capacity and working state.

I was training SDXL lora on Pony's checkpoint.
on 800x800 resolution

Only thing that comes in my mind, is weak RAMs, because i got at least good mid CPU and GPU.

My Specs:
MB: MSI B75MA-P45 (MS-7798)
CPU: I7-3770 3.4GHz
GPU: MSI 1080ti Aero 11GB
RAM: 2x8GB Patriots DDR3-1600MHz (HwInfo showing me that they are running on 800MHz because they are in the same channel])
1x4GB team group team elite-1600 (also running on 800MHz on second channel [i know what you think.. but i decide to use that 4GB one because i needed rams..])
System is on SSD Samsung EVO 840 (500GB)

Just 100 steps took me 1h30min.

So, anyone got explanation? (yes i already think it's in the RAM modules, but i would like to see your opinions)

[b-fission]

A few key differences between your training setup and that guy in the video...

1. GTX 1080ti will be slower than the guy's RTX 3090 due to lack of tensor cores and fp16 support. And it is older.
2. You're training SDXL at 800x800 - he's training on SD1.5 at 512x512 --- SD1.5 is leaner than SDXL, low res is faster to train.
3. Having 20gb of system RAM (did I read that right?) is just enough to do SDXL. (if you have enough vram.)
4. So, about the lack of fp16 support? Without it, vram usage goes way up... like 16gb vram without fp16 to do SDXL at 800x800.
5. If there's not enough VRAM, nvidia's driver will borrow system memory to cover for the VRAM deficit. That causes massive slowdown!
6. And if your system ram is lacking, Windows will hit the swapfile next to cover that memory deficit. More slowdowns!

[MafiapauCZE]

Thanks for some answers.
Next day after i wrote this, i reinstall Kohya_SS, and it get much faster, like 3 to 4 times.
But with one problem, i can't train SDXL lora. if i will set batch and epochs higher than 1,
i will just get error and training will get instantly ended,
and when i'm doing 1 batch, 1 epoch for 2400 steps, i will receive out of memory error (even with 512x512 resolution)
(something like - you got 18GB avaible, but you need 21GB)
So i guess there is end of this for me..

[b-fission]

It should be possible to get vram usage to around 13gb for SDXL, even without fp16.

Try these options?

-Turn on Gradient Checkpointing
-Set the "Text Encoder learning rate" to 0, and turn on "Cache text encoder outputs"
-Use a lower training resolution? 800x800 for SDXL will save some memory, but I'm not sure about 512x512 for quality.