Training speed has increased by 50% after installing the new versions of Torch and Xformer

[wwwffbf]

torch==2.0.0+cu118
torchvision==0.15.1+cu118
xformers==0.0.17rc482
Is this a potential optimization method?Will this be included in the new version of the Kohya_ss ?

[oliverban]

How to install these? Can I just place it in the req txt upon installation and it will take care of it? :)
Just:
pip install torch==2.0.0+cu118 torchvision==0.15.1+cu118 xformers==0.0.17rc482
? :)

Edit, I had to append --extra-url with the correct cu118 adress and it worked, seeing faster speeds on 3090 as well here on my end. Default settings in Lora Dreambooth took 8 min and 58 sec.

[R3xpook]

I installed latest torch 2.0 cu118 and also xformers but latest one so 0.0.19 think is beta or alpha and OMG THE SPEED
Didnt give any problems so far only tested normal lora and 1.5 SD version

[oliverban]

xformers works with torch 2 now?

[Kadah]

Newer xformers requires torch 2.0.

[R3xpook]

Im using xformers 0.0.19.dev516 and pytorch 2 CUDA 11.8 on windows ofc oh my 2060 12 GB i saw a massive improvement right now training at 768x768 models are as fast as 512x512 before updating. Im thinking about rolling back to benchmark effective speed of it...maybe i will do it

[oliverban]

Did you compile it yourself? Or just used a pip install command? :) Mind sharing the line?

[R3xpook]

just pip install easy as that

[Signorlimone]

I know it shouldn't be asked here but, would you be so kind to explain how to update all the required libs?

[R3xpook]

just pip force update torch 2 and latest xformers

[Signorlimone]

thanks, you wouldn't believe it but I just did that few seconds ago 😄

[R3xpook]

ahah right on time

[jonwong666]

do versions of xfomers have to match between kohya trained LORAs and webUI xformers? Meaning if I update xformers in WEBUI I will need to update Kohya?
Also, are there issues with using LORA's in webui that have been trained in older XFORMER version?
Concerned about updating and potentially messing up my older LORAs

[andupotorac]

Be sure to set both "mem_eff_attn": true, and "xformers": true, in your params.

pip3 install torch torchvision

pip install xformers

python -m [xformers.info](http://xformers.info/) (to check if xformers are installed correctly)