Train speed on 4090, Windows. 2.50s/it is not ok, right? #2166

stepahin · 2024-03-27T15:57:26Z

stepahin
Mar 27, 2024

Hi! I'm new to the party. Yesterday I was finally able to run Kohya SS on Win11 for the first time and trained some models. The speed I saw was no higher than 2.30-2.50s/it (XL train, batch size 5) and from what I googled, slower than 3090. But everywhere I've found discussions there is no clear guide and solutions. If you read Reddit there are people who claim to have it/s, there are some who say they have 1.5s/it on their 3090... Please share your speed and experience what can be optimized on 4090? I know the best option go Linux/Ubuntu but I'll leave it for later.

An old draver? Which one?
Triton?
Xformers or SDPA?
Which optimizer?
Bitsandbytes?
System > Display > Graphics > Hardware-accelerated GPU scheduling?
Someone advises scale_parameter=False relative_step=False warmup_init=False
Disabled gradient checkpoint?
Disabled bucketing?

What are some things to try? I really don't want to jump between OS.

Mosfett1975 · 2024-03-27T16:28:29Z

Mosfett1975
Mar 27, 2024

RTX4070TI. For SDXL my fastest result 2,73s/it (768x768, Network Dim. 64, Alpha 32) and 1,53it/s for SD. How I've understood, situation will not change untill sd-script will support CUDNN9

0 replies

Disty0 · 2024-03-28T10:08:56Z

Disty0
Mar 28, 2024

Effective it/s for batch size 5 and 2.5 s/it is 2 it/s (normalized for batch size 1).
Checks out with these benchmark:
https://www.pugetsystems.com/labs/articles/stable-diffusion-lora-training-consumer-gpu-analysis/

Adafactor is slower than AdamW.
Try AdamW with batch size 1 to get it/s range.

4 replies

stepahin Mar 28, 2024
Author

I just switched Adafactor to AdamW and got this. What am I doing wrong?

19:42:52-951895 INFO     Start training LoRA Standard ...
19:42:52-951895 INFO     Validating model file or folder path
                         D:/ComfyUI/ComfyUI/models/checkpoints/sd_xl_base_1.0.safetensors existence...
19:42:52-952894 INFO     ...valid
19:42:52-953895 INFO     Validating output_dir path D:/Dataset/DanWinters-01-Train1\model existence...
19:42:52-954894 INFO     ...valid
19:42:52-954894 INFO     Validating train_data_dir path D:/Dataset/DanWinters-01-Train1\img existence...
19:42:52-955894 INFO     ...valid
19:42:52-956893 INFO     reg_data_dir not specified, skipping validation
19:42:52-956893 INFO     Validating logging_dir path D:/Dataset/DanWinters-01-Train1\log existence...
19:42:52-957893 INFO     ...valid
19:42:52-958893 INFO     log_tracker_config not specified, skipping validation
19:42:52-959893 INFO     resume not specified, skipping validation
19:42:52-959893 INFO     vae not specified, skipping validation
19:42:52-960894 INFO     lora_network_weights not specified, skipping validation
19:42:52-960894 INFO     dataset_config not specified, skipping validation
19:42:52-961893 INFO     Folder 20_DanWinters style: 20 images found
19:42:52-962892 INFO     Folder 20_DanWinters style: 400 steps
19:42:52-962892 INFO     Total steps: 400
19:42:52-963892 INFO     Train batch size: 5
19:42:52-963892 INFO     Gradient accumulation steps: 1
19:42:52-964892 INFO     Epoch: 15
19:42:52-965892 INFO     Regulatization factor: 1
19:42:52-965892 INFO     max_train_steps (400 / 5 / 1 * 15 * 1) = 1200
19:42:52-966892 INFO     stop_text_encoder_training = 0
19:42:52-966892 INFO     lr_warmup_steps = 0
19:42:52-967893 INFO     Saving training config to
                         D:/Dataset/DanWinters-01-Train1\model\DanWinters_20240328-194252.json...
19:42:52-969893 INFO     accelerate launch --num_cpu_threads_per_process=2 "D:\Kohya SS
                         2\kohya_ss/sd-scripts/sdxl_train_network.py"  --bucket_no_upscale --bucket_reso_steps=64
                         --cache_latents --cache_latents_to_disk --caption_extension=".txt" --enable_bucket
                         --min_bucket_reso=256 --max_bucket_reso=2048 --gradient_checkpointing --learning_rate="0.0009"
                         --logging_dir="D:/Dataset/DanWinters-01-Train1\log" --lr_scheduler="constant"
                         --lr_scheduler_num_cycles="15" --max_data_loader_n_workers="0" --max_grad_norm="1"
                         --resolution="1024,1024" --max_token_length=225 --max_train_steps="1200"
                         --mixed_precision="bf16" --network_alpha="1" --network_dim=8 --network_module=networks.lora
                         --no_half_vae --optimizer_args scale_parameter=False relative_step=False warmup_init=False
                         --optimizer_type="AdamW" --output_dir="D:/Dataset/DanWinters-01-Train1\model"
                         --output_name="DanWinters"
                         --pretrained_model_name_or_path="D:/ComfyUI/ComfyUI/models/checkpoints/sd_xl_base_1.0.safetenso
                         rs" --save_every_n_epochs="1" --save_model_as=safetensors --save_precision="bf16"
                         --train_batch_size="5" --train_data_dir="D:/Dataset/DanWinters-01-Train1\img" --sdpa
2024-03-28 19:43:00 INFO     prepare tokenizers                                                   sdxl_train_util.py:135
2024-03-28 19:43:01 INFO     update token length: 225                                             sdxl_train_util.py:160
                    INFO     Using DreamBooth method.                                               train_network.py:173
                    INFO     prepare images.                                                          train_util.py:1469
                    INFO     found directory D:\Dataset\DanWinters-01-Train1\img\20_DanWinters style  train_util.py:1432
                             contains 20 image files
                    INFO     400 train images with repeating.                                         train_util.py:1508
                    INFO     0 reg images.                                                            train_util.py:1511
                    WARNING  no regularization images / 正則化画像が見つかりませんでした              train_util.py:1516
                    INFO     [Dataset 0]                                                              config_util.py:544
                               batch_size: 5
                               resolution: (1024, 1024)
                               enable_bucket: True
                               network_multiplier: 1.0
                               min_bucket_reso: 256
                               max_bucket_reso: 2048
                               bucket_reso_steps: 64
                               bucket_no_upscale: True

                               [Subset 0 of Dataset 0]
                                 image_dir: "D:\Dataset\DanWinters-01-Train1\img\20_DanWinters style"
                                 image_count: 20
                                 num_repeats: 20
                                 shuffle_caption: False
                                 keep_tokens: 0
                                 keep_tokens_separator:
                                 caption_dropout_rate: 0.0
                                 caption_dropout_every_n_epoches: 0
                                 caption_tag_dropout_rate: 0.0
                                 caption_prefix: None
                                 caption_suffix: None
                                 color_aug: False
                                 flip_aug: False
                                 face_crop_aug_range: None
                                 random_crop: False
                                 token_warmup_min: 1,
                                 token_warmup_step: 0,
                                 is_reg: False
                                 class_tokens: DanWinters style
                                 caption_extension: .txt


                    INFO     [Dataset 0]                                                              config_util.py:550
                    INFO     loading image sizes.                                                      train_util.py:794
100%|█████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 499.60it/s]
                    INFO     make buckets                                                              train_util.py:800
                    WARNING  min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is   train_util.py:817
                             set, because bucket reso is defined by image size automatically /
                             bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計
                             算されるため、min_bucket_resoとmax_bucket_resoは無視されます
                    INFO     number of images (including repeats) /                                    train_util.py:846
                             各bucketの画像枚数（繰り返し回数を含む）
                    INFO     bucket 0: resolution (832, 1088), count: 60                               train_util.py:851
                    INFO     bucket 1: resolution (832, 1152), count: 20                               train_util.py:851
                    INFO     bucket 2: resolution (896, 1088), count: 80                               train_util.py:851
                    INFO     bucket 3: resolution (896, 1152), count: 240                              train_util.py:851
                    INFO     mean ar error (without repeats): 0.010911330678104564                     train_util.py:856
                    INFO     preparing accelerator                                                  train_network.py:226
accelerator device: cuda
                    INFO     loading model for process 0/1                                         sdxl_train_util.py:31
                    INFO     load StableDiffusion checkpoint:                                      sdxl_train_util.py:71
                             D:/ComfyUI/ComfyUI/models/checkpoints/sd_xl_base_1.0.safetensors
                    INFO     building U-Net                                                       sdxl_model_util.py:192
                    INFO     loading U-Net from checkpoint                                        sdxl_model_util.py:196
2024-03-28 19:43:03 INFO     U-Net: <All keys matched successfully>                               sdxl_model_util.py:202
                    INFO     building text encoders                                               sdxl_model_util.py:205
                    INFO     loading text encoders from checkpoint                                sdxl_model_util.py:258
                    INFO     text encoder 1: <All keys matched successfully>                      sdxl_model_util.py:272
2024-03-28 19:43:04 INFO     text encoder 2: <All keys matched successfully>                      sdxl_model_util.py:276
                    INFO     building VAE                                                         sdxl_model_util.py:279
                    INFO     loading VAE from checkpoint                                          sdxl_model_util.py:284
                    INFO     VAE: <All keys matched successfully>                                 sdxl_model_util.py:287
                    INFO     Enable SDPA for U-Net                                                    train_util.py:2537
import network module: networks.lora
                    INFO     [Dataset 0]                                                              train_util.py:1948
                    INFO     caching latents.                                                          train_util.py:915
                    INFO     checking cache validity...                                                train_util.py:925
100%|██████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<?, ?it/s]
                    INFO     caching latents...                                                        train_util.py:962
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:07<00:00,  2.53it/s]
2024-03-28 19:43:14 INFO     create LoRA network. base dim (rank): 8, alpha: 1.0                             lora.py:811
                    INFO     neuron dropout: p=None, rank dropout: p=None, module dropout: p=None            lora.py:812
                    INFO     create LoRA for Text Encoder 1:                                                 lora.py:903
                    INFO     create LoRA for Text Encoder 2:                                                 lora.py:903
                    INFO     create LoRA for Text Encoder: 264 modules.                                      lora.py:911
                    INFO     create LoRA for U-Net: 722 modules.                                             lora.py:919
                    INFO     enable LoRA for text encoder                                                    lora.py:962
                    INFO     enable LoRA for U-Net                                                           lora.py:967
prepare optimizer, data loader etc.
                    INFO     use AdamW optimizer | {'scale_parameter': False, 'relative_step': False, train_util.py:3819
                             'warmup_init': False}
Traceback (most recent call last):
  File "D:\Kohya SS 2\kohya_ss\sd-scripts\sdxl_train_network.py", line 184, in <module>
    trainer.train(args)
  File "D:\Kohya SS 2\kohya_ss\sd-scripts\train_network.py", line 349, in train
    optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
  File "D:\Kohya SS 2\kohya_ss\sd-scripts\library\train_util.py", line 3821, in get_optimizer
    optimizer = optimizer_class(trainable_params, lr=lr, **optimizer_kwargs)
TypeError: AdamW.__init__() got an unexpected keyword argument 'scale_parameter'
Traceback (most recent call last):
  File "C:\Users\stepa\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\stepa\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Kohya SS 2\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "D:\Kohya SS 2\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
    args.func(args)
  File "D:\Kohya SS 2\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "D:\Kohya SS 2\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['D:\\Kohya SS 2\\kohya_ss\\venv\\Scripts\\python.exe', 'D:\\Kohya SS 2\\kohya_ss/sd-scripts/sdxl_train_network.py', '--bucket_no_upscale', '--bucket_reso_steps=64', '--cache_latents', '--cache_latents_to_disk', '--caption_extension=.txt', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--gradient_checkpointing', '--learning_rate=0.0009', '--logging_dir=D:/Dataset/DanWinters-01-Train1\\log', '--lr_scheduler=constant', '--lr_scheduler_num_cycles=15', '--max_data_loader_n_workers=0', '--max_grad_norm=1', '--resolution=1024,1024', '--max_token_length=225', '--max_train_steps=1200', '--mixed_precision=bf16', '--network_alpha=1', '--network_dim=8', '--network_module=networks.lora', '--no_half_vae', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--optimizer_type=AdamW', '--output_dir=D:/Dataset/DanWinters-01-Train1\\model', '--output_name=DanWinters', '--pretrained_model_name_or_path=D:/ComfyUI/ComfyUI/models/checkpoints/sd_xl_base_1.0.safetensors', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=bf16', '--train_batch_size=5', '--train_data_dir=D:/Dataset/DanWinters-01-Train1\\img', '--sdpa']' returned non-zero exit status 1.

stepahin Mar 28, 2024
Author

(error because of additional parameters, thanks @Mosfett1975)

Ok good with AdamW with batch size 1 I got 1.78it/s but 6000 steps instead of 1200, and I got a longer total time than on batch size 5 and only 10 GB out of 24 GB vram was used. But on batch size 5 AdamW is really about 12% faster than Adafactor. Thanks!

Mosfett1975 Mar 28, 2024

You can set limit for num of steps in Max train steps and it will be priority over Epoch num

Disty0 Mar 28, 2024

Also don't use --gradient_checkpointing if you don't need it, it is significantly slower with gradient checkpointing.

Mosfett1975 · 2024-03-28T18:51:36Z

Mosfett1975
Mar 28, 2024

It's because you're trying to use this: scale_parameter=False relative_step=False warmup_init=False

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Train speed on 4090, Windows. 2.50s/it is not ok, right? #2166

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Train speed on 4090, Windows. 2.50s/it is not ok, right? #2166

stepahin Mar 27, 2024

Replies: 3 comments · 4 replies

Mosfett1975 Mar 27, 2024

Disty0 Mar 28, 2024

stepahin Mar 28, 2024 Author

stepahin Mar 28, 2024 Author

Mosfett1975 Mar 28, 2024

Disty0 Mar 28, 2024

Mosfett1975 Mar 28, 2024

stepahin
Mar 27, 2024

Replies: 3 comments 4 replies

Mosfett1975
Mar 27, 2024

Disty0
Mar 28, 2024

stepahin Mar 28, 2024
Author

stepahin Mar 28, 2024
Author

Mosfett1975
Mar 28, 2024