Any ideas for Speeding up Flux training with ancient P40s?

[JimikoSK]

I may be barking at the wrong tree, since these GPUs are old, but they were cheap. 😅

We have 2 P40s for training and typically get about 35 s/it which is ok, but not super speedy.

Any idea how to speed things up further with the settings we are using or are we at the limit.

We thought full fp32 would speed up, but couldn't get it working without OOM.

Here are the train settings, any idea is appreciated, if there is something we're not thinking of.

pretrained_model_name_or_path = "models/Stable-Diffusion/flux1-dev.safetensors"
#pretrained_model_name_or_path = "models/Stable-Diffusion/flux1-dev-fp8.safetensors"
clip_l = "models/text_encoder/clip_l.safetensors"
#t5xxl = "models/text_encoder/t5xxl_fp16.safetensors"
t5xxl = "models/text_encoder/t5xxl_fp8_e4m3fn.safetensors"
ae = "models/VAE/ae.sft"

timestep_sampling = "shift"
discrete_flow_shift = 3.1582
#timestep_sampling = "sigmoid"
model_prediction_type = "raw"
guidance_scale = 1.0
loss_type = "l2"
noise_offset = 0.10
sample_every_n_steps = 250
sample_prompts = "prompts.txt"

text_encoder_lr = 0.0
learning_rate = 1

network_module = "networks.lora_flux"
network_dim = 32
network_alpha = 32
cpu_offload = false
save_state = true
cache_info = true
xformers = false
sdpa = true
fp8_base = true
highvram = true
seed = 42
mixed_precision = "bf16"
save_precision = "bf16"
t5xxl_dtype = "float8_e4m3fn"

output_name = "lora"
output_dir = "output"
logging_dir = "output"
dataset_config = "dataset_prodigy.toml"
save_every_n_epochs = 2

cache_latents = true
cache_latents_to_disk = false
cache_text_encoder_outputs = true
cache_text_encoder_outputs_to_disk = false

caption_extension = ".txt"

vae_batch_size = 1

resolution = "512,512"
enable_bucket = true
bucket_no_upscale = true

gradient_accumulation_steps = 1
max_train_epochs 7
max_data_loader_n_workers = 4
persistent_data_loader_workers = true
gradient_checkpointing = true

network_train_unet_only = true
max_grad_norm = 0.0
save_model_as = "safetensors"

optimizer_type = "prodigy"
optimizer_args = ["decouple=True, weight_decay=0.01, betas=[0.9,0.99], d_coef=0.8, use_bias_correction=True, safeguard_warmup=True"]

lr_scheduler = "cosine"
clip_skip = 1
min_snr_gamma = 5

[sdbds]

In fact, in China, it is possible to use the optimized configuration for flux training with a 15$ P104

[JimikoSK]

Very nice, though that does not help my P40s I think. 😅