Update LoRA with Dropout & Conv2d Support

[ExponentialML]

This PR updates LoRA with support. More information on the changes here: cloneofsimo/lora#133 .

Also adds custom scaling of ranks for high quality, as well as descriptions for them.

[FurkanGozukara]

You making awesome updates but really need configuration help to prepare a good tutorial video @ExponentialML :)

[ExponentialML]

You making awesome updates but really need configuration help to prepare a good tutorial video @ExponentialML :)

Thanks! This PR is actually ready to go. Just use Lora as you normally would, and test slightly lower learning rates (5e-6).

[saunderez]

I don't seem to be able to generate checkpoints manually using a LORA. Errors out with tensor size mismatch. I receive this error (Rank 16 for unet and tenc for reference).

RuntimeError: The size of tensor a (16) must match the size of tensor b (320) at non-singleton dimension 1

I'm trying to generate it using the model that created the LORA so tensor sizes should match.

I have checkpoints set to generate at end of job only and I've cancelled the last few jobs prior to getting to that point so can't say if it works that way.

Full error below.

Exception compiling checkpoint: The size of tensor a (16) must match the size of tensor b (320) at non-singleton dimension 1
Traceback (most recent call last):
  File "E:\sd\extensions\sd_dreambooth_extension\dreambooth\diff_to_sd.py", line 403, in compile_checkpoint
    merge_loras_to_pipe(loaded_pipeline, lora_path, lora_alpha=config.lora_weight, lora_txt_alpha=config.lora_txt_weight)
  File "E:\sd\extensions\sd_dreambooth_extension\lora_diffusion\lora.py", line 1060, in merge_loras_to_pipe
    collapse_lora(pipline.unet, lora_alpha)
  File "E:\sd\extensions\sd_dreambooth_extension\lora_diffusion\lora.py", line 588, in collapse_lora
    _child_module.lora_up.weight.data
RuntimeError: The size of tensor a (16) must match the size of tensor b (320) at non-singleton dimension 1

[ExponentialML]

Hey @saunderez. I recently caught this bug and will be pushing a fix fairly quick.