Preparing a Dataset

Multi-concept LoRAs

Multi-Concept Balancing

There may be a case where you want to train two (or more) things into a LoRA, but the amount of images you have is imbalanced, resulting in the LoRA favouring one thing over another. If this is the case, you should have the training put just as many steps into every single thing you want to train. Let's go through a worked example to demonstrate how to do this with kohya-ss.

You have three dog breeds you want to train into a single LoRA: "Borzoi", "Pug", and "Akita".

For the images, you must ensure that no folder is more than double the number of images of any other folder. For example, Borzoi has 50 images while Pug has 10 images. Borzoi is 5 times as much images as Pug, so decrease number of images in Borzoi and/or increase number of images for Pug.

Let's say you ended up with this:

• Borzoi = 43 images (34% greater than Pug, 15% less than Akita)
• Pug = 32 images (25% less than Borzoi, 37% less than Akita)
• Akita = 51 images (18% more than Borzoi, 59% more than Pug)

The amount of training time is going to favour Akita more than the other two breeds, meaning that attempting to gen borzois may gen akita-borzois hybrids instead, while attempting to gen pugs may gen borzoi-akita-pug hybrids instead. Since there's 3 concepts, we need to have one third of each epoch dedicated to each breed. We do this through "repeats", which are the numbers next to your concept folders' names. Your dataset folders may look something like this to begin with:

1_borzoi
1_pug
1_akita

Let's focus on balancing the Borzoi and Pug folders first, through the following steps:

1. Put both image counts into a fraction (X and Y respectively).
2. Slightly adjust one or both integers so they're divisible by the same factor.
3. Simplify the fraction.
4. Repeat 2 and 3 until satisfied.
5. Use the new denominator (from Y) as repeats for the original numerator (on X), and the new numerator (from X) as repeats for the original denominator (from Y).

43/32 ≈ 40/30 = 4/3
Final repeats: [3, 4]

Proof they have same number of steps:
[3, 4] * [43, 32] = [129, 128]
129/128 ≈ 1.008 ≈ 1

So now we have this:

3_borzoi
4_pug
1_akita

Let's balance Borzoi and Pug to Akita. Similar process to before, but we will use the Borzoi folder times by repeats and the Akita folder instead.

(3*43)/51 = 129/51 ≈ 128/52 = 32/13 ≈ 32/12 = 8/3
3 * [3, 4] = [9, 12]
Final repeats: [9, 12, 8]

Proof they have same number of steps:
[9, 12, 8] * [43, 32, 51] = [387, 384, 408]
387/408 ≈ 0.949 ≈ 1
384/408 ≈ 0.941 ≈ 1

After all that, we are left with this:

9_borzoi
12_pug
8_akita

The above folder repeats will ensure that all the concepts in the multi-concept LoRA get equal amounts of training on them.