bug-fix: account for gradient accumulation steps in !sample_packing_eff_est sample_packing case

[kallewoof]

This addresses issues e.g. with warmup_ratio becoming a multiple of the gradient accumulation steps.

See also #977.

[kallewoof]

Actually I'm not comfortable enough to say this is ready for merging, but I'm using this patch locally to address warmup ratio being off. Perhaps it should be specifically applied there for now until this is sorted out.

[winglian]

@kallewoof warmup_ratio is the total number of steps over all epochs/the entire training run. how many gpus are you using, what hyperparams are you using and what step are you seeing warmup peak at?

[kallewoof]

@winglian I can post more detailed examples in some days, but this is what I get, with the above patch applied:

• Peak learning rate (warmup endpoint, where learning rate = the specified value in axolotl.yml): epoch 1.0
• Gradient accumulation steps: 2
• Micro batch size: 2
• Epochs: 4
• Warmup ratio: 0.25

On main I hit peak learning rate at epoch 2.0.

I will try other gas values with and without the patch and post more details if needed (but as noted, it will be some days before I get to it, I'm afraid).

Edit: I should also note that the actual steps to train remains the same with and without the patch.

[kallewoof]

Will revisit this later unless someone fixes in the meantime.