feature: add overlap_len option to completion strategy

[kallewoof]

This is useful with smaller datasets, where the default to split the data into context size length chunks (thus only showing each part of the data a single time).

This PR only adds support to the completion prompt strategy, as that's the only one I've used so far.

[winglian]

great idea! I think we could implement this without needing to modify all the other prompt strategies. Right now completion is pretty isolated to https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/src/axolotl/prompt_strategies/completion.py

You'll notice on line 81:

def load(tokenizer, cfg, ds_cfg: Optional[Dict[str, Any]] = None):

there is an optional ds_cfg var.
so if your yml was

datasets:
  - path: ...
     type: completion
     overlap_len: 128

then you can reference that from ds_cfg.overlap_len

[kallewoof]

Sounds good! Are you sure this is not a desired feature for the other types? It would seem that even instruction sets with responses larger than the context size could benefit as well.,

Edit: looking at your suggestion again, I think that actually can be used in the other types as well, so nevermind.

[kallewoof]

We still do want to include the overlap length in the MD5 sum for the prepared data cache. Isolating it inside the completion strat makes that a bit less straightforward. Still thinking of ways to do that cleanly.

Edit: I think I have a solution; it does mean all existing dataset caches are invalidated though, but that was the case with my initial approach as well.

[winglian]

thanks, looks good. Can you run pre-commit run --all-files to lint the changes? thanks!

[kallewoof]

Done.

[kallewoof]

Is this merge-ready, or is there something else needed?