Fix pretraining with iterable/streaming Dataset

[jphme]

Due to various changes pretraining_dataset didn't work anymore, this should fix it (using it without problems with a streaming dataset, works for local + remote).

[NanoCode012]

What would happen if max_steps is set to a really large number? Would it default to only running through the entire dataset? I'm a bit curious how to determine this value.

[jphme]

What would happen if max_steps is set to a really large number? Would it default to only running through the entire dataset? I'm a bit curious how to determine this value.

No, it would run max_steps (and determine the number of epochs accordingly), see https://github.com/huggingface/transformers/blob/6acc27eea853885270dba5313181443d43e31f2c/src/transformers/trainer.py#L1605 .

For IterableDatasets, max_steps is necessary (see https://github.com/huggingface/transformers/blob/6acc27eea853885270dba5313181443d43e31f2c/src/transformers/trainer.py#L574 ).

[kostum123]

Does this pr fix finetune with raw corpus?
dataset type:
completion: raw corpus
{"text": "..."}

[winglian]

lgtm. thanks!