Better defaults for StreamingDataset subclasses

[snarayan21]

With better defaults and new features being added to Streaming, this PR makes sure those improvements are reflected in llm-foundry. Specifically:

• Default shuffle_algo changed to py1e: More balanced downloads and cache limit performance
• Default partition_algo changed to relaxed: Elastic determinism and resumptions on many more numbers of nodes
• Default predownload changed to None: Lower predownload for more balanced download demand. Will be set by StreamingDataset.
• Default num_canonical_nodes changed to None: Now set to equal the number of physical nodes, unless using py1s and py2s shuffle algorithms, in which case will equal 64 * physical nodes.
• Default shuffle_block_size changed to None: Now set based on num_canonical_nodes to give a consistently good shuffle quality without needing a crazy amount of downloads.

Defaults were changed in Streaming v0.7.0 with this commit: mosaicml/streaming@93bf054

[karan6181]

Overall looks good to me. Do we need to change the defaults in

llm-foundry/llmfoundry/data/denoising.py

Line 473 in 84c86e3

dataset = StreamingTextDataset(

as well as in

llm-foundry/llmfoundry/data/finetuning/dataloader.py

Line 127 in 84c86e3

dataset = dataset_constructor.build_from_streaming(

?

[snarayan21]

Overall looks good to me. Do we need to change the defaults in

llm-foundry/llmfoundry/data/denoising.py

Line 473 in 84c86e3

dataset = StreamingTextDataset(

as well as in

llm-foundry/llmfoundry/data/finetuning/dataloader.py

Line 127 in 84c86e3

dataset = dataset_constructor.build_from_streaming(

?

oop yes we do. Changing that now.

[karan6181]

Can you please also update the epoch_size data type to Optional[Union[int, str]] which is inline with the streaming args? Thanks!

[dakinggg]

There are some more to change in denoising.py. Will approve after that, lgtm otherwise.

[dakinggg]

@snarayan21 please post a PSA in the research slack once you merge this. I'll also include a PSA in the next release notes, but want to make sure research knows that the defaults here are changing.

[karan6181]

LGTM. Thank You!