Support loading datasets saved via save_to_disk

[fozziethebeat]

Description

Fixes

Uses load_from_disk when no data files are listed in a local directory dataset. This is to ensure datasets saved via save_to_disk are properly loaded.

Motivation and Context

Datasets saved via save_to_disk currently can't be loaded without explicitly listing all files.
#1430

How has this been tested?

Created a local dataset saved via save_to_disk and trained a model on it.

Screenshots (if appropriate)

Types of changes

Small code change

Social Handles (Optional)

fozziethebeat on all the platforms

[winglian]

lgtm. @NanoCode012 ?

[winglian]

Looks like there are some issues with this change and the current integration tests we have, especially when loading remote datasets.

[fozziethebeat]

I'll look through the docker tests that failed and add unittests for what's failing.

[fozziethebeat]

I think this should fix the tests. The updates to main require tokenizer_config to be set in the unittests given that its not using the config utils that auto-set the field.

[fozziethebeat]

One E2E test suite failed with this error:

Writing manifest to image destination
Unpacking OCI image
   • unpacking rootfs ...
Runner terminated.
Stopping app - uncaught exception raised locally: RemoteError('Image build for im-RlXPRxFbKAnNrpADzxyalb terminated due to external shut-down. Please try again.').

This seems like it needs a re-run. The other E2E suite completed successfully

[fozziethebeat]

yay~