How to load dataset from MinIO?

[sasakiyori]

I have no idea how to specific the endpoint for MinIO.
If endpoint is not configured, PermissionError: Forbidden is returned.
But if I set endpoint_url like below, it returns FileNotFoundError: Unable to find 's3://datasets/mybucket/mydataset', it seems go to other protocols but not s3.

def load_from_s3():
    storage_options = {
        "key": "minioadmin",
        "secret": "minioadmin",
        "client_kwargs": {
            "endpoint_url": "http://localhost:9000",
        },
    }
    data = load_dataset(
        "json",
        data_files="s3://mybucket/mydataset",
        storage_options=storage_options,
    )

[bmaxdk]

Your issue could be related to dataset path or bucket permissions. is your data_files could be data_files="s3://mybucket/mydataset.json"?

[sasakiyori]

After I upgrade datasets package and revise the data_files param, it works.

def load_from_s3():
    storage_options = {
        "key": "minioadmin",
        "secret": "minioadmin",
        "endpoint_url": "http://localhost:9000",
        "use_ssl": False,
    }
    data = load_dataset(
        "json",
        data_files="s3://mybucket/mydataset/train/myfile.json",
        storage_options=storage_options,
    )

But how can I achieve the result like below? I want to load whole dataset without specific data file paths.

# load from local dir
dataset = load_dataset("/path/to/mydataset", split="train")
# or
dataset = load_dataset("/path/to/mydataset/train")