Reading a webdataset from cloud storage

[omri-cavnue]

Describe the question.

Hello,

I am curious how to read a webdataset from cloud storage (e.g. s3 or gcs)? I have tried using a pipe, providing the gs:// URI, and other options into the sample code provided here but cannot open the webdataset. Is there a way to do this, or do I need to combine the webdataset reader with another reader?

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report

[jantonguirao]

Hi. We do support S3 storage but no gcs at the moment. The example you are referring is relevant, webdataset reader can take S3 uris instead of local paths. Could you please try with a S3 location?

[omri-cavnue]

Hi @jantonguirao we are using GCS, so S3 was just an example in my description, but I would need GCS support. Is the alternative to use this then?

[jantonguirao]

Yes, if you are using GCS, then your only option is to use ExternalSource at the moment.