Reading a single file from s3 with the filesystem resource

[guillaumecherel]

dlt version

1.5.0

Describe the problem

Reading a single file with the filesystem resource don’t work.

Targeting the file directly in the bucket_url yields an empty list:

In [2]: import dlt
   ...: from dlt.sources.filesystem import filesystem
   ...: list(filesystem(bucket_url=f"s3://{bucket}/data/folder/my_file.csv"))
Out[2]: []

Passing the filename to the file_glob parameter crashes:

In [1]: import dlt
   ...: from dlt.sources.filesystem import filesystem
   ...: list(filesystem(bucket_url=f"s3://{bucket}/data/folder/", file_glob="my_file.csv"))
...
ResourceExtractionError: In processing pipe filesystem: extraction of resource filesystem in generator filesystem caused an exception: [Errno 22] Bad Request

Adding a * anywhere in the file glob works:

In [1]: import dlt
   ...: from dlt.sources.filesystem import filesystem
   ...: list(filesystem(bucket_url=f"s3://{bucket}/data/folder/", file_glob="my_file.csv*"))
Out[2]: 
[{'file_name': 'my_file.csv',
  'relative_path': 'my_file.csv',
  ...}]

Expected behavior

Either one of the first two methods above or both should return a list with as only element the expected file metadata.

Steps to reproduce

Start a fresh python interpreter and copy / paste the code above. Change the paths accordingly. Restart the interpreter at each attempt.

Operating system

Linux

Runtime environment

Local

Python version

3.11

dlt data source

filesystem

dlt destination

DuckDB

Other deployment details

No response

Additional information

No response

[mohamedmeqlad99]

he issue might be a bug in the filesystem resource where it doesn't correctly handle direct file paths or specific glob patterns.

A fix could involve updating the resource to correctly parse and handle direct file paths and glob patterns without requiring wildcards.

[rudolfix]

@guillaumecherel we have a test that globs single files on all filesystems. Could you past the full exception trace? maybe there are some details of failed exception