add support for httpfs

[cdoron]

Signed-off-by: [email protected] [email protected]

Hi Roee,
I accidentally found a way to support httpfs.
Please review and check whether we want this support in the-mesh-for-data-flight-module.
Thanks!

[roee88]

Please move out of asset.py. We can create a filesystem package if needed.

[the-mesh-for-data]

done

[roee88]

httpfs seems like a 1 person project with zero adoption. Not saying that we can't use it but we need a requirement here and I'm not sure that we have one.

Also, technically, can you explain how this works? I don't see implementation of any of the FSSpecHandler methods in HttpFs or fs.base.FS

[roee88]

httpfs seems like a 1 person project with zero adoption. Not saying that we can't use it but we need a requirement here and I'm not sure that we have one.

Also, technically, can you explain how this works? I don't see implementation of any of the FSSpecHandler methods in HttpFs or fs.base.FS

Looking around it seems like pyarrow is compatible with fsspec. See http://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems.

fsspec has an HTTPFileSystem implementation so you can use it instead (note that it also requires the aiohttp package):

from fsspec.implementations.http import HTTPFileSystem

[cdoron]

I tried working with the fsspec HTTPFileSystem implementation.
I got the following error:

Traceback (most recent call last):
  File "sample/sample.py", line 39, in <module>
    main(args.port, args.repeat)
  File "sample/sample.py", line 22, in main
    info = client.get_flight_info(
  File "pyarrow/_flight.pyx", line 1237, in pyarrow._flight.FlightClient.get_flight_info
  File "pyarrow/_flight.pyx", line 80, in pyarrow._flight.check_flight_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: 'utf-8' codec can't decode byte 0x86 in position 38: invalid start byte. Detail: Python exception: UnicodeDecodeError

It appears that fsspec HTTPFileSystem is not a friend of the parquet format. I could only get it to work when I applied the following patch to fsspec/implementations/http.py

intake/intake-parquet#18 (comment)

[roee88]

can we set these to fixed versions?

[roee88]

but we don't use config. Is it a placeholder for authentication or something? If so please add a comment