hfsubset CLI failing for v2.2 and v20.1 (Internal error 500)

[taddyb]

Assuming this problem is known, but writing up an issue in case it isn't

Problem:

• Trying to use the HFSubsetCLI (through https://www.lynker-spatial.com/hydrofabric/hfsubset/__docs__/) and getting an internal 500 error running the example from https://noaa-owp.github.io/hydrofabric/articles/05-subsetting.html#hfsubetr

Thoughts on the problem:

• It looks to be a problem with how the v2.2 and v20.1 files are stored within Lynker Spatial's AWS bucket
  • assuming that the subset type refers to the name of the parquet stores
  • Given the weird arrow issue as described in infra/api.dockerfile build fails with latest version of arrow #40 I'm assuming a fix with the output would solve this

Steps to reproduce:

• The following query is successful:
  curl -X GET "https://www.lynker-spatial.com/hydrofabric/hfsubset/subset?identifier=101&identifier_type=comid&layer=divides&layer=flowlines&layer=network&layer=nexus&subset_type=nextgen&version=2.1.1" -H "accept: /"

• The following queries are unsuccessful:

curl -X GET "https://www.lynker-spatial.com/hydrofabric/hfsubset/subset?identifier=101&identifier_type=comid&layer=divides&layer=flowlines&layer=network&layer=nexus&subset_type=nextgen&version=2.2" -H "accept: */*"

curl -X GET "https://www.lynker-spatial.com/hydrofabric/hfsubset/subset?identifier=101&identifier_type=comid&layer=divides&layer=flowlines&layer=network&layer=nexus&subset_type=nextgen&version=20.1" -H "accept: */*"

[mikejohnson51]

You nailed it, this is 100% related to how we distribute the data (gpkg vs parquet) given the 17.0.1 issues we see in arrow. A fix of some sort is coming but I keep hoping arrow just resolves itself 😅

[taddyb]

Fingers crossed Arrow figures out their problems! I'll try some things locally with older versions of arrow in the meantime to get used to the format/managing parquet files.