HLS data query returns incomplete results from 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?'

[ghobart]

This issue began on Sept 4, 2024 when changes were made to the system. I reported this issue through several other channels but it was suggested by USGS EROS User Services that I should post it here as well.

I had been making successful daily queries to the 'https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?' server since early May of 2024 with no issues. The query has not changed in that time, it is composed of a ten day window across 16 utm zones covering the forests of Canada. Prior to Sept 4th my query results were correct spanning the entire ten day window and the entire spatial query bounding box.

Since Sept.4 the server response has been incomplete. The results returned are from only the first day of the ten day temporal query window and only a small segment of the spatial ROI query. The Bounding box query covers UTM7N-UTM22N over Canada's forests but the query result only returns the eastern most zones (20N-22N).

These are the search parameters as passed to the python program SuPer.py from (https://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-super-script/browse/HLS_SuPER.py)

-143.98107185188445,41.522765367128045,-51.013990919045625,71.19280431826473,e:\HLS_Production,2024-10-16T00:00:00Z/2024-10-26T23:59:59Z,{'HLSS30': 'HLSS30_2.0', 'HLSL30': 'HLSL30_2.0'},{'HLSS30': {'COASTAL-AEROSOL': 'B01', 'BLUE': 'B02', 'GREEN': 'B03', 'RED': 'B04', 'RED-EDGE1': 'B05', 'RED-EDGE2': 'B06', 'RED-EDGE3': 'B07', 'NIR-Broad': 'B08', 'NIR1': 'B8A', 'WATER-VAPOR': 'B09', 'CIRRUS': 'B10', 'SWIR1': 'B11', 'SWIR2': 'B12', 'FMASK': 'Fmask', 'VZA': 'VZA', 'VAA': 'VAA', 'SZA': 'SZA', 'SAA': 'SAA'}, 'HLSL30': {'COASTAL-AEROSOL': 'B01', 'BLUE': 'B02', 'GREEN': 'B03', 'RED': 'B04', 'NIR1': 'B05', 'SWIR1': 'B06', 'SWIR2': 'B07', 'CIRRUS': 'B09', 'TIR1': 'B10', 'TIR2': 'B11', 'FMASK': 'Fmask', 'VZA': 'VZA', 'VAA': 'VAA', 'SZA': 'SZA', 'SAA': 'SAA'}},70

[ghobart]

BTW. I noticed that the python scripts have still not been updated on https://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-super-script/browse/HLS_Su.py. The changes on Sept 4th completely broke this code as well.

I submitted the following detailed changes on Sept 5th to USGS Support, for the python scripts to get them running again although the server response is still incorrect. ( My workaround to the server query issue has been to run the query for a single day at a time for each of the 16 utm zones one at a time. Hence 160 queries instead of one query. Far from optimal. :(

Line 164 in HLS_SuPER.py should be
shortname = {'HLSS30': 'HLSS30_2.0', 'HLSL30': 'HLSL30_2.0'}
as per the description in the article. https://lpdaac.usgs.gov/news/important-update-to-cmr-stac-new-identifier-and-search-parameter-format/

IN HLS_Su.py
Line 49 There is no 'numberReturned' key anymore

while search_response['numberReturned'] != 0:
and Line 81
break
It make absolutely no sense to a have break statement at the end of a while loop with no condition so both the while and the break statements can be removed.
Solution : Comment out or delete both line 49 and 81.

Line 59
bandLinks.append(h['assets']['metadata']['href'])
There is no ‘metadata’ key anymore.
Solution : Comment out or delete line 59
*I don’t need the 'metadata' but maybe other people do???

I hope this helps other too.

[waltersdan]

To pile on, we do use the metadata asset (or we did, when it was there).

[ZZMitch]

@ghobart Not sure if these scripts are fixed, but there is a more recently updated HLS_SuPER on the HLS-Data-Resources GitHub.

https://github.com/nasa/HLS-Data-Resources/tree/main/python/scripts/HLS_SuPER

[william-valencia]

@ghobart Are you still having issues with the HLS_SuPER script and CMR Stac? The CMR Stac call seems to work for me.

https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-143.98107185188445,41.522765367128045,-51.013990919045625,71.19280431826473&datetime=2024-10-06T00:00:00Z/2024-10-26T23:59:59Z&collections=HLSL30_2.0,HLSS30_2.0

If you go through the next links, you will get the rest of the data.

[ghobart]

Hi there. Thanks for your message and everyone's contributions. You are all appreciated. :) Yes I am still having issues. Using the latest SuPer.py scripts, only the two coordinate envelope ROI format works. Both the json and 2D shapefile fail for even the smallest test area (a single MGRS tile).I have encountered several issues. The first error involved my shapefile vertices being 3D. An easy fix with geopandas.force2d(). Even then both the 2D json and shapefile fails for even the smallest test area (a single MGRS tile) I get an error RuntimeError: {"errors":["The polygon boundary points are listed in the wrong order. Points must be provided in counter-clockwise order."]} I have tried reversing the order of the polygon points BUT still get the same error. I have tried shapefiles and json files with no success. In the spirit of KISS, I have tried using a single set of 5 2D points to define the polygon and have verified that they are indeed counter-clockwise in order as both shapefiles and json files generated by geopandas. Even then I still get the counter clockwise error. I have even tried making west longitudes positive values but still get the same counter-clockwise order error. Just a thought, could the x and y coordinates be being read backwards? I have not had time to investigate that. The funny thing is that the standard format for ESRI shapefiles is clockwise for the outer ring and counter- clockwise for any INTERIOR rings. https://www.esri.com/content/dam/esrisites/sitecore-archive/Files/Pdfs/library/whitepapers/pdfs/shapefile.pdf *Vertices for a single, ringed polygon are, therefore, always in clockwise order*. Rings defining holes in these polygons have a counterclockwise orientation. " . As I mentioned I did get a two coordinate envelope to work for small areas but I should point out that *every ROI example *on https://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-super-script/browseis incorrect. The examples for 2 coordinate envelopes show no spaces between the values and the comma separators but the script only works when spaces are added. I know this is a small thing but details matter. As I mentioned I can get the SuPer.py scripts to work only for small areas. Prior to Sept 4th changes I was querying with a shapefile of all the forested MGRS tiles for Canada (across 16 UTM zones) for a ten day window which worked flawlessly. Now with these new scripts I am getting several different types of errors sporadically. I am working to track and document these issues and will provide updates when I time permits. Thank you to everyone who has contributed to this project. Your efforts are greatly appreciated. I hope my feedback is helpful.

…

On Tue, Nov 19, 2024 at 6:01 PM william-valencia ***@***.***> wrote: @ghobart <https://github.com/ghobart> Are you still having issues with the HLS_SuPER script and CMR Stac? The CMR Stac call seems to work for me. https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox-143.98107185188445,41.522765367128045,-51.013990919045625,71.19280431826473&datetime=2024-10-06T00:00:00Z/2024-10-26T23:59:59Z&collections=HLSL30_2.0,HLSS30_2.0 If you go through the next links, you will get the rest of the data. — Reply to this email directly, view it on GitHub <#367 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADMMHSK5SV77C6WJBHJN4RD2BPURJAVCNFSM6AAAAABQ2UTC52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBXGE3DEMRYHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[william-valencia]

@ghobart Thanks for your response. Would you be able to provide the underlying cmr-stac GET or POST request that is actually being called when you get the errors?

It would be a call to "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search" I would need the parameters as they are passed to the URL.