-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incidence Angle Layer is not correctly read by ARIA-Tools [edit: buildVRT does not work with new GUNW products] #46
Comments
This leads to an issue in ARIA-tools as the initial E.g. with the newer products if you run the following
The VRT that is generated doesn't contain relative paths:
As opposed to the older products which generate the expected VRT and don't lead to issues in metadata layer extraction:
|
I think its related to #44 . Darn... |
And finally, here is the dockerfile for the older topsApp "plugin" - which installs netcdf library via pip. https://github.com/aria-jpl/topsApp_pge/blob/develop/docker/Dockerfile |
Here are some logs using this gist to demonstrate issues with packaging. make_geocube.log There are pretty standard pyproj warnings. This illustrates the packaging workflow. There are some additional raster layers generated by the CL script |
To summarize, it's hard to say if this netcdf issue persisted throughout the development of this plugin because:
I am not sure of the best way to proceed other than to preserve the existing packaging and manually update the relevant raster layers using If there is some other environment issue that could be at play, it would great to now a roadmap. |
So, I installed
To be clear, this was a change on probing the product, not generating it. Still, the error with |
The buildvrt issue remains and maybe it is isolated there. When I call info on the newer and older products they seem identical. https://gist.github.com/cmarshak/5948c84962c50c4a37e8e8fce358214d |
My guess is that it has something to do with Newer version:
vs. Older version:
|
This is due to an upgrade of the pyproj library, and a "fun" gotacha: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6 Fix is to go through the |
Ok, David brought this to my attention when we updated the packaing scripts for the newest ISCE2 (e.g. 'master' --> 'reference'). Looks like this issue was "sneaking" by to the final products because the product worked in the ways we had outlined to inspect the product (e.g. QGIS and panoply as seen above). I am confident it is occurring in the makeGeocube.py https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/3e9aea8580f98b33c110410c418f9dc8e5124c7a/isce2_topsapp/packaging_utils/makeGeocube.py I will update this and test out the new product. |
The VRT issue that @sssangha still does not work even after updating the pyproj and removing those warnings from |
To determine gdal issue that is causing Specifically, ref: sec: again for this GUNW (2.0.4): https://search.asf.alaska.edu/#/?searchType=List%20Search&searchList=S1-GUNW-A-R-137-top[…]09_20210728-015824-35934N_33889N-PP-5ccf-v2_0_4-amplitude |
I believe that I found the source of this issue. The
That second number should be positive for most gdal geotransforms. Here is a standard one from the other product I hope we can figure out where this is getting written. Relevant pieces of the plugin can roughly be determined from this template: These functions are then in https://github.com/ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/blob/dev/isce2_topsapp/packaging_utils/isce_functions.py |
I have a draft fix for this bug. It's ugly and honestly don't know how it was introduced (by me). Hoping its correct now. We will clearly have to do more tests to assure its correct. May want to make it slightly prettier. |
Yes believe you are correct. If i recall geotrans in gdal when requested
should be consistent top left corner of a product. So second pos and 5th
neg would be north south east west stored image.
@ sim can you also confirm grid spacings to be consistent. Should be
integer increment with dem spacing of old products
…On Mon, Jan 31, 2022 at 6:47 PM Charlie Marshak ***@***.***> wrote:
I have a draft fix for this bug. It's ugly and honestly don't know how it
was introduced (by me).
Hoping its correct now.
We will clearly have to do more tests to assure its correct. May want to
make it slightly prettier.
—
Reply to this email directly, view it on GitHub
<#46 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AESZPSKMPATZLBLQVFXTLY3UY5CUZANCNFSM5M7DADHA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you are subscribed to this thread.Message
ID: <ACCESS-Cloud-Based-InSAR/DockerizedTopsApp/issues/46/1026426383@
github.com>
|
The example secondary scenes are missing a granule resulting in a discontinuous secondary frame (also missing from v204 GUNW metadata). Should be: reference scenes:
secondary scenes:
|
We will make sure with some sample products this works correctly. |
I re-ran the newest dev branch from end-to-end using this notebook: https://gist.github.com/cmarshak/77c251a02c8985153a1c8110b9d25bfe Confident the issue has been resolved. For further inspection, here is the same GUNW 2.0.5 from hyp3 deployment: https://hyp3-isce.asf.alaska.edu/jobs/f07519ee-c239-4b4c-97d0-cd49b0f60566 |
From the above single job - I successfully re-ran the above analysis and verified both ARIA-tools works and that the comparisons with the previous version also works. We can re-open down the road if errors were not properly addressed. |
The issue currently is distilled in that
gdal.buildVRT
does not work on the/science/grids/imagingGeometry
layers of the new GUNW product. Here is the minimum working example:Trying to load
test.vrt
does not work (the xml file does correctly load the various layers). See this comment below for what the GUNW layers should look like.Relevant data:
Older introduction based on ARIA-tools (which needs
gdal.buildVRT
).Based on the minimum working example which compares the previous GUNW products and the hyp3 version, this must be an issue with the current packing scripts in our Hyp3 Plugin (this repo).
This gist shows two minimum working examples:
https://gist.github.com/cmarshak/89c2826bf3334e89da6896d7ccf99f4e
More importantly, here is the data:
And the relevant command line script is:
Also, note that
rasterio
(same env) sees the older products asnetcdf
(correctly) but the newer versions ashdf5
(not correct); moreover, I can't load the transform of the new products. I was able to load the newer version in QGIS so maybe this is arasterio
issue. Still highlighting. This could partially be aboutrasterio
, but maybe not.The error is:
or here in the ARIA-Tools: https://github.com/aria-tools/ARIA-tools/blob/dev/tools/ARIAtools/extractProduct.py#L149-L220
The text was updated successfully, but these errors were encountered: