Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] SET azimuth time interpolation - Interpolation Time Out of Bounds - Track 106 Exclusively #152

Open
cmarshak opened this issue Sep 1, 2023 · 0 comments
Open

[Bug] SET azimuth time interpolation - Interpolation Time Out of Bounds - Track 106 Exclusively #152

cmarshak opened this issue Sep 1, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@cmarshak
Copy link
Collaborator

cmarshak commented Sep 1, 2023

Describe the bug

This accounts for 207 errors of the 94k jobs submitted. The error log looks like this:

Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 233, in gunw_slc
    nc_path = update_gunw_with_solid_earth_tide(nc_path, "secondary", loc_data['secondary_orbits'])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 474, in update_gunw_with_solid_earth_tide
    solid_earth_tide_ds = compute_solid_earth_tide_from_gunw(gunw_path=gunw_path,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 204, in compute_solid_earth_tide_from_gunw
    tide_e, tide_n, tide_u = compute_enu_solid_earth_tide(orbit_xmls=orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 372, in compute_enu_solid_earth_tide
    azimuth_time_arr = get_azimuth_time_array(orbit_xmls,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/solid_earth_tides.py", line 270, in get_azimuth_time_array
    datetime_isce, rng = orb.geo2rdr([lat, lon, hgt])
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/Orbit/Orbit.py", line 1055, in geo2rdr
    raise Exception('Interpolation time out of bounds')
Exception: Interpolation time out of bounds

Here are 10 input dicts for the errors

{'job_id': '9b8cfae5-4f17-47b8-96a7-4c2ed43844cf',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200817T225941_20200817T230008_033953_03F07B_3FDA',
    'S1A_IW_SLC__1SDV_20200817T230006_20200817T230033_033953_03F07B_0550'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/9b8cfae5-4f17-47b8-96a7-4c2ed43844cf/9b8cfae5-4f17-47b8-96a7-4c2ed43844cf.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8386.931]},
 {'job_id': 'ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200817T225941_20200817T230008_033953_03F07B_3FDA',
    'S1A_IW_SLC__1SDV_20200817T230006_20200817T230033_033953_03F07B_0550'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff/ac35c53e-1b4e-4a1d-9560-34e8b09ec3ff.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8647.712]},
 {'job_id': '79b9485e-9d86-4a1b-a634-d00afb93a9ef',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/79b9485e-9d86-4a1b-a634-d00afb93a9ef/79b9485e-9d86-4a1b-a634-d00afb93a9ef.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8434.798]},
 {'job_id': '172d77d1-c524-43c9-8e73-5b53ccc0f0d9',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20201004T225943_20201004T230010_034653_04092E_B85D',
    'S1A_IW_SLC__1SDV_20201004T230008_20201004T230035_034653_04092E_3208'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/172d77d1-c524-43c9-8e73-5b53ccc0f0d9/172d77d1-c524-43c9-8e73-5b53ccc0f0d9.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8500.854]},
 {'job_id': '6602b23b-d1f1-427a-bd04-bf299ae041c9',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:14+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200922T225943_20200922T230010_034478_04030D_928D',
    'S1A_IW_SLC__1SDV_20200922T230007_20200922T230034_034478_04030D_DD2D'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200829T225942_20200829T230009_034128_03F6AB_2357',
    'S1A_IW_SLC__1SDV_20200829T230007_20200829T230034_034128_03F6AB_6ECC'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/6602b23b-d1f1-427a-bd04-bf299ae041c9/6602b23b-d1f1-427a-bd04-bf299ae041c9.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8770.476]},
 {'job_id': '33692e6b-8835-4a26-990a-0ccccec5502e',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20180124T225946_20180124T230013_020303_022ABA_6976',
    'S1A_IW_SLC__1SDV_20180124T230010_20180124T230048_020303_022ABA_CBE6'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20171219T225947_20171219T230014_019778_021A33_184F',
    'S1A_IW_SLC__1SDV_20171219T230012_20171219T230049_019778_021A33_6C12'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/33692e6b-8835-4a26-990a-0ccccec5502e/33692e6b-8835-4a26-990a-0ccccec5502e.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8454.583]},
 {'job_id': '2645f3f9-f796-41ec-afab-45c7cb8c2f6f',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20180406T225946_20180406T230013_021353_024C06_12D5',
    'S1A_IW_SLC__1SDV_20180406T230011_20180406T230048_021353_024C06_57E4'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180301T225945_20180301T230012_020828_023B72_4B9D',
    'S1A_IW_SLC__1SDV_20180301T230010_20180301T230047_020828_023B72_C96D'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/2645f3f9-f796-41ec-afab-45c7cb8c2f6f/2645f3f9-f796-41ec-afab-45c7cb8c2f6f.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8444.457]},
 {'job_id': 'a5358300-2c10-44c2-bf8c-b5750774baa1',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20200910T225942_20200910T230009_034303_03FCC5_F00F',
    'S1A_IW_SLC__1SDV_20200910T230007_20200910T230034_034303_03FCC5_0D86'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20200314T225933_20200314T230000_031678_03A6F1_1983',
    'S1A_IW_SLC__1SDV_20200314T225958_20200314T230025_031678_03A6F1_931E'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/a5358300-2c10-44c2-bf8c-b5750774baa1/a5358300-2c10-44c2-bf8c-b5750774baa1.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [7856.89]},
 {'job_id': 'd5cacfdd-fb3e-4da8-979b-c168d891f743',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20190224T225952_20190224T230019_026078_02E88A_1102',
    'S1A_IW_SLC__1SDV_20190224T230016_20190224T230054_026078_02E88A_A34D'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180205T225945_20180205T230012_020478_02304E_B53C',
    'S1A_IW_SLC__1SDV_20180205T230010_20180205T230047_020478_02304E_3008'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/d5cacfdd-fb3e-4da8-979b-c168d891f743/d5cacfdd-fb3e-4da8-979b-c168d891f743.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8640.922]},
 {'job_id': '9b91341b-8d28-4ee2-9cb6-d24097545928',
  'job_type': 'INSAR_ISCE',
  'request_time': '2023-08-30T20:11:09+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'geo2rdr_2023-08-30',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16422,
   'granules': ['S1A_IW_SLC__1SDV_20181027T225955_20181027T230022_024328_02A9DB_AD58',
    'S1A_IW_SLC__1SDV_20181027T230020_20181027T230057_024328_02A9DB_85D6'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20180406T225946_20180406T230013_021353_024C06_12D5',
    'S1A_IW_SLC__1SDV_20180406T230011_20180406T230048_021353_024C06_57E4'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/9b91341b-8d28-4ee2-9cb6-d24097545928/9b91341b-8d28-4ee2-9cb6-d24097545928.log'],
  'expiration_time': '2023-10-31T00:00:00+00:00',
  'processing_times': [8542.152]}

These are concentrated exclusively in 106.
@cmarshak cmarshak added the bug Something isn't working label Sep 1, 2023
@cmarshak cmarshak changed the title [Bug] SET azimuth time interpolation - Interpolation Time Out of Bounds [Bug] SET azimuth time interpolation - Interpolation Time Out of Bounds - Track 106 Exclusively Sep 1, 2023
@cmarshak cmarshak mentioned this issue Sep 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cmarshak