Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quick Terminations for Hyp3 Plugin #155

Open
cmarshak opened this issue Sep 21, 2023 · 0 comments
Open

Quick Terminations for Hyp3 Plugin #155

cmarshak opened this issue Sep 21, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@cmarshak
Copy link
Collaborator

cmarshak commented Sep 21, 2023 •
edited
Loading

Approximately 200 jobs failed instantly (out of batch of ~10k) with (indecipherable) gdal errors such as:

...
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27438
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27439
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27440
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27441
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27442
2023-09-06 22:37:31,823 - fiona.ogrext - DEBUG - Next index: 27443
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27444
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27445
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27446
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27447
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27448
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27449
2023-09-06 22:37:31,824 - fiona.ogrext - DEBUG - Next index: 27450
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27451
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27452
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27453
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27454
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27455
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27456
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27457
2023-09-06 22:37:31,825 - fiona.ogrext - DEBUG - Next index: 27458
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27459
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27460
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27461
2023-09-06 22:37:31,826 - fiona.ogrext - DEBUG - Next index: 27462
2023-09-06 22:37:31,827 - fiona.ogrext - DEBUG - Next index: 27463
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27464
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27465
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27466
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27467
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27468
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27469
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27470
2023-09-06 22:37:31,828 - fiona.ogrext - DEBUG - Next index: 27471
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27472
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27473
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27474
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27475
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27476
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27477
2023-09-06 22:37:31,829 - fiona.ogrext - DEBUG - Next index: 27478
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27479
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27480
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27481
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27482
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27483
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27484
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27485
2023-09-06 22:37:31,830 - fiona.ogrext - DEBUG - Next index: 27486
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27487
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27488
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27489
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27490
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27491
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27492
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27493
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27494
2023-09-06 22:37:31,831 - fiona.ogrext - DEBUG - Next index: 27495
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27496
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27497
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27498
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27499
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27500
2023-09-06 22:37:31,832 - fiona.ogrext - DEBUG - Next index: 27501
2023-09-06 22:37:31,833 - fiona.ogrext - DEBUG - Next index: 27502
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27503
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27504
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27505
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27506
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27507
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27508
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27509
2023-09-06 22:37:31,834 - fiona.ogrext - DEBUG - Next index: 27510
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27511
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27512
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27513
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27514
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27515
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27516
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27517
2023-09-06 22:37:31,835 - fiona.ogrext - DEBUG - Next index: 27518
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27519
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27520
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27521
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27522
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27523
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27524
2023-09-06 22:37:31,836 - fiona.ogrext - DEBUG - Next index: 27525
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27526
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27527
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27528
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27529
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27530
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27531
2023-09-06 22:37:31,837 - fiona.ogrext - DEBUG - Next index: 27532
2023-09-06 22:37:31,838 - fiona.ogrext - DEBUG - Next index: 27533
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27534
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27535
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27536
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27537
2023-09-06 22:37:31,839 - fiona.ogrext - DEBUG - Next index: 27538
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27539
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27540
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27541
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27542
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27543
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27544
2023-09-06 22:37:31,840 - fiona.ogrext - DEBUG - Next index: 27545
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27546
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27547
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27548
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27549
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27550
2023-09-06 22:37:31,841 - fiona.ogrext - DEBUG - Next index: 27551
2023-09-06 22:37:31,842 - fiona.ogrext - DEBUG - Next index: 27552
2023-09-06 22:37:31,842 - fiona.ogrext - DEBUG - Next index: 27553
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27554
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27555
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27556
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27557
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27558
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27559
2023-09-06 22:37:31,843 - fiona.ogrext - DEBUG - Next index: 27560
2023-09-06 22:37:31,844 - fiona.ogrext - DEBUG - Next index: 27561
2023-09-06 22:37:31,844 - fiona.ogrext - DEBUG - Next index: 27562
2023-09-06 22:37:31,844 - fiona.ogrext - DEBUG - Next index: 27563
2023-09-06 22:37:31,880 - fiona.collection - DEBUG - Flushed buffer
2023-09-06 22:37:31,880 - fiona.collection - DEBUG - Stopped session
2023-09-06 22:37:31,880 - fiona.env - DEBUG - Exiting env context: <fiona.env.Env object at 0x7f233deabb50>
2023-09-06 22:37:31,881 - fiona.env - DEBUG - Cleared existing <fiona._env.GDALEnv object at 0x7f233deaf280> options
2023-09-06 22:37:31,881 - fiona._env - DEBUG - Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f233deaf280>.
2023-09-06 22:37:31,881 - fiona._env - DEBUG - Error handler popped.
2023-09-06 22:37:31,881 - fiona._env - DEBUG - Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f233deaf280>.
2023-09-06 22:37:31,881 - fiona.env - DEBUG - Exiting outermost env
2023-09-06 22:37:31,881 - fiona.env - DEBUG - Exited env context: <fiona.env.Env object at 0x7f233deabb50>
2023-09-06 22:37:32,053 - fiona.env - DEBUG - Entering env context: <fiona.env.Env object at 0x7f234f1e2e80>
2023-09-06 22:37:32,053 - fiona.env - DEBUG - Starting outermost env
2023-09-06 22:37:32,054 - fiona.env - DEBUG - No GDAL environment exists
2023-09-06 22:37:32,054 - fiona.env - DEBUG - New GDAL environment <fiona._env.GDALEnv object at 0x7f233ddddeb0> created
2023-09-06 22:37:32,054 - fiona._env - DEBUG - GDAL_DATA found in environment.
2023-09-06 22:37:32,054 - fiona._env - DEBUG - PROJ data files are available at built-in paths.
2023-09-06 22:37:32,054 - fiona._env - DEBUG - Started GDALEnv: self=<fiona._env.GDALEnv object at 0x7f233ddddeb0>.
2023-09-06 22:37:32,055 - fiona.env - DEBUG - Updated existing <fiona._env.GDALEnv object at 0x7f233ddddeb0> with options {}
2023-09-06 22:37:32,055 - fiona.env - DEBUG - Entered env context: <fiona.env.Env object at 0x7f234f1e2e80>
2023-09-06 22:37:32,055 - fiona._env - ERROR - `/vsimem/d28b756358994cdfb4be51956ad71c98' not recognized as a supported file format.
2023-09-06 22:37:32,055 - fiona.env - DEBUG - Exiting env context: <fiona.env.Env object at 0x7f234f1e2e80>
2023-09-06 22:37:32,056 - fiona.env - DEBUG - Cleared existing <fiona._env.GDALEnv object at 0x7f233ddddeb0> options
2023-09-06 22:37:32,056 - fiona._env - DEBUG - Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f233ddddeb0>.
2023-09-06 22:37:32,056 - fiona._env - DEBUG - Error handler popped.
2023-09-06 22:37:32,056 - fiona._env - DEBUG - Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f233ddddeb0>.
2023-09-06 22:37:32,056 - fiona.env - DEBUG - Exiting outermost env
2023-09-06 22:37:32,056 - fiona.env - DEBUG - Exited env context: <fiona.env.Env object at 0x7f234f1e2e80>
Traceback (most recent call last):
  File "fiona/_shim.pyx", line 83, in fiona._shim.gdal_open_vector
  File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: '/vsimem/d28b756358994cdfb4be51956ad71c98' not recognized as a supported file format.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 365, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 361, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 166, in gunw_slc
    loc_data = localize_data(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 46, in localize_data
    out_slc = download_slcs(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 182, in download_slcs
    percent_water_low_res = get_percent_water_from_ne_land(ifg_geo)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 137, in get_percent_water_from_ne_land
    df_world = get_world_df()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/localize_slc.py", line 131, in get_world_df
    df_world = gpd.read_file(io.BytesIO(resp.content))
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/geopandas/io/file.py", line 281, in _read_file
    return _read_file_fiona(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/geopandas/io/file.py", line 322, in _read_file_fiona
    with reader(path_or_bytes, **kwargs) as features:
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/fiona/collection.py", line 555, in __init__
    super(BytesCollection, self).__init__(self.virtual_file, vsi=filetype, **kwds)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/fiona/collection.py", line 162, in __init__
    self.session.start(self, **kwargs)
  File "fiona/ogrext.pyx", line 540, in fiona.ogrext.Session.start
  File "fiona/_shim.pyx", line 90, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: '/vsimem/d28b756358994cdfb4be51956ad71c98' not recognized as a supported file format.

Here are some sample input parameters

[{'expiration_time': '2024-03-05T00:00:00+00:00',
 'job_id': 'cf25cfe9-7058-4b82-8d3b-cb341a1275da',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 21249,
                    'granules': ['S1B_IW_SLC__1SDV_20190703T015756_20190703T015824_016963_01FEAF_7691',
                                 'S1B_IW_SLC__1SDV_20190703T015821_20190703T015848_016963_01FEAF_0A5E'],
                    'secondary_granules': ['S1B_IW_SLC__1SDV_20190528T015754_20190528T015821_016438_01EF17_2F06',
                                           'S1B_IW_SLC__1SDV_20190528T015819_20190528T015846_016438_01EF17_38B0'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/cf25cfe9-7058-4b82-8d3b-cb341a1275da/cf25cfe9-7058-4b82-8d3b-cb341a1275da.log'],
 'name': 'track137-n3',
 'processing_times': [20.156],
 'request_time': '2023-09-06T22:31:58+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'},
{'expiration_time': '2024-03-05T00:00:00+00:00',
 'job_id': '62c0814b-ad15-4aae-8108-de3f692aae02',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 21255,
                    'granules': ['S1A_IW_SLC__1SDV_20190522T020034_20190522T020102_027334_031536_DDC4',
                                 'S1A_IW_SLC__1SDV_20190522T020100_20190522T020117_027334_031536_A526'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20190404T020032_20190404T020100_026634_02FCEC_248C',
                                           'S1A_IW_SLC__1SDV_20190404T020058_20190404T020115_026634_02FCEC_576A'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/62c0814b-ad15-4aae-8108-de3f692aae02/62c0814b-ad15-4aae-8108-de3f692aae02.log'],
 'name': 'track137-n3',
 'processing_times': [20.674],
 'request_time': '2023-09-06T22:31:58+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'},
{'expiration_time': '2024-03-05T00:00:00+00:00',
 'job_id': '88dc9441-e17f-4c90-9774-4370dde8b339',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 21251,
                    'granules': ['S1B_IW_SLC__1SDV_20190925T015826_20190925T015853_018188_0223F8_D54C',
                                 'S1B_IW_SLC__1SDV_20190925T015851_20190925T015918_018188_0223F8_B611'],
                    'secondary_granules': ['S1B_IW_SLC__1SDV_20190820T015824_20190820T015851_017663_0213AD_8E10',
                                           'S1B_IW_SLC__1SDV_20190820T015849_20190820T015916_017663_0213AD_49C0'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/88dc9441-e17f-4c90-9774-4370dde8b339/88dc9441-e17f-4c90-9774-4370dde8b339.log'],
 'name': 'track137-n3',
 'processing_times': [20.131],
 'request_time': '2023-09-06T22:31:51+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}]

Maybe related to #154 and unusual geometries being passed via the geometric metadata retreived from ASF? Bad frame enumeration? Will investigate later. Could also be transient.
@cmarshak cmarshak added the bug Something isn't working label Sep 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cmarshak