Error in get_bufr for sites where no data is available within the time_limit time range

[BaptisteVandecrux]

2024-04-23 06:11:47,977; INFO; pypromice.postprocess.get_bufr; ####### Processing SDM #######
2024-04-23 06:11:47,977; INFO; pypromice.postprocess.get_bufr; Generating /data/pypromice_aws/pypromice/postprocess/BUFR_out/SDM.bufr from /data/pypromice_aws/aws-l3/tx/SDM/SDM_hour.csv
2024-04-23 06:11:48,216; INFO; pypromice.postprocess.real_time_utilities; TIMESTAMP: 2023-12-27 23:00:00
2024-04-23 06:11:48,216; INFO; pypromice.postprocess.real_time_utilities; finding positions
2024-04-23 06:11:48,217; INFO; pypromice.postprocess.real_time_utilities; last transmission: 2024-04-23 06:00:0
2024-04-23 06:11:48,217; INFO; pypromice.postprocess.real_time_utilities; last transmission: 2024-04-23 06:00:00
2024-04-23 06:11:48,228; ERROR; pypromice.postprocess.get_bufr; Failed processing SDM
Traceback (most recent call last):
  File "pandas/_libs/index.pyx", line 581, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 2606, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 2630, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1703718000000000000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 549, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas/_libs/index.pyx", line 583, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: Timestamp('2023-12-27 23:00:00')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 584, in get_loc
    return Index.get_loc(self, key)
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: Timestamp('2023-12-27 23:00:00')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pypromice/postprocess/get_bufr.py", line 388, in get_bufr
    station_position = process_station(
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pypromice/postprocess/get_bufr.py", line 209, in process_station
    latest_data = get_latest_data(
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pypromice/postprocess/real_time_utilities.py", line 74, in get_latest_data
    s_current = df_limited.loc[last_valid_index]
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1103, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1343, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1293, in _get_label
    return self.obj.xs(label, axis=axis)
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 4095, in xs
    loc = index.get_loc(key)
  File "/home/aws/.local/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 586, in get_loc
    raise KeyError(orig_key) from err
KeyError: Timestamp('2023-12-27 23:00:00')

I have copied the logs to process_stdout_err_SDM and process_stderr_err_SDM

@PennyHow , @ladsmund could you have a look at it?

[BaptisteVandecrux]

For info, this is due to the fact that the last transmission with good *_i data is more than 91 days old.
Code crashes when there are recent coordinates but no recent instantaneous values.

This issue will be fixed:

• when SDM is visited in about 2 weeks
  and/or
• when a smarter estimation of the station position is developed, which would also address:
  Make a postprocess script that update AWS_metadata.csv #142
  Missing "latest_positions" for SCO_U and SCO_L in AWS_latest_position.csv #173
  Interpolate or insert static values for station coordinates when no GPS is available #236

[BaptisteVandecrux]

SDM is fixed but LYN_T has the same issue:

2024-06-25 11:11:53,719; INFO; pypromice.postprocess.get_bufr; ####### Processing LYN_T #######
2024-06-25 11:11:53,719; INFO; pypromice.postprocess.get_bufr; Generating /mnt/data/aws/pypromice_aws/aws-bufr/BUFR_out/LYN_T.bufr from /mnt/data/aws/pypromice_aws/aws-l2/tx/LYN_T/LYN_T_hour.csv
2024-06-25 11:11:53,971; INFO; pypromice.postprocess.real_time_utilities; TIMESTAMP: 2023-12-14 00:00:00
2024-06-25 11:11:53,971; INFO; pypromice.postprocess.real_time_utilities; finding positions
2024-06-25 11:11:53,973; INFO; pypromice.postprocess.real_time_utilities; last transmission: 2024-04-19 11:00:00
2024-06-25 11:11:54,003; ERROR; pypromice.postprocess.get_bufr; Failed processing LYN_T
Traceback (most recent call last):
  File "pandas/_libs/index.pyx", line 548, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 2263, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 2273, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1702512000000000000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3803, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 516, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas/_libs/index.pyx", line 550, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: Timestamp('2023-12-14 00:00:00')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 736, in get_loc
    return Index.get_loc(self, key, method, tolerance)
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    raise KeyError(key) from err
KeyError: Timestamp('2023-12-14 00:00:00')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pypromice/postprocess/get_bufr.py", line 388, in get_bufr
    station_position = process_station(
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pypromice/postprocess/get_bufr.py", line 209, in process_station
    latest_data = get_latest_data(
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pypromice/postprocess/real_time_utilities.py", line 74, in get_latest_data
    s_current = df_limited.loc[last_valid_index]
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexing.py", line 1312, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexing.py", line 1260, in _get_label
    return self.obj.xs(label, axis=axis)
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/generic.py", line 4056, in xs
    loc = index.get_loc(key)
  File "/home/aws/miniconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexes/datetimes.py", line 738, in get_loc
    raise KeyError(orig_key) from err
KeyError: Timestamp('2023-12-14 00:00:00')

[BaptisteVandecrux]

I think this has been fixed in #263
I don't see the error any more in the logs