era5.download-era5 Task Failure Due to API Endpoint Error

[khelfen]

Description

The era5.download-era5 task in the powerd-status-quo-processing-pipeline DAG fails repeatedly due to an error with the Copernicus Climate Data Store (CDS) API. This issue prevents the successful retrieval of ERA5 reanalysis data.

Error Message

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels

Cause

The task attempts to retrieve data from the API endpoint https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels, which results in a 404 Not Found error. Possible reasons include:

• The endpoint URL is incorrect or deprecated.
• Required request parameters are missing or malformed.
• API credentials are invalid or improperly configured.

Expected Behavior

The task should connect to the CDS API successfully and retrieve the specified ERA5 data, storing it in the configured location.

Steps to Reproduce

1. Trigger the powerd-status-quo-processing-pipeline DAG.
2. Wait for the era5.download-era5 task to run.
3. Observe the failure in the logs.

Impact

• The pipeline execution is blocked at this stage, preventing downstream tasks from running.
• Users are unable to process or analyze ERA5 datasets without manual intervention.

Proposed Solution

1. Verify the validity and accessibility of the CDS API endpoint.
2. Test the API call outside of the DAG to isolate and debug issues.
3. If the endpoint has changed or is deprecated, update the script with the correct URL.

Relevant Logs

[2024-11-27, 13:28:10 UTC] {era5.py:393} INFO - Requesting *** for feature height...
[...]
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
[...]
File ".../code/src/egon/data/datasets/era5.py", line 167, in download_era5
  cutout.prepare()