WIP: Add road weather station parser #528

meteoDaniel · 2021-10-17T15:47:50Z

Dear @gutzbenj and @amotl this is a first draft of road weather download and parsing for issue #518.

Here are some notes:

pdbufr and libeccodes is required
The directory short names on the server for the different groups does not make sense to me. I do not know what they are standing for.
You can see the debug code in api.py.
The main issue at the moment is the return type of the parser. I have implemented the ability to pass a list of subgroups. I have to find a way to merge the different dataframes. The columns should be all the same differing only
Only the latest file is downloaded at the moment. Unfortunately pdbufr does not work with in memory bytes file, this is why I have to use a temporary stored file.
Do you know how to link this to the issue and mark this as a draft?
The ThreadPoolExecutor is not a real parallel process, it uses threading. You have to use ProcessPoolExecutor to spread the task over CPU cores.

I am looking forward to your first thoughts and feedback.

…omputing

Rebase

… into add_road_weather_station_parser # Conflicts: # wetterdienst/provider/dwd/index.py

codecov · 2021-10-17T15:56:52Z

Codecov Report

Merging #528 (e4bc391) into main (48fc337) will decrease coverage by 3.07%.
The diff coverage is 88.30%.

❗ Current head e4bc391 differs from pull request most recent head 08b55b0. Consider uploading reports for the commit 08b55b0 to get more accurate results

@@            Coverage Diff             @@
##             main     #528      +/-   ##
==========================================
- Coverage   90.96%   87.88%   -3.08%     
==========================================
  Files          85       99      +14     
  Lines        5255     5958     +703     
  Branches      441      484      +43     
==========================================
+ Hits         4780     5236     +456     
- Misses        368      605     +237     
- Partials      107      117      +10

Flag	Coverage Δ
unittests	`87.88% <88.30%> (-3.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...enst/provider/dwd/forecast/metadata/field_types.py	`100.00% <ø> (ø)`
wetterdienst/provider/dwd/observation/fields.py	`94.73% <ø> (ø)`
...t/provider/dwd/observation/metadata/field_types.py	`100.00% <ø> (ø)`
wetterdienst/provider/dwd/radar/sites.py	`100.00% <ø> (ø)`
wetterdienst/provider/dwd/road_weather/__init__.py	`0.00% <0.00%> (ø)`
wetterdienst/provider/dwd/road_weather/api.py	`0.00% <0.00%> (ø)`
wetterdienst/provider/dwd/road_weather/download.py	`0.00% <0.00%> (ø)`
...etterdienst/provider/dwd/road_weather/fileindex.py	`0.00% <0.00%> (ø)`
...nst/provider/dwd/road_weather/metadata/__init__.py	`0.00% <0.00%> (ø)`
...enst/provider/dwd/road_weather/metadata/dataset.py	`0.00% <0.00%> (ø)`
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3fed23...08b55b0. Read the comment docs.

amotl · 2021-10-17T17:06:50Z

Hi Daniel,

nice work! I specifically like that everything is contained within a single package namespace, wetterdienst.provider.dwd.road_weather.

With kind regards,
Andreas.

gutzbenj · 2021-10-18T10:27:49Z

Dear @meteoDaniel ,

thanks for working on this data integration!

Just some notes:

ProcessPoolExecutor was not working when I was using Windows. Windows in comparison to Mac and Linux uses another technique to spawn multiple tasks. I couldn't get behind this issue which is why I removed the multiprocessing feature.
pdbufr: as I had already tried to integrate pdbufr into the lib with Use pdbufr to read DWD radar data in bufr format #482 I had gone into some difficulty where eccodes wasn't aligned to the python interface via pdbufr. I don't know if this has changed within latest updates but I will check it once again.

I will try out the new api one of the following days.

Thanks so far.

Cheers,
Benjamin

…arser

… into add_road_weather_station_parser

meteoDaniel · 2021-11-04T14:55:11Z

First of all I am still confused because I had to remove some caches before I was able to run my code again. Did not know where these changes come from.

Nevertheless I added a tiny function to generalize the dictionaries into a dataframe.

Next issue:
Some of the variables require additional information like timePeriod and the height of the sensor.
Actually the columns looking like that:

In [3]: data.columns
Out[3]: 
Index(['positionOfRoadSensors', 'roadSurfaceCondition', 'timePeriod',
       'waterFilmThickness', 'timePeriod',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform',
       'airTemperature',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform',
       'dewpointTemperature',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform',
       'relativeHumidity',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', 'windDirection',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', 'windSpeed',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', 'timePeriod',
       'maximumWindGustSpeed',
       'heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', 'timePeriod',
       'maximumWindGustDirection', 'timePeriod', 'precipitationType',
       'timePeriod', 'totalPrecipitationOrTotalWaterEquivalent', 'timePeriod',
       'intensityOfPrecipitation', 'timePeriod', 'intensityOfPhenomena',
       'horizontalVisibility'],
      dtype='object')

My ide is to add that information to the metadata because I think these information are constant over time.

What do you think ?

Best regards from holidays in porto by the way :P

meteoDaniel · 2021-11-29T20:39:31Z

I will try to add the meta information into the meta dataframe for the next step.

gutzbenj · 2021-12-01T07:15:15Z

edited

Dear @meteoDaniel ,

for the correct usage of pdbufr we have to find a way to defined ECCODES_DIR for the different possible installation paths. I'm not yet sure if there's a way to do that but probably best would be to ask for ECCODES_DIR environment variable.

Otherwise I've been googling a bit about the road weather data and found not too much BUT
at the general help page [1] (which btw has some helpful documents) I found a xlsx document [2] which seems to contain the list of stations of the road weather dataset! So we may use this xlsx to get the listing, then from this listing take the group of road weather which we have to acquire to return the data.

[1] https://www.dwd.de/DE/leistungen/opendata/hilfe.html
[2] https://www.dwd.de/DE/leistungen/opendata/help/stationen/sws_stations_xls.xlsx?__blob=publicationFile&v=11

meteoDaniel · 2022-01-06T06:01:13Z

Dear @gutzbenj ,I will go on with this task.

Your suggestion with the file list is very good. thanks for the effort to check this out.

The ECCODES_DIR env could be set via a specific Docker image and we will add another part in the documentation to explain what to do to use this new API.

…arser

meteoDaniel · 2022-01-22T14:55:35Z

Dear @gutzbenj and @amotl ,

today I was able to work on that project again. As @gutzbenj mentioned, I have implemented the download of the stations lists to create a metaindex within the class.

Please tell me whats next ;)

Actually I would like to use the ScalarRequestCore do you think this is doable? And if yes, is it possible that some of you give me a short list of stuff I have to take care and to check ?

Other todo's:

Adding a BUFR column name mapping to the wetterdienst human readable column names

meteoDaniel · 2022-01-22T15:33:06Z

Here is the Dockerfile i used to create an image with eccodes

FROM python:3.8.6-slim

ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux

RUN set -ex \
    && apt update \
    && apt install -y apt-transport-https curl git libeccodes0 wget

RUN apt-get -y dist-upgrade && apt-get update
RUN wget https://cmake.org/files/v3.20/cmake-3.20.1.tar.gz
RUN tar -xzvf cmake-3.20.1.tar.gz
WORKDIR cmake-3.20.1/
RUN apt-get update && apt-get install --fix-missing -yqq libssl-dev build-essential
RUN ./bootstrap
RUN make -j$(nproc) && make install

WORKDIR /
RUN apt-get install -y gfortran protobuf-compiler
RUN curl https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.21.0-Source.tar.gz?api=v2 -o /tmp/eccodes-2.21.0-Source.tar.gz
RUN tar -xzf /tmp/eccodes-2.21.0-Source.tar.gz
RUN ls -lrt /
RUN mkdir build \
    && cd build \
    && cmake /eccodes-2.21.0-Source -DCMAKE_INSTALL_PREFIX=/usr/local/share/eccodes \
    && make install

RUN apt-get -yqq install autoconf libtool musl-dev
RUN ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1

RUN curl https://opendata.dwd.de/weather/lib/bufr/bufrtables_ecCodes-local-dwd.tar.bz2 -o /tmp/bufrtables_ecCodes-local-dwd.tar.bz2
RUN tar -xjf /tmp/bufrtables_ecCodes-local-dwd.tar.bz2

COPY ./requirements.txt /opt/requirements.txt
RUN pip3 install -r /opt/requirements.txt

ENV PYTHONPATH /app
ENV ECCODES_DEFINITION_PATH /usr/local/share/eccodes/share/eccodes/definitions
ENV ECCODES_DIR /usr/local/share/eccodes
ENV BUFR_DUMP_PATH /usr/local/share/eccodes/bin/bufr_dump

WORKDIR /app

gutzbenj · 2022-01-28T17:22:25Z

Dear @meteoDaniel ,

your totally fine using the ScalarRequestCore.

Ideally you should only have to set certain properties in the beginning and implement the ._all() method that returns a cached list of stations. Obviously those header of this stations listing should be aligned with the set of Columns we have defined. Besides that the most important step is to setup Enumerations for Parameters and and their Units.

Besides if the Parameters are nested e.g. there are different resolutions with different parameters, then you'd have to create another mapping between "flattened" parameters and their origin in the "deeper" enumeration.

e.g.

class DwdRoadWeatherTree:
      class Daily:
              class XYZ:
                     Precipitation = "precp"

class DwdRoadWeatherParameter:
      class Daily:
             Precipitation = DwdRoadWeatherTree.Daily.XYZ.Precipitation.value

At the moment, you then have to create another mapping like:

mapping = {
    DwdRoadWeatherParameter.Daily.Precipitation: DwdRoadWeatherTree.Daily.XYZ
}

Unit has to follow the detailed parameter tree

Is there a certain problem with implementing the class? I hope that I have annotated everything with enough text?

Cheers
Benjamin

gutzbenj · 2022-02-02T16:58:24Z

Dear @meteoDaniel ,

you can now simply embed ECCODES_DIR into wetterdienst Settings. Should we together walk through the creation of a new API? This would be a good point for me to write down some documentation on how to approach that.

wetterdienst/provider/dwd/road_weather/api.py

meteoDaniel and others added 8 commits April 21, 2021 13:52

Introducing optional CACHE_DIR environment variable to enable cloud c…

8e78f10

…omputing

Fix spelling

81b956b

Adding requested changes from review process

e81cb64

DRAFT: Adding first code for road weather data API

06d5867

rebase

68400a6

Merge pull request #1 from earthobservations/main

0e54cd8

Rebase

DRAFT: Adding first code for road weather data API

39be64e

Merge remote-tracking branch 'origin/add_road_weather_station_parser'…

a11c387

… into add_road_weather_station_parser # Conflicts: # wetterdienst/provider/dwd/index.py

meteoDaniel and others added 6 commits November 4, 2021 14:56

Merge branch 'earthobservations:main' into add_road_weather_station_p…

ddfdf72

…arser

Merge branch 'earthobservations:main' into main

590a508

DRAFT: Adding first code for road weather data API

844e926

DRAFT: Adding first code for road weather data API

f200255

Merge remote-tracking branch 'origin/add_road_weather_station_parser'…

04f66ca

… into add_road_weather_station_parser

restructure data to one dataframe

8d4c1a7

gutzbenj marked this pull request as draft November 30, 2021 09:34

meteoDaniel and others added 3 commits January 22, 2022 12:02

Merge branch 'earthobservations:main' into add_road_weather_station_p…

1330c4e

…arser

Add metaindex for road_weather Observations and add selection by rank

de148ab

lint tidy

c231352

meteoDaniel added 3 commits January 22, 2022 16:25

add pdbufr

e4bc391

Add openpyxl for reading excl files

e0134b8

re-comment example lines

08b55b0

gutzbenj reviewed Feb 2, 2022

View reviewed changes

wetterdienst/provider/dwd/road_weather/api.py Show resolved Hide resolved

meteoDaniel mentioned this pull request Mar 5, 2022

WIP: Update RoadWeather Project #613

Closed

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add road weather station parser #528

WIP: Add road weather station parser #528

meteoDaniel commented Oct 17, 2021 •

edited

Loading

codecov bot commented Oct 17, 2021 •

edited

Loading

amotl commented Oct 17, 2021 •

edited

Loading

gutzbenj commented Oct 18, 2021

meteoDaniel commented Nov 4, 2021 •

edited by amotl

Loading

meteoDaniel commented Nov 29, 2021

gutzbenj commented Dec 1, 2021 •

edited

Loading

meteoDaniel commented Jan 6, 2022

meteoDaniel commented Jan 22, 2022

meteoDaniel commented Jan 22, 2022 •

edited

Loading

gutzbenj commented Jan 28, 2022 •

edited by amotl

Loading

gutzbenj commented Feb 2, 2022

WIP: Add road weather station parser #528

WIP: Add road weather station parser #528

Conversation

meteoDaniel commented Oct 17, 2021 • edited Loading

codecov bot commented Oct 17, 2021 • edited Loading

Codecov Report

amotl commented Oct 17, 2021 • edited Loading

gutzbenj commented Oct 18, 2021

meteoDaniel commented Nov 4, 2021 • edited by amotl Loading

meteoDaniel commented Nov 29, 2021

gutzbenj commented Dec 1, 2021 • edited Loading

meteoDaniel commented Jan 6, 2022

meteoDaniel commented Jan 22, 2022

meteoDaniel commented Jan 22, 2022 • edited Loading

gutzbenj commented Jan 28, 2022 • edited by amotl Loading

gutzbenj commented Feb 2, 2022

meteoDaniel commented Oct 17, 2021 •

edited

Loading

codecov bot commented Oct 17, 2021 •

edited

Loading

amotl commented Oct 17, 2021 •

edited

Loading

meteoDaniel commented Nov 4, 2021 •

edited by amotl

Loading

gutzbenj commented Dec 1, 2021 •

edited

Loading

meteoDaniel commented Jan 22, 2022 •

edited

Loading

gutzbenj commented Jan 28, 2022 •

edited by amotl

Loading