Check physical limits of nwp data when data is loaded

[peterdudfield]

Detailed Description

Would be good to check the physical limits of nwp data e.g check temperature is between -100 and 100.
This should be only used in inference

Context

• nice to have clear errors, that fail hard
• we could also add this in where the sample has been selected, but not normalized yet

Possible Implementation

• like check for zeros in loader #335 , add an option to check physical limits after loading data. Default to false

[glitch401]

hey @peterdudfield can I take up this issue?

[peterdudfield]

yes please

[glitch401]

are we planning to have a dynamic setting for the limits or are we considering statically typing it in form of a dict like:

limits = {
            "temperature": (-100, 100),
            "specific_humidity": (0, 1),
            "pressure": (0, 2000),
            "u_wind": (-100, 100),
            "v_wind": (-100, 100),
            "geopotential": (0, 10000),
            "total_precipitation": (0, 100),
            "convective_precipitation": (0, 100),
            "snowfall": (0, 100),
            "graupel": (0, 100),
        }

[peterdudfield]

I think a dictionary like that would be good

[peterdudfield]

the means and stds are here. I wonder if you do mean - 5std, mean + 5std, and see what those umbers look like?

[glitch401]

makes sense
The mean and std I'm getting on UKV's values are: 280.0 and 2.023 hence appending the limits as (269.887, 290.112)

[peterdudfield]

Hm, they seems a bit tight. 273 is 0 degrees, so the limits there are -4 degrees to 17 degrees. I wonder for temperature we should just do -20 deg up to 50 deg, or perhaps 250 to 330 to be safe.

Perhaps you could write out the other ones and I can try to review and see if any need adjusting

[glitch401]

gottcha, I'm placing a the limit range at (250, 330)
let me know impacts of the test

[peterdudfield]

Great stuff!