Check for zeros in the data

[peterdudfield]

Detailed Description

It would be great to have a check in place that checks for zeros. A large amount of these is normally an error

Context

• good to catch data problems early and fail hard

Possible Implementation

• add a a check, if number os zeros >20% of the data, error
• @devsjc can you suggest a place in the code for this check?

[devsjc]

Yes, I would add it to the filter here:

nwp-consumer/src/nwp_consumer/internal/service/consumer.py

Lines 410 to 424 in 372359c

	def _dataQualityFilter(ds: xr.Dataset) -> bool:
	"""Filter out data that is not of sufficient quality."""
	if ds == xr.Dataset():
	return False
	# Carry out a basic data quality check
	for data_var in ds.data_vars:
	if ds[f"{data_var}"].isnull().any():
	log.warn(
	event=f"Dataset has NaNs in variable {data_var}",
	initTime=str(ds.coords["init_time"].values[0])[:16],
	variable=data_var,
	)
	return True

[GAMinsect]

Hi, I'm someone who is just starting in the field of open source development.
I would like to know if i can contribute to this, since it' a good first issue.
Thank you in advance.

[peterdudfield]

Hi, I'm someone who is just starting in the field of open source development. I would like to know if i can contribute to this, since it' a good first issue. Thank you in advance.

You can definately contirbute. Are you familiar with python and xarray?

[GAMinsect]

I'm familiar with python and common libraries like pandas, but not with xarray

[peterdudfield]

thanks good @GAMinsect, you migh need to learn a bit of xarray. Your welcome to give it ago.

[GAMinsect]

After 2 weeks of working on it in my free time, here's my implementation.
I formatted the code following your coding style.

def _dataQualityFilter(ds: xr.Dataset) -> bool:
    """Filter out data that is not of sufficient quality."""
    if ds == xr.Dataset():
        return False

    zeroCount = 0
    elementCount = 0
    # Carry out a basic data quality check
    for data_var in ds.data_vars:
        if ds[f"{data_var}"].isnull().any():
            log.warn(
                event=f"Dataset has NaNs in variable {data_var}",
                initTime=str(ds.coords["init_time"].values[0])[:16],
                variable=data_var,
            )

        data = ds[data_var].data
        elementCount += data.size
        zeroCount += (data == 0).sum()

    if zeroCount / elementCount > 0.2:
        raise ValueError("In your dataset more than 20% of your data are 0's")

    return True
 

[GAMinsect]

@peterdudfield is the code fine?

[peterdudfield]

ill let @devsjc review if thats ok.

[GAMinsect]

@devsjc how's the review going?

[devsjc]

Hi @GAMinsect, thanks for your work looking into this, and apologies for the slow response.

I've just merged in a quite comprehensive restructuring of the project in order to improve the speed of the application, which has resulted in the _dataQualityFilter function being removed, in favour of a new - in progress - PostProcess API that I'm in the process of finalising. When complete, I suspect this will be where this check will end up.

The code looks good though, so your work wasn't wasted - it will still be used in the logic, just not in the place I originally suggested!

[GAMinsect]

@devsjc Thank you for the reply (and sorry for my late one) i guess this issue can be closed now, right?