Clip negative FCI radiances #3013

gerritholl · 2024-12-09T15:43:47Z

Optionally clip negative FCI radiances.

Closes artefacts in FCI RGBs using 3.8 µm #3009
Tests added
Fully documented

codecov · 2024-12-09T15:49:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.11%. Comparing base (5984c29) to head (92b9719).
Report is 86 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3013      +/-   ##
==========================================
+ Coverage   96.10%   96.11%   +0.01%     
==========================================
  Files         377      383       +6     
  Lines       55163    55604     +441     
==========================================
+ Hits        53012    53443     +431     
- Misses       2151     2161      +10

Flag	Coverage Δ
behaviourtests	`3.91% <0.00%> (-0.04%)`	⬇️
unittests	`96.20% <100.00%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coveralls · 2024-12-09T16:03:30Z

Pull Request Test Coverage Report for Build 12812804411

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

32 of 32 (100.0%) changed or added relevant lines in 2 files are covered.
8 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.01%) to 96.219%

Files with Coverage Reduction	New Missed Lines	%
satpy/tests/utils.py	2	93.16%
satpy/tests/reader_tests/gms/test_gms5_vissr_l1b.py	3	98.67%
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py	3	97.18%

Totals
Change from base Build 12227072168:	0.01%
Covered Lines:	53691
Relevant Lines:	55801

💛 - Coveralls

gerritholl

start behave test

rymdulf · 2024-12-13T11:02:30Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T11:04:38Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-13T11:08:13Z

rymdulf is lying, the tests were not successful. Failed with:

  Scenario Outline: Compare generated image with reference image -- @1.1         # features/image_comparison.feature:10
    Given I have a airmass reference image file from GOES17 resampled to <area>  # features/steps/image_comparison.py:54
[ WARN:[email protected]] global loadsave.cpp:241 findDecoder imread_('/app/ext_data/reference_images/reference_image_GOES17_airmass_<area>.png'): can't open/read file: check file path/integrity
    When I generate a new airmass image file from GOES17 with abi_l1b for <area> # features/steps/image_comparison.py:64
      Traceback (most recent call last):
        File "/app/venv/lib/python3.10/site-packages/behave/model.py", line 1329, in run
          match.run(runner.context)
        File "/app/venv/lib/python3.10/site-packages/behave/matchers.py", line 98, in run
          self.func(context, *args, **kwargs)
        File "features/steps/image_comparison.py", line 82, in step_when_generate_image
          ls = scn.resample(area)
        File "/app/repository/satpy/scene.py", line 980, in resample
          self._resampled_scene(new_scn, destination, resampler=resampler,
        File "/app/repository/satpy/scene.py", line 869, in _resampled_scene
          destination_area = self._get_finalized_destination_area(destination_area, new_scn)
        File "/app/repository/satpy/scene.py", line 905, in _get_finalized_destination_area
          destination_area = get_area_def(destination_area)
        File "/app/repository/satpy/resample.py", line 231, in get_area_def
          return parse_area_file(get_area_file(), area_name)[0]
        File "/app/venv/lib/python3.10/site-packages/pyresample/area_config.py", line 147, in parse_area_file
          return _parse_yaml_area_file(area_file_name, *regions)
        File "/app/venv/lib/python3.10/site-packages/pyresample/area_config.py", line 201, in _parse_yaml_area_file
          raise AreaNotFound('Area "{0}" not found in file "{1}"'.format(area_name, area_file_name))
      pyresample.area_config.AreaNotFound: 'Area "<area>" not found in file "[\'/app/repository/satpy/etc/areas.yaml\']"'

gerritholl

start behave test

rymdulf · 2024-12-13T11:56:22Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T11:59:38Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T12:11:20Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T12:15:25Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T13:27:59Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T13:32:00Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-13T13:43:41Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T13:47:11Z

The testing process was executed successfully. See the test results for this pull request here!

Trying John Cintineos workaround.

gerritholl

run behave test

gerritholl

start behave test

rymdulf · 2024-12-13T14:59:53Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-13T15:02:46Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl

start behave test

rymdulf · 2024-12-16T17:38:40Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-16T17:42:25Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-16T17:46:22Z

Good news:

Negative radiances are clipped.
No more black holes in the cloudtop RGB.

Bad news:

Still green holes in the night microphysics RGB.

gerritholl

start behave test

rymdulf · 2024-12-16T18:08:37Z

Starting to clone and test the repository pytroll/satpy

rymdulf · 2024-12-16T18:12:28Z

The testing process was executed successfully. See the test results for this pull request here!

gerritholl · 2024-12-17T13:49:16Z

It's clipping space pixels, which we probably don't want? How do ABI and AHI prevent that?

djhoese · 2024-12-17T19:45:20Z

I think ABI, through normal ground processing, have masked space pixels. AHI has space pixels masked by default in Satpy I think (@simonrp84?) so maybe we just don't care? Or is radiance clipping not implemented for AHI?

simonrp84 · 2024-12-18T07:46:04Z

I don't understand your comment, @gerritholl: The space pixels are masked in the raw FCI data so it shouldn't be clipping them. They should already be nodata.

Also, @djhoese IIRc there's no clipping applied to the AHI data. It's the only modern GEO sat that doesn't mask space pixels in the data, but we do by default mask them in satpy.

gerritholl · 2024-12-18T18:34:15Z

I don't understand your comment, @gerritholl: The space pixels are masked in the raw FCI data so it shouldn't be clipping them. They should already be nodata.

Yes, but I had a bug that was not only setting negative radiances to a low radiance, but also nan radiances. It was retaining only pixels where data>lo was True, and this was false for the nan space pixels. Fixed that in d404a79.

satpy/readers/fci_l1c_nc.py

mraspaud

LGTM

Started behave test for FCI cloudtop

576e941

gerritholl added 3 commits December 9, 2024 18:57

Continue development of FCI imagery tests

d4a1e20

Change reference data and area for ABI test

1f562ae

Improve reference data generation script.

2297ee7

gerritholl commented Dec 13, 2024

View reviewed changes

Fix feature syntax and filename setup

02989b9

gerritholl commented Dec 13, 2024

View reviewed changes

import hdf5plugin first despite isort

8ce9a62

gerritholl commented Dec 13, 2024

View reviewed changes

Does it help if I do Meteosat-12 first?

7cd8858

gerritholl commented Dec 13, 2024

View reviewed changes

workaround for failing hdf5lpguni?

4d13d12

Trying John Cintineos workaround.

gerritholl commented Dec 13, 2024

View reviewed changes

gerritholl requested review from djhoese and mraspaud as code owners December 16, 2024 17:38

gerritholl commented Dec 16, 2024

View reviewed changes

gerritholl requested a review from simonrp84 December 17, 2024 07:56

gerritholl marked this pull request as draft December 17, 2024 13:56

gerritholl marked this pull request as ready for review December 17, 2024 14:57

add doc & don't clip space pxiels

d404a79

gerritholl mentioned this pull request Dec 18, 2024

Combine low level moisture with a cloud mask #3022

Draft

3 tasks

gerritholl mentioned this pull request Jan 16, 2025

Use effective_solar_pathlength_corrected for cloud type and cloud phase #3032

Draft

3 tasks

djhoese reviewed Jan 16, 2025

View reviewed changes

satpy/readers/fci_l1c_nc.py Show resolved Hide resolved

djhoese reviewed Jan 16, 2025

View reviewed changes

satpy/readers/fci_l1c_nc.py Show resolved Hide resolved

gerritholl added 2 commits January 16, 2025 16:59

Respect config setting for FCI clipping

08f0087

Add test confirming preservation of dtype

92b9719

djhoese added enhancement code enhancements, features, improvements component:readers labels Jan 16, 2025

djhoese approved these changes Jan 16, 2025

View reviewed changes

mraspaud assigned gerritholl Jan 17, 2025

mraspaud approved these changes Jan 17, 2025

View reviewed changes

mraspaud merged commit 1f46b53 into pytroll:main Jan 17, 2025
17 of 18 checks passed

gerritholl mentioned this pull request Jan 17, 2025

Change ESSL colorisation approach #3021

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clip negative FCI radiances #3013

Clip negative FCI radiances #3013

gerritholl commented Dec 9, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

coveralls commented Dec 9, 2024 •

edited

Loading

gerritholl left a comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl commented Dec 13, 2024

gerritholl left a comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

gerritholl left a comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl commented Dec 16, 2024

gerritholl left a comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl commented Dec 17, 2024

djhoese commented Dec 17, 2024

simonrp84 commented Dec 18, 2024

gerritholl commented Dec 18, 2024 •

edited

Loading

mraspaud left a comment

Clip negative FCI radiances #3013

Clip negative FCI radiances #3013

Conversation

gerritholl commented Dec 9, 2024 • edited Loading

codecov bot commented Dec 9, 2024 • edited Loading

Codecov Report

coveralls commented Dec 9, 2024 • edited Loading

Pull Request Test Coverage Report for Build 12812804411

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 13, 2024

rymdulf commented Dec 13, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl commented Dec 16, 2024

gerritholl left a comment

Choose a reason for hiding this comment

rymdulf commented Dec 16, 2024

rymdulf commented Dec 16, 2024

gerritholl commented Dec 17, 2024

djhoese commented Dec 17, 2024

simonrp84 commented Dec 18, 2024

gerritholl commented Dec 18, 2024 • edited Loading

mraspaud left a comment

Choose a reason for hiding this comment

gerritholl commented Dec 9, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

coveralls commented Dec 9, 2024 •

edited

Loading

gerritholl commented Dec 18, 2024 •

edited

Loading