Use effective_solar_pathlength_corrected for cloud type and cloud phase

[gerritholl]

Use the solar_pathlength_corrected modifier for cloud type and cloud phase RGBs for FCI. This makes the composite usable closer to the terminator but without the saturation seen in the ABI versions.

• Closes #xxxx
• Tests added
• Fully documented

Cloud type with only sunz_corrected (satpy name: cimss_cloud_type):

Cloud type with sunz_corrected and sunz_reduced (satpy name: cloud_type):

Cloud type with solar_pathlength_corrected:

Cloud type with solar_pathlength_corrected and rayleigh_corrected (for VIS 0.6):

Cloud phase with sunz_corrected and sunz_reduced:

Cloud phase with solar_pathlength_corrected:

Cloud phase with solar_pathlength_corrected and rayleigh_corrected (for VIS 0.6):

[strandgren]

Nice work on this, it's clear that there is quite some more information to get from the imagery. But I have two questions:

• in the recipes you use solar_pathlength_corrected, but where is this defined? I can only find effective_solar_pathlength_corrected defined here https://github.com/pytroll/satpy/blob/main/satpy/etc/composites/visir.yaml#L14.
• Could you also provide a couple of example of how your proposed changes look in the day-night blends, i.e. cloud_type_with_night_ir105 and cloud_phase_with_night_ir105? We may need to tweak the blending there as well.

And as a comment, I still find that at least for the thick part of the frontal cloud close to the terminator the level of saturation is a bit too high. But I'm also aware that it's hard to find settings that works well for all conditions.

[gerritholl]

• in the recipes you use solar_pathlength_corrected, but where is this defined?

Oops. Only in my local recipes. Fixed with 6d5fe19. It shows why we need the automated imagery tests, and probably we should indeed merge #3013 first (where I added more on the behave tests) so I can leverage this!

Will check the versions with night_ir105.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.11%. Comparing base (92b9719) to head (c21c1e9).
Report is 69 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3032   +/-   ##
=======================================
  Coverage   96.11%   96.11%           
=======================================
  Files         383      383           
  Lines       55604    55604           
=======================================
  Hits        53443    53443           
  Misses       2161     2161           

Flag	Coverage Δ
behaviourtests	3.91% <ø> (ø)
unittests	96.20% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[coveralls]

Pull Request Test Coverage Report for Build 12830491071

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 96.219%

---

Totals
Change from base Build 12829518226:	0.0%
Covered Lines:	53691
Relevant Lines:	55801

---

💛 - Coveralls

[gerritholl]

Cloud type night IR 10.5, before:

Cloud type night IR, after:

Cloud phase night IR, before:

Cloud phase night IR, after:

For curiosity, our internal solar_pathlength_corrected, that I accidentally used in the first commit, is defined as:

  solar_pathlength_corrected:
    modifier: !!python/name:satpy.modifiers.EffectiveSolarPathLengthCorrector
    optional_prerequisites:
      - solar_zenith_angle
    max_sza: !!null

whereas the satpy one is defined as:

satpy/satpy/etc/composites/visir.yaml

Lines 14 to 17 in 40a62fa

	effective_solar_pathlength_corrected:
	modifier: !!python/name:satpy.modifiers.EffectiveSolarPathLengthCorrector
	optional_prerequisites:
	- solar_zenith_angle

The version I showed in the initial issue description is with max_sza: !!null, which we use internally for all our composites.

Cloud type with the effective solar pathlength corrector, default max_sza:

Cloud type with the solar pathlength corrector, no max_sza:

[strandgren]

For the effective_solar_pathlength_corrected I don't think the reduction is needed and setting max_sza: !!null should be fine (as your plots also confirm). For sunz_corrected I thinks it's useful, or even needed, given that the correction breaks around 90 degrees otherwise. You can see a comparison here: #2955 (comment).

So perhaps you can add your local effective_solar_pathlength_corrected definition in fci.yaml?

Thanks for the addition day/night blends. Doesn't seem to be much affected. But I believe that with these changes we can do the day/night blending at higher sunz angles to retain more of the daytime imagery, but that might be for another PR.

[ameraner]

Thank you for your work and analysis!
I have mixed feelings about this: while I fully agree that trying to include more illuminated daytime pixels is beneficial, both RGBs do not really perform well in that twilight area... In the Cloud Type, the red-toned noise and ESL signal from the NIR1.3 is quite enhanced, and in the Cloud Phase the entire area is dominated by the blue component, making it not too useful.
It seems however that applying the default max_sza removes some of the low-quality and not-so-usable area, while still including a good amout of extra daytime pixels. So maybe that's a good compromise? If I understand the discussion correctly, the current PR code modifications reflects this setup.

[gerritholl]

There is a tradeoff between æsthetics and the ability to extract information. This is probably true for any RGB based on solar channels and also when considering the day/night-transition in geo colour. And any RGB, as well as quantitative products, will lose quality approaching the terminator. Maybe cloud type is more affected here because NIR 1.3 is so dark already.

For our operational users, the ability to extract information is more important than æsthetics. I think they will accept and understand that. Colours are a bit different and noise becomes visible. OK.

If I understand the discussion correctly, the current PR code modifications reflects this setup.

Yes, the current PR code modification uses the default solar_pathlength_corrected (correction_limit=88, max_sza=95). It's a compromise between the status quo (sunz_corrected with or without sunz_reducer) and the DWD version (correction_limit=88, max_sza=None). The compromise looks prettier near the terminator, but the DWD version has more extractable information. What do we want as default in satpy?

[ameraner]

I agree that information content is more important than aesthetics, at least for the default recipe (a user interested in only PR-like nice looking images can still tune a different recipe to their wishes, but I'd say that the general focus should be on meteorological usability and usefulness).
My question is if the low twilight area, with all the artefacts like noise and non-representative color tones (that are at that point not describing the actual physical status of the surface/cloud scene, like the violet terminator of the cloud phase and the saturated red clouds in cloud type) is useful/delivering information to the observer. Admittedly, it likely still delivers information at least for spatial patterns, even if the spectral information is likely not very usable... so I don't have a strong opinion, I'm ok with both solutions in the end.
But at least we can leave a note in the yaml saying that using the default max_sza can help to remove some of the bad quality areas, if any user investigates this in the future.

[mraspaud]

Really nice work!
The discussion re whether to remove the terminator artefacts or not is interesting. Lets no forget though the regions where twilight lasts all day :) for these, I’m suspecting that having data with artefacts can still be better than no data at all...

[gerritholl]

The discussion re whether to remove the terminator artefacts or not is interesting.

I'm all for removing artefacts, but removing the entire area that has artefacts in it has something of throwing out the child with the dirty bathwater...