Ensemble agreement assigns members with exactly no change to the negative fraction

[aulemahal]

Addressing a Problem?

robustness_fractions has the agree output which is the fraction of valid models agreeing on the sign of change. It is currently a simple calculation based on positive, fraction of valid members showing positive change.

In summary:

positive = delta > 0
agree = max(positive, 1 - positive)

If the delta has real zeros, they are not counted towards positive. A dataset where a few members showing real positive change could come out has having low agreement.

Potential Solution

I think it would be more intuitive if the agreement fraction included the members showing no change.

positive = delta > 0
negative = delta < 0
no_change = delta == 0
agree = max(positive + no_change, negative + no_change)

This is also the opinion of the Portraits Climatiques team.

Additional context

The previous value of agree could always be recomputed by a user as positive is an output of the function.

What does @RondeauG and @huard think of this ?

Contribution

• I would be willing/able to open a Pull Request to contribute this feature.

Code of Conduct

• I agree to follow this project's Code of Conduct

[huard]

I would tend to disagree. I consider a model with a change of 2C in disagreement with a model with a change of 0C.

I understand this is beyond the scope of this function, but in my mind you'd have a threshold t such that models can be divided into

negative change < -t <= no significant change <= t < positive change

Here t is 0, and values with 0 change would simply be part of another category of "no change".

My suggestion is thus to say:

positive = delta > 0
negative = delta < 0
agree = max(positive, negative)

[aulemahal]

I understand. The threshold idea is not really beyond the scope of the function, ... it is pretty much what the threshold method does!

In fact, I can indeed avoid my problem by using that method, I didn't think of that! Thanks!

[RondeauG]

Let's say that we have 15 simulations: 5 negative, 4 not significantly negative, 3 with a change of exactly +0, 2 not significantly positive, 1 positive.

• changed: Unless test=None --> 6/15.
• positive: I'd argue that exactly +0 is not positive, thus positive = delta > 0 is correct. --> 3/15.
• changed_positive: A change of +0 will not be counted here, since it will fail both tests. --> 1/15.
• agree: max(3/15, 1 - 3/15) = 12/15 --> This is wrong! --> This should be max(positive, negative), aka 9/15.

[aulemahal]

The scientific consensus has spoken against my idea.
Thanks!

[aulemahal]

Reopen as our agree is indeed incorrect if there is an ambiguity between negative and no change.

Best solution to date:

positive = delta > 0
negative = delta < 0
agree = max(positive, negative, 1 - positive - negative)

With a possibility of most models "agreeing" on exactly no change.