Confidence intervals seem broader than in the MATLAB version

[guillermoaguilar]

For the data in the basic usage example, we have for the threshold:

result.confidence_intervals['threshold']

[[0.0041551724137931035, 0.005110079575596817],
 [0.004218832891246685, 0.005110079575596817],
 [0.004346153846153846, 0.004982758620689655]]

and in MATLAB:

95 % = 0.004282936763270, 0.004964252219404
90% = 0.004340714759236, 0.004907784893587
68 % = 0.004455989969640, 0.004793927224722

This is noticeable when plotting the marginals, that's how I found it.
In python for 95% threshold:

In MATLAB for 95% threshold:

In python the upper bound is >0.005, in MATLAB is <0.005.

[guillermoaguilar]

Same for width

Python, 95 % confidence: [0.00300887573964497, 0.006772189349112426]

MATLAB, 95 % confidence: 0.003532170611945 0.006040905831117

[dekuenstle]

The good news is that apparently, the marginal is computed correctly: Both curves match exactly if overlayed.

@guillermoaguilar
I suspect that the difference can be explained by the default CI method.
Could you please try running the Python version with CI_method='percentiles'?

We support CI calculation with a project and percentiles method. In my understanding, project obtains the cutoff in the N-dimensional grid before projecting to the marginal dimension, while percentile first projects to the dimension and then obtain the cutoff.

While we default to project, the MATLAB version defaults to percentiles. I recommend that we chance our default setting to percentiles, too.

[dekuenstle]

There still is a systematic difference between MATLAB and Python CIs as discussed in the MATLAB test PR.

The python CI still remain systematically smaller (~5%). In my perspective this is not a bug: While the MATLAB sets the border in the middle of two grid points (one within, one outside the CI), while the Python version interpolates the cumulative distribution. In steep regimes like the one below, the interpolation must be narrower.

@guillermoaguilar @otizonaizit @pberkes @lschwetlick What is your opinion, bug or feature?

[guillermoaguilar]

I ran the fits above with defaults, default in MATLAB and default in python. Now running it with 'percentiles' it gives me

result.confidence_intervals['threshold']

[[0.004251168834040448, 0.00493232186010108],
 [0.00430891627918812, 0.004875907476743896],
 [0.004424169210878712, 0.004762087037380426]]

which is much closer to MATLAB:

95 % = 0.004282936763270, 0.004964252219404
90% = 0.004340714759236, 0.004907784893587
68 % = 0.004455989969640, 0.004793927224722

So differences are due to the discrepancy in the default method to compute the CI. IMO we must change the default to 'percentiles', and all defaults should be all the same w.r.t. the MATLAB implementation.

I see that David @dekuenstle updated the PR #66 with the change, but I think this change has to be done ASAP and in isolation. Users will wonder why CI are sufficiently different between implementations.

@otizonaizit do you agree?

[guillermoaguilar]

There still is a systematic difference between MATLAB and Python CIs as discussed in the MATLAB test PR.

The python CI still remain systematically smaller (~5%). In my perspective this is not a bug: While the MATLAB sets the border in the middle of two grid points (one within, one outside the CI), while the Python version interpolates the cumulative distribution. In steep regimes like the one below, the interpolation must be narrower.

@guillermoaguilar @otizonaizit @pberkes @lschwetlick What is your opinion, bug or feature?

I think it boils down to coverage. If the narrower python CI have the appropriate coverage, then these should be kept
Felix said in the presentation of the paper that coverage was sometimes too large than expected (Fig. 6 C and D in the paper), so this might be the reason. But one would have to actually run the simulations to establish coverage.... which is not yet implemented in the python version. I'd keep it as it is.

[otizonaizit]

There still is a systematic difference between MATLAB and Python CIs as discussed in the MATLAB test PR.

The python CI still remain systematically smaller (~5%). In my perspective this is not a bug: While the MATLAB sets the border in the middle of two grid points (one within, one outside the CI), while the Python version interpolates the cumulative distribution. In steep regimes like the one below, the interpolation must be narrower.

@guillermoaguilar @otizonaizit @pberkes @lschwetlick What is your opinion, bug or feature?

I think this is a question for @FelixWichmann , even if my gut feeling is that 5% is small enough that it doesn't really matter.

[otizonaizit]

I ran the fits above with defaults, default in MATLAB and default in python. Now running it with 'percentiles' it gives me

result.confidence_intervals['threshold']

[[0.004251168834040448, 0.00493232186010108],
 [0.00430891627918812, 0.004875907476743896],
 [0.004424169210878712, 0.004762087037380426]]

which is much closer to MATLAB:

95 % = 0.004282936763270, 0.004964252219404 90% = 0.004340714759236, 0.004907784893587 68 % = 0.004455989969640, 0.004793927224722

So differences are due to the discrepancy in the default method to compute the CI. IMO we must change the default to 'percentiles', and all defaults should be all the same w.r.t. the MATLAB implementation.

I see that David @dekuenstle updated the PR #66 with the change, but I think this change has to be done ASAP and in isolation. Users will wonder why CI are sufficiently different between implementations.

@otizonaizit do you agree?

Yes, I have the feeling that it would be better to have a separate PR just to change the default mode for CI calculation, merge the obvious PRs that are already there and do a quick 4.2 release.

What is @FelixWichmann thinking about this? Which method should be default?

[otizonaizit]

@guillermoaguilar : Felix approved the change in an email exchange.

I changed the default CI method in #135 . Could you wait for #132 to be merged and then cut a 4.2 release? In two weeks we can discuss about the further steps, but this fix is urgent enough that it warrants a release right now, I think.

[guillermoaguilar]

Ok!