Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Matlab test #66

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Matlab test #66

wants to merge 16 commits into from

Conversation

dekuenstle
Copy link
Member

No description provided.

@otizonaizit
Copy link
Collaborator

Impressive work, @dekuenstle : Kudos!

Copy link
Collaborator

@pberkes pberkes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty cool, a suggestion about skipping the tests

psignifit/tests/test_matlab.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@pberkes pberkes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with @otizonaizit , let's discuss the open questions

@classmethod
def from_matlab_options(cls, option_dict: Dict[str, Any], **kwargs):
python_configs = config_from_matlab(option_dict, **kwargs)
return cls(**python_configs)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a helper function in the MATLAB tests

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this class function but exposed psignifit.config_from_matlab. This might be handy for users switching from MATLAB to Python

psignifit/_configuration.py Outdated Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file should go to the tests folder

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above -> Matlab compatiblitiy can be useful for people switching from matlab

psignifit/_parameter.py Outdated Show resolved Hide resolved
psignifit/_result.py Show resolved Hide resolved
psignifit/psignifit.py Outdated Show resolved Hide resolved
psignifit/psignifit.py Show resolved Hide resolved
@otizonaizit otizonaizit deleted the branch main September 20, 2024 13:54
@dekuenstle dekuenstle reopened this Sep 20, 2024
@dekuenstle
Copy link
Member Author

  • Configuration should raise Exception if sigmoid name starts with log (notice user, that we do not support logspace, transform on their own)
  • Only run these tests on non-logspace names.
  • Investigate the difference between MATLAB and Python with this datafile. With the demo-file, we get exactly the same results.

@dekuenstle
Copy link
Member Author

dekuenstle commented Sep 22, 2024

I dug deeper into the difference between the MATLAB and psignifit results on these datasets and could resolve many.

  1. In the test I forgot to pool the data blocks. After introducing the pooling the point estimates can typically recovered with <1% error for many datasets.
  2. I changed the default CI method to percentiles (see Confidence intervals seem broader than in the MATLAB version #133 ) to obtain almost the same width (<10% difference).
  3. The python CI still remain systematically smaller (~5%). In my perspective this is not a bug: While the MATLAB sets the border in the middle of two grid points (one within, one outside the CI), while the Python version interpolates the cumulative distribution. In steep regimes like the one below, the interpolation must be more narrow.

Still, the tests fail for some datasets which needs further investigation.

Comment on lines -173 to 175
for parameter, prior in priors.items():
priors[parameter] = normalize_prior(prior, bounds[parameter])
for parameter, bound in bounds.items():
priors[parameter] = normalize_prior(priors[parameter], bound)
return priors
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I understand why this was changed...

Copy link
Member Author

@dekuenstle dekuenstle Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is a relict from a time when bounds might not be defined for fixed items. Therefore, the former version would crash. If I'm right, you changed the behavior last week such that for fixed parameters still bounds are added in the psignifit funciton?!

@otizonaizit otizonaizit changed the base branch from refactor-api to main September 22, 2024 22:18
Comment on lines +99 to +101
if 'gamma' not in fit_dict: # equal asymptotes
fit_dict['gamma'] = fit_dict['lambda']
intervals_dict['gamma'] = intervals_dict['lambda']
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is taken care of, at least partially in the following lines (106-110), so line 101
intervals_dict['gamma'] = intervals_dict['lambda'] should be added after line 110 below

Copy link
Member Author

@dekuenstle dekuenstle Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historically this PR grew too big and does too many things. I currently see it more as a playground to discover if/why there are differenes between the MATLAB and Python version. Afterwards IMHO the essence of this PR should be reimplemented in a separate branch.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historically this PR grew too big and does too many things. I currently see it more as a playground to discover if/why there are differenes between the MATLAB and Python version. Afterwards IMHO the essence of this PR should be reimplemented in a separate branch.

OK. But then it would be useful to get rid of the spurious changes and then merge main into this branch. I understand the playground idea, but given that we are changing things in main that may be relevant here, we need some form of synchronization, I think... Or, really create a new branch and continue the playground there. I am just afraid that at some point in the future no one will remember which changes are relevant and which ones should be discarded...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants