Toy Regression Example

[nilsleh]

Thanks a lot for the interesting paper and open sourcing the code. I am trying to get a better understanding of the method and its sensitivity to hyperparameters. Therefore, I created a toy example notebook contained in this PR because there is no Issues tab, to discuss the observation. The mean fit is okay, albeit not perfect, hower, the estimates for epistemic and aleatoric uncertainty seem rather strange. However, I am not sure, whether I compute it correctly. I was following your implementation in the ood comput step.

posterior, log_prob = module.model(dm.X_test)
mean = posterior.maximum_a_posteriori().mean()
epistemic = torch.clone(log_prob)
aleatoric = -posterior.maximum_a_posteriori().uncertainty()

Given the results, I was wondering if you could point out what I am doing wrong. Cheers.

[borchero]

Hey @nilsleh, happy to see you get your hands dirty with NatPN and thanks for adding the notebook here! ;) On some datasets, fine-tuning is very important which requires an additional "optimization round". Please have a look at the NaturalPosteriorNetwork estimator that is exposed by this package. In its fit method, you can see what fine-tuning (which is enabled by default) uses a different loss to solely optimize the normalizing flow to obtain more accurate uncertainty estimates.

[nilsleh]

Thanks a lot for your reply, I have added a finetuning step, which changes the results but does not really improve them, so I must still be doing something wrong with the hyperparameter tuning. Do yo happen to have a script for the Toy Regression Task Figure 1 of your paper that could maybe help to pinpoint the issue?

Edit for context:

This is before finetuning:

And after finetuning: