Update summation model to reduce wobbles #12
Conversation
|
Example validation results from old model - slightly wobbly predictions |
|
Same example validation results from new model - predicts smoother curves |
dfulu
changed the title
|
Note these tests aren't passing due to the temporary bug fix we put in to compensate for the nwp-consumer bug |
There was a problem hiding this comment.
Looks good. Can we make sure these results are added to the PVnet doc.
Or perhaps there is a seperate doc the the PVnet summation model - again I would push for markdown files as he is an example of a short experiment that produce a postive results, but if you done it in the google doc thats also ok
|
I'll add it to the PVNet doc. I think its really hard to maintain set of markdowns. I have had to edit the PVNet doc a bunch to recontextualise previous experiment write-ups, to update them, or to remove them as we've progressed. And that seems pretty clunky to do with a series of markdowns |
Pull Request
Description
This updates the summation model used.
The current summation model is a bit noisy. In our forecasts there are some high frequency perturbations from a smooth curve which I do not believe are indicative of model skill, but more the model being overly sensitive to the inputs. Although the accuracy of the model is good, I think it makes us look bad from a user's perspective - can we trust an accurate but wobbly line? These high frequency deviations would also likely make our forecasts worse as measured by ramp-rate.
This new model is trained with more effective regularisation (due to rescaling the inputs), and is trained for longer using a slower changing learning rate. This results in smoother forecasts, and a very modest improvement in validation scores.
Both the new and old model were trained and validated on the exact same batches, so their validation results can be directly compared. Validation results, and the wandb training logs for the two models are shown below.