save validation batch results to wandb

[peterdudfield]

Pull Request

Description

Save csv of validation results to weights and biases

This is done by

• results being appended each validation step,
• saved all together at the end of the validation step

How Has This Been Tested?

CI tests

• local test with script

import wandb
import pandas as pd

run = wandb.init(
    project="test",
    notes="My first experiment",
)

# make random dataframe
df = pd.DataFrame({
    "a": [1, 2, 3],
    "b": [4, 5, 6]
})

i=2
validation_artifact = wandb.Artifact(f"validation_results_{i}", type="dataset")
df.to_csv(f"validation_results_{i}.csv", index=False)
validation_artifact.add_file(f"validation_results_{i}.csv")
wandb.log_artifact(validation_artifact)

• Yes

Checklist:

• My code follows OCF's coding style guidelines
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked my code and corrected any misspellings

[dfulu]

Have you done a training run with this? When predicting quantiles, I think y_i and y_hat_i will be different shapes

[peterdudfield]

I havent done training, but the end2end test does go through here.

becasue i take y = batch[self._target_key][:, -self.forecast_len :, 0], it makes it the same length as y_hat.

Perhaps theres a better way to standardised that. something for the future

[dfulu]

Yeh but I think in this case, y_i is a vector with shape (horizon_step,). But y_hat_i can either be a vector with shape (horizon_step,) or (horizon_step, quantile,) depending on whether we are training to predict quantiles or only a central value. The end2end test only tests non-quantile training.

[peterdudfield]

oh good point. ill have a think about the qunalite things. Good catch on that

[peterdudfield]

I should have it now

[dfulu]

I think there are a couple of things to sort out, but otherwise looks alright :)

[codecov]

Codecov Report

Attention: Patch coverage is 94.28571% with 2 lines in your changes missing coverage. Please review.

Project coverage is 59.62%. Comparing base (bdbe00a) to head (156fdfa).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
pvnet/models/base_model.py	94.28%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #252      +/-   ##
==========================================
+ Coverage   59.00%   59.62%   +0.61%     
==========================================
  Files          29       29              
  Lines        1893     1927      +34     
==========================================
+ Hits         1117     1149      +32     
- Misses        776      778       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.