Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restrict saving of validation .csv's to wandb #280

Open
AUdaltsova opened this issue Nov 28, 2024 · 0 comments
Open

Restrict saving of validation .csv's to wandb #280

AUdaltsova opened this issue Nov 28, 2024 · 0 comments

Comments

@AUdaltsova
Copy link
Contributor

We currently have validation .csvs created and saved to wandb here and are quickly running out of storage space on wandb, because these files are saved each epoch for each run, and quickly become massive with horizon and quantile number increases. We should make these optional and cut down on the size/amount of files saved:

  • only save best epoch
  • only save the same 4(?) batches we save val plots for
  • any other way we can trim this?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant