Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sample saving for Site Dataset #290

Open
wants to merge 15 commits into
base: dev-data-sampler
Choose a base branch
from

Conversation

Sukh-P
Copy link
Member

@Sukh-P Sukh-P commented Dec 17, 2024

Pull Request

Description

Inclues:

  • Adding support for creating samples using the ocf-data-sampler Site torch dataset
  • Updating the configs examples to the new data schema used in ocf-data-sampler
  • Removing the datapipe specified datamodules which won't be used after moving to ocf-data-sampler
  • Small documentation updates in the README (may have missed some)

How Has This Been Tested?

This is a sister PR to https://github.com/openclimatefix/ocf-data-sampler/pull/82/files, ran the save_samples script with the changes in that PR and validated the samples being produced looked correct.

pyproject.toml Outdated Show resolved Hide resolved
)
batch = next(iter(dm.train_dataloader()))
return batch
# TODO update this test once we add the loading logic for the Site dataset
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cant work out if we should merge before doing this, im not sure there. Relates to openclimatefix/ocf-data-sampler#99

Copy link
Contributor

@peterdudfield peterdudfield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks all good. I wonder if we should od openclimatefix/ocf-data-sampler#99 first, then add some test cases in here, just to be double sure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants