2025 ocf-data-sampler - let's make it happen

[peterdudfield]

Copying over from ML-notes

Thank you @Sukh-P @dfulu @AndrewLester @felix-e-h-p for all your work on this so far

Here are the tasks to do, in order to fully move over from ocf-datapipes to ocf-data-sampler

Phase 1

• Update example configs - PR
• Satellite scaling Satellite normalization #87
• Site torch dataset update #82
• PVnet ready to use site pipeline PR
• Logic going from netcdfs to tensor batches (Sukh start, maybe review Peter) - sit in ocf-data-sampler - Add training torchdataset #99
• Make separate torch_dataset to load premade samples - Add training torchdataset #99
• Test training for site level dataset
• Update backtest scripts to use ocf-data-sampler, for uk and site. @zakwatts might be to help on this.
• Add time features for sites #77
• Add collate function #100
• Change numpy_batch to numpy_sample #103

Phase 2:

• Train a site level model, and compare with models. This could be an India model as its ready to go.
• Re train UK GSP PVnet model, with satellite scaling.
• Fully Remove ocf_datapipes from PVnet, probably move some functions/class from ocf_datapipes to ocf-data-sampler
• Add torch dataset for training summational model (James)
• Simplify sample/batch interface (with a Sample/batch class) so there is a common interface used between all samples - Create batch class #71
• Pvnet Da: Use data-sampler in new projects going forward - Train PVnet DA with ocf-data-sampler PVNet#289
• GDM: Use data-sampler in new projecy
• Fully move uk-pvnet-app over to ocf-data-sampler - PVnet Intraday: Upgrade to ocf-data-sampler uk-pvnet-app#116
• Fully move india-forecast-app over to ocf-data-sampler

Not do/ do later: Training like to like batches with ocf-data-sampler and ocf-datapipes

Future:

• Add more features that were in ocf_datapipes, support for different NWPs etc
• Save in zipped folder with separate netcdfs per input
• aswell as lots of other issues