Dataloader

[peterdudfield]

Detailed Description

Add a torch dataloader

Context

• nice to add an example of a dataloader, with batchsize 10

Possible Implementation

• need to work out how to collate the samples together, to batches

[dfulu]

Yeh this should be part of creating the batch saving script which is mentioned in #7. Although currently the equivalent script is in PVNet so maybe it should happen over there?

[dfulu]

We also already have the collate function we need to stack the samples together. Its here in datapipes

[peterdudfield]

oh thanks, ill use that collate function

[peterdudfield]

Ill ahve a go, as i need it for openclimatefix/uk-pvnet-app#116, and it seemed better to sit here

[peterdudfield]

Deiceded not to do this, as the code is very simepl

from ocf_datapipes.batch.merge_numpy_examples_to_batch import stack_np_examples_into_batch
from torch.utils.data import DataLoader
DataLoader(
        dataset,
        batch_size=batch_size,
        num_workers=num_workers,
        pin_memory=True,
        collate_fn=stack_np_examples_into_batch,
        **kwargs,
    )