Merge commit '580afb3d11c9686067d007b693c93f62d3bba134' into issue/da…

…-lost

.bumpversion.cfg

@@ -1,7 +1,7 @@
 [bumpversion]
 commit = True
 tag = True
-current_version = 3.0.40
+current_version = 3.0.42
 message = Bump version: {current_version} → {new_version} [skip ci]
 
 [bumpversion:file:pvnet/__init__.py]

.github/workflows/release.yml

@@ -7,6 +7,7 @@ on:
     paths-ignore:
       - "configs.example/**" # ignores all files in configs.example
       - "**/README.md" # ignores all README files
+      - "experiments/**" # ignores all files in experiments directory
 
 jobs:
   release:

README.md

@@ -1,13 +1,13 @@
 # PVNet 2.1
 
-[![test-release](https://github.com/openclimatefix/PVNet/actions/workflows/test-release.yml/badge.svg)](https://github.com/openclimatefix/PVNet/actions/workflows/test-release.yml)
+[![Python Bump Version & release](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml/badge.svg)](https://github.com/openclimatefix/PVNet/actions/workflows/release.yml)
 
-This project is used for training PVNet and running PVnet on live data.
+This project is used for training PVNet and running PVNet on live data.
 
 PVNet2 is a multi-modal late-fusion model that largely inherits the same architecture from
-[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP and
+[PVNet1.0](https://github.com/openclimatefix/predict_pv_yield). The NWP (Numerical Weather Prediction) and
 satellite data are sent through some neural network which encodes them down to
-1D intermediate representations. These are concatenated together with the GSP
+1D intermediate representations. These are concatenated together with the GSP (Grid Supply Point)
 output history, the calculated solar coordinates (azimuth and elevation) and the
 GSP ID which has been put through an embedding layer. This 1D concatenated
 feature vector is put through an output network which outputs predictions of the
@@ -56,7 +56,7 @@ pip install ".[dev]"
 
 ## Getting started with running PVNet
 
-Before running any code in within PVNet, copy the example configuration to a
+Before running any code in PVNet, copy the example configuration to a
 configs directory:
 
 ```
@@ -74,14 +74,14 @@ suggested locations for downloading such datasets below:
 
 **GSP (Grid Supply Point)** - Regional PV generation data\
 The University of Sheffield provides API access to download this data:
-https://www.solar.sheffield.ac.uk/pvlive/api/
+https://www.solar.sheffield.ac.uk/api/
 
 Documentation for querying generation data aggregated by GSP region can be found
 here:
 https://docs.google.com/document/d/e/2PACX-1vSDFb-6dJ2kIFZnsl-pBQvcH4inNQCA4lYL9cwo80bEHQeTK8fONLOgDf6Wm4ze_fxonqK3EVBVoAIz/pub#h.9d97iox3wzmd
 
 **NWP (Numerical weather predictions)**\
-OCF maintains a Zarr formatted version the German Weather Service's (DWD)
+OCF maintains a Zarr formatted version of the German Weather Service's (DWD)
 ICON-EU NWP model here:
 https://huggingface.co/datasets/openclimatefix/dwd-icon-eu which includes the UK
 
@@ -121,212 +121,84 @@ cp -r configs.example configs
 
 ### Set up and config example for batch creation
 
-We will use the example of creating batches using data from gcp:
-`/PVNet/configs/datamodule/configuration/gcp_configuration.yaml`
-Ensure that the file paths are set to the correct locations in
-`gcp_configuration.yaml`.
+We will use the following example config file for creating batches: `/PVNet/configs/datamodule/configuration/example_configuration.yaml`. Ensure that the file paths are set to the correct locations in `example_configuration.yaml`: search for `PLACEHOLDER` to find where to input the location of the files. You will need to comment out or delete the parts of `example_configuration.yaml` pertaining to the data you are not using.
 
-`PLACEHOLDER` is used to indcate where to input the location of the files.
 
-For OCF use cases, file locations can be found in `template_configuration.yaml` located alongside `gcp_configuration.yaml`.
+When creating batches, an additional datamodule config located in `PVNet/configs/datamodule` is passed into the batch creation script: `streamed_batches.yaml`. Like before, a placeholder variable is used when specifying which configuration to use:
 
-In these configurations you can update the train, val & test periods to cover the data you have access to.
-
-
-With your configuration in place, you can proceed to create batches. PVNet uses
-[hydra](https://hydra.cc/) which enables us to pass variables via the command
-line that will override the configuration defined in the `./configs` directory.
-
-When creating batches, an additional config is used which is passed into the batch creation script. This is the datamodule config located `PVNet/configs/datamodule`.
-
-For this example we will be using the `streamed_batches.yaml` config. Like before, a placeholder variable is used when specifing which configuration to use:
-
-`configuration: "PLACEHOLDER.yaml"`
+```yaml
+configuration: "PLACEHOLDER.yaml"
+```
 
-This should be given the whole path to the config on your local machine, such as for our example it should be changed to:
+This should be given the whole path to the config on your local machine, for example:
 
-`configuration: "/FULL-PATH-TO-REPO/PVNet/configs/datamodule/configuration/gcp_configuration.yaml"`
-`
+```yaml
+configuration: "/FULL-PATH-TO-REPO/PVNet/configs/datamodule/configuration/example_configuration.yaml"
+```
 
 Where `FULL-PATH-TO-REPO` represent the whole path to the PVNet repo on your local machine.
 
+This is also where you can update the train, val & test periods to cover the data you have access to.
+
 ### Running the batch creation script
 
-Run the save_batches.py script to create batches if setting parameters in the datamodule config (`streamed_batches.yaml` in this example):
+Run the `save_batches.py` script to create batches with the parameters specified in the datamodule config (`streamed_batches.yaml` in this example):
 
-```
+```bash
 python scripts/save_batches.py
 ```
-or with the following example arguments to override config:
+PVNet uses
+[hydra](https://hydra.cc/) which enables us to pass variables via the command
+line that will override the configuration defined in the `./configs` directory, like this:
 
-```
+```bash
 python scripts/save_batches.py datamodule=streamed_batches datamodule.batch_output_dir="./output" datamodule.num_train_batches=10 datamodule.num_val_batches=5
 ```
 
-In this function the datamodule argument looks for a config under `PVNet/configs/datamodule`. The examples here are either to use "premade_batches" or "streamed_batches".
-
-Its important that the dates set for the training, validation and testing in the datamodule (`streamed_batches.yaml`) config are within the ranges of the dates set for the input features in the configuration (`gcp_configuration.yaml`).
+`scripts/save_batches.py` needs a config under `PVNet/configs/datamodule`. You can adapt `streamed_batches.yaml` or create your own in the same folder.
 
-If downloading private data from a gcp bucket make sure to authenticate gcloud (the public satellite data does not need authentication):
+If downloading private data from a GCP bucket make sure to authenticate gcloud (the public satellite data does not need authentication):
 
 ```
 gcloud auth login
 ```
 
-For files stored in multiple locations they can be added as list. For example from the gcp_configuration.yaml file we can change from satellite data stored on a bucket:
+Files stored in multiple locations can be added as a list. For example, in the `example_configuration.yaml` file we can supply a path to satellite data stored on a bucket:
 
 ```yaml
 satellite:
     satellite_zarr_path: gs://solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr
 ```
 
-To satellite data hosted by Google:
+Or to satellite data hosted by Google:
 
 ```yaml
 satellite:
     satellite_zarr_paths:
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2020_nonhrv.zarr"
       - "gs://public-datasets-eumetsat-solar-forecasting/satellite/EUMETSAT/SEVIRI_RSS/v4/2021_nonhrv.zarr"
 ```
-Datapipes is currently set up to use 11 channels from the satellite data, the 12th of which is HRV and is not included in these.
+
+Datapipes are currently set up to use 11 channels from the satellite data, the 12th of which is HRV and is not included in these.
 
 
 ### Training PVNet
 
 How PVNet is run is determined by the extensive configuration in the config
-files. The following configs have been tested to work using batches of data
-created using the steps and batch creation config mentioned above.
-
-You should create the following configs before trying to train a model locally,
-as so:
-
-In `configs/datamodule/local_premade_batches.yaml`:
-
-```yaml
-_target_: pvnet.data.datamodule.DataModule
-configuration: null
-batch_dir: "./output" # where the batches are saved
-num_workers: 20
-prefetch_factor: 2
-batch_size: 8
-```
-
-In `configs/model/local_multimodal.yaml`:
+files. The configs stored in `PVNet/configs.example` should work with batches created using the steps and batch creation config mentioned above.
 
-```yaml
-_target_: pvnet.models.multimodal.multimodal.Model
-
-output_quantiles: [0.02, 0.1, 0.25, 0.5, 0.75, 0.9, 0.98]
-
-#--------------------------------------------
-# NWP encoder
-#--------------------------------------------
-
-nwp_encoders_dict:
-  ukv:
-    _target_: pvnet.models.multimodal.encoders.encoders3d.DefaultPVNet
-    _partial_: True
-    in_channels: 10
-    out_features: 256
-    number_of_conv3d_layers: 6
-    conv3d_channels: 32
-    image_size_pixels: 24
-
-#--------------------------------------------
-# Sat encoder settings
-#--------------------------------------------
-
-# Ignored as premade batches were created without satellite data
-# sat_encoder:
-#   _target_: pvnet.models.multimodal.encoders.encoders3d.DefaultPVNet
-#   _partial_: True
-#   in_channels: 11
-#   out_features: 256
-#   number_of_conv3d_layers: 6
-#   conv3d_channels: 32
-#   image_size_pixels: 24
-
-add_image_embedding_channel: False
-
-#--------------------------------------------
-# PV encoder settings
-#--------------------------------------------
-
-pv_encoder:
-  _target_: pvnet.models.multimodal.site_encoders.encoders.SingleAttentionNetwork
-  _partial_: True
-  num_sites: 349
-  out_features: 40
-  num_heads: 4
-  kdim: 40
-  pv_id_embed_dim: 20
-
-#--------------------------------------------
-# Tabular network settings
-#--------------------------------------------
-
-output_network:
-  _target_: pvnet.models.multimodal.linear_networks.networks.ResFCNet2
-  _partial_: True
-  fc_hidden_features: 128
-  n_res_blocks: 6
-  res_block_layers: 2
-  dropout_frac: 0.0
-
-embedding_dim: 16
-include_sun: True
-include_gsp_yield_history: False
-
-#--------------------------------------------
-# Times
-#--------------------------------------------
-
-# Foreast and time settings
-history_minutes: 60
-forecast_minutes: 120
-
-min_sat_delay_minutes: 60
-
-sat_history_minutes: 90
-pv_history_minutes: 60
-
-# These must be set for each NWP encoder
-nwp_history_minutes:
-  ukv: 60
-nwp_forecast_minutes:
-  ukv: 120
-
-# ----------------------------------------------
-# Optimizer
-# ----------------------------------------------
-optimizer:
-  _target_: pvnet.optimizers.EmbAdamWReduceLROnPlateau
-  lr: 0.0001
-  weight_decay: 0.01
-  amsgrad: True
-  patience: 5
-  factor: 0.1
-  threshold: 0.002
-```
+Make sure to update the following config files before training your model:
 
-In `configs/local_trainer.yaml`:
-
-```yaml
-_target_: lightning.pytorch.trainer.trainer.Trainer
-
-accelerator: cpu # Important if running on a system without a supported GPU
-devices: auto
-
-min_epochs: null
-max_epochs: null
-reload_dataloaders_every_n_epochs: 0
-num_sanity_val_steps: 8
-fast_dev_run: false
-accumulate_grad_batches: 4
-log_every_n_steps: 50
-```
+1. In `configs/datamodule/local_premade_batches.yaml`:
+    - update `batch_dir` to point to the directory you stored your batches in during batch creation
+2. In `configs/model/local_multimodal.yaml`:
+    - update the list of encoders to reflect the data sources you are using. If you are using different NWP sources, the encoders for these should follow the same structure with two important updates:
+        - `in_channels`: number of variables your NWP source supplies
+        - `image_size_pixels`: spatial crop of your NWP data. It depends on the spatial resolution of your NWP; should match `nwp_image_size_pixels_height` and/or `nwp_image_size_pixels_width` in `datamodule/example_configs.yaml`, unless transformations such as coarsening was applied (e. g. as for ECMWF data)
+3. In `configs/local_trainer.yaml`:
+    - set `accelerator: 0` if running on a system without a supported GPU
 
-And finally update `defaults` in the main `./configs/config.yaml` file to use
+If creating copies of the config files instead of modifying existing ones, update `defaults` in the main `./configs/config.yaml` file to use
 your customised config files:
 
 ```yaml
@@ -350,7 +222,7 @@ python run.py
 
 ## Backtest
 
-If you have succesfully trained a PVNet model and have a saved model checkpoint you can create a backtest using this, e.g. forecasts on historical data to evaluate forecast accuracy/skill. This can be done by running one of the scripts in this repo such as [the UK gsp backtest script](scripts/backtest_uk_gsp.py) or the [the pv site backtest script](scripts/backtest_sites.py), further info on how to run these are in each backtest file.
+If you have successfully trained a PVNet model and have a saved model checkpoint you can create a backtest using this, e.g. forecasts on historical data to evaluate forecast accuracy/skill. This can be done by running one of the scripts in this repo such as [the UK GSP backtest script](scripts/backtest_uk_gsp.py) or the [the pv site backtest script](scripts/backtest_sites.py), further info on how to run these are in each backtest file.
 
 
 ## Testing

configs.example/model/multimodal.yaml

@@ -89,6 +89,10 @@ nwp_history_minutes:
 nwp_forecast_minutes:
   ukv: 480
   ecmwf: 480
+# Optional; defaults to 60, so must be set for data with different time resolution
+nwp_interval_minutes:
+  ukv: 60
+  ecmwf: 60
 
 # ----------------------------------------------
 # Optimizer

experiments/uk/011 - Extending forecast to 36 hours (updated ECMWF data)/PVNet_day_ahead.md

@@ -0,0 +1,22 @@
+PVNet day ahead was retrained to produce a 36 hour forecast, it was given its [previous configuration](https://huggingface.co/openclimatefix/pvnet_uk_region/tree/main) and data except for being given ECMWF NWP data with a longer forecast horizon (max 85 hours but 37 hours given to the model). Longer horizon UKV NWP data was not available at time of training and will be a further addition in the future.
+
+**Results** \
+[The training run](https://wandb.ai/openclimatefix/pvnet_day_ahead_36_hours/runs/m4d3wlft/overview) had 3.15% normalised mean absolute error (NMAE) on validation data (100,000 samples from May 2022 to May 2023), [previous training of PVNet day ahead](https://wandb.ai/openclimatefix/pvnet2.1/runs/2ghzwbxg/overview?) had similar results of 3.19% NMAE.
+
+
+![](PVNets_comparison.png "PVNets comparison")
+
+When comparing the two versions of PVNet day ahead (the new version in green) by forecast accuracy at each step on the validation dataset samples we see some small differences in the model up to 33 hours, such as first the first few steps and between steps 5 and 10, which could be explained by differences in samples seen and evaluated on between the two versions.
+
+However the larger difference is an improvement toward the end of the forecast horizon, from 33 hours onwards which is likely due to ECMWF data now being available for this period, when previously no NWP data was given past 33 hours due to the NWP forecast horizon of previous data and factoring in NWP initialization times and production delays.
+
+UKV NWP data used in the model is currently up to 30 hours, we would expect a further reduction in error from 30+ hours when training with longer horizon UKV data which would cover up to 36 hours.
+
+
+A very rough comparison is also plotted between these two PVNet model versions and the National XG model which is currently used for day ahead predictions in production.
+
+![](PVNEt_national_XG_comparison.png "PVNets national XG comparison")
+
+
+
+This comparison is rough and should not be seen as a fair comparison as the national XG numbers are just an estimate derived from backtest data on different time periods. However, it can show roughly what relative improvement could be achieved from replacing the National XG Day ahead model with a PVNet Day Ahead model.

pvnet/__init__.py

@@ -1,2 +1,2 @@
 """PVNet"""
-__version__ = "3.0.40"
+__version__ = "3.0.42"