Global implementations

[Anmirazik]

Hi , first of all im not someone involved with development of machine learning or AI and that sort of things , Im an IoT engineer who have interest in integrating my home solar with this Quartz Solar Forecast , my question is can this be used in Malaysia or any toher regions outside of UK ? Or do i need to train the model according to my own datasets ? I would like to implement this Quartz Solar Forecast and get a prediction for my home solar on how much I can generate energy for the next 24 / 48 hours . Thank you :)

[peterdudfield]

Hi @Anmirazik, thanks for getting in contact.

We have trained the model with UK datasets, but there is no reason why it wont work in Malaysia. There is a chance that the model is more suited to UK PV weather data, but I would have thought it still gives a reasonable estimate in Malaysia.

It would be great if you gave it a go and see what the results are?
Note: your results might improve after we have done #36 too.

Also, if you willing to share any PV data, this could be used to help evaluate the model, and potentially improve the model in the future

[peterdudfield]

Any luck with this @Anmirazik ?

[pranjalraman03]

@peterdudfield The UK PV dataset uses the latitude and longitude of PV stations in UK for training. If it were to be used for forecasting in Malaysia then the model will not be efficient considering the vast difference in latitude and longitude between the trained data and Malaysia.
Another feature that was used in training that i think can hinder the performance is tilt of PV system. The duration of days and nights largely vary between UK and Malaysia. Hence the intensity of sunlight for same tilt might be very different in both the countries. The UK dataset trained model should forecast well in the neighbouring countries of UK because they do not show high variance from the climate and location of UK.
To make the model global we should train it on features that directly affect the physics of PV system such as average intensity of light throughout the day, etc. If there is no dataset with these features than several separate datsets can be combined to get the desired features. Can I work on this issue?

[peterdudfield]

Yea, that makes total sense. And please yea, would be great if you work on this issue. We probably need to collect some training data from around the world

[pranjalraman03]

@peterdudfield I looked into PV datasets across several countries. All of them have used different features. Some of them used area of PV plant, some used tilt and some both. I don't think a single model can be trained for global implementation given such variance in features across countries. But a model trained on one dataset should work for all its neighbouring countries due to similarity in geographical conditions. What are your thoughts?

[peterdudfield]

Thanks @pranjalraman03. I think we can get away with

• generation (in the same units)
• peak capacity of the site (in the same units)
• Lat and lon
• Optional but of course best to use it is tilt and orientation

Which other features have you seen? I'd also be interested if you can share where these other global PV datasets are

[liamjdavis]

@peterdudfield Looking at the solar atlas below, I think that a model trained on a single continent will scale relatively well to other continents, because each continent holds a roughly similar solar trend. Obviously, the larger the dataset the better, but I, for one, am struggling to find PV datasets from across the globe. The one caveat is that latitude and longitude will be have to be transformed into different features(maybe latitude from equator and longitude from nearest ocean.) I hope this is helpful.

[peterdudfield]

Thanks @liamjdavis . Yea cant use lat long directly, see the thigns we currently use - https://github.com/openclimatefix/Open-Source-Quartz-Solar-Forecast?tab=readme-ov-file#model
Good wait to transform it is to sun elevation and azimuth.

Did you find any global datasets, or bits of data that could be used?

[liamjdavis]

@peterdudfield Solargis has GIS datasets that can be purchased here. Their visualizations are free but datasets are not. The US PV database has publicly available US PV data, but uses z-score instead of raw power output. That is all I have found thus far.

[pranjalraman03]

@peterdudfield Different countries pv dataset were available on different github repos while some were on the website of local organizations in the country. https://github.com/microsoft/solar-farms-mapping/blob/main/data/solar_farms_india_2021.geojson is the link for pv dataset of India and https://github.com/BobbyWong66/PV-plant-dataset-of-China/tree/a8ff5adf4953778c0c9d8fbdf75044a982151671/PV_China/PV_China is for China . I could'nt find any dataset that uses intensity as a feature. I think another method of global implementation is that we train the model on dataset of one country that is large enough compared to its neighbouring country and use it to forecast for the neighbouring countries. This would reduce the number of datasets we would need to train the model on

[peterdudfield]

It would be great to find sone time series production data, where we could compare our model.

[pranjalraman03]

@peterdudfield Since this issue mostly revolves around data selection and not significant development in the model, can you assign some other issue to work on

[peterdudfield]

@pranjalraman03 perhaps #30

[pranjalraman03]

@peterdudfield Is this issue still open to work on? I see a merged PR.

[peterdudfield]

@peterdudfield Is this issue still open to work on? I see a merged PR.

Which issue do you mean? if its #30, do you mind commenting in there, then it is all together

[peterdudfield]

Another example was in #130