Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data
To install the satip
library please run:
pip install satip
Or if you're working in the development environment you can run the following from the directory root:
pip install -e .
Or, if you want to use conda
from a cloned Satip repository:
conda env create -f environment.yml
conda activate satip
pip install -e .
If you plan to work on the development of Satip then also consider installing these development tools:
conda install pytest flake8 jedi mypy black pre-commit
pre-commit install
In order to contribute:
- it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.
- At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.
In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:
- Go to https://eoportal.eumetsat.int and register an account.
- You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
- Please make sure that you added the key and secret to your user's environment variables.
We have moved this to here
scripts/convert_native_to_zarr.py
converts EUMETSAT .nat
files to Zarr datasets, using very mild lossy JPEG-XL compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent bz2
compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range [0, 1]
. NaNs are encoded as the value 0.025
. All "real" values are in the range [0.075, 1]
. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see this comment for more details). Use satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs
to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range [0, 1]
.
The live service uses app.py
as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.
--api-key
or API_KEY
is the EUMETSAT API key
--api-secret
or API_SECRET
is the EUMETSAT API secret
--save-dir
or SAVE_DIR
is the top level directory to save the output files, a latest
subfolder will be added to that directory to contain the latest data
--history
or HISTORY
is the amount of history timesteps to use in the latest.zarr
files
--db-url
or DB_URL
is the URL to the database to save to when a run has finished
--use-rescaler
or USE_RESCALER
tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.
--use-iodc
or USE_IODC
is an option to get the IODC satellite data
To run tests, simply run pytest .
from the root of the repository. To generate the test plots, run python scripts/generate_test_plots.py
.
Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:
EUMETSAT_USER_KEY
: the EUMETSAT API keyEUMETSAT_USER_SECRET
: the EUMETSAT API secret
These can be added using the export
command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).
Thanks goes to these wonderful people (emoji key):
Jacob Bieker 💻 |
Jack Kelly 💻 |
Ayrton Bourn 💻 |
Laurence Watson 💻 |
Notger Heinz 📖 |
Peter Dudfield 📖 |
Azah Norbline 💻 |
Tom Pughe 💻 |
Zhenbang Feng 💻 |
jsbaasi 💻 |
Suleman Karigar 💻 |
Richa 💻 |
Nathan Simpson 🐛 |
peach280 🚧 |
This project follows the all-contributors specification. Contributions of any kind welcome!