Transformation Pipelines

1. GPM weather data to Topoflow Climate :

Data transformation to generate TopoFlow-ready precipitation files (RTS) from Global Precipitation Measurement (GPM) data sources

List of parameters:

* input_dir: path to input directory (GPM netCDF files)
* temp_dir: path to temporary directory
* output_file: path to output file
* var_name: variable name
* DEM_bounds: bounding box
* DEM_xres_arcsecs: x-resolution
* DEM_yres_arcsecs: y-resolution

Sample files

• Configuration file: Download.
• Sample GPM netCDF file: Download.
• Complete GPM dataset (netCDF format): Download

2. ISRIC SoilGirds to Topoflow Soil:

Data transformation to generate TopoFlow-ready precipitation files (RTS) from ISRIC SoilGrids data sources

List of parameters:

* input_dir: path to input directory (ISRIC soil files)
* output_dir: path to output directory
* layer: soil layer
* DEM_bounds: bounding box
* DEM_xres_arcsecs: x-resolution
* DEM_yres_arcsecs: y-resolution

Sample files

• Configuration file: Download.
• Sample ISRIC SoilGrids folder: Download.
• Complete ISRIC SoilGrids dataset: Original | MINT-Server

3.a Select Nearest GLDAS Coordinates for SoilGrids Files:

Data transformation to generate Cycles-ready input files (RTS) from GLDAS weather data sources

List of parameters:

* soil_dataset_id: SoilGrids Dataset ID
* gldas_elevation_file_path: Path to the GLDAS elevation file
* bounding_box: Bounding box of the extracting region (Horn of Africa here)
* num_output_files: Number of output files to create
* output_path: Path to the output directory
* output_prefix: Prefix to be added to the output file names

3.b Select Nearest GLDAS Coordinates for SoilGrids Files:

Data transformation to generate Cycles-ready input files (RTS) from GLDAS weather data sources

List of parameters:

* gldas_dataset_id: GLDAS Dataset ID
* gldas_soil_map_file: Path to the Coordinates Map file from Soil Coordinates to GLDAS Grid Coordinates
* start_date: Start date to filter resources in the dataset
* end_date: End date to filter resources in the dataset
* batch_numdays: Number of days of GLDAS to load in one go. Keep this number higher for machines with more RAM
* output_path: Path to the output directory

3-old. GLDAS to Cycles (Deprecated):

Data transformation to generate Cycles-ready input files (RTS) from GLDAS weather data sources

List of parameters:

* start_date: start date of GLDAS data
* end_date: end date of GLDAS data
* gldas_path: path to GLDAS data directory
* output_prefix: prefix for output files
* latitude: latitude (either lat/long or coord_file need to be provided)
* longitude: longitude
* coord_file: path to coordinate file (contains a list of lat/long for many locations)

Sample files

• Configuration file: Download.
• Complete GLDAS dataset: Download

4. GLDAS woreda daily all variables aggregation:

Data transformation to crop and aggregate daily values of all standard variables for all woredas from GLDAS data sources and return a CSV file.

List of parameters:

* gldas_dataset_id: DataCatalog dataset ID for GLDAS
* agg_function: Operation to be used for aggregation. Values can be ("sum", "average", "count")
* agg_time_period: Time period for aggregation. Values can be ("minute", "hour", "day", "month", "year")
* range_start_time: Start time to filter Resources for DataCatalog GLDAS Dataset (can also be "null" to leave this end open) name
* range_end_time: End time to filter Resources for DataCatalog GLDAS Dataset (can also be "null" to leave this end open)
* range_step_time: ISO 8601 duration string representing the step (or timedelta) to loop from range_start_time to range_end_time
* shapefile_dataset_id: DataCatalog Dataset ID for Shapefile with woredas
* csv_output_file: Filename for output CSV

Sample files

• Configuration file: Download.

5. GLDAS woreda monthly 1 variable aggregation:

Data transformation to crop and aggregate daily values of all standard variables for all woredas from GLDAS data sources and return a CSV file.

List of parameters:

* gldas_dataset_id: DataCatalog dataset ID for GLDAS
* agg_function: Operation to be used for aggregation. Values can be ("sum", "average", "count")
* agg_time_period: Time period for aggregation. Values can be ("minute", "hour", "day", "month", "year")
* range_start_time: Start time to filter Resources for DataCatalog GLDAS Dataset (can also be "null" to leave this end open) name
* range_end_time: End time to filter Resources for DataCatalog GLDAS Dataset (can also be "null" to leave this end open)
* range_step_time: ISO 8601 duration string representing the step (or timedelta) to loop from range_start_time to range_end_time
* gldas_variable_name: GLDAS Standard Variable name for which aggregation is to be performed
* shapefile_dataset_id: DataCatalog Dataset ID for Shapefile with woredas
* crop_region_xmin: Target region bounding box xmin coordinate
* crop_region_ymin: Target region bounding box ymin coordinate
* crop_region_xmax: Target region bounding box xmax coordinate
* crop_region_ymax: Target region bounding box ymax coordinate
* crop_region_label: Target region label
* csv_output_file: Filename for output CSV

Sample files

• Configuration file: Download.