diff --git a/README.md b/README.md index 20aa2b31..ae8d6d57 100644 --- a/README.md +++ b/README.md @@ -9,13 +9,14 @@ This project performs a polygonal summary on tree cover loss and intersecting la Currently the following analysis are implemented * Tree Cover Loss (for ArcPy client) -* Annual Update * Annual Update minimal * Carbon Flux Full Standard Model * Carbon Flux Sensitivity Analysis * Glad Alerts -* Viirs Fire Alerts -* MODIS Fire Alerts +* Viirs/ MODIS Fire Alerts +* Forest Change Diagnostic +* GFW Pro Dashboard +* Integrated Alerts ### Tree Cover Loss @@ -35,18 +36,9 @@ This type of analysis only supports simple features as input. Best used together test:runMain org.globalforestwatch.summarystats.SummaryMain treecoverloss --feature_type feature --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix --tcd 2000 --threshold 0 --threshold 30 --contextual layer is__gfw_plantations --carbon_pools ``` -### Annual Update - -A complex analysis intersecting Tree Cover Loss data with more than 40 layers. This analysis is used for the GFU. Supported input features are GADM features only. -Output are Summary and Change tables for ISO, ADM1 and ADM2 areas. - -```sbt -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis annualupdate --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix -``` - ### Annual Update minimal -This analysis follows the same methodology as the annual update analysis above, just with fewer intersecting layers. +An analysis intersecting Tree Cover Loss data with a number of layers. It is used to compute statistics for the GFW country and user dashboards, including carbon flux outputs (emissions, removals, net flux). Supported input features are @@ -61,10 +53,10 @@ For GADM there will also be summary tables with one row per ISO, ADM1, ADM2 and To produce final spreadsheets you will need to add another [post processing step](https://github.com/wri/write_country_stats). ```sbt -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis annualupdate_minimal --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis annualupdate_minimal --feature_type wdpa --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis annualupdate_minimal --feature_type geostore --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis annualupdate_minimal --feature_type feature --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix +test:runMain org.globalforestwatch.summarystats.SummaryMain annualupdate_minimal --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix +test:runMain org.globalforestwatch.summarystats.SummaryMain annualupdate_minimal --feature_type wdpa --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix +test:runMain org.globalforestwatch.summarystats.SummaryMain annualupdate_minimal --feature_type geostore --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix +test:runMain org.globalforestwatch.summarystats.SummaryMain annualupdate_minimal --feature_type feature --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix ``` ### Carbon Flux Full Standard Model @@ -75,7 +67,7 @@ It also analyzes several contextual layers that are unique to the carbon flux mo It currently only works with GADM features. ```sbt -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis carbonflux --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix +test:runMain org.globalforestwatch.summarystats.SummaryMain carbonflux --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix ``` ### Carbon Flux Sensitivity Analysis @@ -87,8 +79,8 @@ To run this model with the standard flux model output for an analysis using fewe It currently only works with GADM features. ```sbt -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis carbon_sensitivity --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix --sensitivity_type sensitivity_analysis_type -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis carbon_sensitivity --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix --sensitivity_type standard +test:runMain org.globalforestwatch.summarystats.SummaryMain carbon_sensitivity --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix --sensitivity_type sensitivity_analysis_type +test:runMain org.globalforestwatch.summarystats.SummaryMain carbon_sensitivity --feature_type gadm --tcl --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix --sensitivity_type standard ``` ### Glad Alerts @@ -107,10 +99,10 @@ Supported input features are * Simple Feature ```sbt -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis gladalerts --feature_type gadm --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis gladalerts --feature_type wdpa --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis gladalerts --feature_type geostore --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis gladalerts --feature_type feature --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain gladalerts --feature_type gadm --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain gladalerts --feature_type wdpa --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain gladalerts --feature_type geostore --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain gladalerts --feature_type feature --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] ``` ### Viirs/ MODIS Fire Alerts @@ -127,12 +119,59 @@ Supported input features are * Simple Feature ```sbt -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type gadm --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type wdpa --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type geostore --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] -test:runMain org.globalforestwatch.summarystats.SummaryMain --analysis firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type feature --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type gadm --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type wdpa --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type geostore --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +test:runMain org.globalforestwatch.summarystats.SummaryMain firealerts --fire_alert_type MODIS/VIIRS --fire_alert_source s3://bucket/prefix/file.tsv --feature_type feature --glad --features s3://bucket/prefix/file.tsv --output s3://bucket/prefix [--change_only] +``` +### Forest Change Diagnostic + +Forest Change Diagnostic computes forest loss and fire alerts for an input set of +lists of geometries. It is used to compute statistics for lists of a GFW Pro +customer. It requires the `firealerts` options and optionally takes any `feature` +options. The only supported input feature is `gfwpro`. Automatically turns on the +`--split_features` option. + +```sbt +test:runMain org.globalforestwatch.summarystats.SummaryMain forest_change_diagnostic --feature_type gfwpro --features s3://bucket/prefix/file.tsv --fire_alert_source s3://bucket/prefix/file.tsv --output s3://bucket/prefix ``` +There are two test input files available in the source tree, so you can run a sample +forest change diagnostic locally without s3 via a command like: + +```sbt +test:runMain org.globalforestwatch.summarystats.SummaryMain forest_change_diagnostic --split_features --feature_type gfwpro --features src/test/resources/palm-oil-32.tsv --fire_alert_type modis --fire_alert_source src/test/resources/sample-fire-alert.tsv --output testout +``` + +### GFW Pro Dashboard + +GFW Pro Dashboard computes summary statistics for the GFW Pro Dashboard. It +optionally takes `firealert` options and `feature` options. + +Supported input features are: + +* GADM +* Geostore +* WDPA +* Simple Feature +* GFW Pro + + +### Integrated Alerts + +Integrated Alerts computes a combination of deforestation alerts based on GLAD-L, +GLAD-S2, and RADD systems. It is used to compute these alerts for GFW. It optionally +takes any `feature`, `gadm`, or `wdpa` options. + +Supported input features are + +* GADM +* Geostore +* WDPA +* Simple Feature +* GFW Pro + + ## Inputs Use Polygon Features encoded in TSV format. Geometries must be encoded in WKB. You can specify one or many input files using wildcards: @@ -147,6 +186,7 @@ Larger features should be split into smaller features, prior to running the anal Also make sure, that features do not overlap with tile boundaries (we use 10x10 degree tiles). For best performance, intersect input features with a 1x1 degree grid. If you are not sure how to best approach this, simply use the [ArcPY Client](https://github.com/wri/gfw_forest_loss_geotrellis_arcpy_client) +Alternatively, use the `--split_features` opton. ## Options @@ -154,10 +194,9 @@ The following options are supported: |Option |Type |Analysis or Feature Type |Description | |-----------------|------|-------------------------|---------------------------------------------------------------------------------------------------------------------------------| -|analysis |string| |Type of analysis to run `annualupdate`, `annualupdate_minimal`, `carbonflux`, `carbon_sensitivity`, `gladalerts`, `treecoverloss`| |features |string|all (required) |URI of features in TSV format | |output |string|all (required) |URI of output dir for CSV files | -|feature_type |string|all (required) |Feature type: one of 'gadm', 'wdpa', 'geostore' or 'feature | +|feature_type |string|all (required) |Feature type: one of 'gadm', 'wdpa', 'geostore', 'feature', or 'gfwpro' | |limit |int |all |Limit number of records processed | |iso_first |string|`gadm` or `wdpa` features|Filter by first letter of ISO code | |iso_start |string|`gadm` or `wdpa` features|Filter by ISO code larger than or equal to given value | @@ -165,8 +204,10 @@ The following options are supported: |iso |string|`gadm` or `wdpa` features|Filter by country ISO code | |admin1 |string|`gadm` features |Filter by country Admin1 code | |admin2 |string|`gadm` features |Filter by country Admin2 code | -|id_start |int |`feature` analysis |Filter by IDs larger than or equal to given value | +|id_start |int |`feature` analysis |Filter by IDs greater than or equal to given value | +|id_end |int |`feature` analysis |Filter by IDs less than or equal to given value | |wdpa_status |string|`wdpa` features |Filter by WDPA Status | +|iucn_cat |string|`wdpa` features |Filter by IUCS Category | |tcd |int |`treecoverloss` analysis |Select tree cover density year | |threshold |int |`treecoverloss` analysis |Treecover threshold to apply (multiple) | |contextual_layer |string|`treecoverloss` analysis |Include (multiple) selected contextual layers: `is__umd_regional_primary_forest_2001`, `is__gfw_plantations` | @@ -175,8 +216,15 @@ The following options are supported: |glad |flag |all |Filter input feature by GLAD tile extent, requires boolean `glad` field in input feature class | |change_only |flag |all except `treecover` |Process change only | |sensitivity_type |string|`carbon_sensitivity` |Select carbon sensitivity model | -|fire_alert_type |string|`firealerts` |Select Fire alert type | -|fire_alert_source|string|`firealerts` |URI of fire alert TSV file | +|fire_alert_type |string|`firealerts` |Select Fire alert type | +|fire_alert_source|string|`firealerts` |URI of fire alert TSV file | +|overwrite |flag |all |Overwrite output location if already existing | +|split_features |flag |all |Split input features along 1x1 degree grid | +|no_output_path_suffix|flag |all |Do not autogenerate output path suffix at runtime | +|intermediate_list_source|flag |`forest_change_diagnostic` analysis |URI of intermediate list results in TSV format | +|contextual_feature_type|flag |`gfwpro_dashboard` analysis |type of contextual feature | +|contextual_feature_url|flag |`gfwpro_dashboard` analysis |URI of contextual feature in TSV format | + ## Inventory @@ -190,10 +238,8 @@ The following options are supported: ### Local -For local testing input should be limited with `--limit` flag to minimize the time. - ```sbt -sbt:geotrellis-wri> test:runMain org.globalforestwatch.summarystats.SummaryMain --features file:/Users/input/ten-by-ten-gadm36/wdpa__10N_010E.tsv --output file:/User/out/summary --limit 10 +sbt:geotrellis-wri> test:runMain org.globalforestwatch.summarystats.SummaryMain annualupdate_minimal --features file:/Users/input/ten-by-ten-gadm36/wdpa__10N_010E.tsv --output file:/User/out/summary ``` ### EMR @@ -202,6 +248,6 @@ Before running review `sbtlighter` configuration in `build.sbt`, `reload` SBT se ```sbt sbt:geotrellis-wri> sparkCreateCluster -sbt:treecoverloss> sparkSubmitMain org.globalforestwatch.summarystats.SummaryMain --features s3://gfw-files/2018_update/tsv/gadm36_1_1.csv --output s3://gfw-files/2018_update/results/summary --feature_type gadm --analysis annualupdate_minimal --tcl -sbt:treecoverloss> sparkSubmitMain org.globalforestwatch.summarystats.SummaryMain --features s3://gfw-files/2018_update/tsv/wdpa__*.tsv --output s3://gfw-files/2018_update/results/summary --feature_type wdpa --analysis gladalerts --tcl --iso BRA +sbt:treecoverloss> sparkSubmitMain org.globalforestwatch.summarystats.SummaryMain annualupdate_minimal --features s3://gfw-files/2018_update/tsv/gadm36_1_1.csv --output s3://gfw-files/2018_update/results/summary --feature_type gadm --tcl +sbt:treecoverloss> sparkSubmitMain org.globalforestwatch.summarystats.SummaryMain gladalerts --features s3://gfw-files/2018_update/tsv/wdpa__*.tsv --output s3://gfw-files/2018_update/results/summary --feature_type wdpa --tcl --iso BRA ``` diff --git a/src/main/resources/raster-catalog-pro.json b/src/main/resources/raster-catalog-pro.json index 81cf6a12..88ed7f20 100755 --- a/src/main/resources/raster-catalog-pro.json +++ b/src/main/resources/raster-catalog-pro.json @@ -197,12 +197,8 @@ "source_uri":"s3://gfw-data-lake/gfw_managed_forests/v202106/raster/epsg-4326/{grid_size}/{row_count}/is/gdal-geotiff/{tile_id}.tif" }, { - "name":"inpe_amazon_prodes", - "source_uri":"s3://gfw-data-lake/inpe_amazon_prodes/v2021/raster/epsg-4326/{grid_size}/{row_count}/year/geotiff/{tile_id}.tif" - }, - { - "name":"inpe_cerrado_prodes", - "source_uri":"s3://gfw-data-lake/inpe_cerrado_prodes/v2021/raster/epsg-4326/{grid_size}/{row_count}/year/geotiff/{tile_id}.tif" + "name":"inpe_prodes", + "source_uri":"s3://gfw-data-lake/inpe_prodes/v202107/raster/epsg-4326/{grid_size}/{row_count}/is/gdal-geotiff/{tile_id}.tif" }, { "name":"gfw_mining_concessions", @@ -276,6 +272,10 @@ "name":"umd_soy_planted_area", "source_uri": "s3://gfw-data-lake/umd_soy_planted_area/v2/raster/epsg-4326/{grid_size}/{row_count}/is__year_2021/gdal-geotiff/{tile_id}.tif" }, + { + "name":"inpe_prodes", + "source_uri": "s3://gfw-data-lake/inpe_prodes/v202107/raster/epsg-4326/{grid_size}/{row_count}/is/gdal-geotiff/{tile_id}.tif" + }, { "name":"wwf_eco_regions", "source_uri": "s3://gfw-data-lake/wwf_eco_regions/v2012/raster/epsg-4326/{grid_size}/{row_count}/name/gdal-geotiff/{tile_id}.tif" @@ -295,6 +295,26 @@ { "name": "arg_native_forest_land_plan", "source_uri": "s3://gfw-data-lake/arg_native_forest_land_plan/v202212/raster/epsg-4326/{grid_size}/{row_count}/category/gdal-geotiff/{tile_id}.tif" + }, + { + "name": "gadm_adm0", + "source_uri": "s3://gfw-data-lake/gadm_administrative_boundaries/v3.6/raster/epsg-4326/{grid_size}/{row_count}/adm0/geotiff/{tile_id}.tif" + }, + { + "name": "gadm_adm1", + "source_uri": "s3://gfw-data-lake/gadm_administrative_boundaries/v3.6/raster/epsg-4326/{grid_size}/{row_count}/adm1/geotiff/{tile_id}.tif" + }, + { + "name": "gadm_adm2", + "source_uri": "s3://gfw-data-lake/gadm_administrative_boundaries/v3.6/raster/epsg-4326/{grid_size}/{row_count}/adm2/geotiff/{tile_id}.tif" + }, + { + "name": "sbtn_natural_forests_map", + "source_uri": "s3://gfw-data-lake/sbtn_natural_forests_map/v202305/raster/epsg-4326/{grid_size}/{row_count}/class/gdal-geotiff/{tile_id}.tif" + }, + { + "name": "gfwpro_negligible_risk_analysis", + "source_uri": "s3://gfw-data-lake/gfwpro_negligible_risk_analysis/v20230726/raster/epsg-4326/{grid_size}/{row_count}/risk/geotiff/{tile_id}.tif" } ] -} \ No newline at end of file +} diff --git a/src/main/scala/org/globalforestwatch/layers/GadmAdm0.scala b/src/main/scala/org/globalforestwatch/layers/GadmAdm0.scala new file mode 100644 index 00000000..cd881456 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/layers/GadmAdm0.scala @@ -0,0 +1,266 @@ +package org.globalforestwatch.layers + +import org.globalforestwatch.grids.GridTile + +case class GadmAdm0(gridTile: GridTile, kwargs: Map[String, Any]) + extends StringLayer + with OptionalILayer { + + val datasetName: String = "gadm_adm0" + val uri: String = + uriForGrid(gridTile, kwargs) + + def lookup(value: Int): String = value match { + case 4 => "AFG" + case 248 => "ALA" + case 8 => "ALB" + case 12 => "DZA" + case 16 => "ASM" + case 20 => "AND" + case 24 => "AGO" + case 660 => "AIA" + case 10 => "ATA" + case 28 => "ATG" + case 32 => "ARG" + case 51 => "ARM" + case 533 => "ABW" + case 36 => "AUS" + case 40 => "AUT" + case 31 => "AZE" + case 44 => "BHS" + case 48 => "BHR" + case 50 => "BGD" + case 52 => "BRB" + case 112 => "BLR" + case 56 => "BEL" + case 84 => "BLZ" + case 204 => "BEN" + case 60 => "BMU" + case 64 => "BTN" + case 68 => "BOL" + case 535 => "BES" + case 70 => "BIH" + case 72 => "BWA" + case 74 => "BVT" + case 76 => "BRA" + case 86 => "IOT" + case 96 => "BRN" + case 100 => "BGR" + case 854 => "BFA" + case 108 => "BDI" + case 132 => "CPV" + case 116 => "KHM" + case 120 => "CMR" + case 124 => "CAN" + case 136 => "CYM" + case 140 => "CAF" + case 148 => "TCD" + case 152 => "CHL" + case 156 => "CHN" + case 162 => "CXR" + case 166 => "CCK" + case 170 => "COL" + case 174 => "COM" + case 178 => "COG" + case 180 => "COD" + case 184 => "COK" + case 188 => "CRI" + case 384 => "CIV" + case 191 => "HRV" + case 192 => "CUB" + case 531 => "CUW" + case 196 => "CYP" + case 203 => "CZE" + case 208 => "DNK" + case 262 => "DJI" + case 212 => "DMA" + case 214 => "DOM" + case 218 => "ECU" + case 818 => "EGY" + case 222 => "SLV" + case 226 => "GNQ" + case 232 => "ERI" + case 233 => "EST" + case 748 => "SWZ" + case 231 => "ETH" + case 238 => "FLK" + case 234 => "FRO" + case 242 => "FJI" + case 246 => "FIN" + case 250 => "FRA" + case 254 => "GUF" + case 258 => "PYF" + case 260 => "ATF" + case 266 => "GAB" + case 270 => "GMB" + case 268 => "GEO" + case 276 => "DEU" + case 288 => "GHA" + case 292 => "GIB" + case 300 => "GRC" + case 304 => "GRL" + case 308 => "GRD" + case 312 => "GLP" + case 316 => "GUM" + case 320 => "GTM" + case 831 => "GGY" + case 324 => "GIN" + case 624 => "GNB" + case 328 => "GUY" + case 332 => "HTI" + case 334 => "HMD" + case 336 => "VAT" + case 340 => "HND" + case 344 => "HKG" + case 348 => "HUN" + case 352 => "ISL" + case 356 => "IND" + case 360 => "IDN" + case 364 => "IRN" + case 368 => "IRQ" + case 372 => "IRL" + case 833 => "IMN" + case 376 => "ISR" + case 380 => "ITA" + case 388 => "JAM" + case 392 => "JPN" + case 832 => "JEY" + case 400 => "JOR" + case 398 => "KAZ" + case 404 => "KEN" + case 296 => "KIR" + case 408 => "PRK" + case 410 => "KOR" + case 414 => "KWT" + case 417 => "KGZ" + case 418 => "LAO" + case 428 => "LVA" + case 422 => "LBN" + case 426 => "LSO" + case 430 => "LBR" + case 434 => "LBY" + case 438 => "LIE" + case 440 => "LTU" + case 442 => "LUX" + case 446 => "MAC" + case 450 => "MDG" + case 454 => "MWI" + case 458 => "MYS" + case 462 => "MDV" + case 466 => "MLI" + case 470 => "MLT" + case 584 => "MHL" + case 474 => "MTQ" + case 478 => "MRT" + case 480 => "MUS" + case 175 => "MYT" + case 484 => "MEX" + case 583 => "FSM" + case 498 => "MDA" + case 492 => "MCO" + case 496 => "MNG" + case 499 => "MNE" + case 500 => "MSR" + case 504 => "MAR" + case 508 => "MOZ" + case 104 => "MMR" + case 516 => "NAM" + case 520 => "NRU" + case 524 => "NPL" + case 528 => "NLD" + case 540 => "NCL" + case 554 => "NZL" + case 558 => "NIC" + case 562 => "NER" + case 566 => "NGA" + case 570 => "NIU" + case 574 => "NFK" + case 807 => "MKD" + case 580 => "MNP" + case 578 => "NOR" + case 512 => "OMN" + case 586 => "PAK" + case 585 => "PLW" + case 275 => "PSE" + case 591 => "PAN" + case 598 => "PNG" + case 600 => "PRY" + case 604 => "PER" + case 608 => "PHL" + case 612 => "PCN" + case 616 => "POL" + case 620 => "PRT" + case 630 => "PRI" + case 634 => "QAT" + case 638 => "REU" + case 642 => "ROU" + case 643 => "RUS" + case 646 => "RWA" + case 652 => "BLM" + case 654 => "SHN" + case 659 => "KNA" + case 662 => "LCA" + case 663 => "MAF" + case 666 => "SPM" + case 670 => "VCT" + case 882 => "WSM" + case 674 => "SMR" + case 678 => "STP" + case 682 => "SAU" + case 686 => "SEN" + case 688 => "SRB" + case 690 => "SYC" + case 694 => "SLE" + case 702 => "SGP" + case 534 => "SXM" + case 703 => "SVK" + case 705 => "SVN" + case 90 => "SLB" + case 706 => "SOM" + case 710 => "ZAF" + case 239 => "SGS" + case 728 => "SSD" + case 724 => "ESP" + case 144 => "LKA" + case 729 => "SDN" + case 740 => "SUR" + case 744 => "SJM" + case 752 => "SWE" + case 756 => "CHE" + case 760 => "SYR" + case 158 => "TWN" + case 762 => "TJK" + case 834 => "TZA" + case 764 => "THA" + case 626 => "TLS" + case 768 => "TGO" + case 772 => "TKL" + case 776 => "TON" + case 780 => "TTO" + case 788 => "TUN" + case 792 => "TUR" + case 795 => "TKM" + case 796 => "TCA" + case 798 => "TUV" + case 800 => "UGA" + case 804 => "UKR" + case 784 => "ARE" + case 826 => "GBR" + case 840 => "USA" + case 581 => "UMI" + case 858 => "URY" + case 860 => "UZB" + case 548 => "VUT" + case 862 => "VEN" + case 704 => "VNM" + case 92 => "VGB" + case 850 => "VIR" + case 876 => "WLF" + case 732 => "ESH" + case 887 => "YEM" + case 894 => "ZMB" + case 716 => "ZWE" + case _ => "Unknown" + } +} + diff --git a/src/main/scala/org/globalforestwatch/layers/GadmAdm1.scala b/src/main/scala/org/globalforestwatch/layers/GadmAdm1.scala new file mode 100644 index 00000000..78ab46bf --- /dev/null +++ b/src/main/scala/org/globalforestwatch/layers/GadmAdm1.scala @@ -0,0 +1,18 @@ +package org.globalforestwatch.layers + +import org.globalforestwatch.grids.GridTile + +case class GadmAdm1(gridTile: GridTile, kwargs: Map[String, Any]) + extends IntegerLayer + with OptionalILayer { + + val datasetName: String = "gadm_adm1" + val uri: String = + uriForGrid(gridTile, kwargs) + + override def lookup(value: Int): Integer = + if (value == 9999) null else value + + + } + diff --git a/src/main/scala/org/globalforestwatch/layers/GadmAdm2.scala b/src/main/scala/org/globalforestwatch/layers/GadmAdm2.scala new file mode 100644 index 00000000..a7daa7bf --- /dev/null +++ b/src/main/scala/org/globalforestwatch/layers/GadmAdm2.scala @@ -0,0 +1,18 @@ +package org.globalforestwatch.layers + +import org.globalforestwatch.grids.GridTile + +case class GadmAdm2(gridTile: GridTile, kwargs: Map[String, Any]) + extends IntegerLayer + with OptionalILayer { + + val datasetName: String = "gadm_adm2" + val uri: String = + uriForGrid(gridTile, kwargs) + + override def lookup(value: Int): Integer = + if (value == 9999) null else value + + + } + diff --git a/src/main/scala/org/globalforestwatch/layers/NegligibleRisk.scala b/src/main/scala/org/globalforestwatch/layers/NegligibleRisk.scala new file mode 100644 index 00000000..aa3d1670 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/layers/NegligibleRisk.scala @@ -0,0 +1,19 @@ +package org.globalforestwatch.layers + +import org.globalforestwatch.grids.GridTile + +case class NegligibleRisk(gridTile: GridTile, kwargs: Map[String, Any]) + extends StringLayer + with OptionalILayer { + + val datasetName = "gfwpro_negligible_risk_analysis" + val uri: String = + uriForGrid(gridTile, kwargs) + + def lookup(value: Int): String = value match { + case 1 => "NO" + case 2 => "YES" + case 3 => "NA" + case _ => "Unknown" + } +} diff --git a/src/main/scala/org/globalforestwatch/layers/ProdesAmazonLossYear.scala b/src/main/scala/org/globalforestwatch/layers/ProdesAmazonLossYear.scala deleted file mode 100644 index d89deae0..00000000 --- a/src/main/scala/org/globalforestwatch/layers/ProdesAmazonLossYear.scala +++ /dev/null @@ -1,12 +0,0 @@ -package org.globalforestwatch.layers - -import org.globalforestwatch.grids.GridTile - -case class ProdesAmazonLossYear(gridTile: GridTile, kwargs: Map[String, Any]) extends IntegerLayer with OptionalILayer { - val datasetName = "inpe_amazon_prodes" - val uri: String = - uriForGrid(gridTile, kwargs) - - override def lookup(value: Int): Integer = - if (value == 0) null else value -} diff --git a/src/main/scala/org/globalforestwatch/layers/ProdesCerradoLossYear.scala b/src/main/scala/org/globalforestwatch/layers/ProdesCerradoLossYear.scala deleted file mode 100644 index 87a0c5b2..00000000 --- a/src/main/scala/org/globalforestwatch/layers/ProdesCerradoLossYear.scala +++ /dev/null @@ -1,12 +0,0 @@ -package org.globalforestwatch.layers - -import org.globalforestwatch.grids.GridTile - -case class ProdesCerradoLossYear(gridTile: GridTile, kwargs: Map[String, Any]) extends IntegerLayer with OptionalILayer { - val datasetName = "inpe_cerrado_prodes" - val uri: String = - uriForGrid(gridTile, kwargs) - - override def lookup(value: Int): Integer = - if (value == 0) null else value -} diff --git a/src/main/scala/org/globalforestwatch/layers/SBTNNaturalForests.scala b/src/main/scala/org/globalforestwatch/layers/SBTNNaturalForests.scala new file mode 100644 index 00000000..82bc3b24 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/layers/SBTNNaturalForests.scala @@ -0,0 +1,20 @@ +package org.globalforestwatch.layers + +import org.globalforestwatch.grids.GridTile + +case class SBTNNaturalForests(gridTile: GridTile, kwargs: Map[String, Any]) + extends StringLayer + with OptionalILayer { + + val datasetName = "sbtn_natural_forests_map" + val uri: String = uriForGrid(gridTile, kwargs) + + override val externalNoDataValue = "Unknown" + + def lookup(value: Int): String = value match { + case 0 => "Non-Forest" + case 1 => "Natural Forest" + case 2 => "Non-Natural Forest" + case _ => "Unknown" + } +} \ No newline at end of file diff --git a/src/main/scala/org/globalforestwatch/summarystats/SummaryMain.scala b/src/main/scala/org/globalforestwatch/summarystats/SummaryMain.scala index 0d50a3d3..15503712 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/SummaryMain.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/SummaryMain.scala @@ -9,6 +9,7 @@ import org.globalforestwatch.summarystats.gfwpro_dashboard.GfwProDashboardComman import org.globalforestwatch.summarystats.gladalerts.GladAlertsCommand.gladAlertsCommand import org.globalforestwatch.summarystats.treecoverloss.TreeCoverLossCommand.treeCoverLossCommand import org.globalforestwatch.summarystats.integrated_alerts.IntegratedAlertsCommand.integratedAlertsCommand +import org.globalforestwatch.summarystats.afi.AFiCommand.afiCommand import com.monovore.decline._ import org.globalforestwatch.config.GfwConfig @@ -25,7 +26,8 @@ object SummaryMain { gfwProDashboardCommand orElse gladAlertsCommand orElse treeCoverLossCommand orElse - integratedAlertsCommand + integratedAlertsCommand orElse + afiCommand } val command = Command(name, header, true)(main) diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiAnalysis.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiAnalysis.scala new file mode 100644 index 00000000..0b852204 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiAnalysis.scala @@ -0,0 +1,88 @@ +package org.globalforestwatch.summarystats.afi +import org.apache.spark.sql.functions.{col, lit, when, sum, max, concat_ws, collect_list} +import cats.data.Validated.{Invalid, Valid} +import cats.data.{NonEmptyList, Validated} +import geotrellis.vector.{Feature, Geometry} +import geotrellis.store.index.zcurve.Z2 +import org.apache.spark.HashPartitioner +import org.globalforestwatch.features._ +import org.globalforestwatch.summarystats._ +import org.globalforestwatch.util.GeometryConstructor.createPoint +import org.globalforestwatch.util.{RDDAdapter, SpatialJoinRDD} +import org.globalforestwatch.util.RDDAdapter +import org.globalforestwatch.ValidatedWorkflow +import org.apache.spark.rdd.RDD +import org.apache.spark.sql.SparkSession +import org.apache.spark.storage.StorageLevel +import org.apache.spark.sql.RelationalGroupedDataset + +object AFiAnalysis extends SummaryAnalysis { + + val name = "afi" + + def apply( + featureRDD: RDD[ValidatedLocation[Geometry]], + featureType: String, + spark: SparkSession, + kwargs: Map[String, Any] + ): Unit = { + featureRDD.persist(StorageLevel.MEMORY_AND_DISK) + + // TODO invalid should map to job error somehow, probably using ValidatedWorkflow + val validatedRDD = featureRDD.map { + case Validated.Valid(Location(id, geom: Geometry)) => Feature(geom, id) + case Validated.Invalid(Location(id, geom: Geometry)) => Feature(geom, id) + } + + val summaryRDD: RDD[ValidatedLocation[AFiSummary]] = AFiRDD(validatedRDD, AFiGrid.blockTileGrid, kwargs) +// val dataRDD: RDD[ValidatedLocation[AFiData]] = ValidatedWorkflow(summaryRDD).mapValid { summaries => +// summaries +// .mapValues { +// case summary: AFiSummary => summary.toAFiData() +// } +// }.unify + + // TODO somehow convert AFiSummary to AFiData + import spark.implicits._ + + val summaryDF = AFiAnalysis.aggregateResults( + AFiDF + .getFeatureDataFrame(summaryRDD, spark) + .filter(!$"gadm_id".contains("null")) + .withColumn( + "gadm_id", when(col("location_id") =!= -1|| col("gadm_id").contains("null"), lit("") ).otherwise(col("gadm_id")) + ) + .groupBy($"list_id", $"location_id", $"gadm_id") + ) + + val gadmAgg = AFiAnalysis.aggregateResults( + summaryDF + .filter($"location_id" === -1) + .groupBy($"list_id"), + ) + .withColumn("gadm_id", lit("")) + .withColumn("location_id", lit(-1)) + + val combinedDF = summaryDF.unionByName(gadmAgg) + val resultsDF = combinedDF + .withColumn( + "negligible_risk__percent", + $"negligible_risk_area__ha" / $"total_area__ha" * 100 + ) + .drop("negligible_risk_area__ha") + + val runOutputUrl: String = getOutputUrl(kwargs) + AFiExport.export(featureType, resultsDF, runOutputUrl, kwargs) + } + + private def aggregateResults(group: RelationalGroupedDataset) = { + group.agg( + sum("natural_forest__extent").alias("natural_forest__extent"), + sum("natural_forest_loss__ha").alias("natural_forest_loss__ha"), + sum("negligible_risk_area__ha").alias("negligible_risk_area__ha"), + sum("total_area__ha").alias("total_area__ha"), + max("status_code").alias("status_code"), + concat_ws(", ", collect_list(when(col("location_error").isNotNull && col("location_error") =!= "", col("location_error")))).alias("location_error") + ) + } +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiCommand.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiCommand.scala new file mode 100644 index 00000000..af46c216 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiCommand.scala @@ -0,0 +1,49 @@ +package org.globalforestwatch.summarystats.afi + +import cats.data.NonEmptyList +import org.globalforestwatch.summarystats.SummaryCommand +import cats.implicits._ +import com.monovore.decline.Opts +import geotrellis.vector.Geometry +import org.apache.sedona.core.spatialRDD.SpatialRDD +import org.globalforestwatch.config.GfwConfig +import org.globalforestwatch.features._ +import org.locationtech.jts.geom.Geometry +import cats.data.Validated.Valid + + +object AFiCommand extends SummaryCommand { + + val afiCommand: Opts[Unit] = Opts.subcommand( + name = AFiAnalysis.name, + help = "Compute summary statistics for GFW Pro Dashboard." + ) ( + ( + defaultOptions, + featureFilterOptions, + ).mapN { (default, filterOptions) => + val kwargs = Map( + "outputUrl" -> default.outputUrl, + "noOutputPathSuffix" -> default.noOutputPathSuffix, + "overwriteOutput" -> default.overwriteOutput, + "config" -> GfwConfig.get + ) + val featureFilter = FeatureFilter.fromOptions(default.featureType, filterOptions) + + runAnalysis { implicit spark => + val featureRDD = ValidatedFeatureRDD(default.featureUris, default.featureType, featureFilter, default.splitFeatures) + val filteredFeatureRDD = featureRDD.filter{ + case Valid((GfwProFeatureId(_, locationId), _)) => locationId != -2 + case _ => true + } + + AFiAnalysis( + filteredFeatureRDD, + default.featureType, + spark, + kwargs + ) + } + } + ) +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDF.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDF.scala new file mode 100644 index 00000000..a1b49ac0 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDF.scala @@ -0,0 +1,44 @@ +package org.globalforestwatch.summarystats.afi + +import org.apache.spark.rdd.RDD +import org.apache.spark.sql.{DataFrame, SparkSession} +import org.globalforestwatch.features.{CombinedFeatureId, FeatureId, GadmFeatureId, GfwProFeatureId, WdpaFeatureId} +import org.globalforestwatch.summarystats._ +import cats.data.Validated.{Invalid, Valid} +import org.globalforestwatch.summarystats.SummaryDF.{RowError, RowId} +import org.globalforestwatch.util.Util.fieldsFromCol + +object AFiDF extends SummaryDF { + case class RowGadmId(list_id: String, location_id: String, gadm_id: String) + + def getFeatureDataFrame( + summaryRDD: RDD[ValidatedLocation[AFiSummary]], + spark: SparkSession + ): DataFrame = { + import spark.implicits._ + + val rowId: FeatureId => RowId = { + case gfwproId: GfwProFeatureId => + RowId(gfwproId.listId, gfwproId.locationId.toString) + case gadmId: GadmFeatureId => + RowId("GADM 3.6", gadmId.toString) + case wdpaId: WdpaFeatureId => + RowId("WDPA", wdpaId.toString) + case id => + throw new IllegalArgumentException(s"Can't produce DataFrame for $id") + } + + summaryRDD + .flatMap { + case Valid(Location(fid, data)) => + data.stats.map { + case (dataGroup, data) => + (rowId(fid), RowError.empty, dataGroup, data) + } + case Invalid(Location(fid, err)) => + List((rowId(fid), RowError.fromJobError(err), AFiDataGroup.empty, AFiData.empty)) + } + .toDF("id", "error", "dataGroup", "data") + .select($"id.*", $"error.*", $"dataGroup.*", $"data.*") + } +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiData.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiData.scala new file mode 100644 index 00000000..dd4b1eb6 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiData.scala @@ -0,0 +1,34 @@ +package org.globalforestwatch.summarystats.afi + +import cats.Semigroup + +/** Summary data per class + * + * Note: This case class contains mutable values + */ +case class AFiData( + var natural_forest__extent: Double, + var natural_forest_loss__ha: Double, + var negligible_risk_area__ha: Double, + var total_area__ha: Double + ) { + def merge(other: AFiData): AFiData = { + AFiData( + natural_forest__extent + other.natural_forest__extent, + natural_forest_loss__ha + other.natural_forest_loss__ha, + negligible_risk_area__ha + other.negligible_risk_area__ha, + total_area__ha + other.total_area__ha + ) + } +} + +object AFiData { + def empty: AFiData = + AFiData(0, 0, 0, 0) + + implicit val afiDataSemigroup: Semigroup[AFiData] = + new Semigroup[AFiData] { + def combine(x: AFiData, y: AFiData): AFiData = x.merge(y) + } + +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDataGroup.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDataGroup.scala new file mode 100644 index 00000000..1cada84d --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDataGroup.scala @@ -0,0 +1,10 @@ +package org.globalforestwatch.summarystats.afi + +case class AFiDataGroup( + gadm_id: String +) + +object AFiDataGroup { + def empty: AFiDataGroup = + AFiDataGroup("") +} \ No newline at end of file diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDataParser.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDataParser.scala new file mode 100644 index 00000000..bcf80834 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiDataParser.scala @@ -0,0 +1,13 @@ +package org.globalforestwatch.summarystats.afi + +trait AFiDataParser[Self <: AFiDataParser[Self]] { + val value: Any + + def merge(other: Self): Self + + def toJson: String + + protected def round(value: Double, digits: Int = 4): Double = { + Math.round(value * math.pow(10, digits)) / math.pow(10, digits) + } +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiExport.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiExport.scala new file mode 100644 index 00000000..e0a0bc4e --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiExport.scala @@ -0,0 +1,36 @@ +package org.globalforestwatch.summarystats.afi + +import org.apache.spark.sql.{DataFrame, SaveMode} +import org.globalforestwatch.summarystats.SummaryExport +import org.globalforestwatch.util.Util.getAnyMapValue + +object AFiExport extends SummaryExport { + + override val csvOptions: Map[String, String] = Map( + "header" -> "true", + "delimiter" -> "\t", + "quote" -> "\u0000", + "escape" -> "\u0000", + "quoteMode" -> "NONE", + "nullValue" -> null, + "emptyValue" -> null + ) + + override protected def exportGfwPro(summaryDF: DataFrame, + outputUrl: String, + kwargs: Map[String, Any]): Unit = { + val saveMode = + if (getAnyMapValue[Boolean](kwargs, "overwriteOutput")) + SaveMode.Overwrite + else + SaveMode.ErrorIfExists + + summaryDF + .repartition(1) + .write + .mode(saveMode) + .options(csvOptions) + .csv(path = outputUrl + "/final") + } + +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiGrid.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiGrid.scala new file mode 100644 index 00000000..564f0f08 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiGrid.scala @@ -0,0 +1,15 @@ +package org.globalforestwatch.summarystats.afi + +import geotrellis.vector.Extent +import org.globalforestwatch.grids.{GridTile, TenByTen30mGrid} + +object AFiGrid + extends TenByTen30mGrid[AFiGridSources] { + + val gridExtent: Extent = Extent(-180.0000, -90.0000, 180.0000, 90.0000) + + def getSources(gridTile: GridTile, + kwargs: Map[String, Any]): AFiGridSources = + AFiGridSources.getCachedSources(gridTile, kwargs) + +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiGridSources.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiGridSources.scala new file mode 100644 index 00000000..c9f3094a --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiGridSources.scala @@ -0,0 +1,57 @@ +package org.globalforestwatch.summarystats.afi + +import cats.implicits._ +import geotrellis.layer.{LayoutDefinition, SpatialKey} +import geotrellis.raster.Raster +import org.globalforestwatch.grids.{GridSources, GridTile} +import org.globalforestwatch.layers._ + +/** + * @param gridTile top left corner, padded from east ex: "10N_010E" + */ +case class AFiGridSources(gridTile: GridTile, kwargs: Map[String, Any]) extends GridSources { + val treeCoverLoss: TreeCoverLoss = TreeCoverLoss(gridTile, kwargs) + val sbtnNaturalForest: SBTNNaturalForests = SBTNNaturalForests(gridTile, kwargs) + val negligibleRisk: NegligibleRisk = NegligibleRisk(gridTile, kwargs) + val gadmAdm0: GadmAdm0 = GadmAdm0(gridTile, kwargs) + val gadmAdm1: GadmAdm1 = GadmAdm1(gridTile, kwargs) + val gadmAdm2: GadmAdm2 = GadmAdm2(gridTile, kwargs) + + def readWindow( + windowKey: SpatialKey, + windowLayout: LayoutDefinition + ): Either[Throwable, Raster[AFiTile]] = { + for { + lossTile <- Either.catchNonFatal(treeCoverLoss.fetchWindow(windowKey, windowLayout)).right + } yield { + + val sbtnNaturalForestTile = sbtnNaturalForest.fetchWindow(windowKey, windowLayout) + val negligibleRiskTile = negligibleRisk.fetchWindow(windowKey, windowLayout) + val adm0Tile = gadmAdm0.fetchWindow(windowKey, windowLayout) + val adm1Tile = gadmAdm1.fetchWindow(windowKey, windowLayout) + val adm2Tile = gadmAdm2.fetchWindow(windowKey, windowLayout) + + val tile = AFiTile( + lossTile, + sbtnNaturalForestTile, + negligibleRiskTile, + adm0Tile, + adm1Tile, + adm2Tile + ) + Raster(tile, windowKey.extent(windowLayout)) + } + } +} + +object AFiGridSources { + + @transient + private lazy val cache = + scala.collection.concurrent.TrieMap + .empty[String, AFiGridSources] + + def getCachedSources(gridTile: GridTile, kwargs: Map[String, Any]): AFiGridSources = { + cache.getOrElseUpdate(gridTile.tileId, AFiGridSources(gridTile, kwargs)) + } +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiRDD.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiRDD.scala new file mode 100644 index 00000000..d74ec209 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiRDD.scala @@ -0,0 +1,49 @@ +package org.globalforestwatch.summarystats.afi + +import cats.implicits._ +import geotrellis.layer.{LayoutDefinition, SpatialKey} +import geotrellis.raster._ +import geotrellis.raster.rasterize.Rasterizer +import geotrellis.raster.summary.polygonal._ +import geotrellis.vector._ +import org.globalforestwatch.summarystats.ErrorSummaryRDD + +object AFiRDD extends ErrorSummaryRDD { + + type SOURCES = AFiGridSources + type SUMMARY = AFiSummary + type TILE = AFiTile + + def getSources(windowKey: SpatialKey, + windowLayout: LayoutDefinition, + kwargs: Map[String, Any]): Either[Throwable, SOURCES] = { + Either.catchNonFatal { + AFiGrid.getRasterSource( + windowKey, + windowLayout, + kwargs + ) + } + } + + def readWindow( + rs: SOURCES, + windowKey: SpatialKey, + windowLayout: LayoutDefinition + ): Either[Throwable, Raster[TILE]] = + rs.readWindow(windowKey, windowLayout) + + def runPolygonalSummary( + raster: Raster[TILE], + geometry: Geometry, + options: Rasterizer.Options, + kwargs: Map[String, Any] + ): PolygonalSummaryResult[SUMMARY] = { + raster.polygonalSummary( + geometry, + AFiSummary.getGridVisitor(kwargs), + options = options + ) + } + +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiSummary.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiSummary.scala new file mode 100644 index 00000000..78a81e06 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiSummary.scala @@ -0,0 +1,75 @@ +package org.globalforestwatch.summarystats.afi + +import cats.implicits._ +import geotrellis.raster._ +import geotrellis.raster.Raster +import geotrellis.raster.summary.GridVisitor +import org.globalforestwatch.summarystats.{Summary, summarySemigroup} +import org.globalforestwatch.util.Geodesy + +import java.time.LocalDate + +/** LossData Summary by year */ +case class AFiSummary( + stats: Map[AFiDataGroup, AFiData] = Map.empty + ) extends Summary[AFiSummary] { + + /** Combine two Maps and combine their LossData when a year is present in both */ + def merge(other: AFiSummary): AFiSummary = { + // the years.combine method uses LossData.lossDataSemigroup instance to perform per value combine on the map + AFiSummary(stats.combine(other.stats)) + } + + def isEmpty = stats.isEmpty +} + +object AFiSummary { + + def getGridVisitor(kwargs: Map[String, Any]): GridVisitor[Raster[AFiTile], AFiSummary] = + new GridVisitor[Raster[AFiTile], AFiSummary] { + private var acc: AFiSummary = + new AFiSummary() + + def result: AFiSummary = acc + + def visit(raster: Raster[AFiTile], col: Int, row: Int): Unit = { + val lossYear: Integer = raster.tile.treeCoverLoss.getData(col, row) + val naturalForestCategory: String = raster.tile.sbtnNaturalForest.getData(col, row) + val negligibleRisk: String = raster.tile.negligibleRisk.getData(col, row) + + val gadmAdm0: String = raster.tile.gadmAdm0.getData(col, row) + val gadmAdm1: Integer = raster.tile.gadmAdm1.getData(col, row) + val gadmAdm2: Integer = raster.tile.gadmAdm2.getData(col, row) + val gadmId: String = s"$gadmAdm0.$gadmAdm1.$gadmAdm2" + + // pixel Area + val lat: Double = raster.rasterExtent.gridRowToMap(row) + val area: Double = Geodesy.pixelArea( + lat, + raster.cellSize + ) + val areaHa = area / 10000.0 + val isNaturalForest = naturalForestCategory == "Natural Forest" + + + val groupKey = AFiDataGroup(gadmId) + val summaryData = acc.stats.getOrElse(groupKey, AFiData(0, 0, 0, 0)) + summaryData.total_area__ha += areaHa + + if (negligibleRisk == "NO") { + summaryData.negligible_risk_area__ha += areaHa + } + + if (naturalForestCategory == "Natural Forest") { + summaryData.natural_forest__extent += areaHa + } + + if (lossYear >= 2021 && naturalForestCategory == "Natural Forest") { + summaryData.natural_forest_loss__ha += areaHa + } + + val new_stats = acc.stats.updated(groupKey, summaryData) + acc = AFiSummary(new_stats) + } + } +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/afi/AFiTile.scala b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiTile.scala new file mode 100644 index 00000000..af8a8141 --- /dev/null +++ b/src/main/scala/org/globalforestwatch/summarystats/afi/AFiTile.scala @@ -0,0 +1,25 @@ +package org.globalforestwatch.summarystats.afi + +import geotrellis.raster.{CellGrid, CellType, IntCellType} +import org.globalforestwatch.layers._ + +/** + * + * Tile-like structure to hold tiles from datasets required for our summary. + * We can not use GeoTrellis MultibandTile because it requires all bands share a CellType. + */ +case class AFiTile( + treeCoverLoss: TreeCoverLoss#ITile, + sbtnNaturalForest: SBTNNaturalForests#OptionalITile, + negligibleRisk: NegligibleRisk#OptionalITile, + gadmAdm0: GadmAdm0#OptionalITile, + gadmAdm1: GadmAdm1#OptionalITile, + gadmAdm2: GadmAdm2#OptionalITile +) extends CellGrid[Int] { + + def cellType: CellType = treeCoverLoss.cellType + + def cols: Int = treeCoverLoss.cols + + def rows: Int = treeCoverLoss.rows +} diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDF.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDF.scala index b00d3406..4ed014b8 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDF.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDF.scala @@ -29,13 +29,12 @@ object ForestChangeDiagnosticDF extends SummaryDF { throw new IllegalArgumentException(s"Can't produce DataFrame for $id") } - dataRDD - .map { - case Valid(Location(fid, data)) => - (rowId(fid), RowError.empty, data) - case Invalid(Location(fid, err)) => - (rowId(fid), RowError.fromJobError(err), ForestChangeDiagnosticData.empty) - } + dataRDD.map { + case Valid(Location(fid, data)) => + (rowId(fid), RowError.empty, data) + case Invalid(Location(fid, err)) => + (rowId(fid), RowError.fromJobError(err), ForestChangeDiagnosticData.empty) + } .toDF("id", "error", "data") .select($"id.*" :: $"error.*" :: fieldsFromCol($"data", featureFields): _*) } @@ -53,27 +52,25 @@ object ForestChangeDiagnosticDF extends SummaryDF { throw new IllegalArgumentException("Not a CombinedFeatureId") } - dataRDD - .map { - case Valid(Location(fid, data)) => - (rowId(fid), RowError.empty, data) - case Invalid(Location(fid, err)) => - (rowId(fid), RowError.fromJobError(err), ForestChangeDiagnosticData.empty) - } + dataRDD.map { + case Valid(Location(fid, data)) => + (rowId(fid), RowError.empty, data) + case Invalid(Location(fid, err)) => + (rowId(fid), RowError.fromJobError(err), ForestChangeDiagnosticData.empty) + } .toDF("id", "error", "data") .select($"id.*" :: $"error.*" :: fieldsFromCol($"data", featureFields) ::: fieldsFromCol($"data", gridFields): _*) } def readIntermidateRDD( sources: NonEmptyList[String], - spark: SparkSession + spark: SparkSession, ): RDD[ValidatedLocation[ForestChangeDiagnosticData]] = { val df = FeatureDF(sources, GfwProFeature, FeatureFilter.empty, spark) val ds = df.select( colsFor[RowGridId].as[RowGridId], colsFor[RowError].as[RowError], - colsFor[ForestChangeDiagnosticData].as[ForestChangeDiagnosticData] - ) + colsFor[ForestChangeDiagnosticData].as[ForestChangeDiagnosticData]) ds.rdd.map { case (id, error, data) => if (error.status_code == 2) Valid(Location(id.toFeatureID, data)) @@ -104,12 +101,9 @@ object ForestChangeDiagnosticDF extends SummaryDF { "tree_cover_loss_soy_yearly", // treeCoverLossSoyPlanedAreasYearly "tree_cover_loss_idn_legal_yearly", // treeCoverLossIDNForestAreaYearly "tree_cover_loss_idn_forest_moratorium_yearly", // treeCoverLossIDNForestMoratoriumYearly - "tree_cover_loss_prodes_amazon_yearly", // prodesLossAmazonYearly - "tree_cover_loss_prodes_cerrado_yearly", // prodesLossCerradoYearly - "tree_cover_loss_prodes_amazon_wdpa_yearly", // prodesLossAmazonProtectedAreasYearly - "tree_cover_loss_prodes_cerrado_wdpa_yearly", // prodesLossCerradoProtectedAreasYearly - "tree_cover_loss_prodes_amazon_primary_forest_yearly", // prodesLossProdesAmazonPrimaryForestYearly - "tree_cover_loss_prodes_cerrado_primary_forest_yearly", // prodesLossProdesCerradoPrimaryForestYearly + "tree_cover_loss_prodes_yearly", // prodesLossYearly + "tree_cover_loss_prodes_wdpa_yearly", // prodesLossProtectedAreasYearly + "tree_cover_loss_prodes_primary_forest_yearly", // prodesLossProdesPrimaryForestYearly "tree_cover_loss_brazil_biomes_yearly", // treeCoverLossBRABiomesYearly "tree_cover_extent_total", // treeCoverExtent "tree_cover_extent_primary_forest", // treeCoverExtentPrimaryForest @@ -155,4 +149,4 @@ object ForestChangeDiagnosticDF extends SummaryDF { "plantation_in_protected_areas_area" //plantationInProtectedAreasArea ) -} +} \ No newline at end of file diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticData.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticData.scala index 29be9c73..881a302e 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticData.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticData.scala @@ -27,13 +27,10 @@ case class ForestChangeDiagnosticData( /** treeCoverLossIDNForestAreaYearly */ tree_cover_loss_idn_legal_yearly: ForestChangeDiagnosticDataLossYearlyCategory, tree_cover_loss_idn_forest_moratorium_yearly: ForestChangeDiagnosticDataLossYearly, - tree_cover_loss_prodes_amazon_yearly: ForestChangeDiagnosticDataLossYearly, - tree_cover_loss_prodes_cerrado_yearly: ForestChangeDiagnosticDataLossYearly, + tree_cover_loss_prodes_yearly: ForestChangeDiagnosticDataLossYearly, /** prodesLossProtectedAreasYearly */ - tree_cover_loss_prodes_amazon_wdpa_yearly: ForestChangeDiagnosticDataLossYearly, - tree_cover_loss_prodes_cerrado_wdpa_yearly: ForestChangeDiagnosticDataLossYearly, - tree_cover_loss_prodes_amazon_primary_forest_yearly: ForestChangeDiagnosticDataLossYearly, - tree_cover_loss_prodes_cerrado_primary_forest_yearly: ForestChangeDiagnosticDataLossYearly, + tree_cover_loss_prodes_wdpa_yearly: ForestChangeDiagnosticDataLossYearly, + tree_cover_loss_prodes_primary_forest_yearly: ForestChangeDiagnosticDataLossYearly, tree_cover_loss_brazil_biomes_yearly: ForestChangeDiagnosticDataLossYearlyCategory, tree_cover_extent_total: ForestChangeDiagnosticDataDouble, tree_cover_extent_primary_forest: ForestChangeDiagnosticDataDouble, @@ -119,19 +116,12 @@ case class ForestChangeDiagnosticData( tree_cover_loss_idn_forest_moratorium_yearly.merge( other.tree_cover_loss_idn_forest_moratorium_yearly ), - tree_cover_loss_prodes_amazon_yearly.merge(other.tree_cover_loss_prodes_amazon_yearly), - tree_cover_loss_prodes_cerrado_yearly.merge(other.tree_cover_loss_prodes_cerrado_yearly), - tree_cover_loss_prodes_amazon_wdpa_yearly.merge( - other.tree_cover_loss_prodes_amazon_wdpa_yearly + tree_cover_loss_prodes_yearly.merge(other.tree_cover_loss_prodes_yearly), + tree_cover_loss_prodes_wdpa_yearly.merge( + other.tree_cover_loss_prodes_wdpa_yearly ), - tree_cover_loss_prodes_cerrado_wdpa_yearly.merge( - other.tree_cover_loss_prodes_cerrado_wdpa_yearly - ), - tree_cover_loss_prodes_amazon_primary_forest_yearly.merge( - other.tree_cover_loss_prodes_amazon_primary_forest_yearly - ), - tree_cover_loss_prodes_cerrado_primary_forest_yearly.merge( - other.tree_cover_loss_prodes_cerrado_primary_forest_yearly + tree_cover_loss_prodes_primary_forest_yearly.merge( + other.tree_cover_loss_prodes_primary_forest_yearly ), tree_cover_loss_brazil_biomes_yearly.merge(other.tree_cover_loss_brazil_biomes_yearly), tree_cover_extent_total.merge(other.tree_cover_extent_total), @@ -181,27 +171,25 @@ case class ForestChangeDiagnosticData( ) } - /** @see - * https://docs.google.com/presentation/d/1nAq4mFNkv1q5vFvvXWReuLr4Znvr-1q-BDi6pl_5zTU/edit#slide=id.p + /** + * @see https://docs.google.com/presentation/d/1nAq4mFNkv1q5vFvvXWReuLr4Znvr-1q-BDi6pl_5zTU/edit#slide=id.p */ def withUpdatedCommodityRisk(): ForestChangeDiagnosticData = { /* Exclude the last year, limit data to 2021 to sync with palm risk tool: commodity_threat_deforestation, commodity_threat_peat, commodity_threat_protected_areas use year n and year n-1. Including information from the current year would under-represent these values as it's in progress. - */ + */ val minLossYear = ForestChangeDiagnosticDataLossYearly.prefilled.value.keys.min val maxLossYear = 2021 val years: List[Int] = List.range(minLossYear + 1, maxLossYear + 1) val forestValueIndicator: ForestChangeDiagnosticDataValueYearly = - ForestChangeDiagnosticDataValueYearly - .fill( - filtered_tree_cover_extent.value, - filtered_tree_cover_loss_yearly.value, - 2 - ) - .limitToMaxYear(maxLossYear) + ForestChangeDiagnosticDataValueYearly.fill( + filtered_tree_cover_extent.value, + filtered_tree_cover_loss_yearly.value, + 2 + ).limitToMaxYear(maxLossYear) val peatValueIndicator: ForestChangeDiagnosticDataValueYearly = ForestChangeDiagnosticDataValueYearly.fill(peat_area.value).limitToMaxYear(maxLossYear) @@ -212,22 +200,21 @@ case class ForestChangeDiagnosticData( val deforestationThreatIndicator: ForestChangeDiagnosticDataLossYearly = ForestChangeDiagnosticDataLossYearly( SortedMap( - years.map(year => - ( - year, { + years.map( + year => + (year, { // Somehow the compiler cannot infer the types correctly // I hence declare them here explicitly to help him out. val thisYearLoss: Double = - filtered_tree_cover_loss_yearly.value - .getOrElse(year, 0) + filtered_tree_cover_loss_yearly.value + .getOrElse(year, 0) val lastYearLoss: Double = filtered_tree_cover_loss_yearly.value .getOrElse(year - 1, 0) thisYearLoss + lastYearLoss - } - ) + }) ): _* ) ).limitToMaxYear(maxLossYear) @@ -235,14 +222,14 @@ case class ForestChangeDiagnosticData( val peatThreatIndicator: ForestChangeDiagnosticDataLossYearly = ForestChangeDiagnosticDataLossYearly( SortedMap( - years.map(year => - ( - year, { + years.map( + year => + (year, { // Somehow the compiler cannot infer the types correctly // I hence declare them here explicitly to help him out. val thisYearPeatLoss: Double = - filtered_tree_cover_loss_peat_yearly.value - .getOrElse(year, 0) + filtered_tree_cover_loss_peat_yearly.value + .getOrElse(year, 0) val lastYearPeatLoss: Double = filtered_tree_cover_loss_peat_yearly.value @@ -250,8 +237,7 @@ case class ForestChangeDiagnosticData( thisYearPeatLoss + lastYearPeatLoss + plantation_on_peat_area.value - } - ) + }) ): _* ) ).limitToMaxYear(maxLossYear) @@ -259,22 +245,21 @@ case class ForestChangeDiagnosticData( val protectedAreaThreatIndicator: ForestChangeDiagnosticDataLossYearly = ForestChangeDiagnosticDataLossYearly( SortedMap( - years.map(year => - ( - year, { + years.map( + year => + (year, { // Somehow the compiler cannot infer the types correctly // I hence declare them here explicitly to help him out. val thisYearProtectedAreaLoss: Double = - filtered_tree_cover_loss_protected_areas_yearly.value - .getOrElse(year, 0) + filtered_tree_cover_loss_protected_areas_yearly.value + .getOrElse(year, 0) val lastYearProtectedAreaLoss: Double = filtered_tree_cover_loss_protected_areas_yearly.value .getOrElse(year - 1, 0) thisYearProtectedAreaLoss + lastYearProtectedAreaLoss + plantation_in_protected_areas_area.value - } - ) + }) ): _* ) ).limitToMaxYear(maxLossYear) @@ -285,8 +270,7 @@ case class ForestChangeDiagnosticData( commodity_value_protected_areas = protectedAreaValueIndicator, commodity_threat_deforestation = deforestationThreatIndicator, commodity_threat_peat = peatThreatIndicator, - commodity_threat_protected_areas = protectedAreaThreatIndicator - ) + commodity_threat_protected_areas = protectedAreaThreatIndicator) } } @@ -310,9 +294,6 @@ object ForestChangeDiagnosticData { ForestChangeDiagnosticDataLossYearly.empty, ForestChangeDiagnosticDataLossYearly.empty, ForestChangeDiagnosticDataLossYearly.empty, - ForestChangeDiagnosticDataLossYearly.empty, - ForestChangeDiagnosticDataLossYearly.empty, - ForestChangeDiagnosticDataLossYearly.empty, ForestChangeDiagnosticDataLossYearlyCategory.empty, ForestChangeDiagnosticDataDouble.empty, ForestChangeDiagnosticDataDouble.empty, @@ -351,17 +332,17 @@ object ForestChangeDiagnosticData { ForestChangeDiagnosticDataLossYearly.empty, ForestChangeDiagnosticDataLossYearly.empty, ForestChangeDiagnosticDataLossYearly.empty, - ForestChangeDiagnosticDataLossYearly.empty + ForestChangeDiagnosticDataLossYearly.empty, ) implicit val lossDataSemigroup: Semigroup[ForestChangeDiagnosticData] = new Semigroup[ForestChangeDiagnosticData] { - def combine(x: ForestChangeDiagnosticData, y: ForestChangeDiagnosticData): ForestChangeDiagnosticData = + def combine(x: ForestChangeDiagnosticData, + y: ForestChangeDiagnosticData): ForestChangeDiagnosticData = x.merge(y) } implicit def dataExpressionEncoder: ExpressionEncoder[ForestChangeDiagnosticData] = - frameless - .TypedExpressionEncoder[ForestChangeDiagnosticData] + frameless.TypedExpressionEncoder[ForestChangeDiagnosticData] .asInstanceOf[ExpressionEncoder[ForestChangeDiagnosticData]] } diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDataDouble.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDataDouble.scala index 2c7e3369..dda4cb4c 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDataDouble.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticDataDouble.scala @@ -3,6 +3,7 @@ import frameless.Injection import org.globalforestwatch.util.Implicits._ import io.circe.syntax._ + case class ForestChangeDiagnosticDataDouble(value: Double) extends ForestChangeDiagnosticDataParser[ForestChangeDiagnosticDataDouble] { def merge( other: ForestChangeDiagnosticDataDouble diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticGridSources.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticGridSources.scala index 6f359f6c..0fdc566c 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticGridSources.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticGridSources.scala @@ -6,10 +6,11 @@ import geotrellis.raster.Raster import org.globalforestwatch.grids.{GridSources, GridTile} import org.globalforestwatch.layers._ -/** @param gridTile - * top left corner, padded from east ex: "10N_010E" +/** + * @param gridTile top left corner, padded from east ex: "10N_010E" */ -case class ForestChangeDiagnosticGridSources(gridTile: GridTile, kwargs: Map[String, Any]) extends GridSources { +case class ForestChangeDiagnosticGridSources(gridTile: GridTile, kwargs: Map[String, Any]) + extends GridSources { val treeCoverLoss: TreeCoverLoss = TreeCoverLoss(gridTile, kwargs) val treeCoverDensity2000: TreeCoverDensityPercent2000 = TreeCoverDensityPercent2000(gridTile, kwargs) @@ -22,17 +23,17 @@ case class ForestChangeDiagnosticGridSources(gridTile: GridTile, kwargs: Map[Str val isSoyPlantedArea: SoyPlantedAreas = SoyPlantedAreas(gridTile, kwargs) val idnForestArea: IndonesiaForestArea = IndonesiaForestArea(gridTile, kwargs) val isIDNForestMoratorium: IndonesiaForestMoratorium = IndonesiaForestMoratorium(gridTile, kwargs) - val prodesAmazonLossYear: ProdesAmazonLossYear = ProdesAmazonLossYear(gridTile, kwargs) - val prodesCerradoLossYear: ProdesCerradoLossYear = ProdesCerradoLossYear(gridTile, kwargs) + val prodesLossYear: ProdesLossYear = ProdesLossYear(gridTile, kwargs) val braBiomes: BrazilBiomes = BrazilBiomes(gridTile, kwargs) val isPlantation: PlantedForestsBool = PlantedForestsBool(gridTile, kwargs) val gfwProCoverage: GFWProCoverage = GFWProCoverage(gridTile, kwargs) val argOTBN: ArgOTBN = ArgOTBN(gridTile, kwargs) + def readWindow( - windowKey: SpatialKey, - windowLayout: LayoutDefinition - ): Either[Throwable, Raster[ForestChangeDiagnosticTile]] = { + windowKey: SpatialKey, + windowLayout: LayoutDefinition + ): Either[Throwable, Raster[ForestChangeDiagnosticTile]] = { for { // Failure for any of these reads will result in function returning Left[Throwable] @@ -58,13 +59,13 @@ case class ForestChangeDiagnosticGridSources(gridTile: GridTile, kwargs: Map[Str val isSoyPlantedAreasTile = isSoyPlantedArea.fetchWindow(windowKey, windowLayout) val idnForestAreaTile = idnForestArea.fetchWindow(windowKey, windowLayout) val isINDForestMoratoriumTile = isIDNForestMoratorium.fetchWindow(windowKey, windowLayout) - val prodesAmazonLossYearTile = prodesAmazonLossYear.fetchWindow(windowKey, windowLayout) - val prodesCerradoLossYearTile = prodesCerradoLossYear.fetchWindow(windowKey, windowLayout) + val prodesLossYearTile = prodesLossYear.fetchWindow(windowKey, windowLayout) val braBiomesTile = braBiomes.fetchWindow(windowKey, windowLayout) val isPlantationTile = isPlantation.fetchWindow(windowKey, windowLayout) val gfwProCoverageTile = gfwProCoverage.fetchWindow(windowKey, windowLayout) val argOTBNTile = argOTBN.fetchWindow(windowKey, windowLayout) + val tile = ForestChangeDiagnosticTile( lossTile, tcd2000Tile, @@ -77,8 +78,7 @@ case class ForestChangeDiagnosticGridSources(gridTile: GridTile, kwargs: Map[Str isSoyPlantedAreasTile, idnForestAreaTile, isINDForestMoratoriumTile, - prodesAmazonLossYearTile, - prodesCerradoLossYearTile, + prodesLossYearTile, braBiomesTile, isPlantationTile, gfwProCoverageTile, @@ -98,9 +98,9 @@ object ForestChangeDiagnosticGridSources { .empty[String, ForestChangeDiagnosticGridSources] def getCachedSources( - gridTile: GridTile, - kwargs: Map[String, Any] - ): ForestChangeDiagnosticGridSources = { + gridTile: GridTile, + kwargs: Map[String, Any] + ): ForestChangeDiagnosticGridSources = { cache.getOrElseUpdate( gridTile.tileId, diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticRawDataGroup.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticRawDataGroup.scala index bd4676a2..93e95ab7 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticRawDataGroup.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticRawDataGroup.scala @@ -1,34 +1,30 @@ package org.globalforestwatch.summarystats.forest_change_diagnostic -case class ForestChangeDiagnosticRawDataGroup( - umdTreeCoverLossYear: Int, - isUMDLoss: Boolean, - prodesAmazonLossYear: Int, - prodesCerradoLossYear: Int, - isProdesAmazonLoss: Boolean, - isProdesCerradoLoss: Boolean, - isTreeCoverExtent30: Boolean, - isTreeCoverExtent90: Boolean, - isPrimaryForest: Boolean, - isPeatlands: Boolean, - isIntactForestLandscapes2000: Boolean, - isProtectedArea: Boolean, - seAsiaLandCover: String, - idnLandCover: String, - isSoyPlantedAreas: Boolean, - idnForestArea: String, - isIdnForestMoratorium: Boolean, - braBiomes: String, - isPlantation: Boolean, - argOTBN: String, - southAmericaPresence: Boolean, - legalAmazonPresence: Boolean, - braBiomesPresence: Boolean, - cerradoBiomesPresence: Boolean, - seAsiaPresence: Boolean, - idnPresence: Boolean, - argPresence: Boolean -) { +case class ForestChangeDiagnosticRawDataGroup(umdTreeCoverLossYear: Int, + isUMDLoss: Boolean, + prodesLossYear: Int, + isProdesLoss: Boolean, + isTreeCoverExtent30: Boolean, + isTreeCoverExtent90: Boolean, + isPrimaryForest: Boolean, + isPeatlands: Boolean, + isIntactForestLandscapes2000: Boolean, + isProtectedArea: Boolean, + seAsiaLandCover: String, + idnLandCover: String, + isSoyPlantedAreas: Boolean, + idnForestArea: String, + isIdnForestMoratorium: Boolean, + braBiomes: String, + isPlantation: Boolean, + argOTBN: String, + southAmericaPresence: Boolean, + legalAmazonPresence: Boolean, + braBiomesPresence: Boolean, + cerradoBiomesPresence: Boolean, + seAsiaPresence: Boolean, + idnPresence: Boolean, + argPresence: Boolean) { /** Produce a partial ForestChangeDiagnosticData only for the loss year in this data group */ def toForestChangeDiagnosticData(totalArea: Double): ForestChangeDiagnosticData = ForestChangeDiagnosticData( @@ -42,96 +38,93 @@ case class ForestChangeDiagnosticRawDataGroup( totalArea, isUMDLoss && isTreeCoverExtent90 ), - tree_cover_loss_primary_forest_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isPrimaryForest && isUMDLoss - ), + tree_cover_loss_primary_forest_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isPrimaryForest && isUMDLoss + ), tree_cover_loss_peat_yearly = ForestChangeDiagnosticDataLossYearly.fill( umdTreeCoverLossYear, totalArea, isPeatlands && isUMDLoss ), - tree_cover_loss_intact_forest_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isIntactForestLandscapes2000 && isUMDLoss - ), - tree_cover_loss_protected_areas_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isProtectedArea && isUMDLoss - ), - tree_cover_loss_arg_otbn_yearly = ForestChangeDiagnosticDataLossYearlyCategory.fill( - argOTBN, - umdTreeCoverLossYear, - totalArea, - include = isUMDLoss - ), - tree_cover_loss_sea_landcover_yearly = ForestChangeDiagnosticDataLossYearlyCategory.fill( - seAsiaLandCover, - umdTreeCoverLossYear, - totalArea, - include = isUMDLoss - ), - tree_cover_loss_idn_landcover_yearly = ForestChangeDiagnosticDataLossYearlyCategory.fill( - idnLandCover, - umdTreeCoverLossYear, - totalArea, - include = isUMDLoss - ), - tree_cover_loss_soy_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isSoyPlantedAreas && isUMDLoss - ), - tree_cover_loss_idn_legal_yearly = ForestChangeDiagnosticDataLossYearlyCategory.fill( - idnForestArea, + tree_cover_loss_intact_forest_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isIntactForestLandscapes2000 && isUMDLoss + ), + tree_cover_loss_protected_areas_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isProtectedArea && isUMDLoss + ), + tree_cover_loss_arg_otbn_yearly = + ForestChangeDiagnosticDataLossYearlyCategory.fill( + argOTBN, umdTreeCoverLossYear, totalArea, include = isUMDLoss ), - tree_cover_loss_idn_forest_moratorium_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isIdnForestMoratorium && isUMDLoss - ), - tree_cover_loss_prodes_amazon_yearly = ForestChangeDiagnosticDataLossYearly.fill( - prodesAmazonLossYear, - totalArea, - isProdesAmazonLoss - ), - tree_cover_loss_prodes_cerrado_yearly = ForestChangeDiagnosticDataLossYearly.fill( - prodesCerradoLossYear, - totalArea, - isProdesCerradoLoss - ), - tree_cover_loss_prodes_amazon_wdpa_yearly = ForestChangeDiagnosticDataLossYearly.fill( - prodesAmazonLossYear, - totalArea, - isProdesAmazonLoss && isProtectedArea - ), - tree_cover_loss_prodes_cerrado_wdpa_yearly = ForestChangeDiagnosticDataLossYearly.fill( - prodesCerradoLossYear, - totalArea, - isProdesCerradoLoss && isProtectedArea - ), - tree_cover_loss_prodes_amazon_primary_forest_yearly = ForestChangeDiagnosticDataLossYearly.fill( - prodesAmazonLossYear, - totalArea, - isProdesAmazonLoss && isPrimaryForest - ), - tree_cover_loss_prodes_cerrado_primary_forest_yearly = ForestChangeDiagnosticDataLossYearly.fill( - prodesCerradoLossYear, - totalArea, - isProdesCerradoLoss && isPrimaryForest - ), - tree_cover_loss_brazil_biomes_yearly = ForestChangeDiagnosticDataLossYearlyCategory.fill( - braBiomes, - umdTreeCoverLossYear, + tree_cover_loss_sea_landcover_yearly = + ForestChangeDiagnosticDataLossYearlyCategory.fill( + seAsiaLandCover, + umdTreeCoverLossYear, + totalArea, + include = isUMDLoss + ), + tree_cover_loss_idn_landcover_yearly = + ForestChangeDiagnosticDataLossYearlyCategory.fill( + idnLandCover, + umdTreeCoverLossYear, + totalArea, + include = isUMDLoss + ), + tree_cover_loss_soy_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isSoyPlantedAreas && isUMDLoss + ), + tree_cover_loss_idn_legal_yearly = + ForestChangeDiagnosticDataLossYearlyCategory.fill( + idnForestArea, + umdTreeCoverLossYear, + totalArea, + include = isUMDLoss + ), + tree_cover_loss_idn_forest_moratorium_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isIdnForestMoratorium && isUMDLoss + ), + tree_cover_loss_prodes_yearly = ForestChangeDiagnosticDataLossYearly.fill( + prodesLossYear, totalArea, - include = isUMDLoss + isProdesLoss ), + tree_cover_loss_prodes_wdpa_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + prodesLossYear, + totalArea, + isProdesLoss && isProtectedArea + ), + tree_cover_loss_prodes_primary_forest_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + prodesLossYear, + totalArea, + isProdesLoss && isPrimaryForest + ), + tree_cover_loss_brazil_biomes_yearly = + ForestChangeDiagnosticDataLossYearlyCategory.fill( + braBiomes, + umdTreeCoverLossYear, + totalArea, + include = isUMDLoss + ), tree_cover_extent_total = ForestChangeDiagnosticDataDouble .fill(totalArea, isTreeCoverExtent30), tree_cover_extent_primary_forest = ForestChangeDiagnosticDataDouble.fill( @@ -179,30 +172,36 @@ case class ForestChangeDiagnosticRawDataGroup( .fill(braBiomesPresence), cerrado_biome_presence = ForestChangeDiagnosticDataBoolean .fill(cerradoBiomesPresence), - southeast_asia_presence = ForestChangeDiagnosticDataBoolean.fill(seAsiaPresence), - indonesia_presence = ForestChangeDiagnosticDataBoolean.fill(idnPresence), - argentina_presence = ForestChangeDiagnosticDataBoolean.fill(argPresence), + southeast_asia_presence = + ForestChangeDiagnosticDataBoolean.fill(seAsiaPresence), + indonesia_presence = + ForestChangeDiagnosticDataBoolean.fill(idnPresence), + argentina_presence = + ForestChangeDiagnosticDataBoolean.fill(argPresence), filtered_tree_cover_extent = ForestChangeDiagnosticDataDouble .fill( totalArea, isTreeCoverExtent90 && !isPlantation ), - filtered_tree_cover_extent_yearly = ForestChangeDiagnosticDataValueYearly.empty, + filtered_tree_cover_extent_yearly = + ForestChangeDiagnosticDataValueYearly.empty, filtered_tree_cover_loss_yearly = ForestChangeDiagnosticDataLossYearly.fill( umdTreeCoverLossYear, totalArea, isUMDLoss && isTreeCoverExtent90 && !isPlantation ), - filtered_tree_cover_loss_peat_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isUMDLoss && isTreeCoverExtent90 && !isPlantation && isPeatlands - ), - filtered_tree_cover_loss_protected_areas_yearly = ForestChangeDiagnosticDataLossYearly.fill( - umdTreeCoverLossYear, - totalArea, - isUMDLoss && isTreeCoverExtent90 && !isPlantation && isProtectedArea - ), + filtered_tree_cover_loss_peat_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isUMDLoss && isTreeCoverExtent90 && !isPlantation && isPeatlands + ), + filtered_tree_cover_loss_protected_areas_yearly = + ForestChangeDiagnosticDataLossYearly.fill( + umdTreeCoverLossYear, + totalArea, + isUMDLoss && isTreeCoverExtent90 && !isPlantation && isProtectedArea + ), plantation_area = ForestChangeDiagnosticDataDouble .fill(totalArea, isPlantation), plantation_on_peat_area = ForestChangeDiagnosticDataDouble @@ -223,4 +222,6 @@ case class ForestChangeDiagnosticRawDataGroup( commodity_threat_protected_areas = ForestChangeDiagnosticDataLossYearly.empty, commodity_threat_fires = ForestChangeDiagnosticDataLossYearly.empty ) -} + } + + diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticSummary.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticSummary.scala index cd11fed3..76f86e5d 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticSummary.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticSummary.scala @@ -8,8 +8,9 @@ import org.globalforestwatch.util.Geodesy /** LossData Summary by year */ case class ForestChangeDiagnosticSummary( - stats: Map[ForestChangeDiagnosticRawDataGroup, ForestChangeDiagnosticRawData] = Map.empty -) extends Summary[ForestChangeDiagnosticSummary] { + stats: Map[ForestChangeDiagnosticRawDataGroup, + ForestChangeDiagnosticRawData] = Map.empty + ) extends Summary[ForestChangeDiagnosticSummary] { /** Combine two Maps and combine their LossData when a year is present in both */ def merge( @@ -38,23 +39,23 @@ object ForestChangeDiagnosticSummary { def getGridVisitor( kwargs: Map[String, Any] - ): GridVisitor[Raster[ForestChangeDiagnosticTile], ForestChangeDiagnosticSummary] = + ): GridVisitor[Raster[ForestChangeDiagnosticTile], + ForestChangeDiagnosticSummary] = new GridVisitor[Raster[ForestChangeDiagnosticTile], ForestChangeDiagnosticSummary] { private var acc: ForestChangeDiagnosticSummary = new ForestChangeDiagnosticSummary() def result: ForestChangeDiagnosticSummary = acc - def visit(raster: Raster[ForestChangeDiagnosticTile], col: Int, row: Int): Unit = { + def visit(raster: Raster[ForestChangeDiagnosticTile], + col: Int, + row: Int): Unit = { // This is a pixel by pixel operation // pixel Area val lat: Double = raster.rasterExtent.gridRowToMap(row) - val area: Double = Geodesy.pixelArea( - lat, - raster.cellSize - ) // uses Pixel's center coordiate. +- raster.cellSize.height/2 doesn't make much of a difference + val area: Double = Geodesy.pixelArea(lat, raster.cellSize) // uses Pixel's center coordiate. +- raster.cellSize.height/2 doesn't make much of a difference val areaHa = area / 10000.0 // input layers @@ -75,16 +76,8 @@ object ForestChangeDiagnosticSummary { val isIntactForestLandscapes2000: Boolean = raster.tile.isIntactForestLandscapes2000.getData(col, row) val wdpa: String = raster.tile.wdpaProtectedAreas.getData(col, row) - val prodesAmazonLossYear: Int = { - val loss = raster.tile.prodesAmazonLossYear.getData(col, row) - if (loss != null) { - loss.toInt - } else { - 0 - } - } - val prodesCerradoLossYear: Int = { - val loss = raster.tile.prodesCerradoLossYear.getData(col, row) + val prodesLossYear: Int = { + val loss = raster.tile.prodesLossYear.getData(col, row) if (loss != null) { loss.toInt } else { @@ -110,8 +103,8 @@ object ForestChangeDiagnosticSummary { val isTreeCoverExtent90: Boolean = tcd2000 > 90 val isUMDLoss: Boolean = isTreeCoverExtent30 && umdTreeCoverLossYear > 0 val isProtectedArea: Boolean = wdpa != "" - val isProdesAmazonLoss: Boolean = prodesAmazonLossYear > 0 - val isProdesCerradoLoss: Boolean = prodesCerradoLossYear > 0 + val isProdesLoss: Boolean = prodesLossYear > 0 + val southAmericaPresence = gfwProCoverage.getOrElse("South America", false) val legalAmazonPresence = @@ -126,10 +119,8 @@ object ForestChangeDiagnosticSummary { val groupKey = ForestChangeDiagnosticRawDataGroup( umdTreeCoverLossYear, isUMDLoss, - prodesAmazonLossYear, - prodesCerradoLossYear, - isProdesAmazonLoss, - isProdesCerradoLoss, + prodesLossYear, + isProdesLoss, isTreeCoverExtent30, isTreeCoverExtent90, isPrimaryForest, diff --git a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticTile.scala b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticTile.scala index 1051bbaa..26649710 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticTile.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticTile.scala @@ -3,28 +3,29 @@ package org.globalforestwatch.summarystats.forest_change_diagnostic import geotrellis.raster.{CellGrid, CellType} import org.globalforestwatch.layers._ -/** Tile-like structure to hold tiles from datasets required for our summary. We can not use GeoTrellis MultibandTile because it requires - * all bands share a CellType. +/** + * + * Tile-like structure to hold tiles from datasets required for our summary. + * We can not use GeoTrellis MultibandTile because it requires all bands share a CellType. */ case class ForestChangeDiagnosticTile( - loss: TreeCoverLoss#ITile, - tcd2000: TreeCoverDensityPercent2000#ITile, - isPrimaryForest: PrimaryForest#OptionalITile, - isPeatlands: Peatlands#OptionalITile, - isIntactForestLandscapes2000: IntactForestLandscapes2000#OptionalITile, - wdpaProtectedAreas: ProtectedAreas#OptionalITile, - seAsiaLandCover: SEAsiaLandCover#OptionalITile, - idnLandCover: IndonesiaLandCover#OptionalITile, - isSoyPlantedArea: SoyPlantedAreas#OptionalITile, - idnForestArea: IndonesiaForestArea#OptionalITile, - isIDNForestMoratorium: IndonesiaForestMoratorium#OptionalITile, - prodesAmazonLossYear: ProdesAmazonLossYear#OptionalITile, - prodesCerradoLossYear: ProdesCerradoLossYear#OptionalITile, - braBiomes: BrazilBiomes#OptionalITile, - isPlantation: PlantedForestsBool#OptionalITile, - gfwProCoverage: GFWProCoverage#OptionalITile, - argOTBN: ArgOTBN#OptionalITile -) extends CellGrid[Int] { + loss: TreeCoverLoss#ITile, + tcd2000: TreeCoverDensityPercent2000#ITile, + isPrimaryForest: PrimaryForest#OptionalITile, + isPeatlands: Peatlands#OptionalITile, + isIntactForestLandscapes2000: IntactForestLandscapes2000#OptionalITile, + wdpaProtectedAreas: ProtectedAreas#OptionalITile, + seAsiaLandCover: SEAsiaLandCover#OptionalITile, + idnLandCover: IndonesiaLandCover#OptionalITile, + isSoyPlantedArea: SoyPlantedAreas#OptionalITile, + idnForestArea: IndonesiaForestArea#OptionalITile, + isIDNForestMoratorium: IndonesiaForestMoratorium#OptionalITile, + prodesLossYear: ProdesLossYear#OptionalITile, + braBiomes: BrazilBiomes#OptionalITile, + isPlantation: PlantedForestsBool#OptionalITile, + gfwProCoverage: GFWProCoverage#OptionalITile, + argOTBN: ArgOTBN#OptionalITile + ) extends CellGrid[Int] { def cellType: CellType = loss.cellType diff --git a/src/main/scala/org/globalforestwatch/summarystats/gfwpro_dashboard/GfwProDashboardSummary.scala b/src/main/scala/org/globalforestwatch/summarystats/gfwpro_dashboard/GfwProDashboardSummary.scala index a41e9cbc..53abdfab 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/gfwpro_dashboard/GfwProDashboardSummary.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/gfwpro_dashboard/GfwProDashboardSummary.scala @@ -22,7 +22,8 @@ case class GfwProDashboardSummary( def toGfwProDashboardData(): GfwProDashboardData = { stats - .map { case (group, data) => group.toGfwProDashboardData(data.alertCount, data.treeCoverExtentArea) } + .map { case (group, data) => group. + toGfwProDashboardData(data.alertCount, data.treeCoverExtentArea) } .foldLeft(GfwProDashboardData.empty)( _ merge _) } } diff --git a/src/main/scala/org/globalforestwatch/summarystats/integrated_alerts/IntegratedAlertsCommand.scala b/src/main/scala/org/globalforestwatch/summarystats/integrated_alerts/IntegratedAlertsCommand.scala index a15dff15..979c050e 100644 --- a/src/main/scala/org/globalforestwatch/summarystats/integrated_alerts/IntegratedAlertsCommand.scala +++ b/src/main/scala/org/globalforestwatch/summarystats/integrated_alerts/IntegratedAlertsCommand.scala @@ -15,7 +15,7 @@ object IntegratedAlertsCommand extends SummaryCommand { val integratedAlertsCommand: Opts[Unit] = Opts.subcommand( name = IntegratedAlertsAnalysis.name, - help = "Compute GLAD summary statistics for GFW dashboards." + help = "Compute Integrated Alerts summary statistics for GFW dashboards." ) { ( defaultOptions, diff --git a/src/test/resources/palm-32-fcd-output/part-00000-16db04de-3864-40aa-98a5-c6ca58e882ce-c000.csv b/src/test/resources/palm-32-fcd-output/part-00000-4a23d5d0-56c1-4840-8264-9b4a48a83e9c-c000.csv similarity index 72% rename from src/test/resources/palm-32-fcd-output/part-00000-16db04de-3864-40aa-98a5-c6ca58e882ce-c000.csv rename to src/test/resources/palm-32-fcd-output/part-00000-4a23d5d0-56c1-4840-8264-9b4a48a83e9c-c000.csv index 5d76cb71..23a5d78c 100644 --- a/src/test/resources/palm-32-fcd-output/part-00000-16db04de-3864-40aa-98a5-c6ca58e882ce-c000.csv +++ b/src/test/resources/palm-32-fcd-output/part-00000-4a23d5d0-56c1-4840-8264-9b4a48a83e9c-c000.csv @@ -1,2 +1,2 @@ -list_id location_id status_code location_error tree_cover_loss_total_yearly tree_cover_loss_primary_forest_yearly tree_cover_loss_peat_yearly tree_cover_loss_intact_forest_yearly tree_cover_loss_protected_areas_yearly tree_cover_loss_arg_otbn_yearly tree_cover_loss_sea_landcover_yearly tree_cover_loss_idn_landcover_yearly tree_cover_loss_soy_yearly tree_cover_loss_idn_legal_yearly tree_cover_loss_idn_forest_moratorium_yearly tree_cover_loss_prodes_amazon_yearly tree_cover_loss_prodes_cerrado_yearly tree_cover_loss_prodes_amazon_wdpa_yearly tree_cover_loss_prodes_cerrado_wdpa_yearly tree_cover_loss_prodes_amazon_primary_forest_yearly tree_cover_loss_prodes_cerrado_primary_forest_yearly tree_cover_loss_brazil_biomes_yearly tree_cover_extent_total tree_cover_extent_primary_forest tree_cover_extent_protected_areas tree_cover_extent_peat tree_cover_extent_intact_forest natural_habitat_primary natural_habitat_intact_forest total_area protected_areas_area peat_area arg_otbn_area brazil_biomes idn_legal_area sea_landcover_area idn_landcover_area idn_forest_moratorium_area south_america_presence legal_amazon_presence brazil_biomes_presence cerrado_biome_presence southeast_asia_presence indonesia_presence argentina_presence commodity_value_forest_extent commodity_value_peat commodity_value_protected_areas commodity_threat_deforestation commodity_threat_peat commodity_threat_protected_areas commodity_threat_fires -1 31 2 {"2001":1021.7622,"2002":851.014,"2003":310.1835,"2004":2169.8398,"2005":2325.3843,"2006":4162.4968,"2007":2968.7863,"2008":4015.4403,"2009":2002.9194,"2010":1173.7001,"2011":1703.6902,"2012":2838.0498,"2013":1841.7568,"2014":2468.7732,"2015":2028.9672,"2016":3344.8135,"2017":1026.7609,"2018":525.5327,"2019":618.7052,"2020":924.699,"2021":857.8225,"2022":560.0482} {"2001":154.8617,"2002":306.7253,"2003":92.3781,"2004":717.7405,"2005":1202.6952,"2006":1831.5766,"2007":1668.2764,"2008":1753.2317,"2009":797.282,"2010":454.5023,"2011":872.3613,"2012":1251.8543,"2013":1083.6799,"2014":1290.2177,"2015":1360.2574,"2016":2313.5001,"2017":286.2809,"2018":159.8557,"2019":162.3929,"2020":134.2652,"2021":167.4697,"2022":133.6506} {"2001":557.4251,"2002":236.2539,"2003":71.8566,"2004":741.25,"2005":957.52,"2006":1229.3335,"2007":1037.5018,"2008":891.235,"2009":486.4665,"2010":363.5759,"2011":411.9212,"2012":1078.9246,"2013":862.5621,"2014":974.783,"2015":942.4571,"2016":1472.8429,"2017":211.3403,"2018":144.7173,"2019":148.7917,"2020":142.3323,"2021":122.7372,"2022":94.914} {} {"2001":42.2692,"2002":228.1732,"2003":11.3743,"2004":3.9196,"2005":1.614,"2006":15.3711,"2007":4.4576,"2008":2.8437,"2009":4.765,"2010":7.9931,"2011":10.7597,"2012":8.1466,"2013":0.1537,"2014":8.5307,"2015":18.6758,"2016":139.2616,"2017":10.7596,"2018":0.3843,"2019":0.2306,"2020":0.0,"2021":0.0769,"2022":0.0} {} {"Rubber plantation":{"2001":3.0745,"2002":16.5256,"2003":36.0493,"2004":66.1791,"2005":73.4812,"2006":25.9797,"2007":5.9184,"2008":56.571,"2009":47.7317,"2010":33.3581,"2011":21.9825,"2012":52.9583,"2013":11.9137,"2014":42.2742,"2015":34.2038,"2016":63.4883,"2017":10.6839,"2018":24.4423,"2019":22.1363,"2020":10.4533,"2021":25.826,"2022":26.1332},"Secondary forest":{"2001":240.1012,"2002":352.6874,"2003":51.186,"2004":522.8408,"2005":879.6014,"2006":1310.6826,"2007":981.6686,"2008":756.8744,"2009":359.2934,"2010":232.485,"2011":575.4717,"2012":1110.4372,"2013":787.2514,"2014":772.2979,"2015":966.528,"2016":1571.8466,"2017":149.9382,"2018":89.3794,"2019":136.8781,"2020":121.8915,"2021":68.6318,"2022":99.681},"Agriculture":{"2001":3.151,"2002":9.1452,"2003":5.4563,"2004":53.8715,"2005":30.3561,"2006":22.9009,"2007":6.5323,"2008":10.9893,"2009":159.7649,"2010":38.7323,"2011":100.4403,"2012":104.3592,"2013":15.3698,"2014":35.8124,"2015":19.6734,"2016":38.1942,"2017":19.2886,"2018":10.5282,"2019":11.2197,"2020":7.9922,"2021":12.1419,"2022":9.2218},"Oil palm plantation":{"2001":389.5357,"2002":222.339,"2003":103.9797,"2004":96.4524,"2005":67.8614,"2006":368.7244,"2007":440.2632,"2008":428.9814,"2009":151.7946,"2010":184.5942,"2011":113.5139,"2012":263.0128,"2013":147.9443,"2014":88.3878,"2015":58.1061,"2016":70.7105,"2017":44.5029,"2018":31.2823,"2019":233.0475,"2020":526.052,"2021":395.9972,"2022":105.2237},"Swamp":{"2001":265.2372,"2002":112.4372,"2003":38.2726,"2004":648.495,"2005":548.2747,"2006":855.4703,"2007":1129.3025,"2008":2086.08,"2009":484.4962,"2010":300.1085,"2011":526.378,"2012":478.8799,"2013":482.4034,"2014":742.3953,"2015":446.518,"2016":539.8938,"2017":620.8959,"2018":204.1215,"2019":105.5176,"2020":122.2736,"2021":197.9767,"2022":240.6287},"Settlements":{"2001":0.1537,"2002":0.9992,"2003":0.0,"2004":0.6918,"2005":0.1537,"2006":1.1529,"2007":1.1529,"2008":0.538,"2009":1.0761,"2010":0.8455,"2011":1.1529,"2012":0.8454,"2013":0.0,"2014":0.6918,"2015":0.1537,"2016":0.2306,"2017":0.3843,"2018":0.0,"2019":0.1537,"2020":1.1529,"2021":1.3067,"2022":1.691},"Grassland/shrub":{"2001":59.3337,"2002":89.231,"2003":37.5821,"2004":445.7701,"2005":432.4583,"2006":514.3995,"2007":235.9463,"2008":500.7963,"2009":334.6362,"2010":269.6786,"2011":186.2981,"2012":378.8895,"2013":330.4736,"2014":424.3189,"2015":165.2413,"2016":151.5619,"2017":77.7013,"2018":84.6964,"2019":29.59,"2020":91.3842,"2021":53.8004,"2022":41.7326},"Primary forest":{"2001":41.1934,"2002":30.6653,"2003":13.68,"2004":98.6793,"2005":209.8123,"2006":379.429,"2007":115.8962,"2008":96.2208,"2009":368.2156,"2010":47.8819,"2011":42.0413,"2012":228.795,"2013":26.1305,"2014":255.8481,"2015":270.3755,"2016":823.8133,"2017":81.5399,"2018":47.9595,"2019":64.4845,"2020":22.7495,"2021":71.7856,"2022":5.7642},"Water bodies":{"2001":0.8454,"2002":0.0768,"2003":0.0,"2004":0.1537,"2005":0.0,"2006":0.0,"2007":0.0769,"2008":0.0,"2009":0.0,"2010":0.1537,"2011":0.2306,"2012":0.6916,"2013":0.6917,"2014":0.6148,"2015":0.0,"2016":0.2306,"2017":0.0768,"2018":0.0,"2019":0.0,"2020":0.0,"2021":0.2305,"2022":0.1537},"Mixed tree crops":{"2001":19.1363,"2002":16.9073,"2003":23.9776,"2004":236.7062,"2005":83.3852,"2006":683.7575,"2007":52.029,"2008":78.3891,"2009":95.9108,"2010":65.8624,"2011":136.1808,"2012":219.1809,"2013":39.5784,"2014":106.132,"2015":68.1674,"2016":84.8439,"2017":21.749,"2018":33.1229,"2019":15.6777,"2020":20.7498,"2021":30.1257,"2022":29.8185}} {"Bare land":{"2001":3.8428,"2002":35.2766,"2003":5.3801,"2004":14.4491,"2005":17.6005,"2006":39.8116,"2007":99.1447,"2008":141.5687,"2009":59.9482,"2010":20.7508,"2011":136.3415,"2012":129.3478,"2013":94.2991,"2014":83.0794,"2015":280.0642,"2016":735.1371,"2017":28.9729,"2018":36.1198,"2019":8.3774,"2020":7.1477,"2021":8.0699,"2022":2.3824},"Mining":{"2001":7.301,"2002":2.7666,"2003":5.2258,"2004":11.9889,"2005":15.2172,"2006":9.1456,"2007":7.6082,"2008":34.8914,"2009":16.9072,"2010":8.9918,"2011":12.4502,"2012":29.5112,"2013":1.0759,"2014":16.8304,"2015":2.5362,"2016":1.9982,"2017":1.7676,"2018":0.7685,"2019":0.3074,"2020":0.4611,"2021":1.7676,"2022":3.5353},"Settlement":{"2001":30.2802,"2002":84.4598,"2003":6.7627,"2004":5.226,"2005":1.9982,"2006":15.5239,"2007":5.3029,"2008":146.7178,"2009":9.1456,"2010":5.9944,"2011":8.2999,"2012":20.9038,"2013":6.4557,"2014":10.4519,"2015":14.0641,"2016":20.5962,"2017":9.6834,"2018":5.9946,"2019":5.6871,"2020":7.5318,"2021":7.5314,"2022":7.7622},"Secondary forest":{"2001":14.8329,"2002":34.5077,"2003":10.3753,"2004":63.4026,"2005":86.5381,"2006":58.5628,"2007":81.0051,"2008":222.8806,"2009":92.1507,"2010":46.96,"2011":105.6723,"2012":258.1458,"2013":358.5935,"2014":604.2224,"2015":692.3818,"2016":1208.3837,"2017":259.4575,"2018":110.7479,"2019":151.2503,"2020":92.763,"2021":119.8204,"2022":109.8252},"Agriculture":{"2001":87.6883,"2002":42.4224,"2003":18.9819,"2004":289.878,"2005":266.1325,"2006":746.3828,"2007":376.2672,"2008":177.9118,"2009":282.8805,"2010":120.1185,"2011":271.2771,"2012":638.4726,"2013":155.6217,"2014":248.7641,"2015":220.5643,"2016":382.8723,"2017":105.1299,"2018":55.5624,"2019":44.6495,"2020":56.5613,"2021":42.3446,"2022":44.4967},"Swamp":{"2001":110.6023,"2002":161.6235,"2003":30.1276,"2004":349.3156,"2005":345.4651,"2006":346.3096,"2007":162.935,"2008":218.7309,"2009":146.9478,"2010":95.2199,"2011":132.4202,"2012":382.8885,"2013":159.6257,"2014":317.4138,"2015":296.5912,"2016":418.5593,"2017":115.2812,"2018":74.3201,"2019":85.1564,"2020":78.4694,"2021":128.1948,"2022":88.1532},"Grassland/shrub":{"2001":4.9185,"2002":20.2891,"2003":11.7584,"2004":38.7334,"2005":22.748,"2006":135.7978,"2007":15.2937,"2008":74.7011,"2009":35.6594,"2010":20.4429,"2011":50.7993,"2012":105.518,"2013":11.4509,"2014":46.2648,"2015":55.7949,"2016":62.8662,"2017":264.4497,"2018":34.5065,"2019":9.8372,"2020":5.1492,"2021":6.9934,"2022":55.8711},"Estate crop plantation":{"2001":759.8369,"2002":469.6682,"2003":221.0338,"2004":1396.0776,"2005":1569.454,"2006":2808.3496,"2007":2218.0015,"2008":2990.2754,"2009":1359.0495,"2010":854.7606,"2011":983.3553,"2012":1269.5731,"2013":1053.2509,"2014":1138.9796,"2015":466.1251,"2016":509.328,"2017":239.7129,"2018":205.5913,"2019":312.8252,"2020":676.3081,"2021":539.8723,"2022":247.7148},"Body of water":{"2001":2.4593,"2002":0.0,"2003":0.538,"2004":0.7685,"2005":0.2306,"2006":2.6132,"2007":3.228,"2008":7.7625,"2009":0.2306,"2010":0.4611,"2011":3.0743,"2012":3.689,"2013":1.3834,"2014":2.7668,"2015":0.8454,"2016":5.0725,"2017":2.3057,"2018":1.9215,"2019":0.6148,"2020":0.3074,"2021":3.228,"2022":0.3074}} {} {"Other Utilization Area":{"2001":712.0267,"2002":482.1867,"2003":221.5682,"2004":1414.5116,"2005":1126.5942,"2006":2837.7298,"2007":1853.8397,"2008":3013.7624,"2009":1165.5631,"2010":833.4598,"2011":1098.1437,"2012":1865.7614,"2013":971.0994,"2014":1259.078,"2015":657.4037,"2016":999.5492,"2017":622.2031,"2018":279.4429,"2019":376.9137,"2020":705.8893,"2021":595.4311,"2022":299.8162},"Production Forest":{"2001":113.1434,"2002":80.4763,"2003":7.6858,"2004":26.0567,"2005":48.7316,"2006":183.5469,"2007":84.3149,"2008":147.2649,"2009":84.4729,"2010":56.57,"2011":110.1392,"2012":156.3372,"2013":49.1136,"2014":173.4782,"2015":154.1063,"2016":334.2621,"2017":35.1256,"2018":10.4532,"2019":17.4472,"2020":24.9028,"2021":17.5241,"2022":26.3625},"Converted Production Forest":{"2001":151.8635,"2002":60.1778,"2003":69.0172,"2004":724.5834,"2005":1148.2139,"2006":1123.3127,"2007":1023.561,"2008":844.268,"2009":747.8878,"2010":275.2161,"2011":481.5731,"2012":804.1156,"2013":820.0067,"2014":1024.9196,"2015":1197.936,"2016":1866.668,"2017":356.367,"2018":233.3308,"2019":223.499,"2020":193.5994,"2021":241.5624,"2022":233.5622},"Sanctuary Reserves/Nature Conservation Area":{"2001":42.2692,"2002":228.1732,"2003":11.3743,"2004":3.9196,"2005":1.614,"2006":15.3711,"2007":4.4576,"2008":2.8437,"2009":4.765,"2010":7.9931,"2011":10.7597,"2012":8.1466,"2013":0.1537,"2014":8.5307,"2015":18.6758,"2016":139.2616,"2017":10.7596,"2018":0.3843,"2019":0.2306,"2020":0.0,"2021":0.0769,"2022":0.0}} {"2001":85.0014,"2002":248.2325,"2003":18.829,"2004":97.8293,"2005":96.2941,"2006":176.9875,"2007":138.7928,"2008":129.4126,"2009":109.4342,"2010":65.0144,"2011":100.5959,"2012":428.132,"2013":566.3779,"2014":467.2467,"2015":304.2577,"2016":712.6515,"2017":145.3232,"2018":56.2574,"2019":82.8502,"2020":54.0272,"2021":24.2097,"2022":14.7553} {} {} {} {} {} {} {} 76338.8266 34513.678 6014.0986 23301.5138 0.0 34530.8164 0.0 125583.7284 6614.0107 31984.9079 {} {} {"Other Utilization Area":81822.5991,"Production Forest":4848.8827,"Converted Production Forest":28600.5448,"Sanctuary Reserves/Nature Conservation Area":6614.0107} {"Rubber plantation":3542.3364,"Secondary forest":24896.0268,"Agriculture":2763.3729,"Oil palm plantation":24398.5485,"Swamp":29043.6031,"Settlements":851.2415,"Grassland/shrub":22752.6953,"Primary forest":8261.1709,"Water bodies":3133.0348,"Mixed tree crops":5941.6982} {"Bare land":2902.1353,"Mining":1392.3433,"Settlement":5798.0268,"Secondary forest":21966.5842,"Agriculture":9963.5593,"Swamp":7625.7847,"Grassland/shrub":2875.049,"Estate crop plantation":69353.0242,"Body of water":3707.2217} 13342.6717 false false false false true true false {"2002":24579.8084,"2003":24451.4643,"2004":24189.9373,"2005":24132.0676,"2006":23677.7243,"2007":23217.4612,"2008":22247.8139,"2009":21697.0839,"2010":21235.5008,"2011":20791.3746,"2012":20560.122,"2013":20057.1275,"2014":19054.2097,"2015":18742.9578,"2016":18210.5965,"2017":17209.7962,"2018":15452.9753,"2019":15189.6774,"2020":15071.8608,"2021":14916.0005} {"2002":31984.9079,"2003":31984.9079,"2004":31984.9079,"2005":31984.9079,"2006":31984.9079,"2007":31984.9079,"2008":31984.9079,"2009":31984.9079,"2010":31984.9079,"2011":31984.9079,"2012":31984.9079,"2013":31984.9079,"2014":31984.9079,"2015":31984.9079,"2016":31984.9079,"2017":31984.9079,"2018":31984.9079,"2019":31984.9079,"2020":31984.9079,"2021":31984.9079} {"2002":6614.0107,"2003":6614.0107,"2004":6614.0107,"2005":6614.0107,"2006":6614.0107,"2007":6614.0107,"2008":6614.0107,"2009":6614.0107,"2010":6614.0107,"2011":6614.0107,"2012":6614.0107,"2013":6614.0107,"2014":6614.0107,"2015":6614.0107,"2016":6614.0107,"2017":6614.0107,"2018":6614.0107,"2019":6614.0107,"2020":6614.0107,"2021":6614.0107} {"2002":389.8711,"2003":319.3967,"2004":512.213,"2005":914.6063,"2006":1429.9104,"2007":1520.3773,"2008":1012.313,"2009":905.7094,"2010":675.3788,"2011":734.247,"2012":1505.9123,"2013":1314.1698,"2014":843.6132,"2015":1533.1615,"2016":2757.6213,"2017":2020.1188,"2018":381.1145,"2019":273.6768,"2020":272.6016,"2021":245.5511} {"2002":13363.7925,"2003":13309.3807,"2004":13472.9171,"2005":13694.9388,"2006":14063.9863,"2007":14195.4089,"2008":13789.4768,"2009":13613.3257,"2010":13540.0855,"2011":13478.2203,"2012":13889.142,"2013":13891.8339,"2014":13604.4896,"2015":14052.9405,"2016":14710.2098,"2017":14207.349,"2018":13390.077,"2019":13385.3927,"2020":13385.8532,"2021":13355.5725} {"2002":222.102,"2003":200.4294,"2004":9.6067,"2005":0.4611,"2006":3.7659,"2007":3.8427,"2008":0.1537,"2009":1.2297,"2010":2.8437,"2011":2.9205,"2012":3.2279,"2013":1.9982,"2014":0.3843,"2015":13.7571,"2016":74.1656,"2017":63.0214,"2018":2.2287,"2019":0.0,"2020":0.0,"2021":0.0} {} +list_id location_id status_code location_error tree_cover_loss_total_yearly tree_cover_loss_primary_forest_yearly tree_cover_loss_peat_yearly tree_cover_loss_intact_forest_yearly tree_cover_loss_protected_areas_yearly tree_cover_loss_arg_otbn_yearly tree_cover_loss_sea_landcover_yearly tree_cover_loss_idn_landcover_yearly tree_cover_loss_soy_yearly tree_cover_loss_idn_legal_yearly tree_cover_loss_idn_forest_moratorium_yearly tree_cover_loss_prodes_yearly tree_cover_loss_prodes_wdpa_yearly tree_cover_loss_prodes_primary_forest_yearly tree_cover_loss_brazil_biomes_yearly tree_cover_extent_total tree_cover_extent_primary_forest tree_cover_extent_protected_areas tree_cover_extent_peat tree_cover_extent_intact_forest natural_habitat_primary natural_habitat_intact_forest total_area protected_areas_area peat_area arg_otbn_area brazil_biomes idn_legal_area sea_landcover_area idn_landcover_area idn_forest_moratorium_area south_america_presence legal_amazon_presence brazil_biomes_presence cerrado_biome_presence southeast_asia_presence indonesia_presence argentina_presence commodity_value_forest_extent commodity_value_peat commodity_value_protected_areas commodity_threat_deforestation commodity_threat_peat commodity_threat_protected_areas commodity_threat_fires +1 31 2 {"2001":1021.7622,"2002":851.014,"2003":310.1835,"2004":2169.8398,"2005":2325.3843,"2006":4162.4968,"2007":2968.7863,"2008":4015.4403,"2009":2002.9194,"2010":1173.7001,"2011":1703.6902,"2012":2838.0498,"2013":1841.7568,"2014":2468.7732,"2015":2028.9672,"2016":3344.8135,"2017":1026.7609,"2018":525.5327,"2019":618.7052,"2020":924.699,"2021":857.8225,"2022":560.0482} {"2001":154.8617,"2002":306.7253,"2003":92.3781,"2004":717.7405,"2005":1202.6952,"2006":1831.5766,"2007":1668.2764,"2008":1753.2317,"2009":797.282,"2010":454.5023,"2011":872.3613,"2012":1251.8543,"2013":1083.6799,"2014":1290.2177,"2015":1360.2574,"2016":2313.5001,"2017":286.2809,"2018":159.8557,"2019":162.3929,"2020":134.2652,"2021":167.4697,"2022":133.6506} {"2001":557.4251,"2002":236.2539,"2003":71.8566,"2004":741.25,"2005":957.52,"2006":1229.3335,"2007":1037.5018,"2008":891.235,"2009":486.4665,"2010":363.5759,"2011":411.9212,"2012":1078.9246,"2013":862.5621,"2014":974.783,"2015":942.4571,"2016":1472.8429,"2017":211.3403,"2018":144.7173,"2019":148.7917,"2020":142.3323,"2021":122.7372,"2022":94.914} {} {"2001":42.2692,"2002":228.1732,"2003":11.3743,"2004":3.9196,"2005":1.614,"2006":15.3711,"2007":4.4576,"2008":2.8437,"2009":4.765,"2010":7.9931,"2011":10.7597,"2012":8.1466,"2013":0.1537,"2014":8.5307,"2015":18.6758,"2016":139.2616,"2017":10.7596,"2018":0.3843,"2019":0.2306,"2020":0.0,"2021":0.0769,"2022":0.0} {} {"Rubber plantation":{"2001":3.0745,"2002":16.5256,"2003":36.0493,"2004":66.1791,"2005":73.4812,"2006":25.9797,"2007":5.9184,"2008":56.571,"2009":47.7317,"2010":33.3581,"2011":21.9825,"2012":52.9583,"2013":11.9137,"2014":42.2742,"2015":34.2038,"2016":63.4883,"2017":10.6839,"2018":24.4423,"2019":22.1363,"2020":10.4533,"2021":25.826,"2022":26.1332},"Secondary forest":{"2001":240.1012,"2002":352.6874,"2003":51.186,"2004":522.8408,"2005":879.6014,"2006":1310.6826,"2007":981.6686,"2008":756.8744,"2009":359.2934,"2010":232.485,"2011":575.4717,"2012":1110.4372,"2013":787.2514,"2014":772.2979,"2015":966.528,"2016":1571.8466,"2017":149.9382,"2018":89.3794,"2019":136.8781,"2020":121.8915,"2021":68.6318,"2022":99.681},"Agriculture":{"2001":3.151,"2002":9.1452,"2003":5.4563,"2004":53.8715,"2005":30.3561,"2006":22.9009,"2007":6.5323,"2008":10.9893,"2009":159.7649,"2010":38.7323,"2011":100.4403,"2012":104.3592,"2013":15.3698,"2014":35.8124,"2015":19.6734,"2016":38.1942,"2017":19.2886,"2018":10.5282,"2019":11.2197,"2020":7.9922,"2021":12.1419,"2022":9.2218},"Oil palm plantation":{"2001":389.5357,"2002":222.339,"2003":103.9797,"2004":96.4524,"2005":67.8614,"2006":368.7244,"2007":440.2632,"2008":428.9814,"2009":151.7946,"2010":184.5942,"2011":113.5139,"2012":263.0128,"2013":147.9443,"2014":88.3878,"2015":58.1061,"2016":70.7105,"2017":44.5029,"2018":31.2823,"2019":233.0475,"2020":526.052,"2021":395.9972,"2022":105.2237},"Swamp":{"2001":265.2372,"2002":112.4372,"2003":38.2726,"2004":648.495,"2005":548.2747,"2006":855.4703,"2007":1129.3025,"2008":2086.08,"2009":484.4962,"2010":300.1085,"2011":526.378,"2012":478.8799,"2013":482.4034,"2014":742.3953,"2015":446.518,"2016":539.8938,"2017":620.8959,"2018":204.1215,"2019":105.5176,"2020":122.2736,"2021":197.9767,"2022":240.6287},"Settlements":{"2001":0.1537,"2002":0.9992,"2003":0.0,"2004":0.6918,"2005":0.1537,"2006":1.1529,"2007":1.1529,"2008":0.538,"2009":1.0761,"2010":0.8455,"2011":1.1529,"2012":0.8454,"2013":0.0,"2014":0.6918,"2015":0.1537,"2016":0.2306,"2017":0.3843,"2018":0.0,"2019":0.1537,"2020":1.1529,"2021":1.3067,"2022":1.691},"Grassland/shrub":{"2001":59.3337,"2002":89.231,"2003":37.5821,"2004":445.7701,"2005":432.4583,"2006":514.3995,"2007":235.9463,"2008":500.7963,"2009":334.6362,"2010":269.6786,"2011":186.2981,"2012":378.8895,"2013":330.4736,"2014":424.3189,"2015":165.2413,"2016":151.5619,"2017":77.7013,"2018":84.6964,"2019":29.59,"2020":91.3842,"2021":53.8004,"2022":41.7326},"Primary forest":{"2001":41.1934,"2002":30.6653,"2003":13.68,"2004":98.6793,"2005":209.8123,"2006":379.429,"2007":115.8962,"2008":96.2208,"2009":368.2156,"2010":47.8819,"2011":42.0413,"2012":228.795,"2013":26.1305,"2014":255.8481,"2015":270.3755,"2016":823.8133,"2017":81.5399,"2018":47.9595,"2019":64.4845,"2020":22.7495,"2021":71.7856,"2022":5.7642},"Water bodies":{"2001":0.8454,"2002":0.0768,"2003":0.0,"2004":0.1537,"2005":0.0,"2006":0.0,"2007":0.0769,"2008":0.0,"2009":0.0,"2010":0.1537,"2011":0.2306,"2012":0.6916,"2013":0.6917,"2014":0.6148,"2015":0.0,"2016":0.2306,"2017":0.0768,"2018":0.0,"2019":0.0,"2020":0.0,"2021":0.2305,"2022":0.1537},"Mixed tree crops":{"2001":19.1363,"2002":16.9073,"2003":23.9776,"2004":236.7062,"2005":83.3852,"2006":683.7575,"2007":52.029,"2008":78.3891,"2009":95.9108,"2010":65.8624,"2011":136.1808,"2012":219.1809,"2013":39.5784,"2014":106.132,"2015":68.1674,"2016":84.8439,"2017":21.749,"2018":33.1229,"2019":15.6777,"2020":20.7498,"2021":30.1257,"2022":29.8185}} {"Bare land":{"2001":3.8428,"2002":35.2766,"2003":5.3801,"2004":14.4491,"2005":17.6005,"2006":39.8116,"2007":99.1447,"2008":141.5687,"2009":59.9482,"2010":20.7508,"2011":136.3415,"2012":129.3478,"2013":94.2991,"2014":83.0794,"2015":280.0642,"2016":735.1371,"2017":28.9729,"2018":36.1198,"2019":8.3774,"2020":7.1477,"2021":8.0699,"2022":2.3824},"Mining":{"2001":7.301,"2002":2.7666,"2003":5.2258,"2004":11.9889,"2005":15.2172,"2006":9.1456,"2007":7.6082,"2008":34.8914,"2009":16.9072,"2010":8.9918,"2011":12.4502,"2012":29.5112,"2013":1.0759,"2014":16.8304,"2015":2.5362,"2016":1.9982,"2017":1.7676,"2018":0.7685,"2019":0.3074,"2020":0.4611,"2021":1.7676,"2022":3.5353},"Settlement":{"2001":30.2802,"2002":84.4598,"2003":6.7627,"2004":5.226,"2005":1.9982,"2006":15.5239,"2007":5.3029,"2008":146.7178,"2009":9.1456,"2010":5.9944,"2011":8.2999,"2012":20.9038,"2013":6.4557,"2014":10.4519,"2015":14.0641,"2016":20.5962,"2017":9.6834,"2018":5.9946,"2019":5.6871,"2020":7.5318,"2021":7.5314,"2022":7.7622},"Secondary forest":{"2001":14.8329,"2002":34.5077,"2003":10.3753,"2004":63.4026,"2005":86.5381,"2006":58.5628,"2007":81.0051,"2008":222.8806,"2009":92.1507,"2010":46.96,"2011":105.6723,"2012":258.1458,"2013":358.5935,"2014":604.2224,"2015":692.3818,"2016":1208.3837,"2017":259.4575,"2018":110.7479,"2019":151.2503,"2020":92.763,"2021":119.8204,"2022":109.8252},"Agriculture":{"2001":87.6883,"2002":42.4224,"2003":18.9819,"2004":289.878,"2005":266.1325,"2006":746.3828,"2007":376.2672,"2008":177.9118,"2009":282.8805,"2010":120.1185,"2011":271.2771,"2012":638.4726,"2013":155.6217,"2014":248.7641,"2015":220.5643,"2016":382.8723,"2017":105.1299,"2018":55.5624,"2019":44.6495,"2020":56.5613,"2021":42.3446,"2022":44.4967},"Swamp":{"2001":110.6023,"2002":161.6235,"2003":30.1276,"2004":349.3156,"2005":345.4651,"2006":346.3096,"2007":162.935,"2008":218.7309,"2009":146.9478,"2010":95.2199,"2011":132.4202,"2012":382.8885,"2013":159.6257,"2014":317.4138,"2015":296.5912,"2016":418.5593,"2017":115.2812,"2018":74.3201,"2019":85.1564,"2020":78.4694,"2021":128.1948,"2022":88.1532},"Grassland/shrub":{"2001":4.9185,"2002":20.2891,"2003":11.7584,"2004":38.7334,"2005":22.748,"2006":135.7978,"2007":15.2937,"2008":74.7011,"2009":35.6594,"2010":20.4429,"2011":50.7993,"2012":105.518,"2013":11.4509,"2014":46.2648,"2015":55.7949,"2016":62.8662,"2017":264.4497,"2018":34.5065,"2019":9.8372,"2020":5.1492,"2021":6.9934,"2022":55.8711},"Estate crop plantation":{"2001":759.8369,"2002":469.6682,"2003":221.0338,"2004":1396.0776,"2005":1569.454,"2006":2808.3496,"2007":2218.0015,"2008":2990.2754,"2009":1359.0495,"2010":854.7606,"2011":983.3553,"2012":1269.5731,"2013":1053.2509,"2014":1138.9796,"2015":466.1251,"2016":509.328,"2017":239.7129,"2018":205.5913,"2019":312.8252,"2020":676.3081,"2021":539.8723,"2022":247.7148},"Body of water":{"2001":2.4593,"2002":0.0,"2003":0.538,"2004":0.7685,"2005":0.2306,"2006":2.6132,"2007":3.228,"2008":7.7625,"2009":0.2306,"2010":0.4611,"2011":3.0743,"2012":3.689,"2013":1.3834,"2014":2.7668,"2015":0.8454,"2016":5.0725,"2017":2.3057,"2018":1.9215,"2019":0.6148,"2020":0.3074,"2021":3.228,"2022":0.3074}} {} {"Other Utilization Area":{"2001":712.0267,"2002":482.1867,"2003":221.5682,"2004":1414.5116,"2005":1126.5942,"2006":2837.7298,"2007":1853.8397,"2008":3013.7624,"2009":1165.5631,"2010":833.4598,"2011":1098.1437,"2012":1865.7614,"2013":971.0994,"2014":1259.078,"2015":657.4037,"2016":999.5492,"2017":622.2031,"2018":279.4429,"2019":376.9137,"2020":705.8893,"2021":595.4311,"2022":299.8162},"Production Forest":{"2001":113.1434,"2002":80.4763,"2003":7.6858,"2004":26.0567,"2005":48.7316,"2006":183.5469,"2007":84.3149,"2008":147.2649,"2009":84.4729,"2010":56.57,"2011":110.1392,"2012":156.3372,"2013":49.1136,"2014":173.4782,"2015":154.1063,"2016":334.2621,"2017":35.1256,"2018":10.4532,"2019":17.4472,"2020":24.9028,"2021":17.5241,"2022":26.3625},"Converted Production Forest":{"2001":151.8635,"2002":60.1778,"2003":69.0172,"2004":724.5834,"2005":1148.2139,"2006":1123.3127,"2007":1023.561,"2008":844.268,"2009":747.8878,"2010":275.2161,"2011":481.5731,"2012":804.1156,"2013":820.0067,"2014":1024.9196,"2015":1197.936,"2016":1866.668,"2017":356.367,"2018":233.3308,"2019":223.499,"2020":193.5994,"2021":241.5624,"2022":233.5622},"Sanctuary Reserves/Nature Conservation Area":{"2001":42.2692,"2002":228.1732,"2003":11.3743,"2004":3.9196,"2005":1.614,"2006":15.3711,"2007":4.4576,"2008":2.8437,"2009":4.765,"2010":7.9931,"2011":10.7597,"2012":8.1466,"2013":0.1537,"2014":8.5307,"2015":18.6758,"2016":139.2616,"2017":10.7596,"2018":0.3843,"2019":0.2306,"2020":0.0,"2021":0.0769,"2022":0.0}} {"2001":85.0014,"2002":248.2325,"2003":18.829,"2004":97.8293,"2005":96.2941,"2006":176.9875,"2007":138.7928,"2008":129.4126,"2009":109.4342,"2010":65.0144,"2011":100.5959,"2012":428.132,"2013":566.3779,"2014":467.2467,"2015":304.2577,"2016":712.6515,"2017":145.3232,"2018":56.2574,"2019":82.8502,"2020":54.0272,"2021":24.2097,"2022":14.7553} {} {} {} {} 76338.8266 34513.678 6014.0986 23301.5138 0.0 34530.8164 0.0 125583.7284 6614.0107 31984.9079 {} {} {"Other Utilization Area":81822.5991,"Production Forest":4848.8827,"Converted Production Forest":28600.5448,"Sanctuary Reserves/Nature Conservation Area":6614.0107} {"Rubber plantation":3542.3364,"Secondary forest":24896.0268,"Agriculture":2763.3729,"Oil palm plantation":24398.5485,"Swamp":29043.6031,"Settlements":851.2415,"Grassland/shrub":22752.6953,"Primary forest":8261.1709,"Water bodies":3133.0348,"Mixed tree crops":5941.6982} {"Bare land":2902.1353,"Mining":1392.3433,"Settlement":5798.0268,"Secondary forest":21966.5842,"Agriculture":9963.5593,"Swamp":7625.7847,"Grassland/shrub":2875.049,"Estate crop plantation":69353.0242,"Body of water":3707.2217} 13342.6717 false false false false true true false {"2002":24579.8084,"2003":24451.4643,"2004":24189.9373,"2005":24132.0676,"2006":23677.7243,"2007":23217.4612,"2008":22247.8139,"2009":21697.0839,"2010":21235.5008,"2011":20791.3746,"2012":20560.122,"2013":20057.1275,"2014":19054.2097,"2015":18742.9578,"2016":18210.5965,"2017":17209.7962,"2018":15452.9753,"2019":15189.6774,"2020":15071.8608,"2021":14916.0005} {"2002":31984.9079,"2003":31984.9079,"2004":31984.9079,"2005":31984.9079,"2006":31984.9079,"2007":31984.9079,"2008":31984.9079,"2009":31984.9079,"2010":31984.9079,"2011":31984.9079,"2012":31984.9079,"2013":31984.9079,"2014":31984.9079,"2015":31984.9079,"2016":31984.9079,"2017":31984.9079,"2018":31984.9079,"2019":31984.9079,"2020":31984.9079,"2021":31984.9079} {"2002":6614.0107,"2003":6614.0107,"2004":6614.0107,"2005":6614.0107,"2006":6614.0107,"2007":6614.0107,"2008":6614.0107,"2009":6614.0107,"2010":6614.0107,"2011":6614.0107,"2012":6614.0107,"2013":6614.0107,"2014":6614.0107,"2015":6614.0107,"2016":6614.0107,"2017":6614.0107,"2018":6614.0107,"2019":6614.0107,"2020":6614.0107,"2021":6614.0107} {"2002":389.8711,"2003":319.3967,"2004":512.213,"2005":914.6063,"2006":1429.9104,"2007":1520.3773,"2008":1012.313,"2009":905.7094,"2010":675.3788,"2011":734.247,"2012":1505.9123,"2013":1314.1698,"2014":843.6132,"2015":1533.1615,"2016":2757.6213,"2017":2020.1188,"2018":381.1145,"2019":273.6768,"2020":272.6016,"2021":245.5511} {"2002":13363.7925,"2003":13309.3807,"2004":13472.9171,"2005":13694.9388,"2006":14063.9863,"2007":14195.4089,"2008":13789.4768,"2009":13613.3257,"2010":13540.0855,"2011":13478.2203,"2012":13889.142,"2013":13891.8339,"2014":13604.4896,"2015":14052.9405,"2016":14710.2098,"2017":14207.349,"2018":13390.077,"2019":13385.3927,"2020":13385.8532,"2021":13355.5725} {"2002":222.102,"2003":200.4294,"2004":9.6067,"2005":0.4611,"2006":3.7659,"2007":3.8427,"2008":0.1537,"2009":1.2297,"2010":2.8437,"2011":2.9205,"2012":3.2279,"2013":1.9982,"2014":0.3843,"2015":13.7571,"2016":74.1656,"2017":63.0214,"2018":2.2287,"2019":0.0,"2020":0.0,"2021":0.0} {} diff --git a/src/test/resources/sample-fire-alert.tsv b/src/test/resources/sample-fire-alert.tsv new file mode 100644 index 00000000..b768ce84 --- /dev/null +++ b/src/test/resources/sample-fire-alert.tsv @@ -0,0 +1,2 @@ +latitude longitude acq_date acq_time confidence brightness bright_t31 frp +40.115 -83.497 2021-03-08 1845 47 305.5 291.4 4.1 diff --git a/src/test/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticAnalysisSpec.scala b/src/test/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticAnalysisSpec.scala index dab5eab5..016450dc 100644 --- a/src/test/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticAnalysisSpec.scala +++ b/src/test/scala/org/globalforestwatch/summarystats/forest_change_diagnostic/ForestChangeDiagnosticAnalysisSpec.scala @@ -36,8 +36,7 @@ class ForestChangeDiagnosticAnalysisSpec extends TestEnvironment with DataFrameC /** Function to update expected results when this test becomes invalid */ def saveExpectedFcdResult(fcd: DataFrame): Unit = { - fcd - .repartition(1) + fcd.repartition(1) .write .mode(SaveMode.Overwrite) .options(ForestChangeDiagnosticExport.csvOptions)