Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept input layer versions/URIs as a config #126

Open
jterry64 opened this issue Oct 25, 2021 · 0 comments
Open

Accept input layer versions/URIs as a config #126

jterry64 opened this issue Oct 25, 2021 · 0 comments

Comments

@jterry64
Copy link
Member

@echeipesh here's an example of what the config look like for our gfw-raster-analysis-lambda service:

[
    {
        "source_uri": "s3://gfw-data-lake/umd_tree_cover_loss/v1.8/raster/epsg-4326/10/40000/year/geotiff/{tile_id}.tif",
        "tile_scheme": "nw",
        "grid": "10/40000",
        "name": "umd_tree_cover_loss__year",
        "raster_table": {
            "rows": [
                {"value": 1, "meaning": 2001},
                {"value": 2, "meaning": 2002},
                {"value": 3, "meaning": 2003},
                {"value": 4, "meaning": 2004},
                {"value": 5, "meaning": 2005},
                {"value": 6, "meaning": 2006},
                {"value": 7, "meaning": 2007},
                {"value": 8, "meaning": 2008},
                {"value": 9, "meaning": 2009},
                {"value": 10, "meaning": 2010},
                {"value": 11, "meaning": 2011},
                {"value": 12, "meaning": 2012},
                {"value": 13, "meaning": 2013},
                {"value": 14, "meaning": 2014},
                {"value": 15, "meaning": 2015},
                {"value": 16, "meaning": 2016},
                {"value": 17, "meaning": 2017},
                {"value": 18, "meaning": 2018},
                {"value": 19, "meaning": 2019},
                {"value": 20, "meaning": 2020}
            ]
        },
    },
      {
        "source_uri": "s3://gfw-data-lake/umd_tree_cover_gain/v1.6/raster/epsg-4326/10/40000/is/geotiff/{tile_id}.tif",
        "tile_scheme": "nw",
        "grid": "10/40000",
        "name": "is__umd_tree_cover_gain",
        "raster_table": {
            "default_meaning": "false",
            "rows": [
              {"value": 1, "meaning": "true"}
            ]
        }
    },
    {
        "source_uri": "s3://gfw-data-lake/tsc_tree_cover_loss_drivers/v2020/raster/epsg-4326/10/40000/type/geotiff/{tile_id}.tif",
        "tile_scheme": "nw",
        "grid": "10/40000",
        "name": "tsc_tree_cover_loss_drivers__type",
        "raster_table": {
            "default_meaning": "Unknown",
            "rows": [
                {"value": 1, "meaning": "Commodity driven deforestation"},
                {"value": 2, "meaning": "Shifting agriculture"},
                {"value": 3, "meaning": "Forestry"},
                {"value": 4, "meaning": "Wildfire"},
                {"value": 5, "meaning": "Urbanization"}
            ]
        }
    },
    {
        "source_uri": "s3://gfw-data-lake/esa_land_cover_2015/v20160111/raster/epsg-4326/10/40000/class/geotiff/{tile_id}.tif",
        "tile_scheme": "nw",
        "grid": "10/40000",
        "name": "esa_land_cover_2015__class",
        "raster_table": {
            "default_meaning": "Unknown",
            "rows": [
                {"value": 10, "meaning": "Agriculture"},
                {"value": 11, "meaning": "Agriculture"},
                {"value": 12, "meaning": "Agriculture"},
                {"value": 20, "meaning": "Agriculture"},
                {"value": 30, "meaning": "Agriculture"},
                {"value": 40, "meaning": "Agriculture"},
                {"value": 50, "meaning": "Forest"},
                {"value": 60, "meaning": "Forest"},
                {"value": 61, "meaning": "Forest"},
                {"value": 62, "meaning": "Forest"},
                {"value": 70, "meaning": "Forest"},
                {"value": 72, "meaning": "Forest"},
                {"value": 80, "meaning": "Forest"},
                {"value": 81, "meaning": "Forest"},
                {"value": 82, "meaning": "Forest"},
                {"value": 90, "meaning": "Forest"},
                {"value": 100, "meaning": "Forest"},
                {"value": 160, "meaning": "Forest"},
                {"value": 170, "meaning": "Forest"},
                {"value": 110, "meaning": "Grassland"},
                {"value": 130, "meaning": "Grassland"},
                {"value": 180, "meaning": "Wetland"},
                {"value": 190, "meaning": "Settlement"},
                {"value": 120, "meaning": "Shrubland"},
                {"value": 121, "meaning": "Shrubland"},
                {"value": 122, "meaning": "Shrubland"},
                {"value": 140, "meaning": "Sparse vegetation"},
                {"value": 150, "meaning": "Sparse vegetation"},
                {"value": 151, "meaning": "Sparse vegetation"},
                {"value": 152, "meaning": "Sparse vegetation"},
                {"value": 153, "meaning": "Sparse vegetation"},
                {"value": 200, "meaning": "Bare"},
                {"value": 201, "meaning": "Bare"},
                {"value": 202, "meaning": "Bare"},
                {"value": 210, "meaning": "Water"},
                {"value": 220, "meaning": "Permanent snow and ice"}
            ]
        }
    }
]

Some fields that might need some explanation:

  • tile_scheme: tells you how the tile_ids will be named, and this is really only to support the GLAD tiles which haven't been migrated to our data lake yet. I don't think you'll need this since it'll be tied to a case class.
  • default_meaning: usually we will mask out NoData values, but if the layer has a default meaning, all NoData values will be translated to that meaning. This is currently also in this repo, where the lookup function maps _ to some value.

Just to recap:
This config will be used as input to determine where layers should access their data, and allow clients to control the versions. The configs will be saved in S3 and passed as an arguments.

Open question: Pro consumes GLAD, and in the future will consume other alert systems. These create new versions everyday. Either Pro has to create a new config everyday (I was going to do this for flagship), or we do allow you to say in the config to just lookup the latest for specific layers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant