Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Couldn't open hdf files for burned area using osgeo/gdal:ubuntu-small-3.0.4. I think that it just didn't have the hdf driver. I'm trying to build the docker with osgeo/gdal:ubuntu-full-3.0.4 instead to see if that can read hdf files. * Changing to full gdal build for Docker allowed gdal to read hdf. I don't know if that affects anything else in the model. I will see when I do later testing. * Changing to full gdal build for Docker allowed gdal to read hdf. I don't know if that affects anything else in the model. I will see when I do later testing. * Going ahead with burned area analysis now that the gdal-hdf issue is sorted out. * Going ahead with burned area analysis now that the gdal-hdf issue is sorted out. * Going ahead with burned area analysis now that the gdal-hdf issue is sorted out. * Going ahead with burned area analysis now that the gdal-hdf issue is sorted out. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * Working on next step of updating burned area through 2019 for emissions model. * I think I have the clip_year_tiles step working. Running on all tiles now, from 2000 to 2019. * I think I have the clip_year_tiles step working. Running on all tiles now, from 2000 to 2019. * I think I have the clip_year_tiles step working. Running on all tiles now, from 2000 to 2019. * I think I have the clip_year_tiles step working. Running on all tiles now, from 2000 to 2019. * I think I have the clip_year_tiles step working. Running on all tiles now, from 2000 to 2019. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * On the final stage of the burned area analysis update. * Changed burned area final step to skip tiles without loss. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Ran test tile through entire model on my computer (standard model only). Now going to try three test tiles on spot machine. * Still working on getting the carbon pool generation to work again in the full model run context. Testing three tiles on a spot machine. * Still working on getting the carbon pool generation to work again in the full model run context. Testing three tiles on a spot machine. * Still working on getting the carbon pool generation to work again in the full model run context. Testing three tiles on a spot machine. * Still working on getting the carbon pool generation to work again in the full model run context. Issues with soil carbon. Testing three tiles on a spot machine. * Still working on getting the carbon pool generation to work again in the full model run context. Issues with soil carbon. Testing three tiles on a spot machine. * Still working on test run of three tiles on spot machine. Carbon pool generation works now. On to emissions. * Still working on test run of three tiles on spot machine. Carbon pool generation works now. On to emissions. * Still working on test run of three tiles on spot machine. Making sure emissions runs. * Still working on test run of three tiles on spot machine. Making sure emissions runs. * Still working on test run of three tiles on spot machine. Making sure emissions runs. * Still working on test run of three tiles on spot machine. Making sure emissions runs. * Still working on test run of three tiles on spot machine. Making sure emissions runs. * Issue was with where the c++ was being compiled. It wasn't in the project folder, which was messing me up. Think emissions are working now. * Issue was with where the c++ was being compiled. It wasn't in the project folder, which was messing me up. Think emissions are working now. * Issue was with where the c++ was being compiled. It wasn't in the project folder, which was messing me up. Think emissions are working now. * Emissions, net flux, and aggregate worked. Fixing path issue for per pixel output step. * Emissions, net flux, and aggregate worked. Fixing path issue for per pixel output step. * Emissions, net flux, and aggregate worked. Fixing path issue for per pixel output step. * Flux model seems to be ready for full model run for model v.1.2.0 on a spot machine. Ran all stages in Docker locally on one test tile and on Docker on spot machine on three test tiles. * Flux model seems to be ready for full model run for model v.1.2.0 on a spot machine. Ran all stages in Docker locally on one test tile and on Docker on spot machine on three test tiles. Changed output directory dates to reflect intended date run. * Flux model seems to be ready for full model run for model v.1.2.0 on a spot machine. Ran all stages in Docker locally on one test tile and on Docker on spot machine on three test tiles. Changed output directory dates to reflect intended date run. * Realized that metadata tags weren't writing to the aggregated rasters, so I fixed that. Switched from adding tags with rasterio to adding them with gdal_edit. Used the same method for adding metadata tags to emissions outputs. * Changed the number of allocated processors for each model step. Just guessing how much to use for r5d.24xlarge. * Model extent tiles without any data were being uploaded to s3, so adding function to delete the empty ones before uploading. * Fixing issues with full model run as I go and fine tuning number of processors. Had to make exceptions for loss tiles not existing in the age category script. * Fixing issues with full model run as I go and fine tuning number of processors. Had to make exceptions for loss tiles not existing in the age category script. * Fixing issues with full model run as I go and fine tuning number of processors. Had to make exceptions for loss tiles not existing in the age category script. * Fixing issues with full model run as I go and fine tuning number of processors. Had to make exceptions for loss tiles not existing in the age category script. * Fixing issues with full model run as I go and fine tuning number of processors. Had to make exceptions for loss tiles not existing in the age category script. * Fixing issues with full model run as I go and fine tuning number of processors. Had to make exceptions for loss tiles not existing in the age category script. * Model stopped on IPCC default rate creation because one continent-ecozone tile didn't exist. Revised step to skip tiles that don't have ecozone-continent and age category tiles. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Model stopped on merging all the gain year count rasters because some tiles didn't have all four. Fixing that and restarting with gain year count. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Somehow skipped the annual gain rate all forest type step before. Going back to do that now. * Adjusting number of processors for gross removals step. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * Fiddling with carbon pool generation step now. * I got each stage running on all tiles through deadwood/litter emission carbon pool generation. I haven't gotten to testing it beyond that but I hope it works. Decreased allocated processors for remaining steps to reduce likelihood of running out of memory. Trying to run model on all tiles for model v1.2.0 standard run now. * I got each stage running on all tiles through deadwood/litter emission carbon pool generation. I haven't gotten to testing it beyond that but I hope it works. Decreased allocated processors for remaining steps to reduce likelihood of running out of memory. Trying to run model on all tiles for model v1.2.0 standard run now. * I got each stage running on all tiles through deadwood/litter emission carbon pool generation. I haven't gotten to testing it beyond that but I hope it works. Decreased allocated processors for remaining steps to reduce likelihood of running out of memory. Trying to run model on all tiles for model v1.2.0 standard run now. * Full model run ran out of storage during gain year count step. Adding tile deletion before that step to make room. * Full model run ran out of storage during gain year count step. Adding tile deletion before that step to make room. * Full model run ran out of storage during gain year count step. Adding tile deletion before that step to make room. * Running tile statistics on standard v1.2.0 for everything until carbon pools. * Running tile statistics on standard v1.2.0 for everything until carbon pools. * Deadwood/litter c pool creation issue. Restarting model there. * Allocated too many processors for total carbon step. Restarting from there. * Done with carbon pool creation. Issue with cleaning unneeded tiles before emissions step. * Done with carbon pool creation. Issue with cleaning unneeded tiles before emissions step. * Having trouble creating blank tiles for missing emissions inputs. * Having trouble creating blank tiles for missing emissions inputs. * Fixed the tool for making blank tiles for missing gross emissions inputs. Running on all tiles now. * Fixed the tool for making blank tiles for missing gross emissions inputs. Running on all tiles now. * Fixed the tool for making blank tiles for missing gross emissions inputs. Running on all tiles now. * Getting tile statistics for carbon pools in emissions year, gross emissions frombiomass and soil, and net flux. * Need to redo emissions steps onwards. Metadata writing was old version and overwrote results in emissions tiles. * Need to redo emissions steps onwards. Metadata writing was old version and overwrote results in emissions tiles. * Doing more tile statistics. * Need to redo emissions steps onwards. Metadata writing was old version and overwrote results in emissions tiles. * Doing more tile statistics. * Doing more tile statistics. * Needed to fix net flux script. * Needed to fix net flux script. * Needed to fix net flux script. * Needed to fix net flux script. * Needed to fix net flux script. * Needed to fix net flux script. * Running carbon pool 2000 generation and soil only emissions now. * Running carbon pool 2000 generation and soil only emissions now. * Had to fix deadwood and litter in emissions year carbon pool generation. They weren't producing values before. Need to rerun the model from pool generation! * Had to fix deadwood and litter in emissions year carbon pool generation. They weren't producing values before. Need to rerun the model from pool generation! Also, fixed metadata tags for per pixel output step. * Had to fix deadwood and litter in emissions year carbon pool generation. They weren't producing values before. Need to rerun the model from pool generation! Also, fixed metadata tags for per pixel output step. * Had to fix deadwood and litter in emissions year carbon pool generation. They weren't producing values before. Need to rerun the model from pool generation! Also, fixed metadata tags for per pixel output step. * Redoing tile statistics for standard model v1.2.0. * Redoing tile statistics for standard model v1.2.0. * Redoing tile statistics for standard model v1.2.0. * Redoing tile statistics for standard model v1.2.0. * Had to fix deadwood and litter in emissions year carbon pool generation. They weren't producing values before. Need to rerun the model from pool generation! Also, fixed metadata tags for per pixel output step. * Redoing tile statistics for standard model v1.2.0. * Creating carbon pools in 2000 for full model v1.2.0. * Creating carbon pools in 2000 for full model v1.2.0. * Internet died during soil C creation in emissions year for full run of standard modevl v1.2.0. Resuming from there. * Internet died during soil C creation in emissions year for full run of standard modevl v1.2.0. Resuming from there. * Creating carbon pools in 2000 for full standard model v1.2.0. * Redoing tile statistics for standard model v1.2.0. * Creating deadwood C 2000 and onwards. Spot machine ran out of storage, so had to add some lines to clean up tiles more frequently. * Creating deadwood C 2000 and onwards. Spot machine ran out of storage, so had to add some lines to clean up tiles more frequently. * Creating deadwood C 2000 and onwards. Spot machine ran out of storage, so had to add some lines to clean up tiles more frequently. * Creating deadwood C 2000 and onwards. Spot machine ran out of storage, so had to add some lines to clean up tiles more frequently. * Creating deadwood C 2000 and onwards. Spot machine ran out of storage, so had to add some lines to clean up tiles more frequently. * Creating soil only gross emissions for model v1.2.0. * Starting to work on creating uncertainty for soil C using SoilGrids250 CI5 and CI95 data. * Starting to work on creating uncertainty for soil C using SoilGrids250 CI5 and CI95 data. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Working on soil C stdev. All input SoilGrids tiles downloaded. Working on calculating stdev from them. * Creating mangrove removal factor stdev for select tiles again. Some had erroneously high values. * Creating removal factor standard deviation for US tiles and composite RF standard deviation for all forest types. * Creating removal factor standard deviation for US tiles and composite RF standard deviation for all forest types. * Creating removal factor standard deviation for US tiles and composite RF standard deviation for all forest types. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Found problems with US tiles. Rerunning model for US tiles only. * Creating global aggregate maps for standard model v1.2.0 with corrected US tiles. * Rerunning aggregate step for standard model v1.2.0 because I changed which pixels are included in aggregation. Now includes all pixels with tcd>30, all Hansen gain pixels, and all mangrove pixels. * Running no_shifting_ag sensitivity analysis for model v1.2.0 (emissions onwards). * Running no_shifting_ag sensitivity analysis for model v1.2.0 (emissions onwards). * Running no_shifting_ag sensitivity analysis for model v1.2.0 (emissions onwards). * Running no_shifting_ag sensitivity analysis for model v1.2.0 (emissions onwards). * Running no_shifting_ag sensitivity analysis for model v1.2.0 (emissions onwards). * Running no_shifting_ag sensitivity analysis for model v1.2.0 (emissions onwards). * Ready to run maxgain sensitivity analysis for model v1.2.0. * Ready to run maxgain sensitivity analysis for model v1.2.0. * Ready to run maxgain sensitivity analysis for model v1.2.0. * Net flux missed tiles without Hansen loss for no_shifting_ag and convert_to_grassland sensitivity analyses. Fixed to include full tile set and rerunning those models from net flux onwards. * Net flux missed tiles without Hansen loss for no_shifting_ag and convert_to_grassland sensitivity analyses. Fixed to include full tile set and rerunning those models from net flux onwards. * Going to run model v1.2.0 biomass_swap sensitivity analysis now. * Going to run model v1.2.0 biomass_swap sensitivity analysis now. * Going to run model v1.2.0 biomass_swap sensitivity analysis now. * Going to run model v1.2.0 biomass_swap sensitivity analysis now. * Net flux missed tiles without Hansen loss for no_shifting_ag and convert_to_grassland sensitivity analyses. Fixed to include full tile set and rerunning those models from net flux onwards. * Running biomass swap sensitivity analysis again using just the JPL tile set. * Running biomass swap sensitivity analysis again using just the JPL tile set. * Running biomass swap sensitivity analysis again using just the JPL tile set. Reduced number of processors for many steps because biomass_swap seems to take more memory than using WHRC. * Running biomass swap sensitivity analysis again using just the JPL tile set. Reduced number of processors for many steps because biomass_swap seems to take more memory than using WHRC. * Running biomass swap sensitivity analysis again using just the JPL tile set. Reduced number of processors for many steps because biomass_swap seems to take more memory than using WHRC. * Running no_primary_gain sensitivity analysis now. * Running the rest of biomass_swap sensitivity analysis. Had to reduce the number of processors for carbon pools onwards. * Needed to redo no_primary_gain sensitivity analysis. * Running biomass_swap sensitivity analysis again. * Running biomass_swap sensitivity analysis again. * Running biomass_swap sensitivity analysis again. * Running biomass_swap sensitivity analysis again. * Running US_removals sensitivity analysis. * Running US_removals sensitivity analysis. * Trying to finish up biomass_swap sensitivity analysis. Randomly stopped before. * Trying to finish up biomass_swap sensitivity analysis. Randomly stopped before. * Trying to finish up biomass_swap sensitivity analysis. Randomly stopped before. * Trying to finish up biomass_swap sensitivity analysis. Randomly stopped before. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Creating PRODES legal Amazon loss tiles for 2001-2019. * Working on legal_Amazon_loss sensitivity analysis. * Working on legal_Amazon_loss sensitivity analysis. * Working on legal_Amazon_loss sensitivity analysis. * Working on legal_Amazon_loss sensitivity analysis. * Working on legal_Amazon_loss sensitivity analysis. * Working on legal_Amazon_loss sensitivity analysis. * Added a few notes here and there.
- Loading branch information
