Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Starting to work on updating flux model with 2020 tree cover loss. First, updating burned area. * Fixing command for uploading hdf to s3. * Trying again from step 2 of burned area. * Trying again from step 2 of burned area. * Updated c++ scripts for 2020 TCL and made some other 2020 updates. Ran into some issue at the end of step 4 of burned area, so trying that again. * Ran individual model steps locally for test tile 00N_110E. Now going to try to change emissions c++ to use a constants header file. * Ran individual model steps locally for test tile 00N_110E. Added constants.h for emissions model, so standard and sensitivity analyses get their constants and file patterns from the same sources. Easier for updating sensitivity analyses that way. Couldn't figure out how to make equations.cpp also used constants.h, so the model_years in equations.cpp needs to be updated separately for now. Using constants.h compiles and runs locally. * Ran individual model steps locally for test tile 00N_110E. Added constants.h for emissions model, so standard and sensitivity analyses get their constants and file patterns from the same sources. Easier for updating sensitivity analyses that way. Couldn't figure out how to make equations.cpp also used constants.h, so the model_years in equations.cpp needs to be updated separately for now. Using constants.h compiles and runs locally. * Ran individual model steps locally for test tile 00N_110E. Added constants.h for emissions model, so standard and sensitivity analyses get their constants and file patterns from the same sources. Easier for updating sensitivity analyses that way. Couldn't figure out how to make equations.cpp also used constants.h, so the model_years in equations.cpp needs to be updated separately for now. Using constants.h compiles and runs locally. * Ran individual model steps locally for test tile 00N_110E. Added constants.h for emissions model, so standard and sensitivity analyses get their constants and file patterns from the same sources. Easier for updating sensitivity analyses that way. Couldn't figure out how to make equations.cpp also used constants.h, so the model_years in equations.cpp needs to be updated separately for now. Using constants.h compiles and runs locally. * Corrected the calls for checking for empty tiles in annual and gross removals script. * Corrected the calls for checking for empty tiles in annual and gross removals script. Now, annual removals only uploads tiles with data. Gross removals should be able to skip tile_ids that don't have the necessary input tiles (gain year couny and annual removals). * Corrected the calls for checking for empty tiles in annual and gross removals script. Now, annual removals only uploads tiles with data. Gross removals should be able to skip tile_ids that don't have the necessary input tiles (gain year couny and annual removals). * Still trying to figure out whether only annual removal factor tiles that have data are being copied to s3. * Fixing the tile data check function. * Correcting issue with tiles not existing for carbon pool creation. * Changing output folder dates. * Revised readme. Model ran on test tiles through aggregation. Need to fix supplementary output creation script error. * Revised readme. Model ran on test tiles through aggregation. Need to fix supplementary output creation script error. * Revised readme. Model ran on test tiles through aggregation. Need to fix supplementary output creation script error. * Final test of supplementary output creation step. * Going to run the full flux model with 2020 TCL data as model v1.2.1. Running from model_extent to emission-year carbon pool creation with today's date and everything after (emissions onwards) with date 20219999 because I'll need to rerun those steps with the updated drivers model. Tested on a few test tiles but this is my first run on all tiles with the 2020 update. * Going to run the full flux model with 2020 TCL data as model v1.2.1. Running from model_extent to emission-year carbon pool creation with today's date and everything after (emissions onwards) with date 20219999 because I'll need to rerun those steps with the updated drivers model. Tested on a few test tiles but this is my first run on all tiles with the 2020 update. * I've gotten errors a few times during full model runs when the script is trying to upload model logs. It seems to happen towards the end of a tile list and in stages that use more processors (though that could be coincidental). It seems to happen only sometimes, because I ran the same script a few times and sometimes the error happened and sometimes it didn't. My guess is that sometimes different log uploads compete with each other and that causes an error. So I'm reducing the calls for uploading the log by removing uploads when normal statements are printed (in print_log). Instead, I added upload_log() commands at the end of each tile being processed (in end_of_fx_summary), at the end of each model stage (in upload_final_set), and at the very end of the model (last line of run_full_model). Hopefully this'll reduce the conflict between log uploads. Exceptions and subprocess commands still automatically trigger log uploads. * Various small fixes to model. Should run smoothly from model_extent until create_supplementary_outputs. Ran into error with gross removals tile not existing on that stage. * Fixing the supplementary outputs step. Issue with what tile_id_list to use. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * Jimmy MacCarthy updated the TCL 2020 drivers, so ready to process the 2020 drivers and run emissions, net flux, and their aggregation/supplementary outputs for real now. * During emissions metadata creation, the log upload function got overwhelmed. Removed the log upload call in the subprocess.check_call function. * Need to rerun emissions onwards because I have corrected 2020 TCL driver map now (previous run used an erroneous driver map). This should be the final run of the 2020 emissions model (v1.2.1). * Need to rerun emissions onwards because I have corrected 2020 TCL driver map now (previous run used an erroneous driver map). This should be the final run of the 2020 emissions model (v1.2.1). * Need to rerun emissions onwards because I have corrected 2020 TCL driver map now (previous run used an erroneous driver map). This should be the final run of the 2020 emissions model (v1.2.1). * Still got an error from uploading logs too quickly to s3. So, removed log upload call from the subprocess call. * For emissions and final steps using the final, corrected drivers for flux model v1.2.1 (2001-2020 update). * Now with the correctly reprojected driver map. * Ran into issue with deleting extra tiles after creating supplementary outputs. * Successfully ran main flux model with biomass_soil emissions, going to run soil_only gross emissions for model v1.2.1 (2001-2020) now. * Successfully ran main flux model with biomass_soil emissions, going to run soil_only gross emissions for model v1.2.1 (2001-2020) now. * Successfully ran main flux model with biomass_soil emissions, going to run soil_only gross emissions for model v1.2.1 (2001-2020) now. Corrected the output soil_only emissions tile names. * Successfully ran main flux model with biomass_soil emissions, going to run soil_only gross emissions for model v1.2.1 (2001-2020) now. Corrected the output soil_only emissions tile names... needed one final correction.
- Loading branch information