Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated IIJA README #1295

Merged
merged 2 commits into from
Nov 18, 2024
Merged

Updated IIJA README #1295

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7277e64
updated README to reflect notes from most recent script run
csuyat-dot Nov 18, 2024
6ed6272
added more details per comments
csuyat-dot Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 23 additions & 11 deletions dla/iija/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,29 @@
This folder includes the data exploration, cleaning and script for the IIJA program data that is uploaded to [RebuildingCA](https://rebuildingca.ca.gov/map/??).

#### To run the script, run through the following steps:
1. Upload exported data from FMIS (CSV or Excel) to the GCS Bucket
2. Open `run_script.ipynb` and change path to relfect the uploaded exported data
3. Check that the exported file has no empty first rows
4. Run cells up to `Test & Export`
5. In the `Test & Export` section, run the function
<blockquote>`_script_utils.get_clean_data()`</blockquote>
to get the final cleaned data. If you want to get the data aggregated to the program level, use kwargs <blockquote>_script_utils.get_clean_data(df, full_or_agg = 'agg')</blockquote>
If you want the full dataframe where each row is a project phase, use kwarg
<blockquote>_script_utils.get_clean_data(df, full_or_agg = 'full')</blockquote>

Note: In the aggregated data, a project can have more than one row, if the project is funded with more than one IIJA program.
0. Receive list of FMIS IIJA funded projects from DOTP Office of Technical Freight & Project Integration

1. Upload list (CSV or Excel) to the GCS Bucket `/data-analyses/dla/dla-iija/`

2. In terminal, cd to `data-analyses/dla`, then run `make setup_env` to install requirements. cd to `iija` afterwards.

**Note:** install may fail during conda install, but should still be able to continue with the rest of these instructions.

3. Open `run_script.ipynb`, in the `Read in Data and function development` section update the `my_file` path to the latest IIJA project list in GCS bucket.

4. In the `Check Data` section, run the cells to ensure the dataframe has no empty first rows

5. In the `Run Script` section, run the `_script_utils.run_script(*,*,'agg')` function to get the final cleaned data.

The format preferred by stakeholders is for the data to be aggregated to the program level, use this kwarg in the fucntion
<blockquote>_script_utils.run_script(*,*,df_agg_level = 'agg')</blockquote>
Or, If you want the full dataframe where each row is a project phase, use this kwarg in the fucntion
<blockquote>_script_utils.run_script(*,*,df_agg_level = 'full')</blockquote>

6. In the `Export Data` section, Use the current date `(MMDDYYYY)`, (or `(MMDDYYYY_agg)` if the df_agg_level was set to `agg`), to set the filename suffix. Then, run the `_script_utils.export_to_gcs()` function to export the data to GCS.
The data can be found in the same file path stated in the previous steps, with the file name `FMIS_Projects_Universe_IIJA_Reporting_*.csv`

**Note:** In the aggregated data, a project can have more than one row, if the project is funded with more than one IIJA program.


#### Scripts
Expand Down
Loading