[1pt] Update CatFIM metadata API pulls and add duplicate site filtering, add CatFIM vis files

[EmilyDeardorff]

There are many sites in non-CONUS regions (AK, PR, HI) where we would like to run CatFIM but they are being excluded because they are not NWM forecast points. This PR brings back the double API pull and adds in some code to filter out duplicate (and NULL) lids from the metadata lists.

Additions

• inundation-mapping/tools/catfim/vis_categorical_fim.py: Functions for reading in, processing, and visualizing CatFIM results. Functions should be run in vis_catfim_cross_section.ipynb.
• inundation-mapping/tools/catfim/notebooks/vis_catfim_cross_section.ipynb: A new Jupyter notebook for viewing and analyzing CatFIM results at a given site.
• inundation-mapping/tools/catfim/notebooks/eval_catfim_metadata.ipynb: A new Jupyter notebook for evaluating metadata and results from CatFIM runs.
• inundation-mapping/tools/catfim/notebooks/.ipynb_checkpoints/vis_catfim_cross_section-checkpoint.ipynb: Checkpoint file for vis_catfim_cross_section.ipynb.
• inundation-mapping/tools/catfim/notebooks/.ipynb_checkpoints/eval_catfim_metadata-checkpoint.ipynb: Checkpoint file for eval_catfim_metadata.ipynb
• inundation-mapping\config/symbology/qgis/catfim_library.qml: Symbology preset for viewing CatFIM library in QGIS.

Changes

• inundation-mapping/tools/catfim/generate_categorical_fim_flows.py: Re-implements the dual API call, with the first API call getting all forecast points and the second API call getting all points (regardless of forecast point status). Adds code to filter out duplicate sites.

Testing

I ran some tests with the updated API method (two API pulls + filtering out duplicate sites) and summarized the results below. It seems like this update handles the duplicate point issue well while still widening the selection criteria enough to produce more CatFIM results for AK, HI, and PR.

It will be worth separately looking into why HI and PR are not producing any stage-based CatFIM.

	FIM 4.5.2.11 [Currently Live]	FIM 4.5.11.1 [Upcoming Release]	With API Update
Metadata Retrieval Method	two API pulls, no duplicate filtering	one API pull (only forecast points)	two API pulls, filters out duplicate LIDs

---

Flow-based, Number of sites (Mapped | Unmapped)

State/Region	FIM 4.5.2.11 [Currently Live]	FIM 4.5.11.1 [Upcoming Release]	With API Update
Alaska	NA*	14 | 39	31 | 614
Puerto Rico	68 | 177	4 | 1	59 | 181
Hawaii	50 | 441	1 | 1	47 | 442

---

Stage-based, Number of sites (Mapped | Unmapped)

State/Region	FIM 4.5.2.11 [Currently Live]	FIM 4.5.11.1 [Upcoming Release]	With API Update
Alaska	NA*	13 | 40	5 | 209
Puerto Rico	0 | 245	0 | 5	0 | 240
Hawaii	0 | 491	0 | 2	0 | 489

*FIM 4.5.2.11 has no CatFIM points for Alaska because it was added later, in FIM 4.5.11.1
** Unmapped sites weren’t tallied because the test did not produce any mapped sites for the region.

Further Testing

It would be good to test this on stage-based and flow-based a dozen or so HUCs, including some from Alaska, Puerto Rico, and Hawaii. When testing, it would be good to view the CatFIM outputs after the runs to make sure the CatFIM sites and CatFIM libraries look good, especially in AK, PR, and HI.

It would also be good to run inundation-mapping/tools/catfim/notebooks/vis_catfim_cross_section.ipynb to make sure that you can get the visualization scripts to run.

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[RobHanna-NOAA]

Error found.
2025-01-07 23:56:06 | ERROR || An error occured while generating flows for huc 20020000
2025-01-07 23:56:06 | ERROR || Details: local variable 'stream_order' referenced before assignment
2025-01-07 23:56:06 | ERROR || Traceback (most recent call last):
File "/foss_fim/tools/catfim/generate_categorical_fim_flows.py", line 185, in generate_flows_for_huc
segments = filter_nwm_segments_by_stream_order(unfiltered_segments, desired_order, nwm_flows_df)
File "/foss_fim/tools/tools_shared_functions.py", line 76, in filter_nwm_segments_by_stream_order
if stream_order == desired_order:
UnboundLocalError: local variable 'stream_order' referenced before assignment

Can be seen on dev1 at catfim/rob_test/dup_lids_flow_based/logs/

Also take a peak at the warnings for both flow and stage.. did you expect that many?