-
Notifications
You must be signed in to change notification settings - Fork 54
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #463 from freemansw1/v160_merge_main
Merge main into v1.6.0 branch
- Loading branch information
Showing
55 changed files
with
94,487 additions
and
91,581 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,23 +1,19 @@ | ||
| name: check_formatting | ||
| name: Check Python File Formatting with Black | ||
| on: [push, pull_request] | ||
| jobs: | ||
| formatting_job: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: actions/checkout@v4 | ||
| - name: Set up conda | ||
| uses: conda-incubator/setup-miniconda@v2 | ||
| uses: mamba-org/setup-micromamba@v1 | ||
| with: | ||
| miniforge-version: latest | ||
| miniforge-variant: mambaforge | ||
| channel-priority: strict | ||
| channels: conda-forge | ||
| show-channel-urls: true | ||
| use-only-tar-bz2: true | ||
|
|
||
| - name: Install dependencies and check formatting | ||
| shell: bash -l {0} | ||
| environment-file: environment-ci.yml | ||
| generate-run-shell: true | ||
| cache-environment: true | ||
| cache-downloads: true | ||
| - name: Check formatting | ||
| shell: micromamba-shell {0} | ||
| run: | ||
| mamba install --quiet --yes --file requirements.txt black && | ||
| black --version && | ||
| black tobac --check --diff |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,3 +13,4 @@ pre-commit | |
| black | ||
| pytest | ||
| typing_extensions | ||
| nbconvert | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,55 @@ | ||
| Handling Large Datasets | ||
| ------------------------------------- | ||
|
|
||
| Often, one desires to use *tobac* to identify and track features in large datasets ("big data"). This documentation strives to suggest various methods for doing so efficiently. Current versions of *tobac* do not allow for out-of-memory computation, meaning that these strategies may need to be employed for both computational and memory reasons. | ||
| Often, one desires to use *tobac* to identify and track features in large datasets ("big data"). This documentation strives to suggest various methods for doing so efficiently. Current versions of *tobac* do not support out-of-core (e.g., :code:`dask`) computation, meaning that these strategies may need to be employed for both computational and memory reasons. | ||
|
|
||
| .. _Split Feature Detection: | ||
|
|
||
| ======================= | ||
| Split Feature Detection | ||
| Split Feature Detection and Run in Parallel | ||
| ======================= | ||
| Current versions of threshold feature detection (see :doc:`feature_detection_overview`) are time independent, meaning that one can parallelize feature detection across all times (although not across space). *tobac* provides the :py:meth:`tobac.utils.combine_tobac_feats` function to combine a list of dataframes produced by a parallelization method (such as :code:`jug` or :code:`multiprocessing.pool`) into a single combined dataframe suitable to perform tracking with. | ||
| Current versions of threshold feature detection (see :doc:`feature_detection_overview`) are time independent, meaning that one can easily parallelize feature detection across all times (although not across space). *tobac* provides the :py:meth:`tobac.utils.combine_feature_dataframes` function to combine a list of dataframes produced by a parallelization method (such as :code:`jug`, :code:`multiprocessing.pool`, or :code:`dask.bag`) into a single combined dataframe suitable to perform tracking with. | ||
|
|
||
| Below is a snippet from a larger notebook demonstrating how to run feature detection in parallel ( :doc:`big_datasets_examples/notebooks/parallel_processing_tobac`): | ||
|
|
||
| :: | ||
|
|
||
| # build list of tracked variables using Dask.Bag | ||
|
|
||
| b = db.from_sequence( | ||
| [ | ||
| combined_ds["data"][x : x + 1] | ||
| for x in range(len(combined_ds["time"])) | ||
| ], | ||
| npartitions=1, | ||
| ) | ||
| out_feature_dfs = db.map( | ||
| lambda x: tobac.feature_detection_multithreshold( | ||
| x.to_iris(), 4000, **parameters_features | ||
| ), | ||
| b, | ||
| ).compute() | ||
|
|
||
| combined_dataframes = tobac.utils.general.combine_feature_dataframes(out_feature_dfs) | ||
|
|
||
|
|
||
| .. _Split Segmentation: | ||
|
|
||
| ====================================== | ||
| Split Segmentation and Run in Parallel | ||
| ====================================== | ||
| Recall that the segmentation mask (see :doc:`segmentation_output`) is the same size as the input grid, which results in large files when handling large input datasets. The following strategies can help reduce the output size and make segmentation masks more useful for the analysis. | ||
|
|
||
| The first strategy is to only segment on features *after tracking and quality control*. While this will not directly impact performance, waiting to run segmentation on the final set of features (after discarding, e.g., non-tracked cells) can make analysis of the output segmentation dataset easier. | ||
|
|
||
| To enhance the speed at which segmentation runs, one can process multiple segmentation times in parallel independently, similar to feature detection. Unlike feature detection, however, there is currently no built-in *tobac* method to combine multiple segmentation times into a single file. While one can do this using typical NetCDF tools such as :code:`nccat` or with xarray utilities such as :code:`xr.concat`, you can also leave the segmentation mask output as separate files, opening them later with multiple file retrievals such as :code:`xr.open_mfdataset`. | ||
|
|
||
|
|
||
| .. _Tracking Hanging: | ||
|
|
||
| ===================================== | ||
| Tracking Hangs with too many Features | ||
| ===================================== | ||
|
|
||
| When tracking on a large dataset, :code:`tobac.tracking.linking_trackpy` can hang using the default parameters. This is due to the tracking library :code:`trackpy` searching for the next timestep's feature in too large of an area. This can be solved *without impact to scientific output* by lowering the :code:`subnetwork_size` parameter in :code:`tobac.tracking.linking_trackpy`. | ||
|
|
Oops, something went wrong.