Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating Along-slope-velocities.ipynb to use intake from Cosima Coo… #458

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Commits on Sep 16, 2024

  1. Updating Along-slope-velocities.ipynb to use intake from Cosima Coo…

    …kbook.
    
    Couple of issues:
    
    1. Cosima Cookbook seems to allow searching for coordinates: eg (`st_edges_ocean = cc.querying.getvar(experiment, 'st_edges_ocean', session,
                                        start_time=start_time, end_time=end_time, n=1)
    st_edges_array = st_edges_ocean.expand_dims({'yu_ocean': u.yu_ocean, 'xu_ocean': u.xu_ocean}, axis=[1, 2])`
    This doesn't seem to be easily achievable with intake (Cell 11 pre-update)? I can't currently figure out how to do this.
    
    2. Updating from `cosima_cookbook` => `intake` seems to massively increase compute time for `topographic
    _slop_magnitude` calculation:
    `UserWarning: Sending large graph of size 47.19 MiB.` => `Sending large graph of size 123.9 MiB.`.
    
    The source of issue 2 is currently unclear to me & is dramatically increasing computation time.
    charles-turner-1 committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    436a6e4 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Configuration menu
    Copy the full SHA
    de0ddd1 View commit details
    Browse the repository at this point in the history
  2. Updates following comments from Navid & Marc:

      - Where possible, all direct file reads have been removed - still
        requires a bit of a hack to get the grid file (Cell [10]) &
        `st_edges` (Cell [13]).
      - Varnames lowercased.
      - Typos fixed.
      - Direct indexing of catalog.
      - Redundant xarray calls fixed.
    
    Notes:
      - It appears the catalog has several data frequencies available for
        this experiment, unlike `cosima_cookbook`.
        - I made the choice to disambiguate to 3 month freq. to minimise
        	computation size (lowest frequency available).
        - Migrating from cosima_cookbook => intake substantially increased
          dask_graph layers - was necessary to call dask.optimize() to
          flatten these out. All chunking, etc. is the same as it was -
          currently not clear to me where the discrepancy in compute graph
          size originates.
      - Currently doesn't appear that intake supports any good way to search
        for coordinate variables? Leads to aforementioned hacks.
    charles-turner-1 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c77a91e View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. - Added explicit chunking - made most dask.optimize() calls redundant.

    - Extracted depth_slice & data freq to constants & changed default data
      frequency to 1 month (`1mon`)
    - Added a bunch of comments & explanation surrounding chunking.
    - Added some comments surrounding operations & what they're doing
    charles-turner-1 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ff3df89 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Configuration menu
    Copy the full SHA
    3dd9875 View commit details
    Browse the repository at this point in the history