-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updating Along-slope-velocities.ipynb
to use intake from Cosima Coo…
#458
Draft
charles-turner-1
wants to merge
5
commits into
COSIMA:main
Choose a base branch
from
charles-turner-1:INTAKE_along_slope_velocities
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Updating Along-slope-velocities.ipynb
to use intake from Cosima Coo…
#458
charles-turner-1
wants to merge
5
commits into
COSIMA:main
from
charles-turner-1:INTAKE_along_slope_velocities
Commits on Sep 16, 2024
-
Updating
Along-slope-velocities.ipynb
to use intake from Cosima Coo……kbook. Couple of issues: 1. Cosima Cookbook seems to allow searching for coordinates: eg (`st_edges_ocean = cc.querying.getvar(experiment, 'st_edges_ocean', session, start_time=start_time, end_time=end_time, n=1) st_edges_array = st_edges_ocean.expand_dims({'yu_ocean': u.yu_ocean, 'xu_ocean': u.xu_ocean}, axis=[1, 2])` This doesn't seem to be easily achievable with intake (Cell 11 pre-update)? I can't currently figure out how to do this. 2. Updating from `cosima_cookbook` => `intake` seems to massively increase compute time for `topographic _slop_magnitude` calculation: `UserWarning: Sending large graph of size 47.19 MiB.` => `Sending large graph of size 123.9 MiB.`. The source of issue 2 is currently unclear to me & is dramatically increasing computation time.
Configuration menu - View commit details
-
Copy full SHA for 436a6e4 - Browse repository at this point
Copy the full SHA 436a6e4View commit details
Commits on Sep 17, 2024
-
Configuration menu - View commit details
-
Copy full SHA for de0ddd1 - Browse repository at this point
Copy the full SHA de0ddd1View commit details -
Updates following comments from Navid & Marc:
- Where possible, all direct file reads have been removed - still requires a bit of a hack to get the grid file (Cell [10]) & `st_edges` (Cell [13]). - Varnames lowercased. - Typos fixed. - Direct indexing of catalog. - Redundant xarray calls fixed. Notes: - It appears the catalog has several data frequencies available for this experiment, unlike `cosima_cookbook`. - I made the choice to disambiguate to 3 month freq. to minimise computation size (lowest frequency available). - Migrating from cosima_cookbook => intake substantially increased dask_graph layers - was necessary to call dask.optimize() to flatten these out. All chunking, etc. is the same as it was - currently not clear to me where the discrepancy in compute graph size originates. - Currently doesn't appear that intake supports any good way to search for coordinate variables? Leads to aforementioned hacks.
Configuration menu - View commit details
-
Copy full SHA for c77a91e - Browse repository at this point
Copy the full SHA c77a91eView commit details
Commits on Sep 20, 2024
-
- Added explicit chunking - made most dask.optimize() calls redundant.
- Extracted depth_slice & data freq to constants & changed default data frequency to 1 month (`1mon`) - Added a bunch of comments & explanation surrounding chunking. - Added some comments surrounding operations & what they're doing
Configuration menu - View commit details
-
Copy full SHA for ff3df89 - Browse repository at this point
Copy the full SHA ff3df89View commit details
Commits on Oct 12, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 3dd9875 - Browse repository at this point
Copy the full SHA 3dd9875View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.