Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dask.from_map in dask_utils/time_series_utils #1316

Merged
merged 6 commits into from
Dec 3, 2024

Conversation

tiffanychu90
Copy link
Member

@tiffanychu90 tiffanychu90 commented Dec 3, 2024

  • Start with a notebook in rt_segment_speeds/41_dask_from_map.ipynb testing how to use dask.from_map
  • Follow dask tutorial example and add the necessary functions into shared_utils.dask_utils and make sure it supports necessary args/kwargs for time_series_utils
  • Add functions into shared_utils.time_helpers for adding service_date, year, quarter, since these used to be part of time_series_utils. For dask.from_map, put these functions here so we can pipe these additional functions through.
  • Refactor quarter_year_averages.py for segment speed averaging and use these functions
    • Quarterly and yearly averages each take ~10 min, down from 25 min before for yearly averages (quarterly averages blew the kernel)
    • This is better. This can still get refactored to simplify the logic and see if we can stack yearly on top of quarterly so we don't have two, independent, 10 minute steps.

@tiffanychu90 tiffanychu90 linked an issue Dec 3, 2024 that may be closed by this pull request
2 tasks
Copy link

github-actions bot commented Dec 3, 2024

nbviewer URLs for impacted notebooks:

3 similar comments
Copy link

github-actions bot commented Dec 3, 2024

nbviewer URLs for impacted notebooks:

Copy link

github-actions bot commented Dec 3, 2024

nbviewer URLs for impacted notebooks:

Copy link

github-actions bot commented Dec 3, 2024

nbviewer URLs for impacted notebooks:

@tiffanychu90 tiffanychu90 merged commit dc5f6d7 into main Dec 3, 2024
6 checks passed
@tiffanychu90 tiffanychu90 deleted the dask-from-map-switch branch December 4, 2024 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Research Request - switch dask.delayed to dask.from_map
1 participant