Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor nearest neighbor and interpolation steps #1325

Merged
merged 10 commits into from
Dec 18, 2024

Conversation

tiffanychu90
Copy link
Member

@tiffanychu90 tiffanychu90 commented Dec 18, 2024

segment speeds refactor

  • Follow-up to Remove vp nearest neighbor intermediate output #1319 and add the dummy arrival time needed to get cut_stop_segments.py to run successfully
  • Refactor nearest neighbor and interpolation steps
    • Previously, nearest neighbor was broken out into 2 intermediate outputs, nearest 10 vp then filtering to nearest 2 vp
    • Now, we can work all that into the same numpy array and get exactly what we want from the nearest neighbor step
    • Rather than reshaping within interpolation step, just get exactly the wide shape we want when we save out the nearest neighbor output in stage2
    • Clean up the extra entries in gtfs_analytics_data.yml and remove the need for stage2b...we can just use stage2 again
    • 2 nearest neighbor steps took roughly ~30 min, and now swapping it out into 1 step, the times stay the same, but opting for simplicity
    • interpolation step took ~15 min, and now it's 9 min
    • Remove extra functions no longer needed
  • Refactor functions in segment_speed_utils.neighbor
  • Remove test scripts after testing 1 date
  • Add the 2 speed layer URLs to open_data/README.md
  • Epic - GTFS analytics pipeline performance improvements #1315

@tiffanychu90 tiffanychu90 changed the title Refactor nn interpolation Refactor nearest neighbor and interpolation steps Dec 18, 2024
@tiffanychu90 tiffanychu90 merged commit 61e0bd9 into main Dec 18, 2024
2 checks passed
@tiffanychu90 tiffanychu90 deleted the refactor--nn-interpolation branch December 18, 2024 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant