Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve P2P Stream Handling & Stream Sync Enhancements #4762

Merged
merged 32 commits into from
Oct 24, 2024
Merged

Improve P2P Stream Handling & Stream Sync Enhancements #4762

merged 32 commits into from
Oct 24, 2024

Conversation

GheisMohammadi
Copy link
Contributor

@GheisMohammadi GheisMohammadi commented Sep 24, 2024 •
edited
Loading

Issue

This PR introduces several key improvements to the P2P stream handling and Stream Sync mechanisms, focusing on stability, error handling, and code optimization. Key changes include:

  • Enhanced stream error handling for more visibility.
  • Optimized pickAvailableStream to continue searching for other available streams when one fails.
  • Refactored the stream request manager, including a Safe Map to ensure thread-safety.
  • Improved logic for switching between short-range and long-range syncing.
  • Optimized computeLongestHashChain and its whitelist functionality.
  • Improved countHashMaxVote for accurate vote calculations on a specific hash.
  • Added new configurations to better support staged stream sync.
  • Fixed issues with Beacon Node and Epoch Chain detection.
  • Introduced new metrics to monitor stream sync performance.
  • Enhanced lock management and concurrency control.
  • Passing the node roles throughout the stream protocol for better context awareness.
  • Refined stream sync short-range helper functions.
  • Improved the stream shutdown and cool-down processes, ensuring smooth terminations and halting unnecessary discoveries.
  • Comprehensive stream manager refactor for improved readability and maintainability.
  • Added more meaningful logs while removing unnecessary log entries or fields.

@GheisMohammadi GheisMohammadi self-assigned this Sep 24, 2024
@GheisMohammadi GheisMohammadi marked this pull request as draft September 24, 2024 13:42
@GheisMohammadi GheisMohammadi force-pushed the fix/staged_stream_sync_getblock_failure branch from d29a4c4 to 7475e83 Compare September 26, 2024 12:16
@GheisMohammadi GheisMohammadi mentioned this pull request Sep 28, 2024
Open
@GheisMohammadi GheisMohammadi force-pushed the fix/staged_stream_sync_getblock_failure branch 2 times, most recently from 653d129 to 2bea886 Compare October 4, 2024 09:43
@GheisMohammadi GheisMohammadi changed the title Fix localnet stream connections Improve P2P stream layer Oct 4, 2024
@GheisMohammadi
remove error logs if stream already exist
e775f52
@GheisMohammadi
improve pickAvailableStream to continue looking for other available s…
b4a5245 
…treams if any stream fails
@GheisMohammadi
add thread safe map to request manager in p2p stream layer
e95caf8
@GheisMohammadi
improve stream sync switching from long range to short range
fd47604
@GheisMohammadi
improve context for doSync loop in stream sync
9fba37c
@GheisMohammadi
add error handling for context deadline exceed issue
57b5f2a
@GheisMohammadi
fix request manager test
ba70765
@GheisMohammadi
improve computeLongestHashChain and its whitelist
42afaa4
@GheisMohammadi
improve countHashMaxVote
0e1af64
@GheisMohammadi
introduce new configs to StagedStreamSync struct
69eb0e7
@GheisMohammadi
initialize new configs in stream sync and add them to logs
e11a4c3
@GheisMohammadi
pass node config to stream sync downloader, increase stream sync tick…
bf6db5f 
…er time, init sync if not epoch chain
@GheisMohammadi
add two new metrics to stream sync
73a68cd
@GheisMohammadi
pass node configs to service and downloaders to have access to node c…
7cb2cd0 
…onfigurations in stream sync
@GheisMohammadi
using BeaconValidator and BecaonShard instead of BeaconNode, using Ep…
30d0450 
…ochChain config to detect epochcian in stages, add acceptPartially to GetBlockHash to be able to get a part of hashes
@GheisMohammadi
pass nodeconfig to downloaders
21916f6
@GheisMohammadi
remove BeaconNode from protocol, fix shortrange tests
0745b12
@GheisMohammadi
improve lock in request manager test
d1cabbb
@GheisMohammadi
update Failures option names, remove beaconNode from stream protocol
64cc717
@GheisMohammadi
fix shard id detection
1fdeacd
@GheisMohammadi
improve getBlockByMaxVote
f22f4ce
@GheisMohammadi
add one more checking for prestaking
f6ca98c
@GheisMohammadi
improve epoch and beacon node detection for stream sync
773376a
@GheisMohammadi GheisMohammadi force-pushed the fix/staged_stream_sync_getblock_failure branch from 2bea886 to 773376a Compare October 9, 2024 09:53
@GheisMohammadi GheisMohammadi changed the title Improve P2P stream layer Improve P2P Stream Handling & Stream Sync Enhancements Oct 11, 2024
@GheisMohammadi GheisMohammadi marked this pull request as ready for review October 11, 2024 04:47
@sophoah
Copy link
Contributor

sophoah commented Oct 23, 2024

@Frozen please review/approve

@sophoah sophoah merged commit 2274c29 into dev Oct 24, 2024
4 checks passed
@sophoah sophoah mentioned this pull request Jan 16, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sophoah sophoah sophoah approved these changes

@Frozen Frozen Frozen approved these changes

@mur-me mur-me Awaiting requested review from mur-me

Assignees

@GheisMohammadi GheisMohammadi

Labels
stream-sync
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GheisMohammadi @sophoah @Frozen