Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ascending bootstrap dependency resolution #4692

Merged
merged 26 commits into from
Aug 6, 2024
Merged

Ascending bootstrap dependency resolution #4692

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
6647db3
Dependency walking
pwojcikdev Mar 2, 2023
89baf1a
Parallel database scan
pwojcikdev Jul 24, 2024
8f17685
Stats
pwojcikdev Jul 24, 2024
63f2dd1
Throttling & backoff
pwojcikdev Jul 25, 2024
0b8428e
Dependency account scanning
pwojcikdev Jul 29, 2024
70bf7de
Sync dependencies
pwojcikdev Jul 30, 2024
6a699e9
Track source when throttling
pwojcikdev Jul 30, 2024
ede14c4
Count tags
pwojcikdev Jul 30, 2024
4875c41
Priorities backoff
pwojcikdev Jul 30, 2024
d79f979
Timestamps
pwojcikdev Jul 30, 2024
c3ae845
Avoid reprocessing old blocks
pwojcikdev Jul 30, 2024
454d799
Max tags limit
pwojcikdev Jul 30, 2024
861cba5
Handle `gap_previous`
pwojcikdev Jul 30, 2024
4f864e4
Check timestamp
pwojcikdev Jul 30, 2024
430d39d
Tune initial priority
pwojcikdev Jul 31, 2024
dc20ebc
Fix config
pwojcikdev Jul 31, 2024
fb8aa72
Verify response
pwojcikdev Jul 31, 2024
195e692
Use filters
pwojcikdev Jul 31, 2024
1f9dfde
Remove random sampling
pwojcikdev Jul 31, 2024
a083646
Backoff adjustments
pwojcikdev Jul 31, 2024
81ef0bc
Insert genesis on start
pwojcikdev Aug 2, 2024
ec7e7a0
Fix timestamp check performance
pwojcikdev Aug 2, 2024
3396184
Adjust throttle size computation
pwojcikdev Aug 4, 2024
8541e90
Config improvements
pwojcikdev Aug 4, 2024
9f44594
Fix compilation
pwojcikdev Aug 5, 2024
af196cc
Extend test timeout
pwojcikdev Aug 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion nano/lib/stats_enums.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,8 @@ enum class detail
track,
timeout,
nothing_new,
account_info_empty,
loop_dependencies,

// bootstrap ascending accounts
prioritize,
Expand All @@ -429,9 +431,10 @@ enum class detail
unblock,
unblock_failed,

next_none,
next_priority,
next_database,
next_none,
next_dependency,

blocking_insert,
blocking_erase_overflow,
Expand All @@ -442,6 +445,12 @@ enum class detail
deprioritize,
deprioritize_failed,

request_blocks,
request_account_info,

// active
started_hinted,
started_optimistic,
// rep_crawler
channel_dead,
query_target_failed,
Expand Down
37 changes: 27 additions & 10 deletions nano/node/bootstrap_ascending/account_sets.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,23 +153,21 @@ bool nano::bootstrap_ascending::account_sets::check_timestamp (const nano::accou

void nano::bootstrap_ascending::account_sets::trim_overflow ()
{
if (priorities.size () > config.priorities_max)
while (priorities.size () > config.priorities_max)
{
// Evict the lowest priority entry
priorities.get<tag_priority> ().erase (priorities.get<tag_priority> ().begin ());

// Erase the oldest entry
priorities.pop_front ();
stats.inc (nano::stat::type::bootstrap_ascending_accounts, nano::stat::detail::priority_erase_overflow);
}
if (blocking.size () > config.blocking_max)
while (blocking.size () > config.blocking_max)
{
// Evict the lowest priority entry
blocking.get<tag_priority> ().erase (blocking.get<tag_priority> ().begin ());

// Erase the oldest entry
blocking.pop_front ();
stats.inc (nano::stat::type::bootstrap_ascending_accounts, nano::stat::detail::blocking_erase_overflow);
}
}

nano::account nano::bootstrap_ascending::account_sets::next ()
nano::account nano::bootstrap_ascending::account_sets::next_priority ()
{
if (priorities.empty ())
{
Expand Down Expand Up @@ -211,6 +209,25 @@ nano::account nano::bootstrap_ascending::account_sets::next ()
return result;
}

nano::block_hash nano::bootstrap_ascending::account_sets::next_blocking ()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this return optional rather than 0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could and should be done. I didn't have too much time to cleanup everything though.

{
if (blocking.empty ())
{
return { 0 };
}

// Use a dedicated, uniformly distributed field for sampling to avoid problematic corner case when accounts in the queue are very close together
auto search = nano::bootstrap_ascending::generate_id ();
auto iter = blocking.get<tag_id> ().lower_bound (search);
if (iter == blocking.get<tag_id> ().end ())
{
iter = blocking.get<tag_id> ().begin ();
}

auto result = iter->dependency;
return result;
}

bool nano::bootstrap_ascending::account_sets::blocked (nano::account const & account) const
{
return blocking.get<tag_account> ().count (account) > 0;
Expand Down Expand Up @@ -251,4 +268,4 @@ std::unique_ptr<nano::container_info_component> nano::bootstrap_ascending::accou
composite->add_component (std::make_unique<container_info_leaf> (container_info{ "priorities", priorities.size (), sizeof (decltype (priorities)::value_type) }));
composite->add_component (std::make_unique<container_info_leaf> (container_info{ "blocking", blocking.size (), sizeof (decltype (blocking)::value_type) }));
return composite;
}
}
15 changes: 10 additions & 5 deletions nano/node/bootstrap_ascending/account_sets.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ namespace bootstrap_ascending
void timestamp_set (nano::account const & account);
void timestamp_reset (nano::account const & account);

nano::account next ();
nano::account next_priority ();
nano::block_hash next_blocking ();

public:
bool blocked (nano::account const & account) const;
Expand Down Expand Up @@ -78,9 +79,11 @@ namespace bootstrap_ascending

struct blocking_entry
{
nano::account account{ 0 };
nano::block_hash dependency{ 0 };
priority_entry original_entry{ 0, 0 };
nano::account account;
nano::block_hash dependency;
priority_entry original_entry;

id_t id{ generate_id () }; // Uniformly distributed, used for random querying

float priority () const
{
Expand Down Expand Up @@ -115,7 +118,9 @@ namespace bootstrap_ascending
mi::ordered_unique<mi::tag<tag_account>,
mi::member<blocking_entry, nano::account, &blocking_entry::account>>,
mi::ordered_non_unique<mi::tag<tag_priority>,
mi::const_mem_fun<blocking_entry, float, &blocking_entry::priority>>
mi::const_mem_fun<blocking_entry, float, &blocking_entry::priority>>,
mi::ordered_unique<mi::tag<tag_id>,
mi::member<blocking_entry, nano::bootstrap_ascending::id_t, &blocking_entry::id>>
>>;
// clang-format on

Expand Down
Loading