Address Leiden clustering generating too many clusters #4730
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…a requirement during the refinement phase
ChuckHastings
added
bug
naimnv
approved these changes
Oct 22, 2024
seunghwak
approved these changes
Oct 22, 2024
| @@ -230,6 +229,7 @@ refine_clustering( | |||
| cugraph::reduce_op::plus<weight_t>{}, | |||
| weighted_cut_of_vertices_to_louvain.begin()); | |||
|
|
|||
| // FIXME: Consider using bit mask logic here. Would reduce memory by 8x | |||
There was a problem hiding this comment.
Yes, and also a higher chance to fit in L2 cache.
There was a problem hiding this comment.
We have multiple utility functions to support this.
| // a direct lookup in louvain_assignment_of_vertices using | ||
| // leiden - graph_view.local_vertex_partition_range_first() as the | ||
| // index? | ||
| // Changing this would save memory and time |
There was a problem hiding this comment.
Any reason to defer the update?
There was a problem hiding this comment.
Trying to get a fix out to a user question. General Leiden improvements isn't a priority right now.
|
/merge |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our implementation of Leiden was generating too many clusters. This was not obvious in smaller graphs, but as the graphs get larger the problem became more noticeable.
The Leiden loop was terminating if the modularity stopped improving. But the Leiden algorithm as defined in the paper allows the refinement phase to reduce modularity in order to improve the quality of the clusters. The convergence criteria defined in the paper was based on making no changes on the iteration rather than strictly monitoring modularity change.
Updating this criteria results in the Leiden algorithm running more iterations and converging on better answers.
Closes #4529