Start migrating to Stream-based diff iterator
#2501
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
martinvonz
force-pushed
the
push-tokvzroztlxv
branch
2 times, most recently
from
2bd8eb5 to
e60382f
Compare
martinvonz
force-pushed
the
push-tokvzroztlxv
branch
2 times, most recently
from
947e30f to
b998230
Compare
During the transition to using more async code, I keep running into rust-lang/futures-rs#2090. Right now, I want to convert `MergedTree::diff()` into a `Stream`. I don't want to update all call sites at once, so instead I'm adding a `MergedTree::diff_stream()` method, which just wraps `MergedTree::diff()` in a `Stream. However, since the iterator is synchronous, it needs to block on the async `Backend::read_tree()` calls. If we then also block on the `Stream` in the CLI, we run into the panic.
I'm going to implement a `Stream`-based version optimized for high-latency (RPC-based) commit backends. So far, that implementation is about 20% slower in the Linux repo when running `jj diff --ignore-working-copy -s --from v5.0 --to v6.0`. I think that's almost only because the algorithm is different, not because it's async per se. This commit adds a `Stream`-based version of `MergedTree::diff()` that just wraps the regular iterator in stream. I updated `jj diff` to use it. I couldn't measure any difference on the command above in the Linux repo. I think that means we can safely use the same `Stream`-based interface regardless of backend, even if we end up needing two different implementations of the `Stream`. We would then be using the wrapped iterator from this commit for local backends, and the new implementation for remote backends. But ideally we can make the remote-friendly implementation fast enough that we don't need two implementations.
This will make it a little faster to update the working copy at Google once we've made `MergedTree::diff_stream()` fetch trees concurrently. (It only makes it a little faster because we still fetch files serially.)
martinvonz
force-pushed
the
push-tokvzroztlxv
branch
from
b998230 to
2e2d1c6
Compare
yuja
approved these changes
Nov 3, 2023
I think I feel the same way. We'll see if run into a deadlock that |
With that comment the motivation is clear. LGTM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
If applicable:
CHANGELOG.md