Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backend: make read_conflict synchronous again #2453

Merged
merged 1 commit into from
Oct 28, 2023
Merged

Conversation

martinvonz
Copy link
Member

Checklist

If applicable:

  • I have updated CHANGELOG.md
  • I have updated the documentation (README.md, docs/, demos/)
  • I have updated the config schema (cli/src/config-schema.json)
  • I have added tests to cover my changes

lib/src/store.rs Outdated Show resolved Hide resolved
@martinvonz martinvonz force-pushed the push-rxouwowysuqt branch 3 times, most recently from 2019579 to 4658b9d Compare October 28, 2023 12:02
This avoids rust-lang/futures-rs#2090. I
don't think we need to worry about reading legacy conflicts
asynchronously - async is really only useful for Google's backend
right now, and we don't use the legacy format at Google. In
particular, I don't want `MergedTree::value()` to have to be async.
@martinvonz
Copy link
Member Author

I may end up switching to pollster throughout in order to avoid that recursive block_on() limitation in futures. I think it will make transitioning to async simpler. It means that we can simply use block_on() wherever we don't care about async (e.g. because we know it doesn't help with the given backend). I think this PR is reasonable regardless of that, but let me know if you prefer that we instead keep read_conflict() as async and just switch to pollster everywhere.

@yuja
Copy link
Contributor

yuja commented Oct 28, 2023

I may end up switching to pollster throughout in order to avoid that recursive block_on() limitation in futures. I think it will make transitioning to async simpler. It means that we can simply use block_on() wherever we don't care about async (e.g. because we know it doesn't help with the given backend).

I'm okay with switching to pollster (because jj isn't a highly-concurrent app), but I also think the futures' panicking behavior is reasonable. It seems wrong to block async code inside async.

I think this PR is reasonable regardless of that

Yes.

@martinvonz martinvonz merged commit cfcdd71 into main Oct 28, 2023
14 checks passed
@martinvonz martinvonz deleted the push-rxouwowysuqt branch October 28, 2023 23:45
@martinvonz
Copy link
Member Author

I'm okay with switching to pollster (because jj isn't a highly-concurrent app), but I also think the futures' panicking behavior is reasonable. It seems wrong to block async code inside async.

Here's what I was thinking:

  • Our backend traits have async method
  • I want to use an async Stream instead of Iterator for diffing trees
  • The async version of the tree-diffing algorithm is about ~20% slower on a large diff in the Linux repo
  • We can use the current tree-diffing algorithm when the backend is local
  • If we can just wrap the "local" Iterator in a Stream, then we can use the local-friendly algorithm with Git and the remote-friendly algorithm with Google's backend, while still using the same interface and hopefully just paying a small price for the async-ness

Does that make sense?

@martinvonz martinvonz changed the title store: use block_on from pollster when reading conflicts backend: make read_conflict synchronous again Oct 29, 2023
@yuja
Copy link
Contributor

yuja commented Oct 29, 2023

  • Our backend traits have async method
  • I want to use an async Stream instead of Iterator for diffing trees
  • The async version of the tree-diffing algorithm is about ~20% slower on a large diff in the Linux repo
  • We can use the current tree-diffing algorithm when the backend is local
  • If we can just wrap the "local" Iterator in a Stream, then we can use the local-friendly algorithm with Git and the remote-friendly algorithm with Google's backend, while still using the same interface and hopefully just paying a small price for the async-ness

I think that will work, but I don't have async Rust expertise to avoid pitfalls.

If we need to select diff algorithms based on async-ness of backends, it might be better to implement non-async backends without using async fn at all.

  • store.get_tree() calls backend.get_tree()
  • store.get_tree_async() calls backend.get_tree_async()
  • local/git backends implement _async() on top of sync fns, whereas cloud-based backends implement _async() natively

I don't how many layers we have between tree diff and backends, though. If there are many, propagating sync/asnyc-ness wouldn't be feasible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants