Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement jj parallelize #3412

Merged
merged 1 commit into from
Apr 5, 2024
Merged

Implement jj parallelize #3412

merged 1 commit into from
Apr 5, 2024

Conversation

emesterhazy
Copy link
Contributor

@emesterhazy emesterhazy commented Mar 31, 2024

This is an implementation of the simple version of jj parallelize from FR #1079.


Parallelize revisions by making them siblings

Running jj parallelize 1::2 will transform the history like this:

3
|             3
2            / \
|    ->     1   2
1            \ /
|             0
0

Each of the target revisions is rebased onto the parents of the root(s) of
the target revset (not to be confused with the repo root). The children of
the head(s) of the target revset are rebased onto the target revisions.

The target revset is the union of the REVISIONS arguments.

The target revset being parallelized must satisfy several conditions,
otherwise the command will fail.

  1. The heads of the target revset must not have different children.
  2. The roots of the target revset must not have different parents.
  3. The parents of all target revisions except the roots must also be
    parallelized. This means that the target revisions must be connected.

@emesterhazy
Copy link
Contributor Author

I still need to add a change log entry for this, but I'd like to get some feedback first. This also depends a commit from the diffbase that exposes a rebase_descendants function from cli/commands/rebase.rs that is also used for jj split --siblings.

cli/src/commands/mod.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/tests/test_parallelize_command.rs Outdated Show resolved Hide resolved
cli/tests/test_parallelize_command.rs Outdated Show resolved Hide resolved
@emesterhazy emesterhazy force-pushed the push-sxvnsknzvxku branch 2 times, most recently from 985c1f0 to 4c5db0a Compare April 1, 2024 16:13
@emesterhazy
Copy link
Contributor Author

@yuja This draft was broken by #3407 since I was using the RevsetExpression returned by parse_union_revsets to get the heads and roots of the user provided revset along with evaluate_revset. RevsetExpressionEvaluator doesn't appear to expose a way to get the heads and roots of a revset.

I'm not very familiar with these APIs and haven't dug into the details yet to figure out how to fix this. If you have a suggestion please let me know.

@emesterhazy
Copy link
Contributor Author

@yuya I got this working again by making WorkspaceCommandHelper::attach_revset_evaluator a public function, but I'm not sure if this is what you'd recommend. Let me know what you think.

@emesterhazy emesterhazy force-pushed the push-sxvnsknzvxku branch 4 times, most recently from 5049074 to fe267f9 Compare April 1, 2024 23:15
Base automatically changed from push-sxvnsknzvxku to main April 1, 2024 23:22
@martinvonz
Copy link
Member

@yuya I got this working again by making WorkspaceCommandHelper::attach_revset_evaluator a public function, but I'm not sure if this is what you'd recommend. Let me know what you think.

We also have callers at Google that use evaluate_revset() with a revset they build up programmatically with some parts coming from parsed user expressions (and they do more than intersection, so RevsetExpressionEvaluator::intersect_with() is not enough). So we will also want evaluate_revset() to be restore or for attach_revset_evaluator() to become public.

@yuja
Copy link
Contributor

yuja commented Apr 1, 2024

@yuya I got this working again by making WorkspaceCommandHelper::attach_revset_evaluator a public function,

For #3410, I suggested making attach_revset_evaluator() public. I don't think the function name is nice, but it would be useful because callers are likely to do .evaluate_to_commits()/_commit_ids() instead of evaluate()....

@emesterhazy emesterhazy force-pushed the parallelize branch 2 times, most recently from 1c214a2 to dc73e84 Compare April 2, 2024 15:11
@emesterhazy
Copy link
Contributor Author

I think this is in pretty good shape now and ready for another review.

cli/tests/test_parallelize_command.rs Outdated Show resolved Hide resolved
cli/tests/test_parallelize_command.rs Show resolved Hide resolved
cli/src/commands/parallelize.rs Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@yuja yuja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, but I don't have much expertise.

cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@PhilipMetzger PhilipMetzger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor things, as I'm not a expert.

cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Show resolved Hide resolved
@emesterhazy emesterhazy force-pushed the parallelize branch 2 times, most recently from 306d5a0 to ffaabd5 Compare April 4, 2024 01:16
@emesterhazy emesterhazy changed the base branch from main to push-xmkqtumzqspt April 4, 2024 01:34
@emesterhazy emesterhazy force-pushed the push-xmkqtumzqspt branch 2 times, most recently from fa59510 to 8de7a17 Compare April 4, 2024 01:51
Base automatically changed from push-xmkqtumzqspt to main April 4, 2024 13:31
@emesterhazy emesterhazy force-pushed the parallelize branch 2 times, most recently from fc8db61 to a8c29c6 Compare April 4, 2024 23:35
@emesterhazy
Copy link
Contributor Author

Ok I have combed this over and I think I resolved all of the correctness problems. I also adjusted the rules about which revsets can be parallelized. The full description is in the commit description / --help output, but here are the rules:

  • The heads of the target revset must not have different children.
  • The roots of the target revset must not have different parents.
  • The parents of all target revisions except the roots must also be parallelized. This means that the target revisions must be connected.

There are a bunch of comments on this PR, some of which are stale. Happy to open a new PR if would make it easier to review the current version.

Copy link
Member

@martinvonz martinvonz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

cli/tests/test_parallelize_command.rs Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
cli/src/commands/parallelize.rs Outdated Show resolved Hide resolved
@emesterhazy
Copy link
Contributor Author

Ok I addressed the latest round of comments. I'll leave this open for a few days in case anyone else has thoughts. We can always make changes later as well.

@martinvonz
Copy link
Member

Ok I addressed the latest round of comments. I'll leave this open for a few days in case anyone else has thoughts. We can always make changes later as well.

Thanks! I don't think you need to leave it open. It's been open for several days already and I haven't heard any objections. As you say, we can adjust things later if necessary. So feel free to merge as far as I'm concerned.

@emesterhazy emesterhazy force-pushed the parallelize branch 2 times, most recently from cba24a4 to 06726f3 Compare April 5, 2024 16:01
@emesterhazy
Copy link
Contributor Author

Ok sounds good. I forgot to push (actually I accidentally pushed a new branch with -c), so the latest version is up now.

I also added one more test case and tweaked the documentation. I'll merge this soon.

Parallelize revisions by making them siblings

Running `jj parallelize 1::2` will transform the history like this:
```text
3
|             3
2            / \
|    ->     1   2
1            \ /
|             0
0
```

Each of the target revisions is rebased onto the parents of the root(s) of
the target revset (not to be confused with the repo root). The children of
the head(s) of the target revset are rebased onto the target revisions.

The target revset is the union of the REVISIONS arguments.

The target revset being parallelized must satisfy several conditions,
otherwise the command will fail.

1. The heads of the target revset must not have different children.
2. The roots of the target revset must not have different parents.
3. The parents of all target revisions except the roots must also be
   parallelized. This means that the target revisions must be connected.
Copy link
Contributor

@PhilipMetzger PhilipMetzger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG

@emesterhazy emesterhazy merged commit 64e242a into main Apr 5, 2024
16 checks passed
@emesterhazy emesterhazy deleted the parallelize branch April 5, 2024 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants