Josh preparations

[flip1995]

Introduce utils mod in clippy_dev

There was some dependence between the different subcommands of clippy_dev. And
this dependence will increased with the introduction of the sync and release
subcommands. This moves the common functions to a utils module, to decouple
the other modules.

Implement sync subcommand in clippy_dev

Now that JOSH is used to sync, it is much easier to script the sync process.
This introduces the two commands sync pull and sync push. The first one will
pull changes from the Rust repo, the second one will push the changes to the
Rust repo. For details, see the documentation in the book.

Implement release subcommand in clippy_dev

This is a QoL improvement for doing releases. The release bump_version
subcommand increments the version in all relevant Cargo.toml files. The
release commit command will only work with the new Josh syncs. Until then the
old way, using git log has to be used.

---

changelog: none

[flip1995]

flip1995#8

First sync has over 10k commit 😮 See josh-project/josh#1328

[flip1995]

Pro/Con list for Josh:

Pros:

• First sync takes ~1min instead of hours
• Later syncs take seconds, instead of minutes¹
• No more massive merge-commits becoming normal commits on sync-back, messing up git blame and git history.
• Scriptable (this PR), so sync can easily be done by anyone
• No more patched script from some git/git PR (kinda, see cons)

Cons:

• Creates a lot of commits: Josh rustup flip1995/rust-clippy#8 Is there any way to sync fewer merge commits when extracting a subfolder from a large repo? josh-project/josh#1328 (Those would be 1/3 of Clippy's history)
• Patch still required: make rev() detect all parents of the given commits josh-project/josh#1326 (But maintainers said they want to work on merging it and it can easily be installed through cargo, rather than curling the script and placing it in /lib/git-core or similar things)
• SHA1s in Rust repo no longer match SHA1s in Clippy repo. (The only time we needed that was for determining release commits, which can be worked around by including the sync commit in the Rust PR description. This is implemented in this PR)

The only big con is the number of commits. All things considered, the pros outweigh the cons IMO.

Footnotes

1. git subtree push -P src/tools/clippy clippy-local rustup 234.40s user 134.66s system 102% cpu 5:59.71 total (~6 minutes for each sync) ↩

[RalfJung]

I guess it is trading off a few huge commits against a huge number of small merge commits. On the plus side, the josh merge commits do not pollute git blame.

[RalfJung]

josh-project/josh#1329 landed, so a josh patch shouldn't be required any more (but it'd be good to re-test this with the latest josh master to make sure that the final PR that landed still works as intended).

[flip1995]

Thanks, I will re-test this after the next sync on Thursday.

[flip1995]

From the meeting: The move to Josh got accepted. It would be nice to reduce the number of commits though. I will follow up on this and the move once I'm back from my business trip.

[tgross35]

It seems like you have been keeping this very up to date - is this just waiting on a final decision? Or is josh-project/josh#1328 considered a blocker.

[flip1995]

We want to move to Josh, that's already decided. I didn't have the time to follow up with josh-project/josh#1328. But IIUC if that should get implemented, we'd need to force push over the Clippy master, if we want to switch to it later.

[RalfJung]

But IIUC if that should get implemented, we'd need to force push over the Clippy master, if we want to switch to it later.

It's unclear, maybe there are ways to avoid that, e.g. using the rev filter.

[bors]

☔ The latest upstream changes (presumably #13157) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably 0712689) made this pull request unmergeable. Please resolve the merge conflicts.