diff --git a/book/src/development/infrastructure/backport.md b/book/src/development/infrastructure/backport.md index 6920c4e46561..2ebf79fee36d 100644 --- a/book/src/development/infrastructure/backport.md +++ b/book/src/development/infrastructure/backport.md @@ -5,68 +5,107 @@ Backports in Clippy are rare and should be approved by the Clippy team. For example, a backport is done, if a crucial ICE was fixed or a lint is broken to a point, that it has to be disabled, before landing on stable. -Backports are done to the `beta` branch of Clippy. Backports to stable Clippy -releases basically don't exist, since this would require a Rust point release, -which is almost never justifiable for a Clippy fix. +> Note: If you think a PR should be backported you can label it with +> `beta-nominated`. This has to be done before the Thursday the week before the +> release. +## Filtering PRs to backport -## Backport the changes +First, find all labeled PRs using [this filter][beta-accepted-prs]. + +Next, look at each PR individually. There are a few things to check. Those need +some explanation and are quite subjective. Good judgement is required. + +1. **Is the fix worth a backport?** + + This is really subjective. An ICE fix usually is. Moving a lint to a _lower_ + group usually as well. An FP fix usually not (on its own). If a backport is + done anyway, FP fixes might also be included. If the PR has a lot of changes, + backports must be considered more carefully. + +2. **Is the problem that was fixed by the PR already in `beta`?** + + It could be that the problem that was fixed by the PR hasn't made it to the + `beta` branch of the Rust repo yet. If that's the case, and the fix is + already synced to the Rust repo, the fix doesn't need to be backported. If + the fix PR is not synced yet, the fix PR either needs to be "backported" to + the Rust `master` branch or to `beta` in the next backport cycle. + +3. **Is the fix already synced to `master`?** + + The fix must already be synced to the Rust `master` branch. Otherwise, the + next `beta` will be missing this fix again. If it is not yet in `master` it + should probably not be backported. If the backport is really important, do an + out-of-cycle sync first. However, the out-of-cycle sync should be small, + because the changes in that sync will get right into `beta`, without being + tested in `nightly` first. + +[beta-accepted-prs]: https://github.com/rust-lang/rust-clippy/issues?q=label%3Abeta-nominated + +## Preparation + +> Note: All commands in this chapter will be run in the Rust clone. + +Follow the instructions in [defining remotes] to define the `clippy-upstream` +remote in the Rust repository. -Backports are done on the beta branch of the Clippy repository. +After that, fetch the remote with ```bash -# Assuming the current directory corresponds to the Clippy repository -$ git checkout beta -$ git checkout -b backport -$ git cherry-pick # `` is the commit hash of the commit(s), that should be backported -$ git push origin backport +$ git fetch clippy-upstream master ``` -Now you should test that the backport passes all the tests in the Rust -repository. You can do this with: +Then, switch to the `beta` branch: ```bash -# Assuming the current directory corresponds to the Rust repository -$ git checkout beta -# Make sure to change `your-github-name` to your github name in the following command -$ git subtree pull -p src/tools/clippy https://github.com//rust-clippy backport -$ ./x.py test src/tools/clippy +$ git switch beta +$ git fetch upstream +$ git reset --hard upstream/beta ``` -Should the test fail, you can fix Clippy directly in the Rust repository. This -has to be first applied to the Clippy beta branch and then again synced to the -Rust repository, though. The easiest way to do this is: +[defining remotes]: release.md#defining-remotes + +## Backport the changes + +When `bors` merges a PR, the PR is closed with the message + +> `bors` merged commit `` into `rust-lang:master` + +This `` is the commit that needs to be backported. To do that, run the +following command in the clone of the **Rust repository**: ```bash -# In the Rust repository -$ git diff --patch --relative=src/tools/clippy > clippy.patch -# In the Clippy repository -$ git apply /path/to/clippy.patch -$ git add -u -$ git commit -m "Fix rustup fallout" -$ git push origin backport +$ git cherry-pick -m 1 `` ``` -After this, you can open a PR to the `beta` branch of the Clippy repository. +Do this for all PRs that should be backported. +## Open PR in the Rust repository -## Update Clippy in the Rust Repository +Next, open the PR for the backport. Make sure, the PR is opened towards the +`beta` branch and not the `master` branch. The PR description should look like +this: -This step must be done, **after** the PR of the previous step was merged. +``` +[beta] Clippy backports -After the backport landed in the Clippy repository, the branch has to be synced -back to the beta branch of the Rust repository. +r? @Mark-Simulacrum -```bash -# Assuming the current directory corresponds to the Rust repository -$ git checkout beta -$ git checkout -b clippy_backport -$ git subtree pull -p src/tools/clippy https://github.com/rust-lang/rust-clippy beta -$ git push origin clippy_backport +Backports: +- +- ... + + ``` -Make sure to test the backport in the Rust repository before opening a PR. This -is done with `./x.py test src/tools/clippy`. If that passes all tests, open a PR -to the `beta` branch of the Rust repository. In this PR you should tag the -Clippy team member, that agreed to the backport or the `@rust-lang/clippy` team. -Make sure to add `[beta]` to the title of the PR. +Mark is from the release team and they ultimately have to merge the PR before +branching a new `beta` version. Tag them to take care of the backport. Next, +list all the backports and give a short summary what's backported and why it is +worth backporting this. + +## Relabel backported PRs + +When a PR is backported to Rust `beta`, label the PR with `beta-accepted`. This +will then get picked up when [writing the changelog]. + +[writing the changelog]: changelog_update.md#31-include-beta-accepted-prs diff --git a/book/src/development/infrastructure/release.md b/book/src/development/infrastructure/release.md index 98fabf8e89ae..b27270d47180 100644 --- a/book/src/development/infrastructure/release.md +++ b/book/src/development/infrastructure/release.md @@ -7,89 +7,89 @@ Clippy is released together with stable Rust releases. The dates for these releases can be found at the [Rust Forge]. This document explains the necessary steps to create a Clippy release. -1. [Remerge the `beta` branch](#remerge-the-beta-branch) -2. [Update the `beta` branch](#update-the-beta-branch) -3. [Find the Clippy commit](#find-the-clippy-commit) -4. [Tag the stable commit](#tag-the-stable-commit) -5. [Update `CHANGELOG.md`](#update-changelogmd) - -> _NOTE:_ This document is for stable Rust releases, not for point releases. For -> point releases, step 1. and 2. should be enough. +1. [Defining Remotes](#defining-remotes) +1. [Bump Version](#bump-version) +1. [Find the Clippy commit](#find-the-clippy-commit) +1. [Update the `beta` branch](#update-the-beta-branch) +1. [Update the `stable` branch](#update-the-stable-branch) +1. [Tag the stable commit](#tag-the-stable-commit) +1. [Update `CHANGELOG.md`](#update-changelogmd) [Rust Forge]: https://forge.rust-lang.org/ -## Remerge the `beta` branch - -This step is only necessary, if since the last release something was backported -to the beta Rust release. The remerge is then necessary, to make sure that the -Clippy commit, that was used by the now stable Rust release, persists in the -tree of the Clippy repository. +## Defining Remotes -To find out if this step is necessary run +You may want to define the `upstream` remote of the Clippy project to simplify +the following steps. However, this is optional and you can replace `upstream` +with the full URL instead. ```bash -# Assumes that the local master branch of rust-lang/rust-clippy is up-to-date -$ git fetch upstream -$ git branch master --contains upstream/beta +$ git remote add upstream git@github.com:rust-lang/rust-clippy ``` -If this command outputs `master`, this step is **not** necessary. +## Bump Version -```bash -# Assuming `HEAD` is the current `master` branch of rust-lang/rust-clippy -$ git checkout -b backport_remerge -$ git merge upstream/beta -$ git diff # This diff has to be empty, otherwise something with the remerge failed -$ git push origin backport_remerge # This can be pushed to your fork +When a release needs to be done, `cargo test` will fail, if the versions in the +`Cargo.toml` are not correct. During that sync, the versions need to be bumped. +This is done by running: + +``` +$ cargo dev release bump_version ``` -After this, open a PR to the master branch. In this PR, the commit hash of the -`HEAD` of the `beta` branch must exist. In addition to that, no files should be -changed by this PR. +This will increase the version number of each relevant `Cargo.toml` file. After +that, just commit the updated files with: -## Update the `beta` branch +```bash +# XX should be exchanged with the corresponding version +$ git commit -m "Bump Clippy version -> 0.1.XX" **/*Cargo.toml +``` + +## Find the Clippy commit -This step must be done **after** the PR of the previous step was merged. +For both, updating the `beta` and the `stable` branch, the first step is to find +the Clippy commit of the last Clippy sync done in the respective Rust branch. -First, the Clippy commit of the `beta` branch of the Rust repository has to be -determined. +Running the following command will get the commit for the specified branch: ```bash -# Assuming the current directory corresponds to the Rust repository -$ git fetch upstream -$ git checkout upstream/beta -$ BETA_SHA=$(git log --oneline -- src/tools/clippy/ | grep -o "Merge commit '[a-f0-9]*' into .*" | head -1 | sed -e "s/Merge commit '\([a-f0-9]*\)' into .*/\1/g") +$ SHA=$(cargo dev release commit /path/to/rust ) ``` -After finding the Clippy commit, the `beta` branch in the Clippy repository can -be updated. +Where the `/path/to/rust` is a relative path to a Rust clone and the `` +is one of `stable`, `beta`, or `master`. + +## Update the `beta` branch + +After getting the commit of the `beta` branch, the `beta` branch in the Clippy +repository can be updated. ```bash -# Assuming the current directory corresponds to the Clippy repository $ git checkout beta -$ git reset --hard $BETA_SHA +$ git reset --hard $SHA $ git push upstream beta ``` -## Find the Clippy commit +## Update the `stable` branch -The first step is to tag the Clippy commit, that is included in the stable Rust -release. This commit can be found in the Rust repository. +After getting the commit of the `stable` branch, the `stable` branch in the +Clippy repository can be updated. ```bash -# Assuming the current directory corresponds to the Rust repository -$ git fetch upstream # `upstream` is the `rust-lang/rust` remote -$ git checkout 1.XX.0 # XX should be exchanged with the corresponding version -$ SHA=$(git log --oneline -- src/tools/clippy/ | grep -o "Merge commit '[a-f0-9]*' into .*" | head -1 | sed -e "s/Merge commit '\([a-f0-9]*\)' into .*/\1/g") +$ git checkout stable +$ git reset --hard $SHA +$ git push upstream stable ``` -## Tag the stable commit +## Tag the `stable` commit -After finding the Clippy commit, it can be tagged with the release number. +After updating the `stable` branch, tag the HEAD commit and push it to the +Clippy repo. + +> Note: Only push the tag, once the Deploy GitHub action of the `beta` branch is +> finished. Otherwise the deploy for the tag might fail. ```bash -# Assuming the current directory corresponds to the Clippy repository -$ git checkout $SHA $ git tag rust-1.XX.0 # XX should be exchanged with the corresponding version $ git push upstream rust-1.XX.0 # `upstream` is the `rust-lang/rust-clippy` remote ``` @@ -98,22 +98,6 @@ After this, the release should be available on the Clippy [release page]. [release page]: https://github.com/rust-lang/rust-clippy/releases -## Update the `stable` branch - -At this step you should have already checked out the commit of the `rust-1.XX.0` -tag. Updating the stable branch from here is as easy as: - -```bash -# Assuming the current directory corresponds to the Clippy repository and the -# commit of the just created rust-1.XX.0 tag is checked out. -$ git push upstream rust-1.XX.0:stable # `upstream` is the `rust-lang/rust-clippy` remote -``` - -> _NOTE:_ Usually there are no stable backports for Clippy, so this update -> should be possible without force pushing or anything like this. If there -> should have happened a stable backport, make sure to re-merge those changes -> just as with the `beta` branch. - ## Update `CHANGELOG.md` For this see the document on [how to update the changelog]. diff --git a/book/src/development/infrastructure/sync.md b/book/src/development/infrastructure/sync.md index e1fe92f95250..ee62db9f97dc 100644 --- a/book/src/development/infrastructure/sync.md +++ b/book/src/development/infrastructure/sync.md @@ -3,7 +3,7 @@ Clippy currently gets built with a pinned nightly version. In the `rust-lang/rust` repository, where rustc resides, there's a copy of -Clippy that compiler hackers modify from time to time to adapt to changes in the +Clippy that compiler devs modify from time to time to adapt to changes in the unstable API of the compiler. We need to sync these changes back to this repository periodically, and the @@ -15,107 +15,81 @@ done in a bi-weekly basis if there's no urgent changes. This is done starting on the day of the Rust stable release and then every other week. That way we guarantee that we keep this repo up to date with the latest compiler API, and every feature in Clippy is available for 2 weeks in nightly, before it can get -to beta. For reference, the first sync following this cadence was performed the +to beta. For reference, the first sync following this cadence was performed on 2020-08-27. -This process is described in detail in the following sections. For general -information about `subtree`s in the Rust repository see [the rustc-dev-guide][subtree]. +This process is described in detail in the following sections. -## Patching git-subtree to work with big repos +## Installing `josh-proxy` -Currently, there's a bug in `git-subtree` that prevents it from working properly -with the [`rust-lang/rust`] repo. There's an open PR to fix that, but it's -stale. Before continuing with the following steps, we need to manually apply -that fix to our local copy of `git-subtree`. +The sync is done with [JOSH] and fully scripted with `cargo dev sync`. The only +requirement is to install the `josh-proxy` binary from GitHub -You can get the patched version of `git-subtree` from [here][gitgitgadget-pr]. -Put this file under `/usr/lib/git-core` (making a backup of the previous file) -and make sure it has the proper permissions: + -```bash -sudo cp --backup /path/to/patched/git-subtree.sh /usr/lib/git-core/git-subtree -sudo chmod --reference=/usr/lib/git-core/git-subtree~ /usr/lib/git-core/git-subtree -sudo chown --reference=/usr/lib/git-core/git-subtree~ /usr/lib/git-core/git-subtree +```sh +$ RUSTFLAGS="--cap-lints warn" cargo +stable install josh-proxy --git https://github.com/josh-project/josh ``` -> _Note:_ The first time running `git subtree push` a cache has to be built. -> This involves going through the complete Clippy history once. For this you -> have to increase the stack limit though, which you can do with `ulimit -s -> 60000`. Make sure to run the `ulimit` command from the same session you call -> git subtree. - -> _Note:_ If you are a Debian user, `dash` is the shell used by default for -> scripts instead of `sh`. This shell has a hardcoded recursion limit set to -> 1,000. In order to make this process work, you need to force the script to run -> `bash` instead. You can do this by editing the first line of the `git-subtree` -> script and changing `sh` to `bash`. - -## Defining remotes - -You may want to define remotes, so you don't have to type out the remote -addresses on every sync. You can do this with the following commands (these -commands still have to be run inside the `rust` directory): - -```bash -# Set clippy-upstream remote for pulls -$ git remote add clippy-upstream https://github.com/rust-lang/rust-clippy -# Make sure to not push to the upstream repo -$ git remote set-url --push clippy-upstream DISABLED -# Set a local remote -$ git remote add clippy-local /path/to/rust-clippy +[JOSH]: https://josh-project.github.io/josh/ + +## Performing the sync from [`rust-lang/rust`] to Clippy + +Doing the sync now is just running + +``` +$ cargo dev sync pull ``` -> Note: The following sections assume that you have set those remotes with the -> above remote names. +This command will update the nightly toolchain in the `rust-toolchain` file and +will pull the changes from the Rust repository. -## Performing the sync from [`rust-lang/rust`] to Clippy +If there should be merge conflicts, resolve them now and commit with the message +`Merge from rustc`.[^1] -Here is a TL;DR version of the sync process (all the following commands have -to be run inside the `rust` directory): - -1. Clone the [`rust-lang/rust`] repository or make sure it is up-to-date. -2. Checkout the commit from the latest available nightly. You can get it using - `rustup check`. -3. Sync the changes to the rust-copy of Clippy to your Clippy fork: - ```bash - # Be sure to either use a net-new branch, e.g. `sync-from-rust`, or delete the branch beforehand - # because changes cannot be fast forwarded and you have to run this command again. - git subtree push -P src/tools/clippy clippy-local sync-from-rust - ``` - - > _Note:_ Most of the time you have to create a merge commit in the - > `rust-clippy` repo (this has to be done in the Clippy repo, not in the - > rust-copy of Clippy): - ```bash - git fetch upstream # assuming upstream is the rust-lang/rust remote - git checkout sync-from-rust - git merge upstream/master --no-ff - ``` - > Note: This is one of the few instances where a merge commit is allowed in - > a PR. -4. Bump the nightly version in the Clippy repository by changing the date in the - rust-toolchain file to the current date and committing it with the message: - ```bash - git commit -m "Bump nightly version -> YYYY-MM-DD" - ``` -5. Open a PR to `rust-lang/rust-clippy` and wait for it to get merged (to - accelerate the process ping the `@rust-lang/clippy` team in your PR and/or - ask them in the [Zulip] stream.) +> Note: If the version tests fail, refer to [bump version] in the release +> documentation. +Open a PR to `rust-lang/rust-clippy` and if you are a Clippy maintainer, you can +`r+` the PR yourself. If not, change `r? @ghost` to `r? clippy` and a Clippy +maintainer will get assigned. To accelerate the process ping the Clippy team in +[Zulip]. + + +[bump version]: release.md#bump-version [Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/clippy +[^1]: The message is not really important, but consistency is nice. + ## Performing the sync from Clippy to [`rust-lang/rust`] -All the following commands have to be run inside the `rust` directory. +The other direction is done by running + +``` +$ cargo dev sync push /path/to/rust --user +``` + +Where the `/path/to/rust` is a relative path to a Rust clone and the +`` is your GitHub user name. This is required for pushing the sync +to GitHub and opening a PR. + +If everything went right, there will be a GitHub link that has to be used to +open the sync PR in the Rust repository. The PR description must look like this: + +``` +Clippy subtree update + +r? @ghost + +Sync from Clippy commit: rust-lang/rust-clippy@ +``` + +The title must be kept as is, to [tell triagebot] that this is a sync PR. + +The second line must be kept as is, to [find the Clippy commit] during a +release. -1. Make sure you have checked out the latest `master` of `rust-lang/rust`. -2. Sync the `rust-lang/rust-clippy` master to the rust-copy of Clippy: - ```bash - git checkout -b sync-from-clippy - git subtree pull -P src/tools/clippy clippy-upstream master - ``` -3. Open a PR to [`rust-lang/rust`] +[find the Clippy commit]: release.md#find-the-clippy-commit +[tell triagebot]: https://github.com/rust-lang/rust/pull/114157 -[gitgitgadget-pr]: https://github.com/gitgitgadget/git/pull/493 -[subtree]: https://rustc-dev-guide.rust-lang.org/external-repos.html#external-dependencies-subtree [`rust-lang/rust`]: https://github.com/rust-lang/rust