Release Checklist

Nightly Releases

As of the introduction of PR #2409, nightly releases are automatically built on a daily basis. This is accomplished automatically through use of a parameterized invocation of the nightly workflow using CircleCI's Scheduled Pipelines feature.

One-off builds

In the way the schedule is defined, nightly builds are done from the dev branch. However, the functionality that powers nightly builds can be used to also build from any branch (including PRs) and produce a pre-release, "nightly style" build from any desired commit.

This process can only be done by members of the Apollo Router router GitHub repository with contributor permissions on CircleCI.

To invoke a one-off nightly build:

Go to the CircleCI Pipelines view for this repository](https://app.circleci.com/pipelines/github/apollographql/router)
Click on the "All Branches" drop-down menu and choose a branch you'd like to build from.
Press the "Trigger Pipeline" button in the top-right of the navigation (to the left of the "Project Settings" button).
Expand the "Add Parameters" section.
Add one parameter using the following configuration:

Parameter type: boolean Name: nightly Value: true
Press "Trigger Pipeline"
Wait a couple seconds for the pipeline to begin and show in the list.

To obtain the binary builds from the pipeline which was launched:

Note Built nightlies are only available on the Artifacts for a job within 30 days after the CircleCI pipeline that created them is finished. If you need them after this period, you will need to re-run the pipeline and wait for it to finish again. You can do this by clicking the "Rerun from start" option on the pipeline.

Click on the workflow name: nightly of the newly launched pipeline. In the above steps, this is the pipeline that appeared after step 7.
Click on the job representing the system architecture you'd like to obtain the build binary for. For example, to get the macOS binary, click on build_release-macos_build.
If the job hasn't already finished successfully, wait for the job to finish successfully.
Click on the Artifacts tab.
Click on the link to the .tar.gz file to download the tarball of the build distribution. For example, you might click on a link called artifacts/router-v0.0.0-nightly-20230119-abcd1234-x86_64-apple-darwin.tar.gz for a macOS build done on the 19th of January 2023 from commit hash starting with abcd1234.

This is a list of the things that need to happen during a release.

Build a Release

Most of this will be executing some simple commands but here's a high level understanding and some terminology. There will be a total of 3 pull-requests involved:

a Release PR: this will merge into main. It will be a real merge commit and it should NOT be squashed.
a Release Prep PR: this will merge into the release PR above. It SHOULD be squashed.
Reconciliation PR: a PR that merges main back into dev. It will be a real merge commit and it should NOT be squashed.

The examples below will use the GitHub CLI (gh) to simplify the steps. We can automate it further in the future, but feels like the right level of abstraction right now.

A release can be cut from any branch, but we assume you'll be doing it from dev. If you're just doing a release candidate, you can skip merging it back into main.

Make sure you have cargo installed on your machine and in your PATH.
Pick the version number you are going to release. This project uses Semantic Versioning 2.0.0, so analyze the existing changes in the .changesets/ directory to pick the right next version. (e.g., If there are feat_ changes, it must be a minor version bump. If there are breaking_ changes, it must be a major version bump). Do not release a major version without explicit agreement from core team members.
Checkout the branch you want to cut from. Typically, this is dev, but you could do this from another branch as well.
```
git checkout dev
```
We'll set some environment variables for steps that follow this, to simplify copy and pasting. Be sure to customize these for your own conditions, and set the version you picked in the above step as APOLLO_ROUTER_RELEASE_VERSION:
```
APOLLO_ROUTER_RELEASE_VERSION=#.#.#
APOLLO_ROUTER_RELEASE_GIT_ORIGIN=public
APOLLO_ROUTER_RELEASE_GITHUB_REPO=apollographql/router
```
Make sure you have the latest from the remote before releasing, ensuring you're using the right remote!
```
git pull "${APOLLO_ROUTER_RELEASE_GIT_ORIGIN}"
```
Create a new branch #.#.#. (The #.#.# values should be this release's version, and it is perfectly acceptable to use prerelease semantics, e.g., a branch named 1.5.3-rc.9). To do this using the environment variable we just set, we'll just run the following from the same terminal:
```
git checkout -b "${APOLLO_ROUTER_RELEASE_VERSION}"
```
Push this new branch to the appropriate remote. We will open a PR for it later, but this will be the base for the PR created in the next step). (And --set-upstream will of course track this locally. This is commonly abbreviated as -u.)
```
git push --set-upstream "${APOLLO_ROUTER_RELEASE_GIT_ORIGIN}" "${APOLLO_ROUTER_RELEASE_VERSION}"
```
Create another new branch called prep-#.#.# off of #.#.#. This branch will be used for bumping version numbers and getting review on the changelog. We'll do this using the same environment variable, so you can just run:
```
git checkout -b "prep-${APOLLO_ROUTER_RELEASE_VERSION}"
```
On this new prep-#.#.# branch, run the release automation script using this command to use the environment variable set previously:

Note For this command, GITHUB_TOKEN is not used, but it is still required at the moment, so it's set here to prep. This is a bug in the releasing script that needs to be changed.
```
cargo xtask release prepare $APOLLO_ROUTER_RELEASE_VERSION
```
Running this command will:
- Bump the necessary versions to the version specified, including those in the documentation.
- Migrate the current set of /.changesets/*.md files into /CHANGELOG.md using the version specified.
- Run our compliance checks and update the licenses.html file as appropriate.
- Ensure we're not using any incompatible licenses in the release.
MANUALLY CHECK AND UPDATE the federation-version-support.mdx with the latest version info. Use https://github.com/apollographql/version_matrix to generate the version matrix.
Now, review and stage he changes produced by the previous step. This is most safely done using the --patch (or -p) flag to git add (-u ignores untracked files).
```
git add -up .
```

Now commit those changes locally, using a brief message:

git commit -m "prep release: v${APOLLO_ROUTER_RELEASE_VERSION}"

(Optional) Make local edits to the newly rendered CHANGELOG.md entries to do some initial editoral.

These things should typically be resolved earlier in the review process, but need to be double checked:
- There are no breaking changes.
- Entries are in categories (e.g., Fixes vs Features) that make sense.
- Titles stand alone and work without their descriptions.
- You don't need to read the title for the description to make sense.
- Grammar is good. (Or great! But don't let perfect be the enemy of good.)
- Formatting looks nice when rendered as markdown and follows common convention.

Now push the branch up to the correct remote:

git push --set-upstream "${APOLLO_ROUTER_RELEASE_GIT_ORIGIN}" "${APOLLO_ROUTER_RELEASE_VERSION}"

Programatically create a small temporary file called this_release.md with the changelog details of precisely this release from the CHANGELOG.md:

Note: This file could totally be created by the xtask if we merely decide convention for it and whether we want it checked in or not. It will be used again later in process and, in theory, by CI. Definitely not suggesting this should live on as regex.

perl -0777 \
  -sne 'print "$1\n" if m{
    (?:\#\s               # Look for H1 Markdown (line starting with "# ")
    \[v?\Q$version\E\]    # ...followed by [$version] (optionally with a "v")
                          #    since some versions had that in the past.
    \s.*?\n$)             # ... then "space" until the end of the line.
    \s*                   # Ignore PRE-entry-whitespace
    (.*?)                 # Capture the ACTUAL body of the release.  But do it
                          # in a non-greedy way, leading us to stop when we
                          # reach the next version boundary/heading.
    \s*                   # Ignore POST-entry-whitespace
    (?=^\#\s\[[^\]]+\]\s) # Once again, look for a version boundary.  This is
                          # the same bit at the start, just on one line.
  }msx' -- \
    -version="${APOLLO_ROUTER_RELEASE_VERSION}" \
    CHANGELOG.md >  this_release.md

Now, run this command to generate the header and the PR and keep them in an environment variable:

apollo_prep_release_header="$(
cat <<EOM
> **Note**
>
> When approved, this PR will merge into **the \`${APOLLO_ROUTER_RELEASE_VERSION}\` branch** which will — upon being approved itself — merge into \`main\`.
>
> **Things to review in this PR**:
>  - Changelog correctness (There is a preview below, but it is not necessarily the most up to date.  See the _Files Changed_ for the true reality.)
>  - Version bumps
>  - That it targets the right release branch (\`${APOLLO_ROUTER_RELEASE_VERSION}\` in this case!).
>
---
EOM
)"
apollo_prep_release_notes="$(cat ./this_release.md)"

Use the gh CLI to create the PR, using the previously-set environment variables:

echo "${apollo_prep_release_header}\n${apollo_prep_release_notes}" | gh --repo "${APOLLO_ROUTER_RELEASE_GITHUB_REPO}" pr create -B "${APOLLO_ROUTER_RELEASE_VERSION}" --title "prep release: v${APOLLO_ROUTER_RELEASE_VERSION}" --body-file -

Use the gh CLI to enable auto-squash (NOT auto-merge) on the PR you just opened:

gh --repo "${APOLLO_ROUTER_RELEASE_GITHUB_REPO}" pr merge --squash --body "" -t "prep release: v${APOLLO_ROUTER_RELEASE_VERSION}" --auto "prep-${APOLLO_ROUTER_RELEASE_VERSION}"

🗣️ Solicit feedback from the Router team on the prep PR

Once approved, the PR will squash-merge itself into the next branch.
After the PR has auto-merged, change your local branch back to the _non-_prep branch, pull any changes you (or others) may have added on GitHub :
```
git checkout "${APOLLO_ROUTER_RELEASE_VERSION}"
git pull "${APOLLO_ROUTER_RELEASE_GIT_ORIGIN}"
```

Now, from your local final release branch, open the PR from the branch the prep PR already merged into:

apollo_release_pr_header="$(
cat <<EOM

> **Note**
> **This particular PR should be true-merged to \`main\`.**

This PR represents the merge to \`main\` of the v${APOLLO_ROUTER_RELEASE_VERSION} release.

This PR is **primarily a merge commit**, so reviewing every individual commit shown below is **not necessary** since those have been reviewed in their own PR.

**However!** Some things to review on this PR:

- Does this PR target the right branch? (usually, \`main\`)
- Are the appropriate **version bumps** and **release note edits** in the end of the commit list (or within the last few commits).  In other words, "Did the 'release prep' PR actually land on this branch?"

If those things look good, this PR is good to merge.
EOM
)"
echo "${apollo_release_pr_header}" | gh --repo "${APOLLO_ROUTER_RELEASE_GITHUB_REPO}" pr create -B "main" --title "release: v${APOLLO_ROUTER_RELEASE_VERSION}" --body-file -

Use the gh CLI to enable auto-merge (NOT auto-squash):

gh --repo "${APOLLO_ROUTER_RELEASE_GITHUB_REPO}" pr merge --merge --body "" -t "release: v${APOLLO_ROUTER_RELEASE_VERSION}" --auto "${APOLLO_ROUTER_RELEASE_VERSION}"

🗣️ Solicit approval from the Router team, wait for the PR to pass CI and auto-merge into main
After the PR has merged to main, pull main to your local terminal, and Git tag & push the release:

This process will kick off the bulk of the release process on CircleCI, including building each architecture on its own infrastructure and notarizing the macOS binary.
```
git checkout main
git pull "${APOLLO_ROUTER_RELEASE_GIT_ORIGIN}"
git tag -a "v${APOLLO_ROUTER_RELEASE_VERSION}" -m "${APOLLO_ROUTER_RELEASE_VERSION}" && git push "${APOLLO_ROUTER_RELEASE_GIT_ORIGIN}" "v${APOLLO_ROUTER_RELEASE_VERSION}"
```

Open a PR that reconciles dev:

gh --repo "${APOLLO_ROUTER_RELEASE_GITHUB_REPO}" pr create --title "Reconcile \`dev\` after merge to \`main\` for v${APOLLO_ROUTER_RELEASE_VERSION}" -B dev -H main --body "Follow-up to the v${APOLLO_ROUTER_RELEASE_VERSION} being officially released, bringing version bumps and changelog updates into the \`dev\` branch."

👀 Follow along with the process by going to CircleCI for the repository and clicking on release for the Git tag that appears at the top of the list. Wait for publish_github_release to finish on this job before continuing.
After the CI job has finished for the tag, re-run the perl command from Step 15, which will regenerate the this_release.md with changes that happened in the release review.
Change the links from [@username](https://github.com/username) to @username (TODO: Write more perl here. 😄)

This ensures that contribution credit is clearly displayed using the user avatars on the GitHub Releases page when the notes are published in the next step.
Update the release notes on the now-published GitHub Releases (this needs to be moved to CI, but requires this_release.md which we created earlier):
```
gh release edit v"${APOLLO_ROUTER_RELEASE_VERSION}" -F ./this_release.md
```
Publish the Crate from your local computer from the main branch (this also needs to be moved to CI, but requires changing the release containers to be Rust-enabled and to restore the caches):
```
cargo publish -p apollo-router
```
(Optional) To have a "social banner" for this release, run this htmlq command (cargo install htmlq; its jq for HTML), open the link it produces, copy the image to your clipboard:
```
curl -s "https://github.com/apollographql/router/releases/tag/v${APOLLO_ROUTER_RELEASE_VERSION}" | htmlq 'meta[property="og:image"]' --attribute content
```

Review

Most review comments will be about the changelog. Once the PR is finalized and approved:

Always use Squash and Merge GitHub button.

Tag and build release

This part of the release process is handled by CircleCI, and our binaries are distributed as GitHub Releases. When you push a version tag, it kicks off a workflow that checks out the tag, builds release binaries for multiple platforms, and creates a new GitHub release for that tag.

Wait for tests to pass.
Have your PR merged to main.
Once merged, run git checkout main and git pull.
Sync your local tags with the remote tags by running git tag -d $(git tag) && git fetch --tags
Tag the commit by running either git tag -a v#.#.# -m "#.#.#" (release), or git tag -a v#.#.#-rc.# -m "#.#.#-rc.#" (release candidate)
Run git push --tags.
Wait for CI to pass.

Edit the release

After CI builds the release binaries, a new release will appear on the releases page, click Edit, update the release notes, and save the changes to the release.

If this is a stable release (not a release candidate)

Paste the current release notes from NEXT_CHANGELOG.md into the release body.
Reset the content of NEXT_CHANGELOG.md.

If this is a release candidate

CI should already mark the release as a pre-release. Double check that it's listed as a pre-release on the release's Edit page.
If this is a new rc (rc.0), paste testing instructions into the release notes.
If this is a rc.1 or later, the old release candidate testing instructions should be moved to the latest release candidate testing instructions, and replaced with the following message:
```
This beta release is now out of date. If you previously installed this
release, you should reinstall and see what's changed in the latest
[release](https://github.com/apollographql/router/releases).
```
The new release candidate should then include updated testing instructions with a small changelog at the top to get folks who installed the old release candidate up to speed.

Publish the release to Crates.io

To perform these steps, you'll need access credentials which allow you publishing to Crates.io.
Make sure you are on the Git tag you have published and pushed in the previous step by running git checkout v#.#.# (release) or git checkout v#.#.#-rc.# (release candidate). (You are probably still on this commit)
Change into the apollo-router/ directory at the root of the repository.
Make sure that the README.md in this directory is up to date with any necessary or relevant changes. It will be published as the crates README on Crates.io.
Run cargo publish --dry-run if you'd like to smoke test things
Do the real publish with cargo publish.

Troubleshooting a release

Mistakes happen. Most of these release steps are recoverable if you mess up.

I pushed the wrong tag

Tags and releases can be removed in GitHub. First, remove the remote tag:

git push --delete origin vX.X.X

This will turn the release into a draft and you can delete it from the edit page.

Make sure you also delete the local tag:

git tag --delete vX.X.X

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RELEASE_CHECKLIST.md

RELEASE_CHECKLIST.md

Release Checklist

Nightly Releases

One-off builds

Build a Release

Review

Tag and build release

Edit the release

If this is a stable release (not a release candidate)

If this is a release candidate

Publish the release to Crates.io

Troubleshooting a release

I pushed the wrong tag

Files

RELEASE_CHECKLIST.md

Latest commit

History

RELEASE_CHECKLIST.md

File metadata and controls

Release Checklist

Nightly Releases

One-off builds

Build a Release

Review

Tag and build release

Edit the release

If this is a stable release (not a release candidate)

If this is a release candidate

Publish the release to Crates.io

Troubleshooting a release

I pushed the wrong tag