Release Process
This visual outlines our release process, showing from the beginning of the Sprint until the day we release to 100% of users. At the beginning of each Sprint, the previous release will begin to rollout.
Each sprint, we rotate who is responsible for all releng tasks between the team members. This means that the current “Sprint Release Owner” will upgrade a-c, create both LAT’s and manage the staged rollout. All deadlines will be according to the timezone of the current Sprint Release Owner. A releng checklist GitHub issue will be created for each release with all of the required steps. This will allow everyone to see the progress of the release and to be reminded of who the Sprint Release Owner is.
We upgrade our android-components version twice a sprint. The first update is on Wednesday #1 and the second is on Thursday #2. Each upgrade will be to the latest version along with the associated GeckoView nightly version upgrade1. Both upgrades will be included in our weekly beta build (on Wednesday #1) to be tested by QA.
1This version can be found in the a-c changelog under the Gecko link for the current release.
The expectation is that strings will be ready before development on the issue begins. To leave time for localization, the Sprint Release Owner should add and export all new strings on Wednesday # 1.
Instructions for exporting strings can be found here.
We release a LAT every Wednesday. The first LAT will be a beta release that will include two upgrades of a-c. QA will run a subset of regression tests to catch any regressions early in the process. A QA report will be provided with the results of testing at the end of the next day.
The second LAT is after our soft code freeze. This release candidate will go out by the end of the day and then QA will begin testing. Issues should be filed as they are found and any potential release blockers should be raised to the team as soon as possible. A summary (by email) of the testing results should be provided by the last day of the sprint.
Instructions for submitting a LAT through the Amazon Developer Portal can be found here.
Soft code freeze occurs on Wednesday # 2 and is when all sprint work should be complete. If a feature misses soft code freeze, it will be included in the following release. Near the end of the day, a meeting will be held with a UX team member to review the app and receive UX sign-off. If issues are found, the following two days can be used for UX polish. This meeting will also be used to run through new features that are planned for the upcoming sprint to ensure that all UX details are finalized and eng work can begin.
Hard code freeze is on the last day of the sprint and is when the final release build should be ready. The two days between soft and hard code freeze (Thursday and Friday) should be used for, in order of priority:
- Release blockers
- UX polish
- Small follow-up issues
- Small chores
Additionally, on Thursday # 2, if new strings were added, QA will run the screenshot test and send the results to the l10n team for verification. Verification will take up to 4 days.
If any release blockers are not fixed by the end of the day, the issue should be cherry-picked out of the release or if this is not possible, the release will be skipped.
The following are the questions we ask when defining a release blocker:
- Is it a regression? (i.e. is it new to this release?)
- Impact
- Is it a top use case? (e.g. YouTube, navigating the app)
- Does it affect many sites? (as opposed to just one specific site)
- Does it get the user stuck? (as opposed to there being a workaround)
If the answer is ‘yes’ to the majority of these questions, then the issue is likely a release blocker. The Product Manager has the final call on whether it is a release blocker or not.
After a release blocker is fixed, QA will verify the issue is fixed and perform basic sanity testing around the bug.
On the Monday after hard code freeze, the app should be submitted through the Amazon Developer Portal using staged rollout to 1% of users, provided that approval is received from engineering, product, and QA. An email will go out to the wider team to inform of the release, as well as to inform SUMO of new features planned for the upcoming two releases so that any articles needed will be ready. Note: during quarterly planning, all the features that need SUMO will be identified. SUMO needs 1 month of notice for new articles.
Throughout the rollout process, Sentry will be monitored by the QA team for any significant crashes, and they will be raised to the team as they are found.
See here for further information on using Sentry.
On Tuesday, if no significant crashes or issue have occurred, then the staged rollout should be bumped up to 20%. On Wednesday, if the app is still stable, then rollout should increase to 100%. Before each rollout increase, QA should provide their approval.
Instructions for submitting an app through the Amazon Developer Portal can be found here.
If there is a situation where an urgent fix (frequent crasher) is needed to be released, we will create a dot release (x.x.1). To ensure the fix gets out ASAP, this release will not have an associated LAT. The fix will be verified and a subset of regression tests will be run on a local master build. Ideally, the time from creating a PR to release will be no more than 24 hours. The current release owner will be responsible for all release activities.
This describes the current process and future improvements or ideas are gathered here:
https://docs.google.com/document/d/1I_pf7RhR8uC0P_jB8o5Y8teNxUuLlpKiF_FyBW9veFY/edit?usp=sharing (Private Internal Link)