docs: add pre-/post-deployments (#5231)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.

internal/pkg/deploy/pipeline.go

@@ -8,21 +8,17 @@ package deploy
 import (
 	"errors"
 	"fmt"
+	"gopkg.in/yaml.v3"
 	"path"
 	"path/filepath"
 	"regexp"
 	"sort"
 	"strings"
 
-	"gopkg.in/yaml.v3"
-
-	"github.com/aws/copilot-cli/internal/pkg/graph"
-
+	"github.com/aws/aws-sdk-go/aws/arn"
 	"github.com/aws/copilot-cli/internal/pkg/config"
-
+	"github.com/aws/copilot-cli/internal/pkg/graph"
 	"github.com/aws/copilot-cli/internal/pkg/manifest"
-
-	"github.com/aws/aws-sdk-go/aws/arn"
 )
 
 const (
@@ -41,7 +37,7 @@ const (
 	StageFullNamePrefix = "DeployTo-"
 )
 
-// Name of the environment varialbes injected into the CodeBuild projects that support pre/post-deployment actions.
+// Name of the environment variables injected into the CodeBuild projects that support pre/post-deployment actions.
 const (
 	envVarNameEnvironmentName = "COPILOT_ENVIRONMENT_NAME"
 	envVarNameApplicationName = "COPILOT_APPLICATION_NAME"

site/content/docs/concepts/pipelines.en.md

@@ -18,7 +18,7 @@ Copilot can set up a CodePipeline for you with a few commands - but before we ju
 
 1. __Source Stage__ - when you push to a configured GitHub, Bitbucket, or CodeCommit repository branch, a new pipeline execution is triggered.
 2. __Build Stage__ - after your source code is pulled from your repository host, your service's container image is built and published to every environment's ECR repository and any input files, such as [addons](../developing/addons/workload.en.md) templates, lambda function zip files, and [environment variable files](../developing/environment-variables.en.md), are uploaded to S3.
-3. __Deploy Stages__ - after your code is built, you can deploy to any or all of your environments, with optional post-deployment tests or manual approvals.
+3. __Deploy Stages__ - after your code is built, you can deploy to any or all of your environments, with optional manual approvals, pre- and post-deployment actions, and/or test commands.
 
 Once you've set up a CodePipeline using Copilot, all you'll have to do is push to your GitHub, Bitbucket, or CodeCommit repository, and CodePipeline will orchestrate the deployments.
 
@@ -55,9 +55,11 @@ This won't create your pipeline, but it will create some local files under `copi
 
 * __Pipeline name__: We suggest naming your pipeline `[repository name]-[branch name]` (press 'Enter' when asked, to accept the default name). This will distinguish it from your other pipelines, should you create multiple, and works well if you follow a pipeline-per-branch workflow.
 
-* __Release order__: You'll be prompted for environments you want to deploy to – select them based on the order you want them to be deployed in your pipeline (deployments happen one environment at a time). You may, for example, want to deploy to your `test` environment first, and then your `prod` environment.
+* __Pipeline type__: You may select either 'Workloads' or '[Environments](../../blogs/release-v120.en.md#continuous-delivery)'; this determines what your pipeline deploys when triggered.
 
-* __Tracking repository__: After you've selected the environments you want to deploy to, you'll be prompted to select which repository you want your CodePipeline to track. This is the repository that, when pushed to, will trigger a pipeline execution. (If the repository you're interested in doesn't show up, you can pass it in using the `--url` flag.)
+* __Release order__: You'll be prompted for environments you want to deploy or deploy to – select them based on the order you want them to be deployed in your pipeline (deployments happen one environment at a time). You may, for example, want to deploy to your `test` environment first, and then your `prod` environment.
+
+* __Tracking repository__: After you've selected the environments you want to deploy or deploy to, you'll be prompted to select which repository you want your CodePipeline to track. This is the repository that, when pushed to, will trigger a pipeline execution. (If the repository you're interested in doesn't show up, you can pass it in using the `--url` flag.)
 
 * __Tracking branch__: After you've selected the repository, Copilot will designate your current local branch as the branch your pipeline will follow. This can be changed in Step 2.
 
@@ -101,9 +103,9 @@ stages:
 ```
 You can see every available configuration option for `manifest.yml` on the [pipeline manifest](../manifest/pipeline.en.md) page.
 
-There are 3 main parts of this file: the `name` field, which is the name of your pipeline, the `source` section, which details the repository and branch to track, and the `stages` section, which lists the environments you want this pipeline to deploy to. You can update this anytime, but you must commit and push the changed files and run `copilot pipeline deploy` afterwards.
+There are 3 main parts of this file: the `name` field, which is the name of your pipeline, the `source` section, which details the repository and branch to track, and the `stages` section, which lists the environments you want this pipeline to deploy or deploy to. You can update this anytime, but you must commit and push the changed files and run `copilot pipeline deploy` afterward.
 
-Typically, you'll update this file if you add new environments you want to deploy to, or want to track a different branch. If you are using CodeStar Connections to connect to your repository and would like to utilize an existing connection rather than let Copilot generate one for you, you may add the connection name here. The pipeline manifest is also where you may add a manual approval step before deployment or commands to run tests (see "Adding Tests," below) after deployment.
+Typically, you'll update this file to change which environments to deploy or deploy workloads to, specify the order of deployments, add actions for the pipeline to run before or after deployment, or change the branch to track. You also may add a manual approval step before deployment or commands to run tests (see [Customization](#customization)). If you are using CodeStar Connections to connect to your repository and would like to utilize an existing connection rather than let Copilot generate one for you, you may add the connection name here.
 
 ### Step 3: Updating the Buildspec (optional)
 
@@ -146,12 +148,57 @@ After creating your pipeline, you can manage the version of Copilot used by your
 ...
 ```
 
-## Adding Tests
+## Customization
+### Manual approval
+To add an approval step, set the `require_approval` field to 'true'. No pre-deployment
+or deployment actions will run without manual intervention via the CodePipeline console.
+### Pre- and Post-deployment Actions
+As of [v1.30.0](../../blogs/release-v130.en.md), you can insert actions into your pipeline, 
+before and/or after each workload or environment deployment. Add these database migration, 
+testing, or other actions right into your pipeline manifest.
+```yaml
+stages:
+  - name: test
+    require_approval: true
+    pre_deployments:
+      db_migration: # The name of this action.
+        buildspec: copilot/pipelines/demo-api-frontend-main/buildspecs/buildspec.yml # The path to the buildspec.
+    deployments: # Optional, ordering of deployments. 
+      orders:
+      warehouse:
+      frontend:
+        depends_on: [orders, warehouse]
+    post_deployments:
+      db_migration:
+        buildspec: copilot/pipelines/demo-api-frontend-main/buildspecs/post_buildspec.yml
+      integration:
+        buildspec: copilot/pipelines/demo-api-frontend-main/buildspecs/integ-buildspec.yml
+        depends_on: [db_migration] # Optional, ordering of actions.
+```
+In the `buildspec` manifest field, add the path, relative to your project root, 
+of your [buildspec file](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html).
+The Copilot environment variables `$COPILOT_APPLICATION_NAME` and `$COPILOT_ENVIRONMENT_NAME` are available
+for use within these buildspecs.
+
+You may specify the run order of the actions using the `depends_on` subfield, just like you would to indicate your desired [order of deployments](#ordering).  
+
+!!! info
+    The CodeBuild projects generated for pre- and post-deployments and test commands are deployed in the same region as the pipeline and app. To access the VPC of the environment being deployed or deployed to, use Copilot commands in your pre-/post-deployment action buildspec or directly in your test commands.
+
+### Ordering
+The `deployments` field enables you to specify the deployment order of 
+workloads or environments (depending on the type of pipeline). If this 
+is not specified, deployments run in parallel. (See [this blog post](../../blogs/release-v118.en.md#controlling-order-of-deployments-in-a-pipeline) for more info.)
+
+### Testing
+If your post-deployment testing requires only a handful of commands and doesn't necessarily
+warrant wiring up a separate buildspec, utilize the `test_commands` field.
 
-Of course, one of the most important parts of a pipeline is the automated testing. To add tests, such as integration or end-to-end tests, that run after a deployment stage, include those commands in the `test_commands` section. If all the tests succeed, your change is promoted to the next stage.
+!!! warning
+    The `post_deployments` and `test_commands` fields within a stage are mutually exclusive.
 
-Adding `test_commands` generates a CodeBuild project with the [aws/codebuild/amazonlinux2-x86_64-standard:3.0](https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html) image - so most commands from Amazon Linux 2 (including `make`) are available for use.
 
+Pre-deployments, post-deployments, and test commands generate CodeBuild projects with the [aws/codebuild/amazonlinux2-x86_64-standard:4.0](https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html) image, so most commands from Amazon Linux 2 (including `make`) are available for use. 
 Are your tests configured to run inside a Docker container? Copilot's test commands CodeBuild project supports Docker, so `docker build` commands are available as well.
 
 In the example below, the pipeline will run the `make test` command (in your source code directory) and only promote the change to the prod stage if that command exits successfully.
@@ -166,15 +213,18 @@ source:
     repository: https://github.com/kohidave/demo-api-frontend
 
 stages:
-    -
-      name: test
+    - name: test
       # A change will only deploy to the production stage if the
       # make test and echo commands exit successfully.
       test_commands:
         - make test
         - echo "woo! Tests passed"
-    -
-      name: prod
+    - name: prod
 ```
 !!! info
-    AWS's own Nathan Peck provides a great example of a Copilot pipeline, paying special attention to `test_commands`: https://aws.amazon.com/blogs/containers/automatically-deploying-your-container-application-with-aws-copilot/
+    AWS's own Nathan Peck provides a great example of a Copilot pipeline, paying special attention to `test_commands`: https://aws.amazon.com/blogs/containers/automatically-deploying-your-container-application-with-aws-copilot/
+
+### Pipeline Overrides
+If all of these options for custom configuration still don't give you the pipeline you'd like,
+you can use Copilot's "break the glass" solution, [pipeline overrides](../../blogs/release-v129.en.md#pipeline-overrides), with the 
+[CDK](../developing/overrides/cdk.en.md) or [YAML](../developing/overrides/yamlpatch.en.md) to change the pipeline's CloudFormation template.

site/content/docs/manifest/pipeline.en.md

@@ -23,6 +23,9 @@ List of all available properties for a Copilot pipeline manifest. To learn more
         stages: 
           - # By default all workloads are deployed concurrently within a stage.
             name: test
+            pre_deployments:
+              db_migration:
+                buildspec: ./buildspec.yml
             test_commands:
               - make integ-test
               - echo "woo! Tests passed"
@@ -134,10 +137,10 @@ Configuration for CodeBuild project.
 <span class="parent-field">build.</span><a id="build-image" href="#build-image" class="field">`image`</a> <span class="type">String</span>  
 The URI that identifies the Docker image to use for this build project. As of now, `aws/codebuild/amazonlinux2-x86_64-standard:3.0` is used by default.
 
-<span class="parent-field">build.</span><a id="build-buildspec" href="#build-buildspec" class="field">`buildspec`</a> <span class="type">String</span>
-Optional. The URI that identifies a buildspec to use for this build project. By default, Copilot will generate one for you, located at `copilot/pipelines/[your pipeline name]/buildspec.yml`.
+<span class="parent-field">build.</span><a id="build-buildspec" href="#build-buildspec" class="field">`buildspec`</a> <span class="type">String</span>  
+Optional. The path to a buildspec file, relative to the project root, to use for this build project. By default, Copilot will generate one for you, located at `copilot/pipelines/[your pipeline name]/buildspec.yml`.
 
-<span class="parent-field">build.</span><a id="build-additional-policy" href="#build-additional-policy" class="field">`additional_policy.`</a><a id="policy-document" href="#policy-document" class="field">`PolicyDocument`</a> <span class="type">Map</span>
+<span class="parent-field">build.</span><a id="build-additional-policy" href="#build-additional-policy" class="field">`additional_policy.`</a><a id="policy-document" href="#policy-document" class="field">`PolicyDocument`</a> <span class="type">Map</span>  
 Optional. Specify an additional policy document to add to the build project role.
 The additional policy document can be specified in a map in YAML, for example:
 ```yaml
@@ -177,7 +180,29 @@ Ordered list of environments that your pipeline will deploy to.
 The name of an environment to deploy your services to.
 
 <span class="parent-field">stages.</span><a id="stages-approval" href="#stages-approval" class="field">`requires_approval`</a> <span class="type">Boolean</span>  
-Optional. Indicates whether to add a manual approval step before the deployment. Defaults to `false`.
+Optional. Indicates whether to add a manual approval step before the deployment (or the pre-deployment actions, if you have added any). Defaults to `false`.
+
+<span class="parent-field">stages.</span><a id="stages-predeployments" href="#stages-predeployments" class="field">`pre_deployments`</a> <span class="type">Map</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+Optional. Add actions to be executed before deployments.
+```yaml
+stages:
+  - name: <env name>
+    pre_deployments:
+      <action name>:
+        buildspec: <path to local buildspec>
+        depends_on: [<other action's name>, ...]
+```
+<span class="parent-field">stages.pre_deployments.</span><a id="stages-predeployments-name" href="#stages-predeployments-name" class="field">`<name>`</a> <span class="type">Map</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+Name of the pre-deployment action.
+
+<span class="parent-field">stages.pre_deployments.`<name>`.</span><a id="stages-predeployments-buildspec" href="#stages-predeployments-buildspec" class="field">`buildspec`</a> <span class="type">String</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+The path to a buildspec file, relative to the project root, to use for this build project.
+
+<span class="parent-field">stages.pre_deployments.`<name>`.</span><a id="stages-predeployments-dependson" href="#stages-predeployments-dependson" class="field">`depends_on`</a> <span class="type">Array of Strings</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+Optional. Names of other pre-deployment actions that should be deployed prior to deploying this action. Defaults to no dependencies.
+
+!!! info
+    For more on pre- and post-deployments, see the [v1.30.0 blog post](../../blogs/release-v130.en.md) and the [Pipelines](../concepts/pipelines.en.md) page.
 
 <span class="parent-field">stages.</span><a id="stages-deployments" href="#stages-deployments" class="field">`deployments`</a> <span class="type">Map</span>  
 Optional. Control which CloudFormation stacks to deploy and their order.  
@@ -223,8 +248,8 @@ Finally, if `deployments` isn't specified, by default Copilot will deploy all yo
 <span class="parent-field">stages.deployments.</span><a id="stages-deployments-name" href="#stages-deployments-name" class="field">`<name>`</a> <span class="type">Map</span>  
 Name of the job or service to deploy.  
 
-<span class="parent-field">stages.deployments.`<name>`.</span><a id="stages-deployments-dependson" href="#stages-deployments-dependson" class="field">`depends_on`</a> <span class="type">Array of Strings</span>    
-Optional. Name of other job or services that should be deployed prior to deploying this microservice. Defaults to no dependencies.  
+<span class="parent-field">stages.deployments.`<name>`.</span><a id="stages-deployments-dependson" href="#stages-deployments-dependson" class="field">`depends_on`</a> <span class="type">Array of Strings</span>  
+Optional. Name of other jobs or services that should be deployed prior to deploying this microservice. Defaults to no dependencies.  
 
 <span class="parent-field">stages.deployments.`<name>`.</span><a id="stages-deployments-stackname" href="#stages-deployments-stackname" class="field">`stack_name`</a> <span class="type">String</span>  
 Optional. Name of the stack to create or update. Defaults to `<app name>-<stage name>-<deployment name>`.  
@@ -236,5 +261,24 @@ Optional. Path to the CloudFormation template generated during the `build` phase
 <span class="parent-field">stages.deployments.`<name>`.</span><a id="stages-deployments-templateconfig" href="#stages-deployments-templatepath" class="field">`template_config`</a> <span class="type">String</span>  
 Optional. Path to the CloudFormation template configuration generated during the `build` phase. Defaults to `infrastructure/<deployment name>-<stage name>.params.json`.
 
+<span class="parent-field">stages.</span><a id="stages-postdeployments" href="#stages-postdeployments" class="field">`post_deployments`</a> <span class="type">Map</span><span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+Optional. Add actions to be executed after deployments. Mutually exclusive with `stages.test_commands`.
+```yaml
+stages:
+  - name: <env name>
+    post_deployments:
+      <action name>:
+        buildspec: <path to local buildspec>
+        depends_on: [<other action's name>, ...]
+```
+<span class="parent-field">stages.post_deployments.</span><a id="stages-postdeployments-name" href="#stages-postdeployments-name" class="field">`<name>`</a> <span class="type">Map</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+Name of the post-deployment action.
+
+<span class="parent-field">stages.post_deployments.`<name>`.</span><a id="stages-postdeployments-buildspec" href="#stages-postdeployments-buildspec" class="field">`buildspec`</a> <span class="type">String</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>  
+The path to a buildspec file, relative to the project root, to use for this build project.
+
+<span class="parent-field">stages.post_deployments.`<name>`.</span><a id="stages-postdeployments-depends_on" href="#stages-postdeployments-dependson" class="field">`depends_on`</a> <span class="type">Array of Strings</span> <span class="version">Added in [v1.30.0](../../blogs/release-v130.en.md#deployment-actions)</span>   
+Optional. Names of other post-deployment actions that should be deployed prior to deploying this action. Defaults to no dependencies.
+
 <span class="parent-field">stages.</span><a id="stages-test-cmds" href="#stages-test-cmds" class="field">`test_commands`</a> <span class="type">Array of Strings</span>  
-Optional. Commands to run integration or end-to-end tests after deployment. Defaults to no post-deployment validations.
+Optional. Commands to run integration or end-to-end tests after deployment. Defaults to no post-deployment validations. Mutually exclusive with `stages.post_deployment`.