Add overrides section to Atmos stack manifests. Update docs

[aknysh]

what

• Add overrides section to Atmos stack manifests
• Update docs https://pr-464.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/components/overrides/

why

Atmos supports the overrides pattern to modify component(s) configuration and behavior using the overrides section in Atmos stack manifests.

You can override the following sections in the component(s) configuration:

• vars
• settings
• env
• command

The overrides section can be used at the global level or at the Terraform and Helmfile levels.

Overrides Schema

The overrides section schema at the global level is as follows:

overrides:
  # Override the ENV variables for the components in the current stack manifest and all its imports
  env: {}
  # Override the settings for the components in the current stack manifest and all its imports
  settings: {}
  # Override the variables for the components in the current stack manifest and all its imports
  vars: {}
  # Override the command to execute for the components in the current stack manifest and all its imports
  command: "<command to execute>"

The overrides section schemas at the Terraform and Helmfile levels are as follows:

terraform:
  overrides:
    env: {}
    settings: {}
    vars: {}
    command: "<command to execute>"

helmfile:
  overrides:
    env: {}
    settings: {}
    vars: {}
    command: "<command to execute>"

You can include the overrides, terraform.overrides and helmfile.overrides sections in any Atmos stack manifest at any level of inheritance to override the configuration of all the Atmos components defined inline in the manifest and all its imports.

Use-case: Overrides for Teams

The overrides pattern is used to override the components only in a particular Atmos stack manifest and all the imported
manifests. This is different from the other configuration sections (e.g. vars, settings, env). If we define a vars, settings or env section at the global, Terraform or Helmfile levels, all the components in the top-level stack will get the updated configurations. On the other hand, if we define an overrides section in a stack manifest, only the components directly defined in the manifest and its imports will get the overridden values, not all the components in the top-level Atmos stack.

This is especially useful when you have Atmos stack manifests split per Teams, each Team manages a set of components, and you need to define a common configuration (or override the existing one) for the components that only a particular Team manages.

For example, we have two Teams: devops and testing.

The devops Team manifest is defined in stacks/teams/devops.yaml:

import:
  # The `devops` Team manages all the components defined in the following stack manifests:
  - catalog/terraform/top-level-component1

The testing Team manifest is defined in stacks/teams/testing.yaml:

import:
  # The `testing` Team manages all the components defined in the following stack manifests:
  - catalog/terraform/test-component
  - catalog/terraform/test-component-override

We can import the two manifests into a top-level stack manifest, e.g. tenant1/dev/us-west-2.yaml:

import:
  - mixins/region/us-west-2
  - orgs/cp/tenant1/dev/_defaults
  # Import all components that the `devops` Team manages
  - teams/devops
  # Import all components managed by the `testing` Team
  - teams/testing

Suppose that we want to change some variables in the vars and env sections and some config in the settings section for all the components that the testing Team manges, but we don't want to affect any components that the devops Team manages.

If we added a global or Terraform level vars, env or settings sections to the top-level manifest stacks/orgs/cp/tenant1/dev/us-west-2.yaml or to the Team manifest stacks/teams/testing.yaml, then all the components in the tenant1/dev/us-west-2 top-level stack would be modified, including those managed by the devops Team.

To solve this, we could individually modify the vars, env and settings sections in all the components managed by the testing Team, but the entire configuration would not be DRY and reusable. That's where the overrides pattern comes into play. To make the configuration DRY and configured only in one place, use the overrides section.

For example, we want to override some values in the env, vars and setings sections for all the components managed by the testing Team:

import:
  # The `testing` Team manages all the components defined in the following stack manifests:
  - catalog/terraform/test-component
  - catalog/terraform/test-component-override

# Global overrides.
# Override the variables, env, command and settings ONLY in the components managed by the `testing` Team.
overrides:
  env:
    # This ENV variable will be added or overridden in all the components managed by the `testing` Team
    TEST_ENV_VAR1: "test-env-var1-overridden"
  settings: {}
  vars: {}

# Terraform overrides.
# Override the variables, env, command and settings ONLY in the Terraform components managed by the `testing` Team.
# The Terraform `overrides` are deep-merged with the global `overrides`
# and takes higher priority (it will override the same keys from the global `overrides`).
terraform:
  overrides:
    settings:
      spacelift:
        # All the components managed by the `testing` Team will have the Spacelift stacks auto-applied
        # if the planning phase was successful and there are no plan policy warnings
        # https://docs.spacelift.io/concepts/stack/stack-settings#autodeploy
        autodeploy: true
    vars:
      # This variable will be added or overridden in all the Terraform components managed by the `testing` Team
      test_1: 1
    # The `testing` Team uses `tofu` instead of `terraform`
    # https://opentofu.org
    # The commands `atmos terraform <sub-command> ...` will execute the `tofu` binary
    command: tofu

# Helmfile overrides.
# Override the variables, env, command and settings ONLY in the Helmfile components managed by the `testing` Team.
# The Helmfile `overrides` are deep-merged with the global `overrides`
# and takes higher priority (it will override the same keys from the global `overrides`).
helmfile:
  overrides:
    env:
      # This ENV variable will be added or overridden in all the Helmfile components managed by the `testing` Team
      TEST_ENV_VAR2: "test-env-var2-overridden"

In the manifest above, we configure the following:

• The global overrides section to override the TEST_ENV_VAR1 ENV variable in the env section. All the Terraform and Helmfile components managed by the testing Team will get the ENV vars updated to test-env-var1-overridden.

• The Terraform-level terraform.overrides section to override some Spacelift configuration in the settings section, a variable in the vars section, and the tofu command to execute instead of terraform in the command section. All the Terraform components managed by the testing Team will be affected by the new values (but not the Helmfile components). The Terraform overrides are deep-merged with the global overrides and takes higher priority (it will override the same keys from the global overrides).

• The Helmfile-level helmfile.overrides section to override an ENV variable in the env section. All the Helmfile components managed by the testing Team will get the new ENV variable value (but not the Terraform components). The Helmfile overrides are deep-merged with the global overrides and takes higher priority (it will override the same keys from the global overrides).

To confirm that the components managed by the testing Team get the new values from the overrides sections, execute the following
commands:

atmos atmos describe component test/test-component -s tenant1-uw2-dev
atmos atmos describe component test/test-component-override -s tenant1-uw2-dev

You should see the following output:

# Final deep-merged `overrides` from all the global `overrides` and Terraform `overrides` sections
overrides:
  command: tofu
  env:
    TEST_ENV_VAR1: test-env-var1-overridden
  settings:
    spacelift:
      autodeploy: true
  vars:
    test_1: 1

# The `command` was overridden with the value from `terraform.overrides.command`
command: tofu

env:
  # The `TEST_ENV_VAR1` ENV variable was overridden with the value from `overrides.env.TEST_ENV_VAR1`
  TEST_ENV_VAR1: test-env-var1-overridden
  TEST_ENV_VAR2: val2

settings:
  spacelift:
    # The `autodeploy` setting was overridden with the value 
    # from `terraform.overrides.settings.spacelift.autodeploy`
    autodeploy: true
    workspace_enabled: true

vars:
  environment: uw2
  namespace: cp
  region: us-west-2
  stage: dev
  tenant: tenant1
  # The `test_1` variable was overridden with the value from `terraform.overrides.vars.test_1`
  test_1: 1

To confirm that the components managed by the devops Team are not affected by the overrides for the testing Team, execute the following command:

atmos atmos describe component top-level-component1 -s tenant1-uw2-dev

You should see the following output:

# The `command` is not overridden
command: terraform

# The component does not get the `overrides` section since it's not defined 
# for the components managed by the `devops` Team
overrides: {}

vars:
  <variables for 'top-level-component1' in the stack 'tenant1-uw2-dev'>

env:
  <ENV variables for 'top-level-component1' in the stack 'tenant1-uw2-dev'>

settings:
  <settings for 'top-level-component1' in the stack 'tenant1-uw2-dev'>

The top-level-component1 component managed by the devops Team does not get affected by the overrides sections for the testing Team, and the sections vars, env, settings and command are not updated with the values from the overrides configuration.

[nitrocode]

LGTM