Environment Variable and Secrets Management

Configuration follows the Twelve-Factor App methodology Best Practices.

As a best attempt can be made, there is a strict separation of the configuration from code with default values available as Environment Variables. Any default configuration variables are defined in the Dockerfile and are overwritten by project environment files (*.env, *.secret), such as sample_project.env and sample_project.secret, or by secret_getter.

When setting the same environment variable, Docker has an order of precedence for variables.

---

Meta Variables (.env, docker-compose.yml)

Any ${var} found in your docker-compose.yml file can be configured with your bash or .env file. Any variables referenced in the compose file will be replaced by the shell and the .env file.

• If there are duplicate values, the shell has priority over .env.

Order of precedence

• bash

  $ echo $ENV_A
  ENV_A=ENV A from shell
  $ echo $ENV_B
  ENV_B=ENV B from shell

• .env

  ENV_B=ENV B from .env
  ENV_C=ENV C from .env

• docker-compose.yml

  services:
      service_a:
          image: dbmi/myimage:${ENV_A}
          labels:
              - "label1=${ENV_B}"
              - "label2=${ENV_C}"
          environment:
              - CONTAINER_ENV_1=${ENV_A}

• Result: docker-compose config

  services:
      service_a:
          image: myimage:ENV A from shell
          labels:
              - "label1=ENV B from shell"
              - "label2=ENV C from .env"
          environment:
              - CONTAINER_ENV_1=ENV A from shell

NOTE: We do not overwrite container variables with bash or .env variables. See CONTAINER_ENV_1 in the above example.

BEST PRACTICE: Keep container variables in the project's environment or secret file (*.env, *.secret)

We use these meta variables to define service versions, development ports, development volumes, labels, and any other meta stack deployment configurations. Our deployments primarily use .env. We do not use the shell.

Examples

> i2b2/tranSMART docker-compose.yml meta variables

> PIC-SURE docker-compose.yml meta variables

Override with switchenv.sh

You can use the switchenv.sh tool to update the .env meta variables using its service versioning and environment parameters:

Examples

# cd into deployment directory with .env file
$ cd docker-images/deployments/i2b2transmart

# ####
# update service version
# ####
$ ../tools/switchenv.sh --service nginx --version alpine-tmpl.new-version

# -- .env --
# nginx_version=alpine-tmpl.new-version
# ####

---

Service/Container Variables (*.secret, *.env)

---

Any variables used by a service are defined in their respective *.env and *.secret files. For development, any values found in the *.secret are handled as Environment Variables. They are not handled securely.

• If there are duplicate values, order of precedence follows from high to low priority:

  • docker-compose yaml environment tag
  • secret file
  • environment file
  • Dockerfile

Order of precedence

• docker-compose.yml environment tag

  services:
      service_a:
          image: myimage:example
          env_file:
              myproject.env
              myproject.secret
          environment:
              - CONTAINER_ENV_1=ENV 1 from env tag

• *.secret

  CONTAINER_ENV_1=ENV 1 from myproject.secret
  CONTAINER_ENV_2=ENV 2 from myproject.secret

• *.env

  CONTAINER_ENV_2=ENV 2 from myproject.env
  CONTAINER_ENV_3=ENV 3 from myproject.env

• Dockerfile

  FROM alpine:3.7
  
  ENV CONTAINER_ENV_2=ENV 2 from Dockerfile
  ENV CONTAINER_ENV_3=ENV 3 from Dockerfile
  ENV CONTAINER_ENV_4=ENV 4 from Dockerfile
  
  ENTRYPOINT ["env"]

• Result: docker-compose config

  services:
      service_a:
          image: myimage:example
          environment:
              - CONTAINER_ENV_1=ENV 1 from env tag
              - CONTAINER_ENV_2=ENV 2 from myproject.secret
              - CONTAINER_ENV_3=ENV 3 from myproject.env
              - CONTAINER_ENV_4=ENV 4 from Dockerfile

NOTE: Running docker-compose config will not show any Dockerfile ENV defaults. When deploying, if it is not overwritten by the project files, the Dockerfile default value will be used.

To view Dockerfile defaults, run docker inspect on the image, e.g. docker inspect myimage:example

Examples

i2b2/tranSMART

> i2b2/tranSMART sample_project.secret variables

> i2b2/tranSMART sample_project.env variables

PIC-SURE

> PIC-SURE sample_project.secret variables

> PIC-SURE sample_project.env variables

---

Production Secrets (secret_getter)

---

Our production environments uses the golang executable secret_getter.

To handle secrets, the following prerequisites are required:

1. Build your Docker Image appending secret_getter
2. Update the docker-compose yml entrypoint for the service.

Any variables used by a service are defined in their respective *.env and *.secret files. For production, any values found in the *.secret are stored in tmpfs and handled as secrets. secret_getter reads *.secret or uses Vault to retrieve key/value pairs and replace keys found in configuration files.

• If there are duplicate values, order of precedence follows from high to low priority:

  • secret file
  • docker-compose yaml environment tag
  • environment file
  • Dockerfile

For further reading about using Production Secrets, read Use Secret-Getter.

---

Switch Environments (*.env, *.secret) with switchenv.sh

---

You can use the switchenv.sh tool to update which *.env and *.secret files to use for Development and Production deployments:

Examples

# cd into deployment directory with .env file
$ cd docker-images/deployments/i2b2transmart

# ####
# setup to deploy new_project environment
# ####
$ ../tools/switchenv.sh --environment new_project --type transmart

# -- .env --
# COMPOSE_PROJECT_NAME=new_project
# ENV_FILE=new_project.env
# SECRET_FILE=new_project.secret
# STACK_NAME=new_project
# ####

$ ../tools/switchenv.sh --environment other_project --type transmart

# -- .env --
# COMPOSE_PROJECT_NAME=other_project
# ENV_FILE=other_project.env
# SECRET_FILE=other_project.secret
# STACK_NAME=other_project
# ####