Skip to content
Kyle Kamperschroer edited this page Jan 27, 2016 · 19 revisions

The Problem

Managing of secrets in a distributed cloud environment is a challenge. Services need to talk to other services in a safe manner and do so using passwords, keys, certificates, tokens, etc. All can be classified as "secrets." Getting those secrets in the right place at the right time and keeping them secure when they get there is not easy.

To date there are no solutions that have provided all of the following:

  • Ease of use
  • Audit logs
  • Ephemeral secrets
  • Rolling secrets without binary deployments

The old Solutions

The existing solutions can be categorized in a few ways that each have their set of pros and cons. The existing solutions are:

  • Secrets in Source -- Code-time
  • Configuration management deployed secrets (and HSMs) -- Deploy-time
  • Secrets service -- Run-time

Secrets in Source (Code-time)

Bad for so many reasons that it doesn't even feel worth talking about.

Examples

  • Git crypt
  • StackExchange Blackbox
  • Literally writing secrets into the source

Failures of this solution

  • No rolling of secrets without rebuilding and redeploying
  • No easy way without hackery to have unique secrets per environment
  • No auditing of who/what accessed secrets and when
  • Everyone can see the secrets who has access to the source

Advantages

  • The easiest solution to implement

Configuration Management Deployed Secrets (Deploy-time)

This could be using configuration management, deployment tools, or some kind of custom process.

Examples

  • Chef w/ Encrypted Data Bags
  • Puppet
  • Ansible Vault

Failures of this solution

  • No rolling of secrets without deployments
  • No auditing of who/what accessed secrets and when
  • Secrets are on disk, so unauthorized access could expose them

Advantages

  • Still pretty easy to implement
  • Allows for different secrets for different environments

Hardware Security Modules (HSMs) (Deploy-time/run-time)

Encryption keys are backed using actual hardware and can never leave. Encrypted secrets are sent to the HSM to be decrypted when needed.

Examples

  • AWS KMS
  • Azure KeyVault
  • StrongAuth KeyAppliance

Failures of this solution

  • Relatively costly
  • Encrypted secrets still have to be managed in some way
    • Typically just checked in to source, or some kind of variables file per environment
  • Potential single point of failure
  • Rolling keys without deployment not possible

Advantages

  • Encryption keys can never leave the hardware
  • Slightly easier to use

Secrets Service (Run-time)

This is the solution that I believe is safest. Running services call out to some other secret service in some way to request the secret they need.

Examples

  • Square Fuse
  • Lyft Confidant
  • Hashicorp Vault

Failures of this solution

Depends on which one you pick

  • All difficult to setup and install
  • Most are considered a single point of failure
    • Even when clustered, a network partition means your services no longer have access to the secrets they need
  • None have any kind of useful UI

Advantages

  • Rolling secrets without deployments
  • Audit logs are super simple since every secret has to be requested for a centralized location
  • Secrets not on disk (though crappy code could still expose secrets stored in memory in the case of remote code execution)

My Solutions

I'm working on it, but something along the lines of taking Hashicorp Vault and building a UI and simplified installation/configuration process on top of it, and selling that. Vault goes most of the way, but fails at being simple to understand and setup. It's open source and licensed in a way that this is allowable.

Potential Options

Competitor Research

Paid

Free