Testing travis CI workflows locally in docker

Habitat uses the free travis-ci.org service (rather than the paid travis-ci.com) one, we're not able to easily log into a CI instance and debug travis-specific issues. Since spinning up new runs takes awhile and consumes limited resources, debugging travis locally is useful. Travis has documentation for running a container based docker image locally, but it's rather sparse, so this document describes the rest of the details necessary for debugging habitat CI tests.

Prerequisites

• Docker installed

Travis docker image setup

➤ docker run --privileged --name travis-debug -it -u travis travisci/ci-amethyst:packer-1512508255-986baf0 /bin/bash -l

The --privileged option is necessary for the studio to successfully bind mount directories such as /dev. The name travis-debug is arbitrary and the 6dd8f2496746 identifier is automatically generated. The choice of docker image was a best-guess for what matched the instance log output for existing travis builds for habitat. For example:

instance: travis-job-b2e18128-2f18-4c14-b371-f199e1b366aa travis-ci-amethyst-trusty-1512508224-986baf0 (via amqp)

The travis docker images can be found on Docker Hub.

Install travis-build tool

This is broken into chunks to indicate where the commands are expected to generate output.

travis@6dd8f2496746:~$ mkdir .travis
travis@6dd8f2496746:~$ cd .travis/
travis@6dd8f2496746:~/.travis$ git clone https://github.com/travis-ci/travis-build.git

travis@6dd8f2496746:~/.travis$ cd travis-build/
travis@6dd8f2496746:~/.travis/travis-build$ gem install travis

This next set of steps is necessary to work around a travis/support (LoadError) known issue.

travis@6dd8f2496746:~/.travis/travis-build$ bundle install

travis@6dd8f2496746:~/.travis/travis-build$ bundler add travis

travis@6dd8f2496746:~/.travis/travis-build$ bundler binstubs travis
travis@6dd8f2496746:~/.travis/travis-build$ cd

Set up habitat sources

travis@6dd8f2496746:~$ git clone https://github.com/habitat-sh/habitat.git habitat-sh/habitat

travis@6dd8f2496746:~$ cd habitat-sh/habitat/
travis@6dd8f2496746:~/habitat-sh/habitat$ git checkout branch/commit/to/test

• Modify .travis.yml since matrix and env/global sections will be ignored
• Remove branches since we're running the tests manually
• Move the contents of include to the top level since we're explicitly choosing the configuration to run
• Remove the openssl command which generates the builder-github-app.pem since our environment lacks $encrypted_f0fe831d6b31_key and $encrypted_f0fe831d6b31_iv (we'll install this file manually)

For example:

branches:
  only:
    - master
    - /^sentinel.+$/
    - /^acceptance_deploy.+$/
    - /^test_development-.*$/
    - /^\d+\.\d+\.\d+$/

os: linux
env:
  global:
    - PATH=$HOME/.cargo/bin:$PATH
    # Habitat Rust program components
    - _RUST_HAB_BIN_COMPONENTS="components/airlock|components/hab|components/hab-butterfly|components/launcher|components/pkg-export-docker|components/pkg-export-kubernetes|components/pkg-export-helm|components/sup"

matrix:
  include:
    - language: rust
      env:
        - COMPONENTS=srv
        - AFFECTED_DIRS="Cargo\.lock|$_RUST_BLDR_BIN_COMPONENTS|$_RUST_BLDR_LIB_COMPONENTS"
      rust: stable
      sudo: required
      addons:
        apt:
          sources:
            - kalakris-cmake
          packages:
            - build-essential
      cache:
        apt: true
        cargo: true
        directories:
          - "$HOME/pkgs"
      before_install:
        - source ./support/ci/rust_env.sh
        - openssl aes-256-cbc -K $encrypted_f0fe831d6b31_key -iv $encrypted_f0fe831d6b31_iv -in ./support/ci/builder-github-app.pem.enc -out /tmp/builder-github-app.pem -d
      script:
        - ./test/builder-api/test.sh

becomes

os: linux

language: rust
env:
  - COMPONENTS=srv
  - AFFECTED_DIRS="Cargo\.lock|$_RUST_BLDR_BIN_COMPONENTS|$_RUST_BLDR_LIB_COMPONENTS"
rust: stable
sudo: required
addons:
  apt:
    sources:
      - kalakris-cmake
    packages:
      - build-essential
cache:
  apt: true
  cargo: true
  directories:
    - "$HOME/pkgs"
before_install:
  - source ./support/ci/rust_env.sh
script:
  - ./test/builder-api/test.sh

Use travis-build to turn .travis.yml into a bash script

travis@6dd8f2496746:~/habitat-sh/habitat$ ~/.travis/travis-build/bin/travis compile > ci.sh

Fix up ci.sh on account of our .travis.yml modifications:

• Search for branch to find the clone command and add branch/commit/to/test like so:

    travis_cmd git\ clone\ --depth\=50\ --branch\=\'branch/commit/to/test\'\ https://github.com/habitat-sh/habitat.git\ habitat-sh/habitat --echo --retry --timing

• Search for Setting environment variables from .travis.yml and add travis_cmd export commands for any env/global entries removed from the original .travis.yml. For example:

env:
  global:
    # Habitat Rust program components
    - _RUST_HAB_BIN_COMPONENTS="components/airlock|components/hab|components/hab-butterfly|components/launcher|components/pkg-export-docker|components/pkg-export-kubernetes|components/pkg-export-helm|components/sup"

becomes:

travis_cmd export\ _RUST_HAB_BIN_COMPONENTS\=\"components/airlock\|components/hab\|components/hab-butterfly\|components/launcher\|components/pkg-export-docker\|components/pkg-export-kubernetes\|components/sup\" --echo

Note that characters need to be escaped in the context of travis_cmd.

• Create /tmp/builder-github-app.pem and/or /tmp/habitat-srv-admin from the credentials in 1Password to account for the openssl call that was removed
• Optionally replace sudo apt-get update -qq 2>&1 >/dev/null with sudo apt-get update so progress of this slow command is observable

Run ci.sh

For the most readable output, just run bash ci.sh, but for debugging purposes bash -x ci.sh is often helpful