Nightly Tests

Overview

Every night, a set of tests run as part of the TeamCity project Nightlies. These tests have a few common characteristics:

• They set up a temporary CockroachDB cluster and run load against it.
• Their runtime is too long for them to be included in CI.

All nightly tests, except for Jepsen, use Terraform to create and destroy their temporary cluster. It may be wise to remove Terraform in the future, given the cognitive overhead of using a tool that provides much more functionality than we need.

Test Entry Points

TeamCity jobs execute various bash scripts that, in turn, run the relevant tests. These files are named teamcity/build-*.sh. Key files include:

• build/teamcity-nightly-acceptance.sh - Wrapper script for running allocator, continuous load, and backup & restore tests.
• build/teamcity-jepsen.sh - Runs Jepsen tests.

Key Source Files

• pkg/acceptance/terrafarm - terrafarm.(*Farmer) is our thin wrapper around Terraform. It's used by most nightlies to setup, interact with, and destroy the temporary cluster for the test.
• pkg/acceptance/terraform - Contains the Terraform config files. Reference: Terraform configuration docs.
• pkg/acceptance/allocator_test.go - Allocator tests, including the schema change test and test steady 6 nodes.
• pkg/acceptance/continuous_load_test.go - Continuous load tests.

Allocator tests

The allocator tests stress the replica allocator under load. At a high level, they do the following:

1. Create a temporary cluster.
2. Restore tarballs of golden test data on to each node in the cluster.
3. Add new nodes to the cluster. The only current exception to this is the "steady 6 nodes" test.
4. Starts load generators.
5. Wait until the replica allocators reach equilibrium (no replicas added/removed in the last N minutes).
6. The test passes only if the standard deviation of range counts is lower than the threshold (set to 5% of the mean range count). This must happen before TESTTIMEOUT elapses.
7. Destroys the temporary cluster.