Testing

Types of tests:

Automated

• CircleCI:
  • We’re using CircleCI to run automated tests as a requirement before merging any PRs. These tests are configured here, and are currently setup to run:
  • Unit tests - configured here; runs the entire test suite.
  • DB tests - configured here, separate set of tests. we’re planning to combine these with the above test suite but for the time being they are broken out (see #410).
  • End to end tests - run using a special dockerfile; hits this test endpoint that’s tied to the legacy server-syncstorage repo.
• Load Testing:
  • We have a python based load testing script that can be run both in an automated fashion and on demand. We’ve historically run it against the staging environment. We’re defining rough pass/no pass thresholds here, and will refine as needed. Based a few recent data points (here and here) where we saw a 0.0006-.0008% failure rate, we are going to start with:
  • PASS - Failure rate less than 0.001%
  • FAIL - Failure greater than or equal to 0.001%.
• Client Integration Tests:
• FxA automation - FxA + Sync
  • https://mozilla.github.io/ecosystem-platform/docs/process/integration-with-fxa
  • https://github.com/mozilla/fxa/tree/master/packages/fxa-content-server/tests/functional
• App services e2e tests
  • https://github.com/mozilla/application-services/tree/master/testing/sync-test
• Desktop Fenix automation
  • https://github.com/mozilla-mobile/fenix/blob/master/app/src/androidTest/java/org/mozilla/fenix/syncintegration/SyncIntegrationTest.kt
• Sync iOS automation
  • https://github.com/mozilla-mobile/firefox-ios/blob/master/XCUITests/IntegrationTests.swift

Manual

• You can perform sync testing for the environments by adjusting configuration on the various sync clients.
• Manual test plans:
  • Durable Sync test plan - Test plan used by Softvision for both the initial rollout and user migration for Durable Sync.
• Regular sync testing against staging is currently performed on a weekly cadence by Softvision as part of the Firefox train releases. Here’s a list of the tests performed.
• If you open up one of those tests (ie here) you can see under “preconditions” there’s a set of testing instructions. Based on those instructions, we can confirm that SV is already testing against staging instance of Durable Sync here.
• If QA finds an issue in staging, they should contact Rachel or Phil for triage. We’ll identify if the issue is worth rolling a new release for staging, or if it’s safe to allow the issue to roll out to production without being immediately addressed. If a new release needs to be created, we need to also be cautious to understand what else may have been merged to master since the previous release so as not to introduce any unwanted changes that may go untested.