Concurrent Testing

Concurrent Testing

Concurrency is built into the gem as of version 3.0+ It relies on the parallel_tests gem which should help it run across different architectures.

How it works

The rake tasks check your account for your concurrency limit, load a helper file to configure the desired platforms, then divide your tests up among as many threads as you have concurrent Sauce sessions. It runs each test once for each configured platform, then collates the results.

Platform Configuration

Configuration is done the same way as for standard test runs, with a Sauce.config block.

Platforms are read from the "Browsers" option, and each platform is used once for each test:

  # This will run each test 3 times
  Sauce.config do |c|
    c[:browsers] = [
      ["OSX 10.6", "Firefox", 17],
      ["Windows 7", "Opera", 10],
      ["Windows 7", "Firefox", 20]
    ]

See the configuration guide for more details

RSpec will read configuration details from spec/sauce_helper.rb

Cucumber will read configuration details from features/support/sauce_helper.rb

Note that the Browsers array will function normally during non-parallel tests, eg, by running each test in each platform in turn.

Rake Tasks

Set up

$ rake sauce:install:spec     # Create spec/sauce_helper.rb, add a require for it to spec/spec_helper.rb
$ rake sauce:install:features # Create features/support/sauce_helper.rb

RSpec

$ rake sauce:spec                                               # Run all specs in spec at max concurrency
$ rake sauce:spec specs="spec/dynamic_email.spec"               # Run only dynamic_email.spec at max concurrency
$ rake sauce:spec specs="spec/dynamic_email.spec" concurrency=8 # Run only dynamic_email.spec, at most 8 parallel instances

Cucumber

$ rake sauce:features                                                      # Run all features at max concurrency
$ rake sauce:features features="spec/dynamic_email.feature"                # Run only dynamic_email.feature at max concurrency
$ rake sauce:features features="spec/dynamic_email.feature" concurrency=8  # Run only dynamic_email.feature, at most 8 parallel instances

Task options

features                 Files or locations containing features to run      (default: ./features)
specs                    Files or locations containing specs to run         (default: ./spec)
concurrency              How many concurrent tests to run                   (default:  Your account's max)
test_options             Arbitrary options to pass to cucumber or rspec 
                             eg: test_options='--fail-fast'
parallel_test_options    Arbitrary options to pass to the parallel_tests gem 
                             eg: parallel_test_options='--group-by scenarios'

Changes required to your application

Any behaviour that takes place during a non-concurrent run (say, with $ rspec) will occur during concurrent tests. This means you may need to change your test code or application config somewhat to accommodate parallel testing.

Behaviour that can only happen once, regardless of threading

ParallelTests.first_process? ? do_something : sleep(1)

at_exit do
  if ParallelTests.first_process?
    ParallelTests.wait_for_other_processes_to_finish
    undo_something
  end
end

Databases

If your tests are reliant on database state, you may need to have multiple instances of your database.

One way of doing this is to create a database for each concurrent test you'll run, and use the TEST_ENV_NUMBER environment variable to refer to them.

For rails, there's a built in rake task in the parallel_tests gem, rake parallel:create, and to keep your schema up to date after migrations, rake parallel:create.

You'll need to edit your database.yml:

test:
  database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>

Application Server Port

Rails

If the Rails process is started automatically by the Sauce gem (eg, :start_local_application is set to true in your config), it will use the TEST_ENV_NUMBER to start the server on its own Port.

Other application servers

Figure out how to use a unique port -- Turning ENV["TEST_ENV_NUMBER"] into an int and adding it to a base port number is a reasonable approach.

Sauce Connect

If your tests start Sauce Connect by themselves (eg :start_tunnel or :application_host are set to a truthy value in Sauce.config and your tests use the standard integration), only one instance will be started.

If you want to make sure only one instance of Sauce Connect is started per test:

Sauce::Utilities::Connect.start # All options are passed to the Sauce Connect gem - Try {:quiet => true}

Sauce::Utilities::Connect.close # Closes the tunnel once all other threads are finished

These methods will only perform actions for the first thread, so you can safely call them from all test threads without checking if they're the first process.

Environment variables

• TEST_ENV_NUMBER -- The number of the environment of the current thread. Starts with "", and then carries on up from 2, eg "", 2, 3
• SAUCE_PERFILE_BROWSER -- A JSONified hash listing what environment this process should use for which file under test

References

• Parallel Tests gem -- Extra details on parallel support