-
Notifications
You must be signed in to change notification settings - Fork 115
Concurrent Testing
- 'Platform' here means a Sauce Labs platform
- 'Sauce Concurrencies' here means the number of concurrent test slots on Sauce Labs' infrastructure
- 'tests' means specs, tests and features interchangeably
parallel_tests is a sweet gem that lets you run your specs, tests or features in parallel. It's highly configurable and takes away a lot of the concurrency pain on your behalf. You can use the parallel_tests gem as it stands.
The Sauce gem also integrates a modified version of the parallel_tests gem, to take full advantage of the concurrency and multiple platform support Sauce Labs provides. These are accessibly through Rake tasks sauce:spec and sauce:features.
##The Differences The Sauce integration is currently targeted at RSpec and Cucumber, running on a local server you can spin up multiple copies of. It runs a copy of each test for each platform, and divides them up across all the concurrency available to your Sauce Labs account by default
The parallel_tests gem allows you to run TestUnit, RSpec and Cucumber, running on a local server you can spin up multiple copies of. It divides your test files across the cores available to your local machine, by default. It then runs each test file across each platform in serial.
Usually, the limiting factor to how fast your tests perform is your local computational power, so running more tests at once won't provide a speed increase. When testing against a remote server like Sauce, however, the limited factor is the network latency. This means the speed of your tests increases as you run more of them, until you exceed your network bandwidth.
Your local machine has 4 cores, and you've 4 concurrent test slots. You've created 3 tests, Alpha, Beta and Delta. Then, you've specified 4 platforms:
- Chrome on Win7
- Chrome on Linux
- Firefox 18 on Win 8
- Firefox 18 on OS X
| Tool | Core 1 | Core 2 | Core 3 | Core 4 | Sauce Concurrencies Used |
|---|---|---|---|---|---|
| parallel_tests | Alpha on 1, Alpha on 2, Alpha on 3, Alpha on 4 | Beta on 1, Beta on 2, Beta on 3, Beta on 4 | Delta on 1, Delta on 2, Delta on 3, Delta on 4 | 3 | |
| sauce parallel tasks | Alpha on 1, Beta on 1, Delta on 1 | Alpha on 2, Beta on 2, Delta on 3 | Alpha on 3, Beta on 3, Delta on 3 | Alpha on 4, Beta on 4, Delta on 4 | 4 |
If you'd prefer to simply run each test sequentially across each platform, you can use the parallel_tests as it stands.
The parallel_tests gem provides two executables, parallel_rspec and parallel_cucumber, which use the gem's parallelization magic to invoke rspec or cucumber respectively.
If for example, you had a dual core computer, 3 platforms specified, and a features/ directory with 10 .feature files in it, you could run:
% bundle exec parallel_cucumber features
And the command would run two Cucumber processes at the same time, each executing one of the 10 feature files. In those files, each feature would run 3 times, once for each platform.
Don't forget to Check the Parallel Tests documentation.
Concurrency is built into the gem as of version 3.0+
The rake tasks check your account for your concurrency limit, load a helper file to configure the desired platforms, then divide your tests up among as many threads as you have concurrent Sauce sessions. It runs each test once for each configured platform, then collates the results.
Configuration is done the same way as for standard test runs, with a Sauce.config block.
Platforms are read from the "Browsers" option, and each platform is used once for each test:
# This will run each test 3 times
Sauce.config do |c|
c[:browsers] = [
["OSX 10.6", "Firefox", 17],
["Windows 7", "Opera", 10],
["Windows 7", "Firefox", 20]
]See the configuration guide for more details
RSpec will read configuration details from spec/sauce_helper.rb
Cucumber will read configuration details from features/support/sauce_helper.rb
Note that the Browsers array will function normally during non-parallel tests, eg, by running each test in each platform in turn.
$ rake sauce:install:spec # Create spec/sauce_helper.rb, add a require for it to spec/spec_helper.rb
$ rake sauce:install:features # Create features/support/sauce_helper.rb$ rake sauce:spec # Run all specs in spec at max concurrency
$ rake sauce:spec test_files="spec/dynamic_email.spec" # Run only dynamic_email.spec at max concurrency
$ rake sauce:spec test_files="spec/dynamic_email.spec" concurrency=8 # Run only dynamic_email.spec, at most 8 parallel instances$ rake sauce:features # Run all features at max concurrency
$ rake sauce:features test_files="spec/dynamic_email.feature" # Run only dynamic_email.feature at max concurrency
$ rake sauce:features test_files="spec/dynamic_email.feature" concurrency=8 # Run only dynamic_email.feature, at most 8 parallel instancestest_files Files or locations containing features/specs to run (default: ./features or ./spec)
concurrency How many concurrent tests to run (default: Your account's max)
test_options Arbitrary options to pass to cucumber or rspec
eg: test_options='--fail-fast'
parallel_test_options Arbitrary options to pass to the parallel_tests gem
eg: parallel_test_options='--group-by scenarios'
See the Parallel Tests Readme for all available options.
Any behaviour that takes place during a non-concurrent run (say, with $ rspec) will occur during concurrent tests. This means you may need to change your test code or application config somewhat to accommodate parallel testing.
The parallel_tests gem sets the TEST_ENV_NUMBER environment variable for each parallel test group. You can use this to index arrays, make data a bit more unique, keep your logs cleaner and other fun stuff.
For example, sending an informational email informing someone that a test has started and finished
ParallelTests.first_process? ? send_the_email(:start) : sleep(1)
at_exit do
if ParallelTests.first_process?
ParallelTests.wait_for_other_processes_to_finish
send_the_email(:stop)
end
end(*NB: Don't send emails at the start and end of tests from within the tests. CI, baby!)
If your tests are reliant on database state, you may need to have multiple instances of your database.
One way of doing this is to create a database for each concurrent test you'll run, and use the TEST_ENV_NUMBER environment variable to refer to them.
For rails, there's a built in rake task in the parallel_tests gem, rake parallel:create, and to keep your schema up to date after migrations, rake parallel:create.
You'll need to edit your database.yml:
test:
database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>If the Rails process is started automatically by the Sauce gem (eg, :start_local_application is set to true in your config), it will use the TEST_ENV_NUMBER to start the server on its own Port.
Figure out how to use a unique port -- Turning ENV["TEST_ENV_NUMBER"] into an int and adding it to a base port number is a reasonable approach.
If your tests use Sauce Connect, you'll need to use the Sauce Connect compatible ports. They're available as Sauce::Config::POTENTIAL_PORTS. Again, using ENV["TEST_ENV_NUMBER"] as an index is a good approach.
If your tests start Sauce Connect by themselves (eg :start_tunnel or :application_host are set to a truthy value in Sauce.config and your tests use the standard integration), only one instance will be started.
If you want to make sure only one instance of Sauce Connect is started per test:
Sauce::Utilities::Connect.start # All options are passed to the Sauce Connect gem - Try {:quiet => true}
Sauce::Utilities::Connect.close # Closes the tunnel once all other threads are finishedThese methods will only perform actions for the first thread, so you can safely call them from all test threads without checking if they're the first process.
- TEST_ENV_NUMBER -- The number of the environment of the current thread. Starts with "", and then carries on up from 2, eg "", 2, 3
- SAUCE_PERFILE_BROWSER -- A JSONified hash listing what environment this process should use for which file under test
In order to run parallel tests effectively, your test code must be able to run at the same time as other tests.
- Avoid shared or hard-coded data in your tests, if you need an email address or some other form of fake data, generate it yourself (check out the Faker gem).
- If you can use a remote staging server and not have test data collisions, do it - ideally one that closely represents production and can handle many clients simultaneously. Spinning servers up and down is a sucker's game.
- If you have to share a staging server, consider using
ENV[TEST_ENV_NUMBER]to make unique test data; Appending it to strings or using it to index into arrays. - Make sure your tests pass on a single platform first. When testing with Sauce, Chrome is the fastest browser, so if you run your tests with Chrome first then you can confidently say that the app isn't fundamentally broken before spending more time running the tests on slower browsers.
- Are these docs wrong? Can you explain better? Open an issue, we love feedback!
- Parallel Tests gem -- The gem we're driving everything with (with judicious monkey patching)
The sauce_ruby gem will refuse to run in parallel if there are zombie processes from previous parallel test runs. On OSX and Linux, an easy way to terminate these is with the following shell command:
for pid in $(ps -ef | awk '/[T]EST_ENV_NUMBER=/ {print $2}'); do kill -9 $pid; done