Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changing topology and custom clusters in integration tests #1097

Open
Lorak-mmk opened this issue Oct 21, 2024 · 0 comments
Open

Changing topology and custom clusters in integration tests #1097

Lorak-mmk opened this issue Oct 21, 2024 · 0 comments
Assignees
Labels
area/testing Related to tests - unit, integration, or even out of repo

Comments

@Lorak-mmk
Copy link
Collaborator

Lorak-mmk commented Oct 21, 2024

Currently all our integration tests are executed on the same 3-node cluster.
That means we are not testing topology changes at all.

Side note: I think that we are also not adequately testing schema changes - there are 5 occurences of ALTER statement and 6 occurences of DROP statement in tests.

There are ideas like #1078 but it doesn't fully address the issue:

  • Any SCT run takes a loot of time and resources and is not easy to set up. That means it can't be used locally by developers - tests to run locally must finish in reasonable time and be easy to run.
  • It uses cql-stress: it won't t thoroughly test drivers APIs correctness like in-repo tests can.
  • Being out of repo means it is harder to edit / fix: you need to makes changes here, in cql stress and in sct. That's a lot of friction.

For those reasons I mostly view SCT tests as benchmarks (which is something we severely miss!) and as a guard against some regressions.

What I propose to have in-repo topology tests, in steps:

  1. Move all tests that require running Scylla/C* from unit tests to integration tests.
    1.5 Step 1 may require exposing more private stuff in test_utils module. To avoid having it compiled in user code (and to prevent users from accidentaly modifying it) we should move it behind a feature - or even better, a cfg, like Tokio does: https://docs.rs/tokio/latest/tokio/#unstable-features
  2. Create an API for tests to choose the required cluster configuration and other parameters - more on that later.
    For that I'd see a macro that describes the cluster. It could look a bit like this:
#[scylla::integration_test(cluster=Cluster(nodes=3, tablets=off), ks=( parameters for created keyspace), table=(parameters for created table), session = (parameters for created session))]
test_fn(session, ks_name, table_name) {

}
  1. Add a test runner that would collect the test, decide on execution order (to minimize cluster changes) and execute the tests.
    It would use ccm underneath for cluster provisioning. It would also create a keyspace(s) and table(s) required by the test and clean them up after.
    Relevant links:
    https://www.infinyon.com/blog/2021/04/rust-custom-test-harness/
    https://crates.io/crates/inventory
    https://github.com/infinyon/fluvio/tree/master/crates/fluvio-test/src
    https://github.com/infinyon/fluvio/blob/6635b8aae20df796604d60d2706289984bcc8da4/tests/runner/src/fluvio-integration-derive/src/lib.rs
@Lorak-mmk Lorak-mmk self-assigned this Oct 21, 2024
@Lorak-mmk Lorak-mmk added the area/testing Related to tests - unit, integration, or even out of repo label Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/testing Related to tests - unit, integration, or even out of repo
Projects
None yet
Development

No branches or pull requests

1 participant