improve script for running e2e source tests

[xxchan]

          Yeah I encounter the same issue, and run my backfill tests separately from e2e tests in `run-backfill-tests.sh`

Originally posted by @kwannoel in #15602 (comment)

[xxchan]

I noticed there are both ci/scripts/run-backfill-tests.sh and src/tests/simulation/tests/integration_tests/backfill_tests.rs. This looks a little repetitive. Is this just to test in two different environments at the same time? Or are there any differences?

I'm thinking maybe we can use the same interface to write tests, and let it run in different environments. (i.e., support non-sim environment for the integration test. In other words, reuse Cluster, Configuration apis) @kwannoel @wangrunji0408 How do you think?

[xxchan]

Some aspects I want to improve

• we can run tests easily both in CI and locally. (Should we utilize docker-compose or risedev?
• we can use similar methods to setup tests, instead of writing shell scripts for each kind of tests..

[xxchan]

Thanks @BugenZhao 's idea

It can be as easy as

• Write system commands in scripts for setup. Use env vars to specify ports
  • We can further write e.g., run_mysql.sh, which uses envvars. Then in slt, we reuse the script, and don't need to copy-paste ports.
• The external services are started via risedev. Note that their env vars are stored in risedev-env. We use risedev slt to execute tests, which will load risedev-env first.
  • Ideally, CI can also use risedev to start services. But we can also just manually specify env vars.
  • optional: Maybe we can use docker-compose to start services.

[wangrunji0408]

Generally I think it's a good idea to have a unified interface for both sim and non-sim env. Maybe risedev can be used as a backend for this API. But I'm not sure how much work it would take.

[BugenZhao]

Related #12451

[kwannoel]

I noticed there are both ci/scripts/run-backfill-tests.sh and src/tests/simulation/tests/integration_tests/backfill_tests.rs. This looks a little repetitive. Is this just to test in two different environments at the same time? Or are there any differences?

I'm thinking maybe we can use the same interface to write tests, and let it run in different environments. (i.e., support non-sim environment for the integration test. In other words, reuse Cluster, Configuration apis) @kwannoel @wangrunji0408 How do you think?

Madsim's environment has some quirks. Running those tests in run-backfill-tests.sh is intentional, I don't want to execute them in madsim's environment.
For instance, time ticks randomly and depends on IO. So when there's sleep involved in the tests, I prefer to use a real-time environment, instead of one that is simulated.

[xxchan]

close in favor of #12451