Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make ci e2e scripts meta backend agnostic #19527

Open
BugenZhao opened this issue Nov 21, 2024 · 0 comments
Open

make ci e2e scripts meta backend agnostic #19527

BugenZhao opened this issue Nov 21, 2024 · 0 comments
Labels
component/ci component/meta Meta related issue. component/test Test related issue. type/enhancement Improvements to existing implementation.
Milestone

Comments

@BugenZhao
Copy link
Member

BugenZhao commented Nov 21, 2024

In #19152, we enable running regular e2e tests with Postgres and MySQL meta backend in CI. This is achieved by passing different risedev profile in different CI steps.

However, there are other e2e tests like e2e-source-test, e2e-sink-test, and slow-e2e-test, where multiple profiles might be used in the same script. This makes it difficult to cover them using the same approach. It's also easy to miss a step.


We may consider making e2e scripts meta backend agnostic to address this. Here's my proposal:

  1. Specify meta backend to env and do not use concrete meta backend service.
      ci-kafka:
        config-path: src/config/ci.toml
        steps:
          - use: minio
    -     - use: sqlite
          - use: meta-node
    -       meta-backend: sqlite
    +       meta-backend: env
          - use: compute-node
            enable-tiered-cache: true
          - use: frontend
          - use: compactor
          - use: kafka
            user-managed: true
            address: message_queue
            port: 29092
          - use: schema-registry
            user-managed: true
            address: schemaregistry
            port: 8082
  2. Run e2e tests with different env var RISEDEV_SQL_ENDPOINT. Use DB services from the docker-compose CI environment.
  3. risedev-dev will parse the env var to generate CLI args for meta-node service.
  4. Before launching services, risedev-dev executes database-specific commands to reset the database for meta store.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/ci component/meta Meta related issue. component/test Test related issue. type/enhancement Improvements to existing implementation.
Projects
None yet
Development

No branches or pull requests

1 participant