Skip to content

Integration Tests for Extensions

Jump to bottom
Paul Rogers edited this page Feb 24, 2023 · 1 revision

Much work has been done to revise the Druid integration test framework. Druid has many extensions. At present, the integration test framework works only with core Druid; there is no way for an extension to write a test that uses the framework. This note explains the problem and proposes a fix.

Challenges

  • Maven scripts assume everything is in integration-tests-ex/cases.
  • The Docker compose generator assumes files are in cases/templates and writes to cases/target.
  • Code uses test files that are not (yet) build as a test jar.
  • Docker compose scripts include base files and environment files which reside in cases/cluster.

Simplest Solution

Just allow tests in cases for extensions. Isolate the cases by a category. Since the test exercise only client functionality, they can exist without compile dependencies on the extension.

However, gRPC is special: it must depend on the shaded gRPC library, and we probably don't want cases to have this dependency. So, we do need a special test project.

Possible Solution

  • Establish a IT_MODULE_DIR which points to the Maven module that has a test. It defaults to integration-tests-ex/cases.
  • All scripts honor the above to find specific test files.
  • Scripts are aware of the base integration-tests/cases directory for the common things. (Need a reliable way to find that directory.)
  • Need a mechanism (or reuse an existing one?) to load the extension into the container (don't want to include it in the image)

Steps

  • For each extension that wants to define a test, define an <extension>-it project.
  • Generate the docker-compose.yaml file for an extension. (No dependencies)
  • Build the test jar for cases.
  • The IT project structure is a simplified version of the cases structure.
  • Modify it.sh to be aware of extensions (probably via the IT_MODULE_DIR env var, or via a new command line arg.)
  • Modify cluster.sh to use the env var.
  • Define the extension IT project to depend on the extension and cases.

Details

To be added to a doc file in cases.

Create an Extension IT Module

  • Create a directory called <extension-name>-it in the same directory as <extension-name> (typically extensions-contrib .)
  • Create a pom.xml file. (See below.)
  • Create /src/test/java and templates.
  • Add the new project to the root pom.xml file. (See example below.) Add it after the integration-tests-ex/cases module.
  • Import the project into your IDE.
  • Define a name for your test category as a Java class name version of the extension name: ExtensionName.
  • Create the Category class: org.apache.druid.testsEx.categories.<ExtensionName>.java. (See below.)
  • Create the templates/<ExtensionName>.py template file for your cluster.
    • At the least, add you extension to the load list.
  • Set the IT_MODULE_DIR environment variable to your extension dir:
    export IT_MODULE_DIR=<path-to-your-extension-name-it-dir>
  • Generate the docker-compose.yaml file and inspect that it is correct: .it.sh gen <ExtensionName>
  • Verify that the cluster starts: ./it.sh up <ExtensionName>. Shut it down again with ./it.sh down <ExtensionName>
  • Create a class org.apache.druid.testsEx.Test<Something> to hold your tests.

Example pom.xml file:

TBD

Example change in the root pom.xml file:

        <module>integration-tests-ex/cases</module>
        ...
        <module>extensions-contrib/<extension-name>-it</module>
    </modules>
Clone this wiki locally