#250: Install Script Language Containers (#255)

doc/changes/changes_7.1.0.md

@@ -1,17 +1,18 @@
-# Test Containers for Exasol on Docker 7.1.0, released 2024-??-??
+# Test Containers for Exasol on Docker 7.1.0, released 2024-04-18
 
-Code name:
+Code name: Install Custom SLC
 
 ## Summary
 
+This release allows installing custom Script Language Containers (SLC) into the Exasol Docker container. This is useful for testing UDFs with a SLC containing additional libraries or a newer Java version. See the [user guide](https://github.com/exasol/exasol-testcontainers/blob/main/doc/user_guide/user_guide.md#installing-custom-script-language-containers-slc) for detailed usage instructions.
+
 ### Refactoring
 
 This release updates the following methods to throw the runtime exception `UncheckedSqlException` instead of the checked exception `SqlException`. This simplifies exception handling in clients.
 
 * `ExasolContainer.createConnectionForUser()`
 * `ExasolContainer.createConnection()`
 
-
 ## Features
 
 * #250: Added support for installing custom Script Language Containers
@@ -38,7 +39,9 @@ This release updates the following methods to throw the runtime exception `Unche
 
 ### Test Dependency Updates
 
+* Added `com.exasol:hamcrest-resultset-matcher:1.6.5`
 * Updated `com.exasol:udf-api-java:1.0.4` to `1.0.5`
+* Added `com.jparams:to-string-verifier:1.4.8`
 * Added `nl.jqno.equalsverifier:equalsverifier:3.16.1`
 * Updated `org.mockito:mockito-junit-jupiter:5.10.0` to `5.11.0`
 * Updated `org.testcontainers:junit-jupiter:1.19.5` to `1.19.7`

doc/design.md

@@ -86,6 +86,10 @@ The `HostIpDetector` detects the IP address of the host the container is running
 
 The `SupportInformationRetriever` module provides access to cluster logs, configuration and core-dumps.
 
+## `ScriptLanguageContainerInstaller`
+
+The `ScriptLanguageContainerInstaller` is responsible for installing custom Script Language Containers (SLC).
+
 # Runtime
 
 This section describes the runtime behavior of the software.
@@ -412,6 +416,56 @@ Covers:
 
 Needs: impl, utest, itest
 
+## Install Custom Script Language Containers (SLC)
+`dsn~install-custom-slc~1`
+
+The `ExasolContainer` lets integrators install custom SLCs during container startup.
+
+Covers:
+* [`req~install-custom-slc~1`](system_requirements.md#install-custom-slc)
+
+Needs: impl, utest, itest
+
+### Only Install SLC When Not Yet Done
+`dsn~install-custom-slc.only-if-required~1`
+
+The `ExasolContainer` checks the container status if a given SLC was already installed and skips installation.
+
+Rationale:
+In case a container is reused we want to skip installation to speed-up tests.
+
+Needs: impl, utest, itest
+
+### Install SLC From Local File
+`dsn~install-custom-slc.local-file~1`
+
+ETC allows installing SLCs from a local file.
+
+Rationale:
+This is useful during development when testing an SLC that was not yet released.
+
+Needs: impl, utest
+
+### Install SLC From a URL
+`dsn~install-custom-slc.url~1`
+
+ETC allows installing SLCs from an URL.
+
+Rationale:
+This is useful for running tests during CI with a released SLC.
+
+Needs: impl, utest, itest
+
+### Verify SLC Checksum
+`dsn~install-custom-slc.verify-checksum~1`
+
+ETC verifies the Sha512 checksum of SLCs downloaded from the internet.
+
+Rationale:
+This is required to ensure the integrity of the downloaded file.
+
+Needs: impl, utest, itest
+
 ## Log Access
 
 ### Mapping the Cluster Log Directory to the Host

doc/system_requirements.md

@@ -31,9 +31,9 @@ Integrators integrate their solution with Exasol. To test this, they need a fram
 
 The following list gives you an overview of terms and abbreviations commonly used in ETC documents.
 
-* Container: (Docker container)[https://docs.docker.com/glossary/#container]
+* Container: [Docker container](https://docs.docker.com/glossary/#container)
 * Docker: Container-based virtualization framework
-* Image: (Docker image)[https://docs.docker.com/glossary/#image]
+* Image: [Docker image](https://docs.docker.com/glossary/#image)
 
 ## Features
 
@@ -60,6 +60,13 @@ ETC provides access to the BucketFS service(s) of the Exasol database.
 
 Needs: req
 
+### Install Custom Script Language Containers (SLC)
+`feat~install-custom-slc~1`
+
+ETC allows installing custom SLCs during.
+
+Needs: req
+
 ### Log Access
 `feat~log-access~1`
 
@@ -484,6 +491,22 @@ Covers:
 
 Needs: dsn
 
+### Install Custom SLC
+`req~install-custom-slc~1`
+
+ETC lets Integrators install custom SLCs.
+
+Rationale:
+
+Integrators want to test their integration with a custom Script Language Container (SLC). This is necessary in the following cases:
+* Integrators want to use a later Java version than supported by the default SLC that comes with the Exasol Docker container
+* Integrators want to use additional Python libraries and build a custom SLC with the required libraries
+
+Covers:
+* [`feat~install-custom-slc~1`](#install-custom-script-language-containers-slc)
+
+Needs: dsn
+
 ### Log Access
 
 During the installation process and while running the Exasol installation inside the Docker container produces a lot of log files. Integrators need access to those logs in order to see what is going on inside the Exasol platform.

doc/user_guide/user_guide.md

@@ -363,6 +363,46 @@ The test container knows the cluster setup, including the [bucket credentials](#
 
 Creating and deleting of buckets and BucketFS services is not yet supported by the Exasol test container.
 
+## Installing Custom Script Language Containers (SLC)
+
+Exasol User Defined Functions (UDF) run inside a Script Language Container. The Exasol DB contains SLC that is used by default. If you need to test UDFs against a custom SLC (e.g. containing a newer Java version or additional libraries), you can install a custom SLC.
+
+```java
+// Create Script Language Container definition
+ScriptLanguageContainer slc = ScriptLanguageContainer.builder()
+    .language(Language.JAVA)
+    .localFile(Path.of("/path/to/java17-slc.tar.gz"))
+    .build();
+// Pass definition to ExasolContainer
+ExasolContainer<? extends ExasolContainer<?>> container = new ExasolContainer<>().withReuse(true).withScriptLanguageContainer(slc);
+```
+
+You can also specify a released SLC from [script-languages-release](https://github.com/exasol/script-languages-release/releases):
+
+```java
+ScriptLanguageContainer slc = ScriptLanguageContainer.builder()
+    .language(Language.PYTHON)
+    .slcRelease("7.1.0", "template-Exasol-all-python-3.10_release.tar.gz")
+    .sha512sum("db19e16b9cb5b3d02c44ad6e401eb9296b0483c0078a2e23ac00ad7f26473d115febb0c799a86aed33f49252947a86aa7e8927571a2679131da52e5fc090939c")
+    .build();
+```
+
+You can also specify the URL directly:
+
+```java
+ScriptLanguageContainer slc = ScriptLanguageContainer.builder()
+    .language(Language.PYTHON)
+    .url("https://github.com/exasol/script-languages-release/releases/download/7.1.0/template-Exasol-all-python-3.10_release.tar.gz")
+    .sha512sum("db19e16b9cb5b3d02c44ad6e401eb9296b0483c0078a2e23ac00ad7f26473d115febb0c799a86aed33f49252947a86aa7e8927571a2679131da52e5fc090939c")
+    .build();
+```
+
+Please note:
+* The SHA512 checksum is required to ensure the validity of downloaded files.
+* When reusing a Docker Container, ETC installs the SLC only once.
+* By default, ETC uses SLC aliases `JAVA`, `R` and `PYTHON3` depending on the chosen language. This will overwrite the default SLC for the language. If necessary you can specify a custom alias, e.g. `.alias("JAVA17")`.
+* See the [Exasol DB documentation](https://docs.exasol.com/db/latest/database_concepts/udf_scripts/adding_new_packages_script_languages.htm) for details about installing SLCs.
+
 ## Viewing Cluster Logs
 
 A running Exasol cluster produces a lot of logs. In the Docker version those logs are located internally under `/exa/logs`. Since this happens inside the Docker container, logs by default are not visible in an integration test.

error_code_config.yml

@@ -5,4 +5,4 @@ error-tags:
       - com.exasol.support
       - com.exasol.clusterlogs
       - com.exasol.bucketfs.testcontainers
-    highest-index: 26
+    highest-index: 42

pom.xml

@@ -146,6 +146,18 @@
             <version>3.16.1</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>com.jparams</groupId>
+            <artifactId>to-string-verifier</artifactId>
+            <version>1.4.8</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.exasol</groupId>
+            <artifactId>hamcrest-resultset-matcher</artifactId>
+            <version>1.6.5</version>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <!-- org.testcontainers:testcontainers uses org.slf4j:slf4j-api for logging.
             To get the complete log during tests we redirect slf4j to JUL with this dependency. -->