Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: split java connector tests #19382

Merged
merged 14 commits into from
Nov 14, 2024
Merged

chore: split java connector tests #19382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
cb92b52
split out EsSinkTest
fuyufjh Nov 12, 2024
6c9f23c
split FlinkMockTest
fuyufjh Nov 12, 2024
989fa6a
split HttpSinkTest
fuyufjh Nov 12, 2024
690bfb5
split JDBCSinkTest
fuyufjh Nov 13, 2024
ad14dcd
split FileSinkTest into risingwave-connector-service
fuyufjh Nov 13, 2024
13baf39
rename risingwave-connector-test to risingwave-sink-deltalake-test
fuyufjh Nov 13, 2024
15ad3f3
clean unused dep in risingwave-sink-deltalake-test
fuyufjh Nov 13, 2024
006b461
rename risingwave-source-test to risingwave-source-cdc-test
fuyufjh Nov 13, 2024
3d7aabf
dedup TestUtils.java
fuyufjh Nov 13, 2024
f92f362
avoid recompilation for proto
fuyufjh Nov 13, 2024
62f675f
move tracing to assembly to let UT pass
fuyufjh Nov 13, 2024
f645d22
fix: ClassNotFoundException: org.slf4j.LoggerFactory
fuyufjh Nov 14, 2024
19b8535
fix: NoClassDefFoundError: org/apache/commons/lang3/StringUtils
fuyufjh Nov 14, 2024
5416524
Merge remote-tracking branch 'origin/main' into eric/split_java-conne…
fuyufjh Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions java/connector-node/assembly/assembly.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
<includes>
<!-- The connector service -->
<include>*:risingwave-connector-service</include>
<include>*:tracing</include>

<!-- Source connectors -->
<include>*:risingwave-source-cdc</include>
Expand Down
4 changes: 4 additions & 0 deletions java/connector-node/assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
<groupId>com.risingwave</groupId>
<artifactId>s3-common</artifactId>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>tracing</artifactId>
</dependency>
</dependencies>

<build>
Expand Down
0 ...a/com/risingwave/connector/TestUtils.java → ...a/com/risingwave/connector/TestUtils.java
View file
Open in desktop
File renamed without changes.
16 changes: 10 additions & 6 deletions java/connector-node/risingwave-connector-service/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,6 @@
<groupId>com.risingwave</groupId>
<artifactId>proto</artifactId>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>tracing</artifactId>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>java-binding</artifactId>
Expand All @@ -42,12 +38,18 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>


<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
Expand All @@ -64,6 +66,8 @@
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
</dependency>

<!-- test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
Expand Down
2 changes: 2 additions & 0 deletions ...or-node/risingwave-connector-service/src/main/java/com/risingwave/connector/FileSink.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@
import java.util.UUID;
import java.util.stream.IntStream;

// `FileSink` only serves for testing purpose. It is NOT the file sink in RisingWave docs.
// TODO: consider rename or remove it to avoid confusion
public class FileSink extends SinkWriterBase {
private final FileWriter sinkWriter;

Expand Down
0 ...singwave/connector/sink/FileSinkTest.java → ...singwave/connector/sink/FileSinkTest.java
View file
Open in desktop
File renamed without changes.
0 ...onnector/sink/SinkStreamObserverTest.java → ...onnector/sink/SinkStreamObserverTest.java
View file
Open in desktop
File renamed without changes.
0 ...test/src/test/resources/log4j2.properties → ...vice/src/test/resources/log4j2.properties
View file
Open in desktop
File renamed without changes.
90 changes: 18 additions & 72 deletions ...or-node/risingwave-connector-test/pom.xml → ...de/risingwave-sink-deltalake-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,15 @@
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<name>risingwave-connector-test</name>
<name>risingwave-sink-deltalake-test</name>
<packaging>jar</packaging>
<artifactId>risingwave-connector-test</artifactId>
<artifactId>risingwave-sink-deltalake-test</artifactId>
<description>
This module contains the unit test of Delta Lake Sink. Usually we just put the unit test alongside
the main code, but Delta Lake's test relies on Apache Spark, which contains numerous dependencies and
can easily conflict with ours. By splitting it as a dedicated module, these Spark's dependencies
(especially the stale ones) only affect the test itself.
</description>

<properties>
<iceberg.version>1.0.0</iceberg.version>
Expand All @@ -28,11 +34,6 @@
<artifactId>risingwave-sink-deltalake</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-iceberg</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>s3-common</artifactId>
Expand All @@ -53,14 +54,22 @@
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
Expand Down Expand Up @@ -101,30 +110,8 @@
<version>3.24.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>5.0.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>mysql</artifactId>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>elasticsearch</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>

<!-- Force using jackson-databind 2.13.5, as required by spark-3.2_2.12 -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
Expand All @@ -138,46 +125,5 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-connector-service</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>com.risingwave</groupId>
<artifactId>tracing</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-jdbc</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-es-7</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-cassandra</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-mock-flink-runtime</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-mock-flink-common</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.risingwave</groupId>
<artifactId>risingwave-sink-mock-flink-http-sink</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
0 ...ink/deltalake/DeltaLakeLocalSinkTest.java → ...ink/deltalake/DeltaLakeLocalSinkTest.java
View file
Open in desktop
File renamed without changes.
0 ...k/deltalake/DeltaLakeSinkFactoryTest.java → ...k/deltalake/DeltaLakeSinkFactoryTest.java
View file
Open in desktop
File renamed without changes.
10 changes: 10 additions & 0 deletions java/connector-node/risingwave-sink-deltalake-test/src/test/resources/log4j2.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
rootLogger.level=ERROR
# declare the appender to use
appenders=console
# appender properties
appender.console.type=Console
appender.console.name=stdout
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%t] %c{2}:%L - %m%n
rootLogger.appenderRefs=console
rootLogger.appenderRef.console.ref=stdout
24 changes: 24 additions & 0 deletions java/connector-node/risingwave-sink-es-7/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,30 @@
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>

<!-- test dependencies -->
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>elasticsearch</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>

</dependencies>

</project>
0 ...nector/sink/elasticsearch/EsSinkTest.java → ...nector/sink/elasticsearch/EsSinkTest.java
View file
Open in desktop
File renamed without changes.
10 changes: 10 additions & 0 deletions java/connector-node/risingwave-sink-es-7/src/test/resources/log4j2.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
rootLogger.level=ERROR
# declare the appender to use
appenders=console
# appender properties
appender.console.type=Console
appender.console.name=stdout
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%t] %c{2}:%L - %m%n
rootLogger.appenderRefs=console
rootLogger.appenderRef.console.ref=stdout
38 changes: 38 additions & 0 deletions java/connector-node/risingwave-sink-jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,43 @@
<artifactId>mysql-connector-j</artifactId>
</dependency>


<!-- test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>mysql</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>postgresql</artifactId>
<scope>test</scope>
</dependency>

</dependencies>
</project>
0 ...ave/connector/sink/jdbc/JDBCSinkTest.java → ...ave/connector/sink/jdbc/JDBCSinkTest.java
View file
Open in desktop
File renamed without changes.
10 changes: 10 additions & 0 deletions java/connector-node/risingwave-sink-jdbc/src/test/resources/log4j2.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
rootLogger.level=ERROR
# declare the appender to use
appenders=console
# appender properties
appender.console.type=Console
appender.console.name=stdout
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%t] %c{2}:%L - %m%n
rootLogger.appenderRefs=console
rootLogger.appenderRef.console.ref=stdout
22 changes: 22 additions & 0 deletions java/connector-node/risingwave-sink-mock-flink/risingwave-sink-mock-flink-http-sink/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,28 @@
<artifactId>risingwave-sink-mock-flink-runtime</artifactId>
<scope>provided</scope>
</dependency>

<!-- test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

</project>
0 ...ingwave/connector/sink/http/HttpTest.java → ...ingwave/connector/sink/http/HttpTest.java
View file
Open in desktop
File renamed without changes.
10 changes: 10 additions & 0 deletions ...sink-mock-flink/risingwave-sink-mock-flink-http-sink/src/test/resources/log4j2.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
rootLogger.level=ERROR
# declare the appender to use
appenders=console
# appender properties
appender.console.type=Console
appender.console.name=stdout
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%t] %c{2}:%L - %m%n
rootLogger.appenderRefs=console
rootLogger.appenderRef.console.ref=stdout
22 changes: 22 additions & 0 deletions java/connector-node/risingwave-sink-mock-flink/risingwave-sink-mock-flink-runtime/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,28 @@
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-base</artifactId>
</dependency>

<!-- test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

</project>
0 ...e/connector/sink/flink/FlinkMockTest.java → ...e/connector/sink/flink/FlinkMockTest.java
View file
Open in desktop
File renamed without changes.
10 changes: 10 additions & 0 deletions ...e-sink-mock-flink/risingwave-sink-mock-flink-runtime/src/test/resources/log4j2.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
rootLogger.level=ERROR
# declare the appender to use
appenders=console
# appender properties
appender.console.type=Console
appender.console.name=stdout
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%t] %c{2}:%L - %m%n
rootLogger.appenderRefs=console
rootLogger.appenderRef.console.ref=stdout
Loading
Loading