From c98e02917a1c8d416dd4deae17f3471a2a817645 Mon Sep 17 00:00:00 2001 From: ClownXC Date: Sun, 31 Mar 2024 16:56:36 +0800 Subject: [PATCH 1/4] [Doc] Fix some spell errors --- .../engine-jar-storage-mode.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/en/seatunnel-engine/engine-jar-storage-mode.md b/docs/en/seatunnel-engine/engine-jar-storage-mode.md index cdb30e9dc18..2e06d511a8f 100644 --- a/docs/en/seatunnel-engine/engine-jar-storage-mode.md +++ b/docs/en/seatunnel-engine/engine-jar-storage-mode.md @@ -15,7 +15,7 @@ We are committed to ongoing efforts to enhance and stabilize this functionality, We can enable the optimization job submission process, which is configured in the `seatunel.yaml`. After enabling the optimization of the Seatunnel job submission process configuration item, users can use the Seatunnel Zeta engine as the execution engine without placing the connector Jar packages required for task execution or the third-party Jar packages that the connector relies on in each engine `connector` directory. Users only need to place all the Jar packages for task execution on the client that submits the job, and the client will automatically upload the Jars required for task execution to the Zeta engine. It is necessary to enable this configuration item when submitting jobs in Docker or k8s mode, -which can fundamentally solve the problem of large container images caused by the heavy weight of the Seatunnrl Zeta engine. In the image, only the core framework package of the Zeta engine needs to be provided, +which can fundamentally solve the problem of large container images caused by the heavy weight of the Seatunnel Zeta engine. In the image, only the core framework package of the Zeta engine needs to be provided, and then the jar package of the connector and the third-party jar package that the connector relies on can be separately uploaded to the pod for distribution. After enabling the optimization job submission process configuration item, you do not need to place the following two types of Jar packages in the Zeta engine: @@ -26,7 +26,7 @@ COMMON_ PLUGIN_ JARS refers to the third-party Jar package that the connector re When common jars do not exist in Zeta's `lib`, it can upload the local common jars of the client to the `lib` directory of all engine nodes. This way, even if the user does not place a jar on all nodes in Zeta's `lib`, the task can still be executed normally. However, we do not recommend relying on the configuration item of opening the optimization job submission process to upload the third-party Jar package that the connector relies on. -If you use Zeta Engine, please add the the third-party jar package files that the connector relies on to `$SEATUNNEL_HOME/lib/` directory on each node, such as jdbc drivers. +If you use Zeta Engine, please add the third-party jar package files that the connector relies on to `$SEATUNNEL_HOME/lib/` directory on each node, such as jdbc drivers. # ConnectorJar storage strategy @@ -36,13 +36,13 @@ Two different storage strategies provide a more flexible storage mode for Jar fi ## Related configuration -| paramemter | default value | describe | -|-------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------| -| connector-jar-storage-enable | false | Whether to enable uploading the connector Jar package to the engine. The default enabled state is false. | -| connector-jar-storage-mode | SHARED | Engine-side Jar package storage mode selection. There are two optional modes, SHARED and ISOLATED. The default Jar package storage mode is SHARED. | -| connector-jar-storage-path | " " | User-defined Jar package storage path. | -| connector-jar-cleanup-task-interval | 3600s | Engine-side Jar package cleaning scheduled task execution interval. | -| connector-jar-expiry-time | 600s | Engine-side Jar package storage expiration time. | +| parameter | default value | describe | +|--------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------| +| connector-jar-storage-enable | false | Whether to enable uploading the connector Jar package to the engine. The default enabled state is false. | +| connector-jar-storage-mode | SHARED | Engine-side Jar package storage mode selection. There are two optional modes, SHARED and ISOLATED. The default Jar package storage mode is SHARED. | +| connector-jar-storage-path | " " | User-defined Jar package storage path. | +| connector-jar-cleanup-task-interval | 3600s | Engine-side Jar package cleaning scheduled task execution interval. | +| connector-jar-expiry-time | 600s | Engine-side Jar package storage expiration time. | ## IsolatedConnectorJarStorageStrategy @@ -79,7 +79,7 @@ Example: ```yaml jar-storage: - connector-jar-storage-enable：true + connector-jar-storage-enable: true connector-jar-storage-mode: SHARED connector-jar-storage-path: "" connector-jar-cleanup-task-interval: 3600 From 7c2ec52758a086f8f79fd20eda537a064fd73532 Mon Sep 17 00:00:00 2001 From: ClownXC Date: Sun, 31 Mar 2024 17:00:34 +0800 Subject: [PATCH 2/4] code style --- .../en/seatunnel-engine/engine-jar-storage-mode.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/en/seatunnel-engine/engine-jar-storage-mode.md b/docs/en/seatunnel-engine/engine-jar-storage-mode.md index 2e06d511a8f..eff40eac9b4 100644 --- a/docs/en/seatunnel-engine/engine-jar-storage-mode.md +++ b/docs/en/seatunnel-engine/engine-jar-storage-mode.md @@ -36,13 +36,13 @@ Two different storage strategies provide a more flexible storage mode for Jar fi ## Related configuration -| parameter | default value | describe | -|--------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------| -| connector-jar-storage-enable | false | Whether to enable uploading the connector Jar package to the engine. The default enabled state is false. | -| connector-jar-storage-mode | SHARED | Engine-side Jar package storage mode selection. There are two optional modes, SHARED and ISOLATED. The default Jar package storage mode is SHARED. | -| connector-jar-storage-path | " " | User-defined Jar package storage path. | -| connector-jar-cleanup-task-interval | 3600s | Engine-side Jar package cleaning scheduled task execution interval. | -| connector-jar-expiry-time | 600s | Engine-side Jar package storage expiration time. | +| parameter | default value | describe | +|-------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------| +| connector-jar-storage-enable | false | Whether to enable uploading the connector Jar package to the engine. The default enabled state is false. | +| connector-jar-storage-mode | SHARED | Engine-side Jar package storage mode selection. There are two optional modes, SHARED and ISOLATED. The default Jar package storage mode is SHARED. | +| connector-jar-storage-path | " " | User-defined Jar package storage path. | +| connector-jar-cleanup-task-interval | 3600s | Engine-side Jar package cleaning scheduled task execution interval. | +| connector-jar-expiry-time | 600s | Engine-side Jar package storage expiration time. | ## IsolatedConnectorJarStorageStrategy From 3f1184d4b682375e08455a31152cfa8f954046f6 Mon Sep 17 00:00:00 2001 From: ClownXC Date: Mon, 1 Apr 2024 19:20:21 +0800 Subject: [PATCH 3/4] triggerrebuild From 93497f6f2b7f33879baa4baa60be117e8e9de339 Mon Sep 17 00:00:00 2001 From: ClownXC Date: Mon, 1 Apr 2024 20:10:05 +0800 Subject: [PATCH 4/4] code style --- .../rest/RestHttpGetCommandProcessor.java | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java index 5c79838bf0b..79f29575a14 100644 --- a/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java +++ b/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java @@ -353,7 +353,7 @@ private SeaTunnelServer getSeaTunnelServer(boolean shouldBeMaster) { this.textCommandService.getNode().getNodeExtension().createExtensionServices(); SeaTunnelServer seaTunnelServer = (SeaTunnelServer) extensionServices.get(Constant.SEATUNNEL_SERVICE_NAME); - if (shouldBeMaster && !seaTunnelServer.isMasterNode()) { + if (!seaTunnelServer.isMasterNode() && shouldBeMaster) { return null; } return seaTunnelServer; @@ -374,6 +374,17 @@ private JsonObject convertToJson(JobInfo jobInfo, long jobId) { .getSerializationService() .toObject(jobInfo.getJobImmutableInformation())); + ClassLoaderService classLoaderService = getSeaTunnelServer(false).getClassLoaderService(); + ClassLoader classLoader = + classLoaderService.getClassLoader( + jobId, jobImmutableInformation.getPluginJarsUrls()); + LogicalDag logicalDag = + CustomClassLoadedObject.deserializeWithCustomClassLoader( + this.textCommandService.getNode().getNodeEngine().getSerializationService(), + classLoader, + jobImmutableInformation.getLogicalDag()); + classLoaderService.releaseClassLoader(jobId, jobImmutableInformation.getPluginJarsUrls()); + SeaTunnelServer seaTunnelServer = getSeaTunnelServer(true); String jobMetrics; JobStatus jobStatus; @@ -390,25 +401,12 @@ private JsonObject convertToJson(JobInfo jobInfo, long jobId) { getNode().nodeEngine, new GetJobStatusOperation(jobId)) .join()]; - seaTunnelServer = getSeaTunnelServer(false); - } else { jobMetrics = seaTunnelServer.getCoordinatorService().getJobMetrics(jobId).toJsonString(); jobStatus = seaTunnelServer.getCoordinatorService().getJobStatus(jobId); } - ClassLoaderService classLoaderService = seaTunnelServer.getClassLoaderService(); - ClassLoader classLoader = - classLoaderService.getClassLoader( - jobId, jobImmutableInformation.getPluginJarsUrls()); - LogicalDag logicalDag = - CustomClassLoadedObject.deserializeWithCustomClassLoader( - this.textCommandService.getNode().getNodeEngine().getSerializationService(), - classLoader, - jobImmutableInformation.getLogicalDag()); - classLoaderService.releaseClassLoader(jobId, jobImmutableInformation.getPluginJarsUrls()); - jobInfoJson .add(RestConstant.JOB_ID, String.valueOf(jobId)) .add(RestConstant.JOB_NAME, logicalDag.getJobConfig().getName())