Skip to content

Commit

Permalink
Merge pull request #5 from TongchengOpenSource/jar_doc
Browse files Browse the repository at this point in the history
[Fix][Doc] Fix some spell errors
  • Loading branch information
xiaochen-zhou authored Apr 1, 2024
2 parents 15a981c + 93497f6 commit cb97c2c
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 18 deletions.
8 changes: 4 additions & 4 deletions docs/en/seatunnel-engine/engine-jar-storage-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ We are committed to ongoing efforts to enhance and stabilize this functionality,
We can enable the optimization job submission process, which is configured in the `seatunel.yaml`. After enabling the optimization of the Seatunnel job submission process configuration item,
users can use the Seatunnel Zeta engine as the execution engine without placing the connector Jar packages required for task execution or the third-party Jar packages that the connector relies on in each engine `connector` directory.
Users only need to place all the Jar packages for task execution on the client that submits the job, and the client will automatically upload the Jars required for task execution to the Zeta engine. It is necessary to enable this configuration item when submitting jobs in Docker or k8s mode,
which can fundamentally solve the problem of large container images caused by the heavy weight of the Seatunnrl Zeta engine. In the image, only the core framework package of the Zeta engine needs to be provided,
which can fundamentally solve the problem of large container images caused by the heavy weight of the Seatunnel Zeta engine. In the image, only the core framework package of the Zeta engine needs to be provided,
and then the jar package of the connector and the third-party jar package that the connector relies on can be separately uploaded to the pod for distribution.

After enabling the optimization job submission process configuration item, you do not need to place the following two types of Jar packages in the Zeta engine:
Expand All @@ -26,7 +26,7 @@ COMMON_ PLUGIN_ JARS refers to the third-party Jar package that the connector re
When common jars do not exist in Zeta's `lib`, it can upload the local common jars of the client to the `lib` directory of all engine nodes.
This way, even if the user does not place a jar on all nodes in Zeta's `lib`, the task can still be executed normally.
However, we do not recommend relying on the configuration item of opening the optimization job submission process to upload the third-party Jar package that the connector relies on.
If you use Zeta Engine, please add the the third-party jar package files that the connector relies on to `$SEATUNNEL_HOME/lib/` directory on each node, such as jdbc drivers.
If you use Zeta Engine, please add the third-party jar package files that the connector relies on to `$SEATUNNEL_HOME/lib/` directory on each node, such as jdbc drivers.

# ConnectorJar storage strategy

Expand All @@ -36,7 +36,7 @@ Two different storage strategies provide a more flexible storage mode for Jar fi

## Related configuration

| paramemter | default value | describe |
| parameter | default value | describe |
|-------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
| connector-jar-storage-enable | false | Whether to enable uploading the connector Jar package to the engine. The default enabled state is false. |
| connector-jar-storage-mode | SHARED | Engine-side Jar package storage mode selection. There are two optional modes, SHARED and ISOLATED. The default Jar package storage mode is SHARED. |
Expand Down Expand Up @@ -79,7 +79,7 @@ Example:

```yaml
jar-storage:
connector-jar-storage-enabletrue
connector-jar-storage-enable: true
connector-jar-storage-mode: SHARED
connector-jar-storage-path: ""
connector-jar-cleanup-task-interval: 3600
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ private SeaTunnelServer getSeaTunnelServer(boolean shouldBeMaster) {
this.textCommandService.getNode().getNodeExtension().createExtensionServices();
SeaTunnelServer seaTunnelServer =
(SeaTunnelServer) extensionServices.get(Constant.SEATUNNEL_SERVICE_NAME);
if (shouldBeMaster && !seaTunnelServer.isMasterNode()) {
if (!seaTunnelServer.isMasterNode() && shouldBeMaster) {
return null;
}
return seaTunnelServer;
Expand All @@ -374,6 +374,17 @@ private JsonObject convertToJson(JobInfo jobInfo, long jobId) {
.getSerializationService()
.toObject(jobInfo.getJobImmutableInformation()));

ClassLoaderService classLoaderService = getSeaTunnelServer(false).getClassLoaderService();
ClassLoader classLoader =
classLoaderService.getClassLoader(
jobId, jobImmutableInformation.getPluginJarsUrls());
LogicalDag logicalDag =
CustomClassLoadedObject.deserializeWithCustomClassLoader(
this.textCommandService.getNode().getNodeEngine().getSerializationService(),
classLoader,
jobImmutableInformation.getLogicalDag());
classLoaderService.releaseClassLoader(jobId, jobImmutableInformation.getPluginJarsUrls());

SeaTunnelServer seaTunnelServer = getSeaTunnelServer(true);
String jobMetrics;
JobStatus jobStatus;
Expand All @@ -390,25 +401,12 @@ private JsonObject convertToJson(JobInfo jobInfo, long jobId) {
getNode().nodeEngine,
new GetJobStatusOperation(jobId))
.join()];
seaTunnelServer = getSeaTunnelServer(false);

} else {
jobMetrics =
seaTunnelServer.getCoordinatorService().getJobMetrics(jobId).toJsonString();
jobStatus = seaTunnelServer.getCoordinatorService().getJobStatus(jobId);
}

ClassLoaderService classLoaderService = seaTunnelServer.getClassLoaderService();
ClassLoader classLoader =
classLoaderService.getClassLoader(
jobId, jobImmutableInformation.getPluginJarsUrls());
LogicalDag logicalDag =
CustomClassLoadedObject.deserializeWithCustomClassLoader(
this.textCommandService.getNode().getNodeEngine().getSerializationService(),
classLoader,
jobImmutableInformation.getLogicalDag());
classLoaderService.releaseClassLoader(jobId, jobImmutableInformation.getPluginJarsUrls());

jobInfoJson
.add(RestConstant.JOB_ID, String.valueOf(jobId))
.add(RestConstant.JOB_NAME, logicalDag.getJobConfig().getName())
Expand Down

0 comments on commit cb97c2c

Please sign in to comment.