Merge branch 'master' into 29159

yydeng626 · Jan 30, 2024 · 5ab48b2 · 5ab48b2
2 parents 27e2749 + 8646c49
commit 5ab48b2
Show file tree

Hide file tree

Showing 122 changed files with 1,245 additions and 750 deletions.
diff --git a/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md b/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.cn.md
@@ -13,7 +13,7 @@ chapter = true
 在解析并加载 YAML 文件为 ShardingSphere 的元数据后，
 会再次通过[模式配置](../../../java-api/mode)的相关配置决定下一步行为。讨论两种情况：
 1. 元数据持久化仓库中不存在 ShardingSphere 的元数据，本地元数据将被存储到元数据持久化仓库。
-2. 元数据持久化仓库中已存在与本地元数据不同的 ShardingSphere 的元数据，本地元数据将被元数据持久化仓库的元数据覆盖。
+2. 元数据持久化仓库中已存在 ShardingSphere 的元数据，无论是否与本地元数据相同，本地元数据将被元数据持久化仓库的元数据覆盖。
 
 对元数据持久化仓库的配置需参考[元数据持久化仓库](../../../../common-config/builtin-algorithm/metadata-repository)。
 
@@ -33,5 +33,51 @@ chapter = true
 用例：
 - `jdbc:shardingsphere:absolutepath:/path/to/config.yaml`
 
+### 从类路径中加载包含环境变量的配置文件
+
+加载 classpath 中包含环境变量的 config.yaml 配置文件的 JDBC URL，通过 `jdbc:shardingsphere:classpath-environment:` 前缀识别。
+配置文件为 `xxx.yaml`，配置文件格式与 [YAML 配置](../../../yaml-config)基本一致。
+在涉及的 YAML 文件中，允许通过环境变量设置特定YAML属性的值，并配置可选的默认值。这常用于 Docker Image 的部署场景。
+环境变量的名称和其可选的默认值通过`::`分割，在最外层通过`$${`和`}`包裹。
+
+讨论两种情况。
+1. 当对应的环境变量不存在时，此 YAML 属性的值将被设置为`::`右侧的默认值。
+2. 当对应的环境变量和`::`右侧的默认值均不存在时，此属性将被设置为空。
+
+假设存在以下一组环境变量，
+1. 存在环境变量`FIXTURE_JDBC_URL`为`jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`。
+2. 存在环境变量`FIXTURE_USERNAME`为`sa`。
+
+则对于以下 YAML 文件的截取片段，
+```yaml
+ds_1:
+    dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+    driverClassName: $${FIXTURE_DRIVER_CLASS_NAME::org.h2.Driver}
+    jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use}
+    username: $${FIXTURE_USERNAME::}
+    password: $${FIXTURE_PASSWORD::}
+```
+此 YAML 截取片段将被解析为，
+```yaml
+ds_1:
+    dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+    driverClassName: org.h2.Driver
+    jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
+    username: sa
+    password:
+```
+
+用例：
+- `jdbc:shardingsphere:classpath-environment:config.yaml`
+
+### 从绝对路径中加载包含环境变量的配置文件
+
+加载绝对路径中包含环境变量的 config.yaml 配置文件的 JDBC URL，通过 `jdbc:shardingsphere:absolutepath-environment:` 前缀识别。
+配置文件为 `xxx.yaml`，配置文件格式与`jdbc:shardingsphere:classpath-environment:`一致。
+与 `jdbc:shardingsphere:classpath-environment:` 的区别仅在于 YAML 文件的加载位置。
+
+用例：
+- `jdbc:shardingsphere:absolutepath-environment:/path/to/config.yaml`
+
 ### 其他实现
 具体可参考 https://github.com/apache/shardingsphere-plugin 。
diff --git a/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md b/...l/shardingsphere-jdbc/yaml-config/jdbc-driver/known-implementation/_index.en.md
@@ -12,10 +12,11 @@ allows YAML configuration files to be fetched from multiple sources and File Sys
 
 After parsing and loading the YAML file into ShardingSphere's metadata,
 The next behavior will be determined again through the relevant configuration of [Mode Configuration](../../../java-api/mode). Discuss two situations:
-1. ShardingSphere’s metadata does not exist in the metadata persistence warehouse, and local metadata will be stored in the metadata persistence warehouse.
-2. Metadata of ShardingSphere that is different from local metadata already exists in the metadata persistence warehouse, and the local metadata will be overwritten by the metadata of the metadata persistence warehouse.
+1. ShardingSphere’s metadata does not exist in the metadata repository, and local metadata will be stored in the metadata repository.
+2. ShardingSphere’s metadata already exists in the metadata repository, regardless of whether it is the same as the local metadata, 
+the local metadata will be overwritten by the metadata of the metadata repository.
 
-For the configuration of the metadata persistence warehouse, please refer to [Metadata Persistence Warehouse](../../../../common-config/builtin-algorithm/metadata-repository).
+For the configuration of the metadata repository, please refer to [Metadata Repository](../../../../common-config/builtin-algorithm/metadata-repository).
 
 ## URL configuration
 
@@ -33,5 +34,53 @@ The configuration file is `xxx.yaml`, and the configuration file format is consi
 Example:
 - `jdbc:shardingsphere:absolutepath:/path/to/config.yaml`
 
+### Load configuration file containing environment variables from classpath
+
+JDBC URL to load the config.yaml configuration file that contains environment variables in classpath, identified by the `jdbc:shardingsphere:classpath-environment:` prefix.
+The configuration file is `xxx.yaml`, and the configuration file format is basically the same as [YAML configuration](../../../yaml-config).
+Allows setting the value of specific YAML properties via environment variables and configuring optional default values in the involved YAML files. 
+This is commonly used in Docker Image deployment scenarios.
+The name of an environment variable and its optional default value are separated by `::` and wrapped in the outermost layer by `$${` and `}`.
+
+Discuss two situations.
+1. When the corresponding environment variable does not exist, the value of this YAML attribute will be set to the default value on the right side of `::`.
+2. When the corresponding environment variable and the default value on the right side of `::` do not exist, this property will be set to empty.
+
+Assume that the following set of environment variables exists,
+1. The existing environment variable `FIXTURE_JDBC_URL` is `jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL`.
+2. The existing environment variable `FIXTURE_USERNAME` is `sa`.
+
+Then for the intercepted fragment of the following YAML file,
+```yaml
+ds_1:
+    dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+    driverClassName: $${FIXTURE_DRIVER_CLASS_NAME::org.h2.Driver}
+    jdbcUrl: $${FIXTURE_JDBC_URL::jdbc:h2:mem:foo_ds_do_not_use}
+    username: $${FIXTURE_USERNAME::}
+    password: $${FIXTURE_PASSWORD::}
+```
+This YAML snippet will be parsed as,
+```yaml
+ds_1:
+    dataSourceClassName: com.zaxxer.hikari.HikariDataSource
+    driverClassName: org.h2.Driver
+    jdbcUrl: jdbc:h2:mem:foo_ds_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MySQL
+    username: sa
+    password:
+```
+
+Example:
+- `jdbc:shardingsphere:classpath-environment:config.yaml`
+
+### Load configuration file containing environment variables from absolute path
+
+JDBC URL to load the config.yaml configuration file that contains environment variables in absolute path, 
+identified by the `jdbc:shardingsphere:absolutepath-environment:` prefix.
+The configuration file is `xxx.yaml`, and the configuration file format is consistent with `jdbc:shardingsphere:classpath-environment:`.
+The difference from `jdbc:shardingsphere:classpath-environment:` is only where the YAML file is loaded.
+
+Example:
+- `jdbc:shardingsphere:absolutepath-environment:/path/to/config.yaml`
+
 ### Other implementations
 For details, please refer to https://github.com/apache/shardingsphere-plugin .
diff --git a/...che/shardingsphere/broadcast/distsql/handler/update/CreateBroadcastTableRuleExecutor.java b/...che/shardingsphere/broadcast/distsql/handler/update/CreateBroadcastTableRuleExecutor.java
@@ -46,7 +46,7 @@ public void checkBeforeUpdate(final CreateBroadcastTableRuleStatement sqlStateme
     }
 
     @Override
-    public BroadcastRuleConfiguration buildToBeCreatedRuleConfiguration(final BroadcastRuleConfiguration currentRuleConfig, final CreateBroadcastTableRuleStatement sqlStatement) {
+    public BroadcastRuleConfiguration buildToBeCreatedRuleConfiguration(final CreateBroadcastTableRuleStatement sqlStatement, final BroadcastRuleConfiguration currentRuleConfig) {
         Collection<String> tables = sqlStatement.getTables();
         if (sqlStatement.isIfNotExists()) {
             Collection<String> duplicatedRuleNames = getDuplicatedRuleNames(sqlStatement, currentRuleConfig);

diff --git a/...pache/shardingsphere/broadcast/distsql/handler/update/DropBroadcastTableRuleExecutor.java b/...pache/shardingsphere/broadcast/distsql/handler/update/DropBroadcastTableRuleExecutor.java
@@ -21,9 +21,10 @@
 import org.apache.shardingsphere.broadcast.api.config.BroadcastRuleConfiguration;
 import org.apache.shardingsphere.broadcast.distsql.statement.DropBroadcastTableRuleStatement;
 import org.apache.shardingsphere.distsql.handler.exception.rule.MissingRequiredRuleException;
+import org.apache.shardingsphere.distsql.handler.required.DistSQLExecutorCurrentRuleRequired;
 import org.apache.shardingsphere.distsql.handler.type.rdl.rule.spi.database.DatabaseRuleDropExecutor;
-import org.apache.shardingsphere.infra.metadata.database.ShardingSphereDatabase;
 import org.apache.shardingsphere.infra.exception.core.ShardingSpherePreconditions;
+import org.apache.shardingsphere.infra.metadata.database.ShardingSphereDatabase;
 
 import java.util.Collection;
 import java.util.HashSet;
@@ -32,28 +33,20 @@
 /**
  * Drop broadcast table rule executor.
  */
+@DistSQLExecutorCurrentRuleRequired("Broadcast")
 @Setter
 public final class DropBroadcastTableRuleExecutor implements DatabaseRuleDropExecutor<DropBroadcastTableRuleStatement, BroadcastRuleConfiguration> {
 
     private ShardingSphereDatabase database;
 
     @Override
     public void checkBeforeUpdate(final DropBroadcastTableRuleStatement sqlStatement, final BroadcastRuleConfiguration currentRuleConfig) {
-        if (!isExistRuleConfig(currentRuleConfig) && sqlStatement.isIfExists()) {
-            return;
+        if (!sqlStatement.isIfExists()) {
+            checkBroadcastTableRuleExist(sqlStatement, currentRuleConfig);
         }
-        checkCurrentRuleConfiguration(currentRuleConfig);
-        checkBroadcastTableRuleExist(sqlStatement, currentRuleConfig);
-    }
-
-    private void checkCurrentRuleConfiguration(final BroadcastRuleConfiguration currentRuleConfig) {
-        ShardingSpherePreconditions.checkNotNull(currentRuleConfig, () -> new MissingRequiredRuleException("Broadcast", database.getName()));
     }
 
     private void checkBroadcastTableRuleExist(final DropBroadcastTableRuleStatement sqlStatement, final BroadcastRuleConfiguration currentRuleConfig) {
-        if (sqlStatement.isIfExists()) {
-            return;
-        }
         Collection<String> currentRules = currentRuleConfig.getTables();
         Collection<String> notExistRules = sqlStatement.getTables().stream().filter(each -> !containsIgnoreCase(currentRules, each)).collect(Collectors.toList());
         ShardingSpherePreconditions.checkState(notExistRules.isEmpty(), () -> new MissingRequiredRuleException("Broadcast", database.getName(), notExistRules));
@@ -69,7 +62,7 @@ public boolean hasAnyOneToBeDropped(final DropBroadcastTableRuleStatement sqlSta
     }
 
     @Override
-    public BroadcastRuleConfiguration buildToBeAlteredRuleConfiguration(final BroadcastRuleConfiguration currentRuleConfig, final DropBroadcastTableRuleStatement sqlStatement) {
+    public BroadcastRuleConfiguration buildToBeAlteredRuleConfiguration(final DropBroadcastTableRuleStatement sqlStatement, final BroadcastRuleConfiguration currentRuleConfig) {
         BroadcastRuleConfiguration result = new BroadcastRuleConfiguration(new HashSet<>(currentRuleConfig.getTables()));
         result.getTables().removeIf(each -> containsIgnoreCase(sqlStatement.getTables(), each));
         return result;

diff --git a/...shardingsphere/broadcast/distsql/handler/update/CreateBroadcastTableRuleExecutorTest.java b/...shardingsphere/broadcast/distsql/handler/update/CreateBroadcastTableRuleExecutorTest.java
@@ -65,7 +65,7 @@ void assertBuildToBeCreatedRuleConfiguration() {
         CreateBroadcastTableRuleStatement sqlStatement = new CreateBroadcastTableRuleStatement(false, Collections.singleton("t_address"));
         executor.setDatabase(mockShardingSphereDatabase());
         executor.checkBeforeUpdate(sqlStatement, currentConfig);
-        BroadcastRuleConfiguration toBeCreatedRuleConfig = executor.buildToBeCreatedRuleConfiguration(currentConfig, sqlStatement);
+        BroadcastRuleConfiguration toBeCreatedRuleConfig = executor.buildToBeCreatedRuleConfiguration(sqlStatement, currentConfig);
         executor.updateCurrentRuleConfiguration(currentConfig, toBeCreatedRuleConfig);
         assertThat(currentConfig.getTables().size(), is(1));
         assertThat(currentConfig.getTables().iterator().next(), is("t_address"));

diff --git a/...e/shardingsphere/broadcast/distsql/handler/update/DropBroadcastTableRuleExecutorTest.java b/...e/shardingsphere/broadcast/distsql/handler/update/DropBroadcastTableRuleExecutorTest.java
@@ -44,12 +44,6 @@ void setUp() {
         executor.setDatabase(database);
     }
 
-    @Test
-    void assertCheckSQLStatementWithoutCurrentRule() {
-        DropBroadcastTableRuleStatement sqlStatement = new DropBroadcastTableRuleStatement(false, Collections.singleton("t_address"));
-        assertThrows(MissingRequiredRuleException.class, () -> executor.checkBeforeUpdate(sqlStatement, null));
-    }
-
     @Test
     void assertCheckSQLStatementWithoutToBeDroppedRule() {
         DropBroadcastTableRuleStatement sqlStatement = new DropBroadcastTableRuleStatement(false, Collections.singleton("t_address"));

diff --git a/...va/org/apache/shardingsphere/encrypt/distsql/handler/update/AlterEncryptRuleExecutor.java b/...va/org/apache/shardingsphere/encrypt/distsql/handler/update/AlterEncryptRuleExecutor.java
@@ -21,6 +21,7 @@
 import lombok.Setter;
 import org.apache.shardingsphere.distsql.handler.exception.rule.InvalidRuleConfigurationException;
 import org.apache.shardingsphere.distsql.handler.exception.rule.MissingRequiredRuleException;
+import org.apache.shardingsphere.distsql.handler.required.DistSQLExecutorCurrentRuleRequired;
 import org.apache.shardingsphere.distsql.handler.type.rdl.rule.spi.database.DatabaseRuleAlterExecutor;
 import org.apache.shardingsphere.distsql.segment.AlgorithmSegment;
 import org.apache.shardingsphere.encrypt.api.config.EncryptRuleConfiguration;
@@ -43,23 +44,19 @@
 /**
  * Alter encrypt rule executor.
  */
+@DistSQLExecutorCurrentRuleRequired("Encrypt")
 @Setter
 public final class AlterEncryptRuleExecutor implements DatabaseRuleAlterExecutor<AlterEncryptRuleStatement, EncryptRuleConfiguration> {
 
     private ShardingSphereDatabase database;
 
     @Override
     public void checkBeforeUpdate(final AlterEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
-        checkCurrentRuleConfiguration(currentRuleConfig);
         checkToBeAlteredRules(sqlStatement, currentRuleConfig);
         checkColumnNames(sqlStatement);
         checkToBeAlteredEncryptors(sqlStatement);
     }
 
-    private void checkCurrentRuleConfiguration(final EncryptRuleConfiguration currentRuleConfig) {
-        ShardingSpherePreconditions.checkNotNull(currentRuleConfig, () -> new MissingRequiredRuleException("Encrypt", database.getName()));
-    }
-
     private void checkToBeAlteredRules(final AlterEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
         Collection<String> currentEncryptTableNames = currentRuleConfig.getTables().stream().map(EncryptTableRuleConfiguration::getName).collect(Collectors.toList());
         Collection<String> notExistEncryptTableNames = getToBeAlteredEncryptTableNames(sqlStatement).stream().filter(each -> !currentEncryptTableNames.contains(each)).collect(Collectors.toList());
@@ -99,7 +96,7 @@ private void checkToBeAlteredEncryptors(final AlterEncryptRuleStatement sqlState
     }
 
     @Override
-    public EncryptRuleConfiguration buildToBeAlteredRuleConfiguration(final EncryptRuleConfiguration currentRuleConfig, final AlterEncryptRuleStatement sqlStatement) {
+    public EncryptRuleConfiguration buildToBeAlteredRuleConfiguration(final AlterEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
         return EncryptRuleStatementConverter.convert(sqlStatement.getRules());
     }
 

diff --git a/...a/org/apache/shardingsphere/encrypt/distsql/handler/update/CreateEncryptRuleExecutor.java b/...a/org/apache/shardingsphere/encrypt/distsql/handler/update/CreateEncryptRuleExecutor.java
@@ -59,6 +59,19 @@ public void checkBeforeUpdate(final CreateEncryptRuleStatement sqlStatement, fin
         checkDataSources();
     }
 
+    private void checkDuplicateRuleNames(final CreateEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
+        Collection<String> duplicatedRuleNames = getDuplicatedRuleNames(sqlStatement, currentRuleConfig);
+        ShardingSpherePreconditions.checkState(duplicatedRuleNames.isEmpty(), () -> new DuplicateRuleException("encrypt", database.getName(), duplicatedRuleNames));
+    }
+
+    private Collection<String> getDuplicatedRuleNames(final CreateEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
+        Collection<String> currentRuleNames = new LinkedHashSet<>();
+        if (null != currentRuleConfig) {
+            currentRuleNames = currentRuleConfig.getTables().stream().map(EncryptTableRuleConfiguration::getName).collect(Collectors.toSet());
+        }
+        return sqlStatement.getRules().stream().map(EncryptRuleSegment::getTableName).filter(currentRuleNames::contains).collect(Collectors.toSet());
+    }
+
     private void checkColumnNames(final CreateEncryptRuleStatement sqlStatement) {
         for (EncryptRuleSegment each : sqlStatement.getRules()) {
             ShardingSpherePreconditions.checkState(isColumnNameNotConflicts(each),
@@ -104,19 +117,6 @@ private void checkAssistedAlgorithmType(final EncryptColumnItemSegment itemSegme
                 () -> new InvalidAlgorithmConfigurationException("assisted encrypt", encryptAlgorithm.getType()));
     }
 
-    private void checkDuplicateRuleNames(final CreateEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
-        Collection<String> duplicatedRuleNames = getDuplicatedRuleNames(sqlStatement, currentRuleConfig);
-        ShardingSpherePreconditions.checkState(duplicatedRuleNames.isEmpty(), () -> new DuplicateRuleException("encrypt", database.getName(), duplicatedRuleNames));
-    }
-
-    private Collection<String> getDuplicatedRuleNames(final CreateEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
-        Collection<String> currentRuleNames = new LinkedHashSet<>();
-        if (null != currentRuleConfig) {
-            currentRuleNames = currentRuleConfig.getTables().stream().map(EncryptTableRuleConfiguration::getName).collect(Collectors.toSet());
-        }
-        return sqlStatement.getRules().stream().map(EncryptRuleSegment::getTableName).filter(currentRuleNames::contains).collect(Collectors.toSet());
-    }
-
     private void checkToBeCreatedEncryptors(final CreateEncryptRuleStatement sqlStatement) {
         Collection<AlgorithmSegment> encryptors = new LinkedHashSet<>();
         sqlStatement.getRules().forEach(each -> each.getColumns().forEach(column -> {
@@ -136,7 +136,7 @@ private void checkDataSources() {
     }
 
     @Override
-    public EncryptRuleConfiguration buildToBeCreatedRuleConfiguration(final EncryptRuleConfiguration currentRuleConfig, final CreateEncryptRuleStatement sqlStatement) {
+    public EncryptRuleConfiguration buildToBeCreatedRuleConfiguration(final CreateEncryptRuleStatement sqlStatement, final EncryptRuleConfiguration currentRuleConfig) {
         Collection<EncryptRuleSegment> segments = sqlStatement.getRules();
         if (sqlStatement.isIfNotExists()) {
             Collection<String> duplicatedRuleNames = getDuplicatedRuleNames(sqlStatement, currentRuleConfig);