From 842aa2e08674104db66ed638bb5e4529409f9fae Mon Sep 17 00:00:00 2001 From: yangyuxia <13853186257@139.com> Date: Mon, 27 May 2024 17:39:53 +0800 Subject: [PATCH] Add personalized configuration parameters for each metastore. (#315) * Add personalized configuration parameters for each metastore. * Add personalized configuration parameters for each metastore * Recover * Update junit test * Update Junit Test * Update Junit Test * Update Junit Test * Format the code and update the readme * Revert * Update FederatedMetaStoreTest.java * Update PrimaryMetaStoreTest.java * Update AbstractMetaStore.java using new HashMap so the generated Yaml doesn't generate an anchor (reference &id001) * Update YamlFederatedMetaStoreStorageTest.java fixing test --------- Co-authored-by: yangyx <360508847@qq.com> Co-authored-by: Patrick Duin --- README.md | 57 +++++++------ .../api/model/AbstractMetaStore.java | 13 ++- .../api/model/FederatedMetaStoreTest.java | 2 +- .../api/model/PrimaryMetaStoreTest.java | 2 +- ...ThriftHiveMetastoreIfaceClientFactory.java | 4 + ...ftHiveMetastoreIfaceClientFactoryTest.java | 8 +- .../YamlFederatedMetaStoreStorageTest.java | 83 ++++++++++--------- 7 files changed, 98 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index 4757b33e2..926bcf775 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,8 @@ Example: - my_writable_db2 - user_db_.* - ... + configuration-properties: + hive.metastore.kerberos.principal: hive/_HOST@HADOOP.COM federated-meta-stores: # List of read only metastores to federate - remote-meta-store-uris: thrift://10.0.0.1:9083 name: secondary @@ -147,35 +149,38 @@ Example: - database: prod_db2 mapped-tables: - tbl2 + configuration-properties: + hive.metastore.kerberos.principal: hive/clustername@HADOOP.COM - ... - The table below describes all the available configuration values for Waggle Dance federations: -| Property | Required | Description | -|:----|:----:|:----| -| `primary-meta-store` | No | Primary MetaStore config. Can be empty but it is advised to configure it. | -| `primary-meta-store.remote-meta-store-uris` | Yes | Thrift URIs of the federated read-only metastore. | -| `primary-meta-store.name` | Yes | Database name that uniquely identifies this metastore. Used internally. Cannot be empty. | -| `primary-meta-store.database-prefix` | No | Prefix used to access the primary metastore and differentiate databases in it from databases in another metastore. The default prefix (i.e. if this value isn't explicitly set) is empty string.| -| `primary-meta-store.access-control-type` | No | Sets how the client access controls should be handled. Default is `READ_ONLY` Other options `READ_AND_WRITE_AND_CREATE`, `READ_AND_WRITE_ON_DATABASE_WHITELIST` and `READ_AND_WRITE_AND_CREATE_ON_DATABASE_WHITELIST` see Access Control section below. | -| `primary-meta-store.writable-database-white-list` | No | White-list of databases used to verify write access used in conjunction with `primary-meta-store.access-control-type`. The list of databases should be listed without any `primary-meta-store.database-prefix`. This property supports both full database names and (case-insensitive) [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).| -| `primary-meta-store.metastore-tunnel` | No | See metastore tunnel configuration values below. | -| `primary-meta-store.latency` | No | Indicates the acceptable slowness of the metastore in **milliseconds** for increasing the default connection timeout. Default latency is `0` and should be changed if the metastore is particularly slow. If you get an error saying that results were omitted because the metastore was slow, consider changing the latency to a higher number.| -| `primary-meta-store.mapped-databases` | No | List of databases to federate from the primary metastore; all other databases will be ignored. This property supports both full database names and [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) (both being case-insensitive). By default, all databases from the metastore are federated. | -| `primary-meta-store.mapped-tables` | No | List of mappings from databases to tables to federate from the primary metastore, similar to `mapped-databases`. By default, all tables are available. See `mapped-tables` configuration below. | -| `primary-meta-stores.hive-metastore-filter-hook` | No | Name of the class which implements the `MetaStoreFilterHook` interface from Hive. This allows a metastore filter hook to be applied to the corresponding Hive metastore calls. Can be configured with the `configuration-properties` specified in the `waggle-dance-server.yml` configuration. They will be added in the HiveConf object that is given to the constructor of the `MetaStoreFilterHook` implementation you provide. | -| `primary-meta-stores.database-name-mapping` | No | BiDirectional Map of database names and mapped name, where key=`` and value=``. See the [Database Name Mapping](#database-name-mapping) section.| -| `federated-meta-stores` | No | Possible empty list of read only federated metastores. | -| `federated-meta-stores[n].remote-meta-store-uris` | Yes | Thrift URIs of the federated read-only metastore. | -| `federated-meta-stores[n].name` | Yes | Name that uniquely identifies this metastore. Used internally. Cannot be empty. | -| `federated-meta-stores[n].database-prefix` | No | Prefix used to access this particular metastore and differentiate databases in it from databases in another metastore. Typically used if databases have the same name across metastores but federated access to them is still needed. The default prefix (i.e. if this value isn't explicitly set) is {federated-meta-stores[n].name} lowercased and postfixed with an underscore. For example if the metastore name was configured as "waggle" and no database prefix was provided but `PREFIXED` database resolution was used then the value of `database-prefix` would be "waggle_". | -| `federated-meta-stores[n].metastore-tunnel` | No | See metastore tunnel configuration values below. | -| `federated-meta-stores[n].latency` | No | Indicates the acceptable slowness of the metastore in **milliseconds** for increasing the default connection timeout. Default latency is `0` and should be changed if the metastore is particularly slow. If you get an error saying that results were omitted because the metastore was slow, consider changing the latency to a higher number.| -| `federated-meta-stores[n].mapped-databases` | No | List of databases to federate from this federated metastore, all other databases will be ignored. This property supports both full database names and [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) (both being case-insensitive). By default, all databases from the metastore are federated. | -| `federated-meta-stores[n].mapped-tables` | No | List of mappings from databases to tables to federate from this federated metastore, similar to `mapped-databases`. By default, all tables are available. See `mapped-tables` configuration below. | -| `federated-meta-stores[n].hive-metastore-filter-hook` | No | Name of the class which implements the `MetaStoreFilterHook` interface from Hive. This allows a metastore filter hook to be applied to the corresponding Hive metastore calls. Can be configured with the `configuration-properties` specified in the `waggle-dance-server.yml` configuration. They will be added in the HiveConf object that is given to the constructor of the `MetaStoreFilterHook` implementation you provide. | -| `federated-meta-stores[n].database-name-mapping` | No | BiDirectional Map of database names and mapped names where key=`` and value=``. See the [Database Name Mapping](#database-name-mapping) section.| -| `federated-meta-stores[n].writable-database-white-list` | No | White-list of databases used to verify write access used in conjunction with `federated-meta-stores[n].access-control-type`. The list of databases should be listed without a `federated-meta-stores[n].database-prefix`. This property supports both full database names and (case-insensitive) [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html).| +| Property | Required | Description | +|:----------------------------------------------------------|:----:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `primary-meta-store` | No | Primary MetaStore config. Can be empty but it is advised to configure it. | +| `primary-meta-store.remote-meta-store-uris` | Yes | Thrift URIs of the federated read-only metastore. | +| `primary-meta-store.name` | Yes | Database name that uniquely identifies this metastore. Used internally. Cannot be empty. | +| `primary-meta-store.database-prefix` | No | Prefix used to access the primary metastore and differentiate databases in it from databases in another metastore. The default prefix (i.e. if this value isn't explicitly set) is empty string. | +| `primary-meta-store.access-control-type` | No | Sets how the client access controls should be handled. Default is `READ_ONLY` Other options `READ_AND_WRITE_AND_CREATE`, `READ_AND_WRITE_ON_DATABASE_WHITELIST` and `READ_AND_WRITE_AND_CREATE_ON_DATABASE_WHITELIST` see Access Control section below. | +| `primary-meta-store.writable-database-white-list` | No | White-list of databases used to verify write access used in conjunction with `primary-meta-store.access-control-type`. The list of databases should be listed without any `primary-meta-store.database-prefix`. This property supports both full database names and (case-insensitive) [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html). | +| `primary-meta-store.metastore-tunnel` | No | See metastore tunnel configuration values below. | +| `primary-meta-store.latency` | No | Indicates the acceptable slowness of the metastore in **milliseconds** for increasing the default connection timeout. Default latency is `0` and should be changed if the metastore is particularly slow. If you get an error saying that results were omitted because the metastore was slow, consider changing the latency to a higher number. | +| `primary-meta-store.mapped-databases` | No | List of databases to federate from the primary metastore; all other databases will be ignored. This property supports both full database names and [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) (both being case-insensitive). By default, all databases from the metastore are federated. | +| `primary-meta-store.mapped-tables` | No | List of mappings from databases to tables to federate from the primary metastore, similar to `mapped-databases`. By default, all tables are available. See `mapped-tables` configuration below. | +| `primary-meta-stores.hive-metastore-filter-hook` | No | Name of the class which implements the `MetaStoreFilterHook` interface from Hive. This allows a metastore filter hook to be applied to the corresponding Hive metastore calls. Can be configured with the `configuration-properties` specified in the `waggle-dance-server.yml` configuration. They will be added in the HiveConf object that is given to the constructor of the `MetaStoreFilterHook` implementation you provide. | +| `primary-meta-stores.database-name-mapping` | No | BiDirectional Map of database names and mapped name, where key=`` and value=``. See the [Database Name Mapping](#database-name-mapping) section. | +| `primary-meta-stores.configuration-properties` | No | Map of the primary metastore personalized properties that will be added to the HiveConf used when creating the Thrift clients (they will be effect only on this client),the priority is higher than the properites of the same name in waggle-dance-server.yml. | +| `federated-meta-stores` | No | Possible empty list of read only federated metastores. | +| `federated-meta-stores[n].remote-meta-store-uris` | Yes | Thrift URIs of the federated read-only metastore. | +| `federated-meta-stores[n].name` | Yes | Name that uniquely identifies this metastore. Used internally. Cannot be empty. | +| `federated-meta-stores[n].database-prefix` | No | Prefix used to access this particular metastore and differentiate databases in it from databases in another metastore. Typically used if databases have the same name across metastores but federated access to them is still needed. The default prefix (i.e. if this value isn't explicitly set) is {federated-meta-stores[n].name} lowercased and postfixed with an underscore. For example if the metastore name was configured as "waggle" and no database prefix was provided but `PREFIXED` database resolution was used then the value of `database-prefix` would be "waggle_". | +| `federated-meta-stores[n].metastore-tunnel` | No | See metastore tunnel configuration values below. | +| `federated-meta-stores[n].latency` | No | Indicates the acceptable slowness of the metastore in **milliseconds** for increasing the default connection timeout. Default latency is `0` and should be changed if the metastore is particularly slow. If you get an error saying that results were omitted because the metastore was slow, consider changing the latency to a higher number. | +| `federated-meta-stores[n].mapped-databases` | No | List of databases to federate from this federated metastore, all other databases will be ignored. This property supports both full database names and [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html) (both being case-insensitive). By default, all databases from the metastore are federated. | +| `federated-meta-stores[n].mapped-tables` | No | List of mappings from databases to tables to federate from this federated metastore, similar to `mapped-databases`. By default, all tables are available. See `mapped-tables` configuration below. | +| `federated-meta-stores[n].hive-metastore-filter-hook` | No | Name of the class which implements the `MetaStoreFilterHook` interface from Hive. This allows a metastore filter hook to be applied to the corresponding Hive metastore calls. Can be configured with the `configuration-properties` specified in the `waggle-dance-server.yml` configuration. They will be added in the HiveConf object that is given to the constructor of the `MetaStoreFilterHook` implementation you provide. | +| `federated-meta-stores[n].database-name-mapping` | No | BiDirectional Map of database names and mapped names where key=`` and value=``. See the [Database Name Mapping](#database-name-mapping) section. | +| `federated-meta-stores[n].writable-database-white-list` | No | White-list of databases used to verify write access used in conjunction with `federated-meta-stores[n].access-control-type`. The list of databases should be listed without a `federated-meta-stores[n].database-prefix`. This property supports both full database names and (case-insensitive) [Java RegEx patterns](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html). | +| `federated-meta-stores[n].configuration-properties` | No | Map of the federate metastore personalized properties that will be added to the HiveConf used when creating the Thrift clients (they will be effect only on this client),the priority is higher than the properites of the same name in waggle-dance-server.yml. | #### Metastore tunnel The table below describes the metastore tunnel configuration values: diff --git a/waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java b/waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java index fdc91fa88..c78b63342 100644 --- a/waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java +++ b/waggle-dance-api/src/main/java/com/hotels/bdp/waggledance/api/model/AbstractMetaStore.java @@ -51,7 +51,7 @@ public abstract class AbstractMetaStore { private List writableDatabaseWhitelist; private List mappedDatabases; private @Valid List mappedTables; - private Map databaseNameMapping = Collections.emptyMap(); + private Map databaseNameMapping = new HashMap<>(); private @NotBlank String name; private @NotBlank String remoteMetaStoreUris; private @Valid MetastoreTunnel metastoreTunnel; @@ -59,6 +59,7 @@ public abstract class AbstractMetaStore { private transient @JsonProperty @NotNull MetaStoreStatus status = MetaStoreStatus.UNKNOWN; private long latency = 0; private transient @JsonIgnore HashBiMap databaseNameBiMapping = HashBiMap.create(); + private Map configurationProperties = new HashMap<>(); public AbstractMetaStore(String name, String remoteMetaStoreUris, AccessControlType accessControlType) { this.name = name; @@ -201,6 +202,15 @@ public HashBiMap getDatabaseNameBiMapping() { return databaseNameBiMapping; } + public Map getConfigurationProperties() { + return configurationProperties; + } + + public void setConfigurationProperties( + Map configurationProperties) { + this.configurationProperties = configurationProperties; + } + @Transient public MetaStoreStatus getStatus() { return status; @@ -242,5 +252,4 @@ public String toString() { .add("status", status) .toString(); } - } diff --git a/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/FederatedMetaStoreTest.java b/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/FederatedMetaStoreTest.java index 86048bb53..dcbe47103 100644 --- a/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/FederatedMetaStoreTest.java +++ b/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/FederatedMetaStoreTest.java @@ -72,7 +72,7 @@ public void nullDatabasePrefix() { @Test public void toJson() throws Exception { - String expected = "{\"accessControlType\":\"READ_ONLY\",\"connectionType\":\"DIRECT\",\"databaseNameMapping\":{},\"databasePrefix\":\"name_\",\"federationType\":\"FEDERATED\",\"hiveMetastoreFilterHook\":null,\"latency\":0,\"mappedDatabases\":null,\"mappedTables\":null,\"metastoreTunnel\":null,\"name\":\"name\",\"remoteMetaStoreUris\":\"uri\",\"status\":\"UNKNOWN\",\"writableDatabaseWhiteList\":[]}"; + String expected = "{\"accessControlType\":\"READ_ONLY\",\"configurationProperties\":{},\"connectionType\":\"DIRECT\",\"databaseNameMapping\":{},\"databasePrefix\":\"name_\",\"federationType\":\"FEDERATED\",\"hiveMetastoreFilterHook\":null,\"latency\":0,\"mappedDatabases\":null,\"mappedTables\":null,\"metastoreTunnel\":null,\"name\":\"name\",\"remoteMetaStoreUris\":\"uri\",\"status\":\"UNKNOWN\",\"writableDatabaseWhiteList\":[]}"; ObjectMapper mapper = new ObjectMapper(); // Sorting to get deterministic test behaviour mapper.enable(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY); diff --git a/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/PrimaryMetaStoreTest.java b/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/PrimaryMetaStoreTest.java index 8c1634eb6..1e1b1dc67 100644 --- a/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/PrimaryMetaStoreTest.java +++ b/waggle-dance-api/src/test/java/com/hotels/bdp/waggledance/api/model/PrimaryMetaStoreTest.java @@ -89,7 +89,7 @@ public void nonEmptyDatabasePrefix() { @Test public void toJson() throws Exception { - String expected = "{\"accessControlType\":\"READ_ONLY\",\"connectionType\":\"DIRECT\",\"databaseNameMapping\":{},\"databasePrefix\":\"\",\"federationType\":\"PRIMARY\",\"hiveMetastoreFilterHook\":null,\"latency\":0,\"mappedDatabases\":null,\"mappedTables\":null,\"metastoreTunnel\":null,\"name\":\"name\",\"remoteMetaStoreUris\":\"uri\",\"status\":\"UNKNOWN\",\"writableDatabaseWhiteList\":[]}"; + String expected = "{\"accessControlType\":\"READ_ONLY\",\"configurationProperties\":{},\"connectionType\":\"DIRECT\",\"databaseNameMapping\":{},\"databasePrefix\":\"\",\"federationType\":\"PRIMARY\",\"hiveMetastoreFilterHook\":null,\"latency\":0,\"mappedDatabases\":null,\"mappedTables\":null,\"metastoreTunnel\":null,\"name\":\"name\",\"remoteMetaStoreUris\":\"uri\",\"status\":\"UNKNOWN\",\"writableDatabaseWhiteList\":[]}"; ObjectMapper mapper = new ObjectMapper(); // Sorting to get deterministic test behaviour mapper.enable(MapperFeature.SORT_PROPERTIES_ALPHABETICALLY); diff --git a/waggle-dance-core/src/main/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java b/waggle-dance-core/src/main/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java index ca62e4063..cd60bfb44 100644 --- a/waggle-dance-core/src/main/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java +++ b/waggle-dance-core/src/main/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactory.java @@ -47,6 +47,10 @@ public CloseableThriftHiveMetastoreIface newInstance(AbstractMetaStore metaStore if (waggleDanceConfiguration.getConfigurationProperties() != null) { properties.putAll(waggleDanceConfiguration.getConfigurationProperties()); } + if (metaStore.getConfigurationProperties() != null) { + properties.putAll(metaStore.getConfigurationProperties()); + } + return newHiveInstance(metaStore, properties); } diff --git a/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactoryTest.java b/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactoryTest.java index d011d50cd..73ca5d968 100644 --- a/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactoryTest.java +++ b/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/client/CloseableThriftHiveMetastoreIfaceClientFactoryTest.java @@ -24,10 +24,12 @@ import static com.hotels.bdp.waggledance.api.model.AbstractMetaStore.newFederatedInstance; +import java.util.Collections; import java.util.HashMap; import java.util.Map; import java.util.concurrent.TimeUnit; +import com.hotels.bdp.waggledance.api.model.FederatedMetaStore; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; import org.junit.Before; @@ -68,8 +70,9 @@ public void setUp() { @Test public void defaultFactory() { ArgumentCaptor hiveConfCaptor = ArgumentCaptor.forClass(HiveConf.class); - - factory.newInstance(newFederatedInstance("fed1", THRIFT_URI)); + FederatedMetaStore fed1 = newFederatedInstance("fed1", THRIFT_URI); + fed1.setConfigurationProperties(Collections.singletonMap(ConfVars.METASTORE_KERBEROS_PRINCIPAL.varname, "hive/_HOST@HADOOP.COM")); + factory.newInstance(fed1); verify(defaultMetaStoreClientFactory).newInstance(hiveConfCaptor.capture(), eq( "waggledance-fed1"), eq(3), eq(2000)); verifyNoInteractions(tunnelingMetaStoreClientFactory); @@ -80,6 +83,7 @@ public void defaultFactory() { assertThat(hiveConf.getTimeVar(ConfVars.METASTORE_CLIENT_CONNECT_RETRY_DELAY, TimeUnit.SECONDS), is(5L)); assertThat(hiveConf.getBoolVar(ConfVars.METASTORE_USE_THRIFT_FRAMED_TRANSPORT), is(true)); assertThat(hiveConf.getBoolVar(ConfVars.METASTORE_USE_THRIFT_COMPACT_PROTOCOL), is(false)); + assertThat(hiveConf.getVar(ConfVars.METASTORE_KERBEROS_PRINCIPAL), is("hive/_HOST@HADOOP.COM")); } @Test diff --git a/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/mapping/service/impl/YamlFederatedMetaStoreStorageTest.java b/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/mapping/service/impl/YamlFederatedMetaStoreStorageTest.java index bb99938ae..a39434a8e 100644 --- a/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/mapping/service/impl/YamlFederatedMetaStoreStorageTest.java +++ b/waggle-dance-core/src/test/java/com/hotels/bdp/waggledance/mapping/service/impl/YamlFederatedMetaStoreStorageTest.java @@ -23,6 +23,7 @@ import java.io.File; import java.nio.charset.StandardCharsets; import java.nio.file.Files; +import java.util.Collections; import java.util.List; import javax.validation.ConstraintViolationException; @@ -211,10 +212,11 @@ public void saveFederationWriteFederations() throws Exception { MappedTables mappedTables2 = new MappedTables("db2", Lists.newArrayList("tbl2")); newFederatedInstance.setMappedTables(Lists.newArrayList(mappedTables1, mappedTables2)); newFederatedInstance.setHiveMetastoreFilterHook("filter.hook.class"); + newFederatedInstance.setConfigurationProperties(Collections.singletonMap("hive.metastore.kerberos.principal", "hive/_HOST@REALM")); storage.insert(newFederatedInstance); storage.saveFederation(); List lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); - assertThat(lines.size(), is(24)); + assertThat(lines.size(), is(25)); assertThat(lines.get(0), is("primary-meta-store:")); assertThat(lines.get(1), is(" access-control-type: READ_ONLY")); assertThat(lines.get(2), is(" database-prefix: ''")); @@ -223,22 +225,23 @@ public void saveFederationWriteFederations() throws Exception { assertThat(lines.get(5), is(" remote-meta-store-uris: thrift://localhost:19083")); assertThat(lines.get(6), is("federated-meta-stores:")); assertThat(lines.get(7), is("- access-control-type: READ_ONLY")); - assertThat(lines.get(8), is(" database-name-mapping: {}")); - assertThat(lines.get(9), is(" database-prefix: hcom_2_")); - assertThat(lines.get(10), is(" hive-metastore-filter-hook: filter.hook.class")); - assertThat(lines.get(11), is(" latency: 0")); - assertThat(lines.get(12), is(" mapped-databases:")); - assertThat(lines.get(13), is(" - db1")); - assertThat(lines.get(14), is(" - db2")); - assertThat(lines.get(15), is(" mapped-tables:")); - assertThat(lines.get(16), is(" - database: db1")); - assertThat(lines.get(17), is(" mapped-tables:")); - assertThat(lines.get(18), is(" - tbl1")); - assertThat(lines.get(19), is(" - database: db2")); - assertThat(lines.get(20), is(" mapped-tables:")); - assertThat(lines.get(21), is(" - tbl2")); - assertThat(lines.get(22), is(" name: hcom_2")); - assertThat(lines.get(23), is(" remote-meta-store-uris: thrift://localhost:29083")); + assertThat(lines.get(8), is(" configuration-properties:")); + assertThat(lines.get(9), is(" hive.metastore.kerberos.principal: hive/_HOST@REALM")); + assertThat(lines.get(10), is(" database-prefix: hcom_2_")); + assertThat(lines.get(11), is(" hive-metastore-filter-hook: filter.hook.class")); + assertThat(lines.get(12), is(" latency: 0")); + assertThat(lines.get(13), is(" mapped-databases:")); + assertThat(lines.get(14), is(" - db1")); + assertThat(lines.get(15), is(" - db2")); + assertThat(lines.get(16), is(" mapped-tables:")); + assertThat(lines.get(17), is(" - database: db1")); + assertThat(lines.get(18), is(" mapped-tables:")); + assertThat(lines.get(19), is(" - tbl1")); + assertThat(lines.get(20), is(" - database: db2")); + assertThat(lines.get(21), is(" mapped-tables:")); + assertThat(lines.get(22), is(" - tbl2")); + assertThat(lines.get(23), is(" name: hcom_2")); + assertThat(lines.get(24), is(" remote-meta-store-uris: thrift://localhost:29083")); } @Test @@ -292,34 +295,36 @@ public void savePrimaryWriteFederations() throws Exception { MappedTables mappedTables1 = new MappedTables("db1", Lists.newArrayList("tbl1")); MappedTables mappedTables2 = new MappedTables("db2", Lists.newArrayList("tbl2")); primaryMetaStore.setMappedTables(Lists.newArrayList(mappedTables1, mappedTables2)); + primaryMetaStore.setConfigurationProperties(Collections.singletonMap("hive.metastore.kerberos.principal", "hive/_HOST@REALM")); storage.insert(primaryMetaStore); storage.insert(newFederatedInstance("hcom_2", "thrift://localhost:29083")); storage.saveFederation(); List lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); - assertThat(lines.size(), is(23)); + assertThat(lines.size(), is(24)); assertThat(lines.get(0), is("primary-meta-store:")); assertThat(lines.get(1), is(" access-control-type: READ_ONLY")); - assertThat(lines.get(2), is(" database-prefix: ''")); - assertThat(lines.get(3), is(" latency: 0")); - assertThat(lines.get(4), is(" mapped-databases:")); - assertThat(lines.get(5), is(" - db1")); - assertThat(lines.get(6), is(" - db2")); - assertThat(lines.get(7), is(" mapped-tables:")); - assertThat(lines.get(8), is(" - database: db1")); - assertThat(lines.get(9), is(" mapped-tables:")); - assertThat(lines.get(10), is(" - tbl1")); - assertThat(lines.get(11), is(" - database: db2")); - assertThat(lines.get(12), is(" mapped-tables:")); - assertThat(lines.get(13), is(" - tbl2")); - assertThat(lines.get(14), is(" name: hcom_1")); - assertThat(lines.get(15), is(" remote-meta-store-uris: thrift://localhost:19083")); - assertThat(lines.get(16), is("federated-meta-stores:")); - assertThat(lines.get(17), is("- access-control-type: READ_ONLY")); - assertThat(lines.get(18), is(" database-name-mapping: {}")); - assertThat(lines.get(19), is(" database-prefix: hcom_2_")); - assertThat(lines.get(20), is(" latency: 0")); - assertThat(lines.get(21), is(" name: hcom_2")); - assertThat(lines.get(22), is(" remote-meta-store-uris: thrift://localhost:29083")); + assertThat(lines.get(2), is(" configuration-properties:")); + assertThat(lines.get(3), is(" hive.metastore.kerberos.principal: hive/_HOST@REALM")); + assertThat(lines.get(4), is(" database-prefix: ''")); + assertThat(lines.get(5), is(" latency: 0")); + assertThat(lines.get(6), is(" mapped-databases:")); + assertThat(lines.get(7), is(" - db1")); + assertThat(lines.get(8), is(" - db2")); + assertThat(lines.get(9), is(" mapped-tables:")); + assertThat(lines.get(10), is(" - database: db1")); + assertThat(lines.get(11), is(" mapped-tables:")); + assertThat(lines.get(12), is(" - tbl1")); + assertThat(lines.get(13), is(" - database: db2")); + assertThat(lines.get(14), is(" mapped-tables:")); + assertThat(lines.get(15), is(" - tbl2")); + assertThat(lines.get(16), is(" name: hcom_1")); + assertThat(lines.get(17), is(" remote-meta-store-uris: thrift://localhost:19083")); + assertThat(lines.get(18), is("federated-meta-stores:")); + assertThat(lines.get(19), is("- access-control-type: READ_ONLY")); + assertThat(lines.get(20), is(" database-prefix: hcom_2_")); + assertThat(lines.get(21), is(" latency: 0")); + assertThat(lines.get(22), is(" name: hcom_2")); + assertThat(lines.get(23), is(" remote-meta-store-uris: thrift://localhost:29083")); } private PrimaryMetaStore newPrimaryInstance(String name, String remoteMetaStoreUris) {