From 8b7775f45a6a9b0fa6079aba130d6ab1de899e75 Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 16 Sep 2024 14:13:44 +0100 Subject: [PATCH 01/14] Doc: Proposal for FederatedCatalog distribution. --- .../README.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md new file mode 100644 index 000000000..7ec42ecb3 --- /dev/null +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -0,0 +1,39 @@ +# Proposal for FederatedCatalog with Tractus-X distribution + +## Decision + +The **proposed solution is to have a FederatedCatalog as own standalone service**. +Choosing a solution that decouples from Control Plane (like the one used for the Data Plane) and also able to be scalable, we believe future proofs the Federated Catalog as a feature and embraces wider usage. Like so, having a FederatedCatalog service, scalable on its own based on user needs and not on Control Plane allows to best resource management in the long term and allows to add features to the Federated Catalog without the need to change the Control Plane logic (assuming no API usage change). +The Network Latency expected may be a concern, but not more than any other downstream dependencies of the service. + +## Rationale + +Currently two main options are being considered regarding the distribution of Tractus-X (TX) with the FederatedCatalog (FC): +- FederatedCatalog **running as a standalone service** with its own API providing the catalogs' data it contains upon request, similar to what happens currently with the DataPlane (DP). +- FederatedCatalog as **a ControlPlane (CP) extension**, i.e. embedded and shipped as any other extension. + +Taking under consideration both options mentioned above, the following topics were analyzed. + +To reduce verbosity, the next acronyms are used +**FCSS**: Federated Catalog as a Standalone Service +**FCE**: Federated Catalog as a Control Plane Extension + + +### Coupling +Aiming at decoupling the FCSS is a better option, since changes on Control Plane do not demand action on FC and vice versa. Additionally, with this solution, **independent upgrades can be assured**, unless they include API changes. + +### Scalability +FCSS can be better scalable since has no other service dependencies. If several instances of Control Plane are needed, there would be a need to either disable the extension on some tasks or accept several FederatedCatalog extentions, either being a scalability concern and showing limitations of FCE. So, if each instance of the Control Plane boots up a FederatedCatalog extension that can be an overkill and have undesired overhead. + +### Network Latency +Since FCE does not require HTTP requests (being the FederatedCatalog embedded part of the client) the network latency expected is lower, even if FCSS is deployed in a participant own private network. The impact should be minimal, but FCSS would have more latency. + +### Resources +FCSS and FCE would both have a store, which would represent the majority of computing resources (directly related with storage cost) so that is the most relevant consideration either way. +It also is important to highlight that FCE may require less computing resources usage since FederatedCatalog does not have own instance. + + + +## Approach + +Running the FederatedCatalog own application should be straightforward. If need to test locally, the [Readme.md](https://github.com/eclipse-edc/FederatedCatalog/blob/main/README.md) includes a very helpful guide. In terms of infrastructure applicability, similar approach to Control Plane or Data Plane can be used. \ No newline at end of file From 40f9a94cc20e0e1900264b6973c9787608ec42cf Mon Sep 17 00:00:00 2001 From: bmg13 Date: Tue, 24 Sep 2024 15:04:26 +0100 Subject: [PATCH 02/14] Restructure based on PR. --- .../README.md | 36 ++++++++----------- 1 file changed, 15 insertions(+), 21 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 7ec42ecb3..e4d83170e 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -2,38 +2,32 @@ ## Decision -The **proposed solution is to have a FederatedCatalog as own standalone service**. -Choosing a solution that decouples from Control Plane (like the one used for the Data Plane) and also able to be scalable, we believe future proofs the Federated Catalog as a feature and embraces wider usage. Like so, having a FederatedCatalog service, scalable on its own based on user needs and not on Control Plane allows to best resource management in the long term and allows to add features to the Federated Catalog without the need to change the Control Plane logic (assuming no API usage change). -The Network Latency expected may be a concern, but not more than any other downstream dependencies of the service. +The **proposed solution is to have a Federated Catalog as own runnable service** capable of crawling all the wanted catalogs and expose that data. + ## Rationale -Currently two main options are being considered regarding the distribution of Tractus-X (TX) with the FederatedCatalog (FC): -- FederatedCatalog **running as a standalone service** with its own API providing the catalogs' data it contains upon request, similar to what happens currently with the DataPlane (DP). -- FederatedCatalog as **a ControlPlane (CP) extension**, i.e. embedded and shipped as any other extension. +The Federated Catalog aims at providing the aggregation of catalogs from multiple participants in a centralized point to reduce latency. -Taking under consideration both options mentioned above, the following topics were analyzed. +Choosing a solution that decouples from Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. +Like so, having a Federated Catalog service, scalable on its own based on user needs, and not on Control Plane, allows the best resource management in the long term. It also enables adding features to the Federated Catalog without the need to change the Control Plane logic (assuming no API usage change). -To reduce verbosity, the next acronyms are used -**FCSS**: Federated Catalog as a Standalone Service -**FCE**: Federated Catalog as a Control Plane Extension +Additionally, the Federated Catalog deployment flow can be very similar to EDC components, ensuring the consistency and no need for additional know-how to handle the CI. +Overall, aiming at independent upgrades and instance scalability, the decision above is more suitable when compared with the Federated Catalog embedded and shipped as any other extension in the ControlPlane. -### Coupling -Aiming at decoupling the FCSS is a better option, since changes on Control Plane do not demand action on FC and vice versa. Additionally, with this solution, **independent upgrades can be assured**, unless they include API changes. -### Scalability -FCSS can be better scalable since has no other service dependencies. If several instances of Control Plane are needed, there would be a need to either disable the extension on some tasks or accept several FederatedCatalog extentions, either being a scalability concern and showing limitations of FCE. So, if each instance of the Control Plane boots up a FederatedCatalog extension that can be an overkill and have undesired overhead. -### Network Latency -Since FCE does not require HTTP requests (being the FederatedCatalog embedded part of the client) the network latency expected is lower, even if FCSS is deployed in a participant own private network. The impact should be minimal, but FCSS would have more latency. +## Approach -### Resources -FCSS and FCE would both have a store, which would represent the majority of computing resources (directly related with storage cost) so that is the most relevant consideration either way. -It also is important to highlight that FCE may require less computing resources usage since FederatedCatalog does not have own instance. +The Federated Catalog running as own instance can be performed either by running respective Docker image in a container or running a generated `jar`. Both approaches should be very straightforward +> In case of aiming at testing locally, the [Readme.md](https://github.com/eclipse-edc/FederatedCatalog/blob/main/README.md) includes a very helpful guide for both approaches mentioned. +Regarding infrastructure concerns, this service deployment approach can be very similar to the Data Plane, like a creation of a `deployment-federatedcatalog.yaml` very similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47). Considering hosting and other user infrastructure requirements, once again, the Data Plane can be used as a baseline comparison. -## Approach +Relevant to highlight some potential challenges of the proposed approach. +- The Network Latency expected may be a concern, but not more than any other downstream dependencies of the service; +- Having its own instance (that itself contains a cache) may use considerable computing resources (related with storage cost). -Running the FederatedCatalog own application should be straightforward. If need to test locally, the [Readme.md](https://github.com/eclipse-edc/FederatedCatalog/blob/main/README.md) includes a very helpful guide. In terms of infrastructure applicability, similar approach to Control Plane or Data Plane can be used. \ No newline at end of file +Finally, and also working as a reminder, to enable this flow in any usage, it must be enabled. [See this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem) for additional information. \ No newline at end of file From a452459467e7d78d60ce263a2fecf6ea2c14c205 Mon Sep 17 00:00:00 2001 From: bmg13 Date: Thu, 26 Sep 2024 14:33:03 +0100 Subject: [PATCH 03/14] Changes from PR. --- .../README.md | 20 ++++++------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index e4d83170e..ad5ebb4a2 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -2,32 +2,24 @@ ## Decision -The **proposed solution is to have a Federated Catalog as own runnable service** capable of crawling all the wanted catalogs and expose that data. - +The proposed solution is to have a Federated Catalog as own runnable service capable of crawling all the chosen catalogs and expose that data. ## Rationale The Federated Catalog aims at providing the aggregation of catalogs from multiple participants in a centralized point to reduce latency. -Choosing a solution that decouples from Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. -Like so, having a Federated Catalog service, scalable on its own based on user needs, and not on Control Plane, allows the best resource management in the long term. It also enables adding features to the Federated Catalog without the need to change the Control Plane logic (assuming no API usage change). - -Additionally, the Federated Catalog deployment flow can be very similar to EDC components, ensuring the consistency and no need for additional know-how to handle the CI. - -Overall, aiming at independent upgrades and instance scalability, the decision above is more suitable when compared with the Federated Catalog embedded and shipped as any other extension in the ControlPlane. - - +Choosing a solution that decouples it from the Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. ## Approach -The Federated Catalog running as own instance can be performed either by running respective Docker image in a container or running a generated `jar`. Both approaches should be very straightforward +The Federated Catalog running as own instance can be performed either by running respective Docker image in a container or directly running a generated `jar`. -> In case of aiming at testing locally, the [Readme.md](https://github.com/eclipse-edc/FederatedCatalog/blob/main/README.md) includes a very helpful guide for both approaches mentioned. +> In case of aiming at testing locally, the [Readme.md](https://github.com/eclipse-edc/FederatedCatalog/blob/main/README.md) includes a very helpful guide. -Regarding infrastructure concerns, this service deployment approach can be very similar to the Data Plane, like a creation of a `deployment-federatedcatalog.yaml` very similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47). Considering hosting and other user infrastructure requirements, once again, the Data Plane can be used as a baseline comparison. +Regarding deployment, the Tractus-Connector Helm charts will be updated to include this service. Relevant to highlight some potential challenges of the proposed approach. - The Network Latency expected may be a concern, but not more than any other downstream dependencies of the service; - Having its own instance (that itself contains a cache) may use considerable computing resources (related with storage cost). -Finally, and also working as a reminder, to enable this flow in any usage, it must be enabled. [See this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem) for additional information. \ No newline at end of file +To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). \ No newline at end of file From 960a5f9582e71b0d9fdac923f2e70619d9a7e7f8 Mon Sep 17 00:00:00 2001 From: bmg13 Date: Fri, 27 Sep 2024 11:13:55 +0100 Subject: [PATCH 04/14] Small doc improvements --- .../README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index ad5ebb4a2..6a075db2c 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -18,8 +18,7 @@ The Federated Catalog running as own instance can be performed either by running Regarding deployment, the Tractus-Connector Helm charts will be updated to include this service. -Relevant to highlight some potential challenges of the proposed approach. -- The Network Latency expected may be a concern, but not more than any other downstream dependencies of the service; +Relevant to highlight a potential challenge of the proposed approach. - Having its own instance (that itself contains a cache) may use considerable computing resources (related with storage cost). To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). \ No newline at end of file From 48b1900fa819dbf0ec0f2652fc4fc07ef5cd8525 Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 30 Sep 2024 12:12:38 +0100 Subject: [PATCH 05/14] Changes from PR. --- .../README.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 6a075db2c..74f57f85c 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -2,23 +2,17 @@ ## Decision -The proposed solution is to have a Federated Catalog as own runnable service capable of crawling all the chosen catalogs and expose that data. +Federated Catalog will be deployable as a standalone component capable of crawling all the chosen catalogs and expose that data. The Tractus-Connector Helm charts will be updated to feature a new Federated Catalog deployment template. ## Rationale -The Federated Catalog aims at providing the aggregation of catalogs from multiple participants in a centralized point to reduce latency. - Choosing a solution that decouples it from the Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. -## Approach - -The Federated Catalog running as own instance can be performed either by running respective Docker image in a container or directly running a generated `jar`. +Having a specific runtime incurs on additional overhead (new Helm Chart, as example) and results in additional configuration complexity. Also, periodical crawling results in increased remote calls over time. However, being decoupled from the Control Plane, this solution permits the Federated Catalog to scale independently (based on own demand). -> In case of aiming at testing locally, the [Readme.md](https://github.com/eclipse-edc/FederatedCatalog/blob/main/README.md) includes a very helpful guide. +## Approach -Regarding deployment, the Tractus-Connector Helm charts will be updated to include this service. +Since the Federated Catalog will be a standalone runtime, the Tractus-Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity. -Relevant to highlight a potential challenge of the proposed approach. -- Having its own instance (that itself contains a cache) may use considerable computing resources (related with storage cost). To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). \ No newline at end of file From 4247446ab8dbbc67f0058e95bcd668fd6250f97e Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 30 Sep 2024 14:57:56 +0100 Subject: [PATCH 06/14] Merge both Distribution and TargetNodeDirectory decision proposals. --- .../README.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 74f57f85c..6a26d5514 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -1,18 +1,33 @@ -# Proposal for FederatedCatalog with Tractus-X distribution +# Proposal for FederatedCatalog with Tractus-X distribution and its TargetNodeDirectory ## Decision Federated Catalog will be deployable as a standalone component capable of crawling all the chosen catalogs and expose that data. The Tractus-Connector Helm charts will be updated to feature a new Federated Catalog deployment template. +Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will have a db/cache containing the BPNL's and Connectors' URL's of each partner a member wants the offers from. ## Rationale -Choosing a solution that decouples it from the Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. +Considering the Federated Catalog distribution, choosing a solution that decouples it from the Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. +Having a specific runtime incurs on additional overhead (new Helm Chart, as example) and results in additional configuration complexity. Also, periodical crawling results in increased remote calls over time. However, being decoupled from the Control Plane, this solution permits the Federated Catalog to scale independently (based on own demand). + +For TargetNodeDirectory it will be set by a new extension responsible for exposing an API, where a member can input the BPNL's of the participants from which the catalogs are wanted, and then it will retrieve and store the respective Connector URL's. This new extension would get the data from the Discovery Service, provided a BPNL, and will be named `DiscoveryServiceRetrieverExtension`. This solution allows the member to choose precisely the Target Catalog Nodes that interests them, resulting in reduced network calls and latency. +Additionally, if a Connector URL is registered (or unregistered) in the Discovery Service, the retriever will reflect it since it requests based on BPNL (which should not change) and the registered URL's will be returned. + +Other solution for the TargetNodeDirectory was also considered +- File in a S3 bucket (or different cloud provider's solution) + - This solution was discarded due to one file for all instead of each partner having the data that respectively needs does not match the requirement and this solution would lock the usage of a proprietary tool (cloud provider) being harder to sustain in the long run. -Having a specific runtime incurs on additional overhead (new Helm Chart, as example) and results in additional configuration complexity. Also, periodical crawling results in increased remote calls over time. However, being decoupled from the Control Plane, this solution permits the Federated Catalog to scale independently (based on own demand). ## Approach Since the Federated Catalog will be a standalone runtime, the Tractus-Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity. +To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). + +For its TargetNodeDirectory, the user is able to obtain the Connectors' URL's through the Discovery Service and store them in the new extension through its API. The API will allow to save a list of BPNLs (and Connectors' URL's if desired) and the `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. +This solution improves on the default one of having the data in a static file since a dynamic approach would avoid downtime when a change is required. -To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). \ No newline at end of file +Some limitations of this TargetNodeDirectory solution are: +- Each partner must have the BPNLs beforehand. If a new Partner registers and an existing partner would want their catalog, the BPNL (or Connector URL's) of the new partner must be obtained first; +- Deal with the overhead an additional persistence store; +- The usage of the Discovery Service requires a technical user account to access it (must be requested). \ No newline at end of file From b851b927095cda2442fe8cb15981c1f49ea98cec Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 7 Oct 2024 15:54:55 +0100 Subject: [PATCH 07/14] Changes from PR. --- .../README.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 6a26d5514..134d3a5b5 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -13,6 +13,8 @@ Having a specific runtime incurs on additional overhead (new Helm Chart, as exam For TargetNodeDirectory it will be set by a new extension responsible for exposing an API, where a member can input the BPNL's of the participants from which the catalogs are wanted, and then it will retrieve and store the respective Connector URL's. This new extension would get the data from the Discovery Service, provided a BPNL, and will be named `DiscoveryServiceRetrieverExtension`. This solution allows the member to choose precisely the Target Catalog Nodes that interests them, resulting in reduced network calls and latency. Additionally, if a Connector URL is registered (or unregistered) in the Discovery Service, the retriever will reflect it since it requests based on BPNL (which should not change) and the registered URL's will be returned. +This solution improves on the default one of having the data in a static file since a dynamic approach would avoid downtime when a change is required. + Other solution for the TargetNodeDirectory was also considered - File in a S3 bucket (or different cloud provider's solution) - This solution was discarded due to one file for all instead of each partner having the data that respectively needs does not match the requirement and this solution would lock the usage of a proprietary tool (cloud provider) being harder to sustain in the long run. @@ -25,7 +27,12 @@ Since the Federated Catalog will be a standalone runtime, the Tractus-Connector To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). For its TargetNodeDirectory, the user is able to obtain the Connectors' URL's through the Discovery Service and store them in the new extension through its API. The API will allow to save a list of BPNLs (and Connectors' URL's if desired) and the `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. -This solution improves on the default one of having the data in a static file since a dynamic approach would avoid downtime when a change is required. + +The retrieval of Connector URL's through the Discovery Service is enabled by the endpoint: +``` +POST: /api/administration/connectors/discovery +``` +In which, the body of the request can contain the BPNL's related with participants from which the catalogs want to be obtained. If no BPNL is provided (empty list) then all Connector URL's will be returned by the Discovery Service. Some limitations of this TargetNodeDirectory solution are: - Each partner must have the BPNLs beforehand. If a new Partner registers and an existing partner would want their catalog, the BPNL (or Connector URL's) of the new partner must be obtained first; From b4b00b005dcbc5943738eeb04f4dfd9ec40722aa Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 7 Oct 2024 18:36:41 +0100 Subject: [PATCH 08/14] Changes from PR. --- .../README.md | 50 ++++++++++++++++++- 1 file changed, 49 insertions(+), 1 deletion(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 134d3a5b5..b25f0356c 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -33,8 +33,56 @@ The retrieval of Connector URL's through the Discovery Service is enabled by the POST: /api/administration/connectors/discovery ``` In which, the body of the request can contain the BPNL's related with participants from which the catalogs want to be obtained. If no BPNL is provided (empty list) then all Connector URL's will be returned by the Discovery Service. +Information regarding the related API can be found [here](https://catenax-ev.github.io/docs/standards/CX-0001-EDCDiscoveryAPI#22-api-specification). Some limitations of this TargetNodeDirectory solution are: - Each partner must have the BPNLs beforehand. If a new Partner registers and an existing partner would want their catalog, the BPNL (or Connector URL's) of the new partner must be obtained first; - Deal with the overhead an additional persistence store; -- The usage of the Discovery Service requires a technical user account to access it (must be requested). \ No newline at end of file +- The usage of the Discovery Service requires a technical user account to access it (must be requested). + + +As indicated, the new service would have own API capable of: + +#### Save BPNL's +Request body would contain a list of BPNL's, allowing to store in bulk. +``` +[POST] /api/target-nodes +``` +Request Body Example +```json +[ "BPNL000000000001","BPNL000000000002" ] +``` + +#### Remove a stored BPNL +BPNL to be remvoed is sent as a path param. +``` +[DELETE] /api/target-nodes/{bpnl} +``` +#### Retrieve BPNL's +Get BPNL's (value and connectors associated with it). +``` +[POST] /api/target-nodes +``` +Request Body Example +```json +[ "BPNL000000000001","BPNL000000000002" ] +``` +Response Example +```json +[ + { + "bpn": "BPNL000000000001", + "connectorEndpoint": [ + "https://connector1/api/v1/dsp" + ] + }, + { + "bpn": "BPNL000000000002", + "connectorEndpoint": [ + "https://connector2/api/v1/dsp", + "https://connector3/api/v1/dsp", + "https://connector4/api/v1/dsp" + ] + } +] +``` \ No newline at end of file From 473e800aa85b950fea6044b316d2fbc012efbd7a Mon Sep 17 00:00:00 2001 From: bmg13 Date: Tue, 8 Oct 2024 08:29:45 +0100 Subject: [PATCH 09/14] Update retrieve path. --- .../README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index b25f0356c..46e2affca 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -61,7 +61,7 @@ BPNL to be remvoed is sent as a path param. #### Retrieve BPNL's Get BPNL's (value and connectors associated with it). ``` -[POST] /api/target-nodes +[POST] /api/target-nodes/request ``` Request Body Example ```json From b3c8ce28f715b6ee77b4260101327953b3c1696e Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 14 Oct 2024 11:05:18 +0100 Subject: [PATCH 10/14] Changes from PR. --- .../README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 46e2affca..e63ba3546 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -2,13 +2,12 @@ ## Decision -Federated Catalog will be deployable as a standalone component capable of crawling all the chosen catalogs and expose that data. The Tractus-Connector Helm charts will be updated to feature a new Federated Catalog deployment template. -Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will have a db/cache containing the BPNL's and Connectors' URL's of each partner a member wants the offers from. +The Federated Catalog will be deployed as a standalone component. The Tractus-X EDC Connector Helm charts will be updated to feature a new Federated Catalog deployment template. +Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will have a db/cache containing the BPNL's and Connectors' URL's of each partner a member wants the offers from. The member defines the BPNL's through a new API exposed by the extension. ## Rationale -Considering the Federated Catalog distribution, choosing a solution that decouples it from the Control Plane (like the one used for the Data Plane) and able to be scalable will future-proof the Federated Catalog as a feature and embraces wider usage. -Having a specific runtime incurs on additional overhead (new Helm Chart, as example) and results in additional configuration complexity. Also, periodical crawling results in increased remote calls over time. However, being decoupled from the Control Plane, this solution permits the Federated Catalog to scale independently (based on own demand). +While a standalone component (= K8S deployment) brings a slight increase in configuration complexity, its ability to be managed and scaled independently makes up for that. For TargetNodeDirectory it will be set by a new extension responsible for exposing an API, where a member can input the BPNL's of the participants from which the catalogs are wanted, and then it will retrieve and store the respective Connector URL's. This new extension would get the data from the Discovery Service, provided a BPNL, and will be named `DiscoveryServiceRetrieverExtension`. This solution allows the member to choose precisely the Target Catalog Nodes that interests them, resulting in reduced network calls and latency. Additionally, if a Connector URL is registered (or unregistered) in the Discovery Service, the retriever will reflect it since it requests based on BPNL (which should not change) and the registered URL's will be returned. @@ -22,11 +21,11 @@ Other solution for the TargetNodeDirectory was also considered ## Approach -Since the Federated Catalog will be a standalone runtime, the Tractus-Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity. +Since the Federated Catalog will be a standalone runtime, the Tractus-X EDC Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity. To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). -For its TargetNodeDirectory, the user is able to obtain the Connectors' URL's through the Discovery Service and store them in the new extension through its API. The API will allow to save a list of BPNLs (and Connectors' URL's if desired) and the `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. +For its TargetNodeDirectory, the user is able to obtain the Connectors' URL's through the Discovery Service and store them in the new extension through the new extensions' API. The API will allow to save a list of BPNLs (and Connectors' URL's if desired) and the `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. The retrieval of Connector URL's through the Discovery Service is enabled by the endpoint: ``` @@ -41,9 +40,10 @@ Some limitations of this TargetNodeDirectory solution are: - The usage of the Discovery Service requires a technical user account to access it (must be requested). -As indicated, the new service would have own API capable of: +As indicated, the new extension would have own API capable of: #### Save BPNL's +A member can add a BPNL through this API from which the Connector URL's are needed. This extension will iterate over the listed BPNL's and query the Discovery Service. Request body would contain a list of BPNL's, allowing to store in bulk. ``` [POST] /api/target-nodes @@ -54,6 +54,7 @@ Request Body Example ``` #### Remove a stored BPNL +Once a member understands that they do not need the Catalogs from a certain BPNL, this can be removed. BPNL to be remvoed is sent as a path param. ``` [DELETE] /api/target-nodes/{bpnl} From 800dbc1962a149029b91bda9a459c82979d26408 Mon Sep 17 00:00:00 2001 From: bmg13 Date: Mon, 14 Oct 2024 16:12:13 +0100 Subject: [PATCH 11/14] Changes from PR. --- .../README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index e63ba3546..1bd9167c1 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -26,18 +26,19 @@ Since the Federated Catalog will be a standalone runtime, the Tractus-X EDC Conn To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). For its TargetNodeDirectory, the user is able to obtain the Connectors' URL's through the Discovery Service and store them in the new extension through the new extensions' API. The API will allow to save a list of BPNLs (and Connectors' URL's if desired) and the `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. +If no BPNL is saved (to be crawled) then the extension will not request data from the Discovery Service, since the number of catalogs in the space can be significant. The retrieval of Connector URL's through the Discovery Service is enabled by the endpoint: ``` POST: /api/administration/connectors/discovery ``` -In which, the body of the request can contain the BPNL's related with participants from which the catalogs want to be obtained. If no BPNL is provided (empty list) then all Connector URL's will be returned by the Discovery Service. +In which, the body of the request can contain the BPNL's related with participants from which the catalogs want to be obtained. Although the DiscoveryService allows to perform a request without providing BPNL's (empty list) it will not be done by the extension. Information regarding the related API can be found [here](https://catenax-ev.github.io/docs/standards/CX-0001-EDCDiscoveryAPI#22-api-specification). Some limitations of this TargetNodeDirectory solution are: -- Each partner must have the BPNLs beforehand. If a new Partner registers and an existing partner would want their catalog, the BPNL (or Connector URL's) of the new partner must be obtained first; +- Each partner must have the BPNLs beforehand. If a new Partner is registered and an existing partner would want their catalog, the BPNL (or Connector URL's) of the new partner must be obtained first and added through the new extension API; - Deal with the overhead an additional persistence store; -- The usage of the Discovery Service requires a technical user account to access it (must be requested). +- The usage of the Discovery Service requires a technical user account to access it (must be requested). After obtaining them, the credentials can be stored in the vault. As indicated, the new extension would have own API capable of: From 4b75211ca9ffac01da973e2f29f110cb18fc3e76 Mon Sep 17 00:00:00 2001 From: bmg13 Date: Tue, 29 Oct 2024 11:23:21 +0000 Subject: [PATCH 12/14] Update documentation to include the DID input. --- .../README.md | 45 ++++++++++--------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 1bd9167c1..1bb36b118 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -3,14 +3,14 @@ ## Decision The Federated Catalog will be deployed as a standalone component. The Tractus-X EDC Connector Helm charts will be updated to feature a new Federated Catalog deployment template. -Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will have a db/cache containing the BPNL's and Connectors' URL's of each partner a member wants the offers from. The member defines the BPNL's through a new API exposed by the extension. +Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will have a db/cache containing the DID's of each partner a member wants the offers from. The member defines the DID's through a new API exposed by the extension. ## Rationale While a standalone component (= K8S deployment) brings a slight increase in configuration complexity, its ability to be managed and scaled independently makes up for that. -For TargetNodeDirectory it will be set by a new extension responsible for exposing an API, where a member can input the BPNL's of the participants from which the catalogs are wanted, and then it will retrieve and store the respective Connector URL's. This new extension would get the data from the Discovery Service, provided a BPNL, and will be named `DiscoveryServiceRetrieverExtension`. This solution allows the member to choose precisely the Target Catalog Nodes that interests them, resulting in reduced network calls and latency. -Additionally, if a Connector URL is registered (or unregistered) in the Discovery Service, the retriever will reflect it since it requests based on BPNL (which should not change) and the registered URL's will be returned. +For TargetNodeDirectory it will be set by a new extension responsible for exposing an API, where a member can input the DID's of the participants from which the catalogs are wanted, and then it will retrieve and store the respective Connector URL's. This new extension would get the data from the Discovery Service, and will be named `DiscoveryServiceRetrieverExtension`. This solution allows the member to choose precisely the Target Catalog Nodes that interests them, resulting in reduced network calls and latency. +Additionally, if a Connector URL is registered (or unregistered) in the Discovery Service, the retriever will reflect it since it requests based on BPN and the registered URL's will be returned. This solution improves on the default one of having the data in a static file since a dynamic approach would avoid downtime when a change is required. @@ -25,61 +25,64 @@ Since the Federated Catalog will be a standalone runtime, the Tractus-X EDC Conn To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). -For its TargetNodeDirectory, the user is able to obtain the Connectors' URL's through the Discovery Service and store them in the new extension through the new extensions' API. The API will allow to save a list of BPNLs (and Connectors' URL's if desired) and the `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. -If no BPNL is saved (to be crawled) then the extension will not request data from the Discovery Service, since the number of catalogs in the space can be significant. +For its TargetNodeDirectory, the extension is able to obtain the Connectors' URL's through the Discovery Service and store them. Two API's will be provided in this new extension, at least during alpha stage, one to allow the user to input a list of DID's and other for BPN's. The `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. +If no DID is saved (to be crawled) then the extension will not request data from the Discovery Service, since the number of catalogs in the space can be significant. + +A DID added through the `DiscoveryServiceRetrieverExtension` API will be resolved with the BDRS client to obtain the BPN which will be used to query the Discovery Service. the BDRS client must be updated [since only allows to resolve a BPN to a DID and not the other way around](https://github.com/eclipse-tractusx/tractusx-edc/blob/8e1a3202be77d6374731dee5aaf6847feec8963a/spi/bdrs-client-spi/src/main/java/org/eclipse/tractusx/edc/spi/identity/mapper/BdrsClient.java). A change to resolve a BPN given the respective DID has to be done prior to the new extension. The retrieval of Connector URL's through the Discovery Service is enabled by the endpoint: ``` POST: /api/administration/connectors/discovery ``` -In which, the body of the request can contain the BPNL's related with participants from which the catalogs want to be obtained. Although the DiscoveryService allows to perform a request without providing BPNL's (empty list) it will not be done by the extension. +In which, the body of the request can contain the BPN's related with participants from which the catalogs want to be obtained. Although the DiscoveryService allows to perform a request without providing BPN's (empty list) it will not be done by the extension. Information regarding the related API can be found [here](https://catenax-ev.github.io/docs/standards/CX-0001-EDCDiscoveryAPI#22-api-specification). Some limitations of this TargetNodeDirectory solution are: -- Each partner must have the BPNLs beforehand. If a new Partner is registered and an existing partner would want their catalog, the BPNL (or Connector URL's) of the new partner must be obtained first and added through the new extension API; +- Each partner must have the DID's beforehand. If a new Partner is registered and an existing partner would want their catalog, the DID (or BPN) of the new partner must be obtained first and added through the new extension API; - Deal with the overhead an additional persistence store; -- The usage of the Discovery Service requires a technical user account to access it (must be requested). After obtaining them, the credentials can be stored in the vault. +- The usage of the Discovery Service requires a technical user account to access it (must be requested). After obtaining them, the credentials can be stored in the vault; +- Change in the BDRS client to allow resolve a BPN provided the DID. As indicated, the new extension would have own API capable of: -#### Save BPNL's -A member can add a BPNL through this API from which the Connector URL's are needed. This extension will iterate over the listed BPNL's and query the Discovery Service. -Request body would contain a list of BPNL's, allowing to store in bulk. +#### Save DID's +A member can add a DID (or BPN while two API's are maintained) through this API from which the Connector URL's are needed. This extension will iterate over the listed DID's, resolved them and query the Discovery Service. +Request body would contain a list of BPN's, allowing to store in bulk. ``` [POST] /api/target-nodes ``` Request Body Example ```json -[ "BPNL000000000001","BPNL000000000002" ] +[ "did:web:info:api:administration:staticdata:did:BPNL000000000001","did:web:info:api:administration:staticdata:did:BPNL000000000002" ] ``` -#### Remove a stored BPNL -Once a member understands that they do not need the Catalogs from a certain BPNL, this can be removed. -BPNL to be remvoed is sent as a path param. +#### Remove a stored DID +Once a member understands that they do not need the Catalogs from a certain DID, this can be removed. +DID to be removed is sent as a path param. ``` -[DELETE] /api/target-nodes/{bpnl} +[DELETE] /api/target-nodes/{did} ``` -#### Retrieve BPNL's -Get BPNL's (value and connectors associated with it). +#### Retrieve DID's +Get DID's (value and connectors associated with it). ``` [POST] /api/target-nodes/request ``` Request Body Example ```json -[ "BPNL000000000001","BPNL000000000002" ] +[ "did:web:info:api:administration:staticdata:did:BPNL000000000001","did:web:info:api:administration:staticdata:did:BPNL000000000002" ] ``` Response Example ```json [ { - "bpn": "BPNL000000000001", + "did": "did:web:info:api:administration:staticdata:did:BPNL000000000001", "connectorEndpoint": [ "https://connector1/api/v1/dsp" ] }, { - "bpn": "BPNL000000000002", + "bpn": "did:web:info:api:administration:staticdata:did:BPNL000000000002", "connectorEndpoint": [ "https://connector2/api/v1/dsp", "https://connector3/api/v1/dsp", From b082df185f85ccd0aa5d1c64fa19514becee57ea Mon Sep 17 00:00:00 2001 From: bmg13 Date: Fri, 8 Nov 2024 17:06:23 +0000 Subject: [PATCH 13/14] Changes from PR. --- .../README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index 1bb36b118..e0dc7f660 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -3,7 +3,7 @@ ## Decision The Federated Catalog will be deployed as a standalone component. The Tractus-X EDC Connector Helm charts will be updated to feature a new Federated Catalog deployment template. -Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will have a db/cache containing the DID's of each partner a member wants the offers from. The member defines the DID's through a new API exposed by the extension. +Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will expose an API to allow adding participant's identifiers which will be used to obtain the respective data from the Discovery Service. ## Rationale From 8a3f859f388cbff83e476f248a1952eb8a32ee9e Mon Sep 17 00:00:00 2001 From: bmg13 Date: Wed, 13 Nov 2024 15:02:56 +0000 Subject: [PATCH 14/14] Changes from PR. --- .../README.md | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md index e0dc7f660..de8acf0e3 100644 --- a/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md +++ b/docs/development/decision-records/2024-09-16-proposal-federated-catalog-distribution/README.md @@ -2,13 +2,15 @@ ## Decision -The Federated Catalog will be deployed as a standalone component. The Tractus-X EDC Connector Helm charts will be updated to feature a new Federated Catalog deployment template. +The Federated Catalog Cache will be deployed as a standalone component. The Tractus-X EDC Connector Helm charts will be updated to feature a new Federated Catalog deployment template. Regarding the TargetNodeDirectory, a new extension in the FederatedCatalog will expose an API to allow adding participant's identifiers which will be used to obtain the respective data from the Discovery Service. ## Rationale While a standalone component (= K8S deployment) brings a slight increase in configuration complexity, its ability to be managed and scaled independently makes up for that. +## Approach + For TargetNodeDirectory it will be set by a new extension responsible for exposing an API, where a member can input the DID's of the participants from which the catalogs are wanted, and then it will retrieve and store the respective Connector URL's. This new extension would get the data from the Discovery Service, and will be named `DiscoveryServiceRetrieverExtension`. This solution allows the member to choose precisely the Target Catalog Nodes that interests them, resulting in reduced network calls and latency. Additionally, if a Connector URL is registered (or unregistered) in the Discovery Service, the retriever will reflect it since it requests based on BPN and the registered URL's will be returned. @@ -18,15 +20,10 @@ Other solution for the TargetNodeDirectory was also considered - File in a S3 bucket (or different cloud provider's solution) - This solution was discarded due to one file for all instead of each partner having the data that respectively needs does not match the requirement and this solution would lock the usage of a proprietary tool (cloud provider) being harder to sustain in the long run. +Since the Federated Catalog Cache will be a standalone runtime, the Tractus-X EDC Connector Helm charts will be updated to include the Federated Catalog Cache as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity. -## Approach - -Since the Federated Catalog will be a standalone runtime, the Tractus-X EDC Connector Helm charts will be updated to include the Federated Catalog as a separated deployment. The update will include the creation of a specific `deployment-federatedcatalog.yaml`, similar [to this one](https://github.com/eclipse-tractusx/tractusx-edc/blob/a263bf71a110245657131509d4b37d058a1d220d/charts/tractusx-connector-azure-vault/templates/deployment-dataplane.yaml#L47) (for `ingress` and `hpa` as well), for different scenarios (InMemory, PostreSQL, etc.). This results in added configuration complexity. - -To enable the Federated Catalog flow, please [see this table](https://github.com/eclipse-tractusx/tractusx-edc/blob/75bdacbad43e2cad352204ea28a359c6aac7adea/docs/development/management-domains/README.md#enable-and-configure-the-crawler-subsystem). - -For its TargetNodeDirectory, the extension is able to obtain the Connectors' URL's through the Discovery Service and store them. Two API's will be provided in this new extension, at least during alpha stage, one to allow the user to input a list of DID's and other for BPN's. The `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog. -If no DID is saved (to be crawled) then the extension will not request data from the Discovery Service, since the number of catalogs in the space can be significant. +For its TargetNodeDirectory, the extension is able to obtain the Connectors' URL's through the Discovery Service and store them. Two API's will be provided in this new extension, at least during alpha stage, one to allow the user to input a list of DID's and other for BPN's. The `DiscoveryServiceRetrieverExtension` is responsible to retrieve the data and store it (in memory or in a database). The URL's can later be retrieved and crawled by the Federated Catalog Cache. +By default no TargetNodes are stored, so the extension will not request data from the Discovery Service. A DID added through the `DiscoveryServiceRetrieverExtension` API will be resolved with the BDRS client to obtain the BPN which will be used to query the Discovery Service. the BDRS client must be updated [since only allows to resolve a BPN to a DID and not the other way around](https://github.com/eclipse-tractusx/tractusx-edc/blob/8e1a3202be77d6374731dee5aaf6847feec8963a/spi/bdrs-client-spi/src/main/java/org/eclipse/tractusx/edc/spi/identity/mapper/BdrsClient.java). A change to resolve a BPN given the respective DID has to be done prior to the new extension. @@ -39,7 +36,6 @@ Information regarding the related API can be found [here](https://catenax-ev.git Some limitations of this TargetNodeDirectory solution are: - Each partner must have the DID's beforehand. If a new Partner is registered and an existing partner would want their catalog, the DID (or BPN) of the new partner must be obtained first and added through the new extension API; -- Deal with the overhead an additional persistence store; - The usage of the Discovery Service requires a technical user account to access it (must be requested). After obtaining them, the credentials can be stored in the vault; - Change in the BDRS client to allow resolve a BPN provided the DID.