Introduce support for Ranger Hub deployed in 2 DCs replicate nodes from each other

[ravidesai47]

Ranger Hub Server allows pointing to multiple upstreams.

In a 2 DC setup, if we want to setup a Ranger Hub Server providing details of nodes across both DCs it requires below deployments.

• Setting up Ranger Hub Server in each DC pointing to ZK cluster of each DC as upstream. (RangerZKDC1 and RangerZKDC2)
• Setting up another Ranger Hub Server in each DC pointing to Ranger Hub Servers setup in previous step (RangerZKDC1 and RangerZKDC2) from each deployment fetching information of nodes from both DCs.

This requires us to configure 4 Ranger Hub Server deployments. To reduce the maintenance overhead of 4 deployments, we can reduce the required deployments to only 2.

Expected Setup:

• Ranger Hub Server in each DC pointing to 2 upstreams
  • ZK cluster of own DC
  • Ranger Hub Server of another DC

In this setup, ZK cluster will provide nodes from the current DC and Ranger Hub Server from another DC will provide details of the nodes from another DC.

In the current implementation the same is not possible because it is resulting in an infinite loop of calls across both instances. R1 server calling R2 for resolving nodes and when R2 receives a call, R2 server calling R1 for resolving nodes and this continues infinitely due to cyclic dependency in the configuration.

If we can break the loop then the above setup can help in reducing maintenance overhead.

[santanusinha]

Configuring multiple upstreams is already supported. The server will deduplicate common nodes as well

…

On Fri, 13 Dec, 2024, 19:13 Ravi Desai, ***@***.***> wrote: Ranger Hub Server allows pointing to multiple upstreams. In a 2 DC setup, if we want to setup a Ranger Hub Server providing details of nodes across both DCs it requires below deployments. - Setting up Ranger Hub Server in each DC pointing to ZK cluster of each DC as upstream. (RangerZKDC1 and RangerZKDC2) - Setting up another Ranger Hub Server in each DC pointing to Ranger Hub Servers setup in previous step (RangerZKDC1 and RangerZKDC2) from each deployment fetching information of nodes from both DCs. This requires us to configure 4 Ranger Hub Server deployments. To reduce the maintenance overhead of 4 deployments, we can reduce the required deployments to only 2. Expected Setup: - Ranger Hub Server in each DC pointing to 2 upstreams - ZK cluster of own DC - Ranger Hub Server of another DC In this setup, ZK cluster will provide nodes from the current DC and Ranger Hub Server from another DC will provide details of the nodes from another DC. In the current implementation the same is not possible because it is resulting in an infinite loop of calls across both instances. R1 server calling R2 for resolving nodes and when R2 receives a call, R2 server calling R1 for resolving nodes and this continues infinitely due to cyclic dependency in the configuration. If we can break the loop then the above setup can help in reducing maintenance overhead. — Reply to this email directly, view it on GitHub <#52>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAME5Z6TNPUR5IEQUHR5WXT2FLQAJAVCNFSM6AAAAABTSAQK2WVHI2DSMVQWIX3LMV43ASLTON2WKOZSG4ZTQNBTGU2TCNA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>