Investigate next gen architecture for ecchronos

[tommystendahl]

The current architecture with one ecchronos node for each cassandra node in a cluster doesn't scale well for large clusters. It would be very good if this could be changed so we could use one or two ecchronos nodes for each DC or maybe a small number of ecchronos nodes that handles the the whole cluster.

The goal of this issue would be to propose how a new architecture could work and a plan for how it could be implemented.

Expected outcome:

• a proposal for a new architecture
• a plan with tasks or milestones to implement the new atchitecture

[patrickclee0207]

really curious about this myself. I started trying ecchronos recently and so far having better success than the way I've been using Cassandra-reaper but I've wondered how this would work on larger clusters that I have. Largest I've tested so far is 60 node cluster.

[tommystendahl]

@patrickclee0207 Thanks for showing interest and I'm happy to here that you had some success trying out ecchronos. It works fine for smaller cluster and from our experience I think it still scales ok beyond 60 nodes. But when the clusters grow in size the number of connections each ecchronos node has to the cluster becomes a bottleneck. I hope we can find some good way to redesign ecchronos to address this.

[itskarlsson]

really curious about this myself. I started trying ecchronos recently and so far having better success than the way I've been using Cassandra-reaper but I've wondered how this would work on larger clusters that I have. Largest I've tested so far is 60 node cluster.

Just to expand on what Tommy said. Smaller clusters in this case are clusters below 100 nodes or close to 100 nodes. These will largely never see this issue. However problems arise when there are many hundreds of nodes or even thousands.