I want to know can every Datanode use same s3 as storage?

[Murasame233]

I want to know can every Datanode use same s3 as storage?
Here's my use example:

• three physical server
• every server's GreptimeDB connect to the same s3.
• I want to make sure when one or two server goes offline (Shutdown/Network disconnected) and all of the data still available.

[waynexia]

Yes. Datanodes in one cluster can use the same S3 bucket and root. They'll coordinate things like access permission and directory by themselves.

I want to make sure when one or two server goes offline (Shutdown/Network disconnected) and all of the data still available.

This is also supported after v0.6. When a node is down, meta server will failover data to other available nodes. You can also schedule workload manually, see region migration. Notice these functionalities rely on shared storage services. Like kafka for WAL and S3 for storage.

[Murasame233]

I think it's Yes about when server offline and the data still available based on your answer.

And I want to know, is there any feature can work on this condition:

• Normal the data will write to s3 and local.
• when server loss internet connection, data only writes to local.
• when server's internet reconnected, the data on the local which s3 don't have will write to s3. And the new data will write to s3 and local at same time.

[Murasame233]

I think I have to deploy frontend and Datanode on each server.
For example I want to feed each servers' CPU usage to the DB.

[waynexia]

Writing to multiple storage backends is not supported on the server side. Maybe an S3-compatible middleware proxy can meet your expectations.

But before that, if a datanode server cannot connect to the meta server, it will lose write permission soonly to keep data consistent. In this situation all write requests to that server will be rejected.

[waynexia]

I think I have to deploy frontend and Datanode on each server. For example I want to feed each servers' CPU usage to the DB.

There is a metric process_cpu_seconds_total that counts CPU usage per process. You can still monitor CPU usage by components on the same namespace

[Murasame233]

Now I know How Can I do it. I should write a data middleware proxy.
When remote GreptimeDB available, use remote.
When remote DB not available, use local.
When remote DB is available again, push local data and use remote. Then clear local data.

Give every server a standalone GreptimeDB. And deploy a main GrepTimeDB Cluster on cloud.