feat: start services after first heartbeat response processed

[WenyXu]

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Start services after the first heartbeat response is processed

Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

[codecov]

Codecov Report

Merging #2424 (f5d4a2d) into develop (4a82926) will decrease coverage by 0.03%.
Report is 1 commits behind head on develop.
The diff coverage is 17.39%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2424      +/-   ##
===========================================
- Coverage    77.72%   77.70%   -0.03%     
===========================================
  Files          724      724              
  Lines       114092   114126      +34     
===========================================
- Hits         88679    88676       -3     
- Misses       25413    25450      +37     

[fengjiachun]

Will it encounter a readonly region error if we start services directly?
But blocking datanode start does not seem to be a good idea, consider providing a parameter such as start(blocking: bool), we can call start(true) on testing.

[fengjiachun]

I am afraid that may not be effective to blocking datanode from starting, because the route information of metasrv will guide the Frontend to access the datanode, and the Frontend will still encounter errors, but not readonly, but errors such as no service.

[WenyXu]

I am afraid that may not be effective to blocking datanode from starting, because the route information of metasrv will guide the Frontend to access the datanode, and the Frontend will still encounter errors, but not readonly, but errors such as no service.

1. Waits for the first heartbeat response should be very fast. The first thing (without any wait) the datanode will do is send a heartbeat to metasrv.
2. No Service means the datanode is still under coordination, and the service is currently unavailable. Then, the gateway or other component can know it. If we skip the coordination, No components will know this except the Enduesr (Then they may scream, why my table is readonly🤣).
3. Retrying writes on the "read-only" table is problematic. It's better to let upper service know the datanode is currently unavailable

[killme2008]

The datanode must try its best to start and provide services properly at startup. If it can't for some reason (such as metasrv availability or network issues etc.), it can retry in the background.

[killme2008]

But my question is do we need to provide an explicit synchronous RPC for such requests or just wait for the first heartbeat response asynchronously in an implicit way?

I think the former is better for understanding and debugging.

[MichaelScofield]

I suggest going the full async way, and combining region's open/close state with table route. Like this:

• Datanode starts with heartbeat enabled, but not waiting on the "first heartbeat" to decide which regions to open;
• Metasrv decides which regions a Datanode can open, and carry this infomation in heartbeat's response;
• When regions are opened in a Datanode, it's reflected in the Datanode's heartbeats. Metasrv knows it, then modify the table route to let the regions visible to Frontend(clients).

The essential of this idea is fully reverse the region open/close management from "Datanode requests regions from Metasrv" to "regions are assigned to Datanode by Metasrv, dynamically". Also combining region open/close state with table route makes clients happy.

[WenyXu]

But my question is do we need to provide an explicit synchronous RPC for such requests or just wait for the first heartbeat response asynchronously in an implicit way?

I think the former is better for understanding and debugging.

The synchronous RPC is too heavy.