ZIti Controller in HA mode returns empty list on /edge/management/v1/sessions endpoint

[nenkoru]

Hello there!

After a thorough investigation of an issue outlined in this[1] thread I found out that in ziti-sdk-c:/library/ziti.c:1442 there is a change of state of a session when a new ER is observed making it to send a request to a /edge/client/v1/sessions/{id} but for whatever reason current raft leader controller doesn't respond with a session. Nor does the management API /sessions endpoint. Nor does this info exist in a bolt ctrl.db.
Could be replicated at current HEAD at openziti/ziti@1932a6d

Steps to replicate[2]:

1. Have a raft cluster and local ziti-edge-tunnel ready
2. Register two ERs with a controller, run one of them and put the other one offline
3. Run a demo echo server ziti demo ...
4. Run a curl against an echo server
5. Shut down one of the ERs
6. Start another ER and wait for a terminator for an echo-server to be created
7. Run a curl against an echo server and observe that the request is getting stuck
8. Observe in logs of ziti-edge-tunnel the following breadcrumbs suggesting there is an issue with the session refresh, making it unable to use the fresh ER(as per my understanding)

 (77700)[       73.500] VERBOSE ziti-sdk:ziti_ctrl.c:206 ctrl_resp_cb() ctrl[127.0.0.1:443] received headers GET[/sessions/ea78bff3-c674-4ecc-afd9-ae12559d7a99]
(77700)[       73.500]   DEBUG ziti-sdk:ziti_ctrl.c:485 ctrl_body_cb() ctrl[127.0.0.1:443] completed GET[/sessions/ea78bff3-c674-4ecc-afd9-ae12559d7a99] in 0.016 s
(77700)[       73.500]    WARN ziti-sdk:connect.c:453 refresh_session_cb() failed to refresh session

[1] https://openziti.discourse.group/t/ziti-edge-tunnel-doesnt-connect-to-a-service-using-another-er-after-the-initial-one-went-offline/3586
[2] https://asciinema.org/a/8khQh2sMsNQbdLCrJG86Xitvb

[ekoby]

SDK should not try to refresh session when network is in HA mode. transferring to c-sdk repo

[nenkoru]

SDK should not try to refresh session when network is in HA mode. transferring to c-sdk repo

What should be done instead?
SDK might request the edge-routers, try to connect through them. But it won't know whether those edge routers are able to route traffic to the service based on policies.
Should SDK try to request /services/{id}/edge-routers[1] and merge them with the ones identity is able to connect through?

[1] https://openziti.io/docs/reference/developer/api/edge-client-reference#tag/Service/operation/listServiceEdgeRouters