Unable to access service: service xxx has no terminators for instanceId xxx

[theheckwr]

Using the compiled ziti-edge-tunnel binary on Manjaro to connect to a service leads to a connection error and logging the following lines:

[       76.038]    INFO tunnel-cbs:ziti_dns.c:506 format_resp() found record[100.64.0.4] for query[1:<address>]
[       76.183]   ERROR ziti-sdk:connect.c:941 connect_reply_cb() conn[0.0/Connecting] failed to connect, reason=service <serviceid> has no terminators for instanceId <identity>
[       76.183]   ERROR tunnel-cbs:ziti_tunnel_cbs.c:104 on_ziti_connect() ziti dial failed: connection is closed

However the same service is accessible using the Android tunneler app.

• Environment: Manjaro (Kernel: 5.19.7-1-MANJARO)
• Using the latest ziti-edge-tunnel binaries (v0.19.11-local) and compiling it from main (v0.19.11-1-g971f4fc)

Some additional info:
I can see terminators getting created in the ZAC when I update the services.
But the issue on the client tunneler's side still persists.

[dovholuknf]

Hi @theheckwr . Thanks for the report. This could be a policy issue. If you run ziti edge policy-advisor identities does your identity have dial capability? I'm thinking it might have bind but not dial?

[scareything]

If the policy check doesn't turn up anything, could you also please share your service configurations, and show a list of terminators at the time the dial fails?

[theheckwr]

I checked the policy advisor and everything looks fine.
As for the service configuration, this is what I'm able to dump via the CLI:

The terminators at the time of the dial failure:

In the case it matters: I only did the dumps with the CLI, for the configuration I used the ZAC.
Thanks in advance.

[scareything]

Thanks for the info! I was hoping to see the actual service configuration (json), e.g. via ziti show config:

 ziti edge show config 3d7TgvPfhSC1RfiFrgsBx3
 ziti edge show config 658owvlrnClMBJgZEbVuZ8

Feel free to sanitize the content to your liking. I'm mostly interested in the dialOptions.identity value from the intercept.v1, and the listenOptions.identity value from the host.v1.

Thanks!

[theheckwr]

Oh okay, I'm new to ZITI, so I don't know all the different sections that well.
But here are the results of the two commands you asked for:


Does this help or did I sanitize too much?
If I can help with anything else, don't hesitate to ask again.

[scareything]

This helps!

tl;dr:

I think you want to remove the dialOptions.identity value from your intercept.v1 configuration. I haven't checked, but I'm pretty sure that the android tunneler simply doesn't look at this field yet.

cursory explanation that you are welcome to ignore

The dialOptions.identity and listenOptions.identity fields work hand in hand. Basically they give you a way to connect to a specific hosting tunneler when more than one tunneler is hosting the same service.

If you use "hard-coded" values in these fields then they won't be much help. But you can use variable references in these fields. dialOptions.identity can contain and combination of the following (intermixed with text):

• "$dst_ip" (the IP that was intercepted by the intercepting tunneler)
• "$dst_port" (the port that was intercepted)
• "$dst_hostname" (the hostname that was intercepted - this is determined by looking up the intercepted IP in the tunneler's internal DNS server's mappings)

listenOptions.identity only supports a single variable right now - "$tunneler_id.name". So for example if you named your hosting tunneler identities as IPs or hostnames that they represent, you could define a single service configuration that connects to any hosting tunneler. (in this case your intercept address(es) will probably be a CIDR and/or wildcard domain.

[dovholuknf]

Given the displayed hosting config has no "listenOptions" set - it seems like it's a sure thing that really what @theheckwr wants is to remove that "identity" from dialOptions. :)

And - we found a bug in the android tunneler !

[dovholuknf]

Android bug filed openziti/ziti-sdk-android#2

[theheckwr]

Huge thanks to you both!
Yes, removing the dialOptions.identity did bring the two services to life.
Somehow the ZAC dialog led me down the wrong path and I wasn't able to track that issue down.
With the Android bug filed, I'm closing this issue.
Thanks again.

[dovholuknf]

If you have a flow in the zac that would have helped you or made more sense - and want to file an enhancement request over at https://github.com/openziti/ziti-console that'd be much appreciated. tell us how we could have made it better/easier :)

glad you're sorted!