Only one node at a time working behind NAT?

[justyns]

I am not sure where I'm going wrong, so hoping someone has some suggestions.

I'm using Omni to manage several Talos nodes. Some of these nodes are regular VPS with only a public ip, but most of them are behind a firewall/NAT.

When using kubespan, it seems like only a single node behind each NAT is reachable at a time.

For example:

❯ talosctl get kubespanpeerstatuses -n 100.64.0.71 | grep 136.2.3.4
100.64.0.71   kubespan    KubeSpanPeerStatus   DPpTH1uGBArGXXFroT0hSil+GgCO+Z5SfuHa02xPH2A=   80        talos-home-2          136.2.3.4:51820     unknown   0            51356
100.64.0.71   kubespan    KubeSpanPeerStatus   E5ppzD4YV6fskh0EF2q8QTjR0WoolbOwJ3AqoqZvkmQ=   75        talos-home-1          136.2.3.4:51820     up        189919316    25174956
100.64.0.71   kubespan    KubeSpanPeerStatus   frzfjmAustopLIbe+/yNkxA4WTKUgIUjSQ1U2wIGIi8=   81        talos-home-3          136.2.3.4:51820     unknown   0            51504

IP changed, but all three of those servers are behind a NAT with the IP 136.2.3.4. They are able to connect to each other, but the node at 100.64.0.71 is only able to connect to talos-home-1 at the moment.

If I check from talos-home-2, it lists pubvm-1 (100.64.0.71) as down.

❯ talosctl get kubespanpeerstatuses -n 100.64.0.46
NODE          NAMESPACE   TYPE                 ID                                             VERSION   LABEL                 ENDPOINT                STATE     RX         TX
100.64.0.46   kubespan    KubeSpanPeerStatus   648nI+gX6tzO4n/w4z0uuTp624lNZqoSsAoxKncz714=   79        pubvm-1       75.2.3.4:51820      down      0          49432

75.2.3.4 is a vm with a public ip and no firewall between it and the internet. I don't have any firewall rules/network policies configured in talos/k8s either.

This is my kubespan configuration:

machine:
    network:
        kubespan:
            enabled: true
            filters:
                endpoints:
                    - 0.0.0.0/0
                    - 192.168.0.0/16
                    - '!100.64.0.0/10'
                    - '!fd7a:115c:a1e0::/48'

100.64.0.0/10 and fd7a:115c:a1e0::/48 are tailscale cidrs, so I'm filtering them out so that kubespan doesn't use them. 192.168.0.0/16 is the LAN where the first 3 servers are able to connect locally. I also have other servers at a different location with a site-to-site vpn within this range. They're also able to connect to these servers using 192.168.0.0/16 addresses.

Am I missing anything obvious? Also, if I am using the tailscale extension, does kubespan actually help me? I'm wondering if I should just try disabling it and relying on tailscale for node-to-node communication? Assuming flannel would also work on top of it.

[smira]

I don't know about Tailscale - you probably will need to figure that out.

As for the KubeSpan - KubeSpan requires a direct one-way connection for each pair of machines in the cluster.

E.g. if you have one set of machines in a private network and another in a public cloud with public IPs, that would work:

• machines in private network can connect to each other
• machines from private network can connect to public cloud machines

But KubeSpan won't work if you have two set of machines, both behind each own NAT.

[justyns]

Ahh thank you, that may be part of my problem at least.

I assume there's no routing taking place either, so if I have two sets of machines behind their own NAT, and also a set of machines with public ips - the two sets behind nat still won't be able to connect to each other?

I'll do some tests without kubespan enabled, and only using tailscale as the mesh network to see if it's easy to get working or not.

[smira]

I assume there's no routing taking place either, so if I have two sets of machines behind their own NAT, and also a set of machines with public ips - the two sets behind nat still won't be able to connect to each other?

Yes, basically KubeSpan doesn't have DERP servers like Tailscale does.

[justyns]

In case anyone else runs into this - I ended up disabling Kubespan and just relying on tailscale for connectivity instead.

Yes, basically KubeSpan doesn't have DERP servers like Tailscale does.

@smira is there any chance of adding some sort of hole-punching or nat traversal to kubespan in the future? Is it worth opening a feature request? I wouldn't expect or want a relay or derp-like server, but maybe a public hostable STUN server to help machines find each other

[smira]

The chance is always there, as long as we have enough interest from someone to get such feature implemented.