Validate/Tshoot k8s hostNetwork

[nerdalert]

Ran into an issue where hostNetwork: true was not getting wg traffic off the host. The issue was in an EC2 OCP deployment so we should look at that as well as vanilla k8s.

The following was failing. If you remove hostNetwork: true everything works in the symmetric NAT environment:

apiVersion: v1
kind: Pod
metadata:
  name: nexd
  namespace: <NAMESPACE>
spec:
  hostNetwork: true
  hostAliases:
    - ip: <API_SERVER>
      hostnames:
        - try.nexodus.127.0.0.1.nip.io
        - api.try.nexodus.127.0.0.1.nip.io
        - auth.try.nexodus.127.0.0.1.nip.io
  containers:
  - name: nexd-container
    image: quay.io/nexodus/nexd:latest
    command:
      - nexd
    args:
      - --username
      - <USERNAME>
      - --password
      - <PASSWORD>
      - --insecure-skip-tls-verify
      - https://try.nexodus.127.0.0.1.nip.io
    securityContext:
      privileged: true

Tested so far:

• (Works) Docker docker run --rm -it --network host --cap-add SYS_MODULE --cap-add NET_ADMIN --cap-add NET_RAW quay.io/nexodus/nexd
• (Works) KIND with the above yaml

[nerdalert]

The isue here is the FirewallMark for exit node that is conflicting with some k8s implementation with hostNetworking: true. Removing the mark resolves the issue. Need a patch to resolve this without breaking exit-node client PBR on linux.

nexodus/internal/nexodus/nexodus_linux.go

Line 97 in 30b7ec5

fwMark := wgFwMark

:

//fwMark := wgFwMark

err = c.ConfigureDevice(nx.tunnelIface, wgtypes.Config{
	PrivateKey:   &privateKey,
	ListenPort:   &listenPort,
	ReplacePeers: true,
	Peers:        nil,
	//FirewallMark: &fwMark,
})

[russellb]

The isue here is the FirewallMark for exit node

@nerdalert exit node wasn't being used here, right? so why is the mark used?

[nerdalert]

@russellb since exit node can be toggled on and off its pre-provisioned there. I can't really come up with a work around other than tearing the interface down if exit node client is toggled. Any ideas?