Connecting to talos cluster from remote machine - x509 cert issue

[rakhbari]

Hi all. I've fully gone through the quickstart and have a Talos provisioned Kube cluster up & running on single machine. I can do kubectl on the same machine where the Talos cluster is installed just fine. But I wanted to reach the cluster from a different machine so I copied the clusters/users portion of the .kube/config file from the Talos installed node to a remote client machine I have. Of course, in the .kube/config of the remote client machine I changed the server address to the DNS entry of the Talos installed node:

    server: https://mainserv03.lan:6443

When I try to do anything with kubectl I get:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not mainserv03.lan

If I simply add --insecure-skip-tls-verify to the kubectl cmd it's able to reach the Talos node just fine, so I know it's not an issue with network access and/or connectivity:

$ kubectl --insecure-skip-tls-verify version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

I know all I have to do is find the Talos cluster API server certificate and add it as a trusted cert on my remote machine, but I'm not seeing a guide on how to retrieve the cluster API server cert with taloscfg. Is there a guide or how-to doc on this I might have missed?

Thank you in advance.

[smira]

Hi @rakhbari, first of all you can retrieve kubeconfig (admin) from Talos API using talosctl -n <IP> kubeconfig.

Answering your first question: Talos generates certificate for the Kubernetes API server based on the control plane endpoint, hostname, etc.

Looks like in your case you're trying to use multiple control plane endpoints (changed from whatever it was to mainserv03.lan), so you need to make sure Talos includes mainserv03.lan into the Kubernetes API certificate. This can be configured via machine configuration:
https://www.talos.dev/docs/v0.10/reference/configuration/#apiserverconfig

You need to add certSANs to the cluster.apiServer section of the configuration.

This change can be applied immediately (without a reboot) with: talosctl -n <IP> edit mc --immediate

[rakhbari]

Hi @smira, thanks a bunch for your timely reply. I have to first apologize that although I'm pretty familiar with Kubernetes in general, I'm a novice with Talos. I used talosctl cluster create (from Quickstart) to create/boostrap Talos onto my bare metal server, which already had Ubuntu 20.04 installed. So I did go through the process of generating the various config files.

But I understand now that in order to add my server's DNS name (mainserv03.lan) to certSANs to cluster.apiServer I first have to have the controlplane.yaml file. So I ran the following command to generate it:

talosctl gen config --version v1alpha1 talos-default https://10.5.0.2:6443

I then edited the controlplane.yaml file and added my server's DNS name to the cluster.apiServer section:

    apiServer:
        # Extra certificate subject alternative names for the API server's certificate.
        certSANs:
            - 10.5.0.2
            - mainserv03.lan

Now all that's left is to apply the change to Talos. So tried running the cmd:

talosctl --talosconfig=./talosconfig -n https://10.5.0.2:6443 edit mc --immediate

Got output:

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.5.0.2, 10.5.0.2, not 127.0.0.1"

I then opened talosconfig and saw:

contexts:
    talos-default:
        endpoints:
            - 127.0.0.1

I changed 127.0.0.1 to 10.5.0.2 and saved the file and re-ran the talosctl cmd and now I get:

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"x509: Ed25519 verification failure\" while trying to verify candidate authority certificate \"talos\")"

So I'm not sure where exactly I'm going wrong, but obviously something is still amiss with the x509 cert config. Any pointers would be appreciated.

Thanks.

[smira]

@rakhbari you don't have to generate new machine configuration with talosctl gen config, your Docker cluster already has one (it was generated with talosctl gen config).

So in your case, you can do talosctl -n 10.5.0.2 edit mc --immediate to add the certSANs. This will regenerate the kube-apiserver certificated in a few seconds. Command talosctl edit mc launches the editor to update the machine configuraiton on the node itself (it fetches machine configuration from Talos node and then applies the changes).

(you should have talosconfig, which is a client-side configuration, like kubeconfig in the default location ~/.talos/config, so you don't need to use explict --talosconfig argument)

[rakhbari]

@smira Ah, ok. That makes sense. I do see the .talos/ dir under my username on the Ubuntu server where I ran the initial talos cluster create command (mainserv03.lan).

Ok, so I just ran: talosctl -n 10.5.0.2 edit mc --immediate, and I get:

error constructing client: failed to determine endpoints

I examined .talos/config:

$ cat .talos/config
context: ""
contexts: {}

...which obviously doesn't look right. I should have more metadata about my Talos cluster in my talosconfig file, shouldn't I?

Then I took a look at the talosconfig file that was created when I ran talosctl gen config and saw that it contains a context and ca/crt/key entries:

context: talos-default
contexts:
    talos-default:
        endpoints:
            - 127.0.0.1
        ca: <<REDACTED>>
        crt: <<REDACTED>>
        key: <<REDACTED>>

So I copied its content to my .talos/config file. Then tried running the cmd again:

$ talosctl -n 10.5.0.2 edit mc --immediate
rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for 10.5.0.2, 10.5.0.2, not 127.0.0.1"

So I really don't know where to go from here. I'm just wondering if I missed a step somewhere.

[smira]

Looks like your ~/.talos/config is empty?

I don't know what exactly is going on, but probably start from scratch:

# if you still have docker cluster running:
$ talosctl cluster destroy

$ talosctl cluster create
# wait for the cluster to finish booting
$ talosctl -n 10.5.0.2 edit mc --immediate

[rakhbari]

@smira Hi Andrey. That did it! 👍

Something must have gone wrong when I first did talosctl cluster create, even though there were no error messages and the cmd completed successfully. But destroying the cluster & recreating it seemed to have solved it. I can now run kubectl from a remote machine just fine.

Thank you very much for all your help. Much appreciated. 😄

[steverfrancis]

Glad you got it resolved, Let us know if there are other issues!

[QuinnBast]

Hey, I know this is a very old thread, but I am having the same issue here but against a production cluster.

I have 3 machines booted from the ISO image. I run the command to generate configuration on a remote machine that can ping the machines:

talosctl gen config talos-dev-test https://myDesiredVirtualIPAddress:6443

But the generated talosconfig is bad and is the same issue as what was pointed out above:

context: talos-dev-test
contexts:
    talos-dev-test:
        endpoints:
            - 127.0.0.1
        ca: redacted
        crt: redacted
        key: redacted

Even if I change the endpoints to be my server's IP address, I am getting the same issue

$ talosctl version --nodes 172.29.2.21 --talosconfig talosconfig 
Client:
	Tag:         v1.6.7
	SHA:         46c8ac10
	Built:       
	Go version:  go1.21.8 X:loopvar
	OS/Arch:     linux/amd64
Server:
error getting version: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

Am I missing something? Is there a way to get a talosconfig from machines that were booted from an ISO?

I saw someone run these commands after running the talosctl gen command, but these also did not work for me (they seem to just update the talosconfig file accordingly):

$ talosctl --talosconfig talosconfig config endpoint <hostIP>
$ talosctl --talosconfig talosconfig config node <hostIP>

[frezbo]

have you applied the config and bootstrapped the node yet, talos booted of iso doesn't have a config, so no PKI is set up, in this stage only limited operations are available using talos --insecure mode, once a config is applied, the node has PKI and talosconfig can be used to authenticate against it

[QuinnBast]

Ohhh, I was using talosctl apply-patch but didn't know I needed --insecure, thanks. Is there a list of what commands can be run with --insecure while in maintenance mode?

After using talosctl apply-patch --insecure to get the initial configuration onto the machines, the bootstrap command does not have --insecure but I am still getting the x509 error:

$ talosctl bootstrap --nodes <ipAddress> --talosconfig talosconfig
error executing bootstrap: 1 error occurred:
        * <ipAddress>: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

I found this page about PKIs like you mentioned, but I don't use a CA and would be using self-signed certs. I feel like maybe I did something wrong as the Getting started guide doesn't require interacting with PKIs

[Alkhazrajy]

I have the same issue :
authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: Ed25519 verification failure" while trying to verify candidate authority certificate "talos")"

to fix this issue

I open the terminal in the directory where exist talosconfig file
I write in terminal export TALOSCONFIG=talosconfig
then it work for me

[xyhhx]

yeah. talosctl will default to using ${HOME}/.talos to place your talos configs, but you can override that with the $TALOSCONFIG environment variable¹. (pro tip: this works really nicely with https://direnv.net)

Footnotes

1. https://www.talos.dev/v1.8/learn-more/talosctl/#client-configuration ↩