Weird "sshkeys" SSH public key validation error

[justinclift]

I'm trying to create a VM with a cloud-init ssh key, but the server keeps on returning 500 SSH public key validation error. 😕

It seems pretty weird, as the exact same key works fine when using qm on the server itself to create the VM.

The code in question for creating the VM:

vmProps := []proxmox.VirtualMachineOption{
	{Name: "name", Value: "test1"},
	{Name: "memory", Value: 1024 * 8},
	{Name: "cores", Value: 6},
	{Name: "cpu", Value: "host"},
	{Name: "net0", Value: "model=virtio,bridge=" + publicBridge + ",firewall=1"},
	{Name: "scsihw", Value: "virtio-scsi-single"},
	{Name: "virtio1", Value: "local-zfs:16,cache=" + cacheMode + ",discard=on,iothread=1"},
	{Name: "agent", Value: 1},
	{Name: "ostype", Value: "l26"},
	{Name: "localtime", Value: "0"},
	{Name: "ide0", Value: "local-zfs:cloudinit"},
	{Name: "ipconfig0", Value: "gw=10.1.1.1,ip=10.1.248." + strconv.Itoa(vmID) + "/16"},
	{Name: "sshkeys", Value: url.QueryEscape("/root/.ssh/id_rsa.pub")},
}
vmTask, err := node.NewVirtualMachine(ctx, vmID, vmProps...)
if err != nil {
	log.Fatal(err)
}

If I remove that last vmProps line (the sshkeys one), then the vm creation works. With that in place though, I'm getting:

2024/04/30 04:12:44 500 SSH public key validation error

The working qm version of it:

# Create a new VM
qm create ${VMID} --name "test1" \
  --cpu host \
  --cores 6 \
  --memory 8192 \
  --net0 virtio,bridge=${PUBLIC_BRIDGE},firewall=1 \
  --scsihw virtio-scsi-single \
  --virtio1 local-zfs:16,cache=${CACHE_MODE},discard=on,iothread=1 \
  --agent 1 \
  --ostype l26 \
  --localtime 0 \
  --ide0 local-zfs:cloudinit \
  --ipconfig0 gw=10.1.1.1,ip=10.1.248.${VMID}/16 \
  --sshkeys "~/.ssh/id_rsa.pub"

Anyone have ideas what could be going wrong?

[justinclift]

Not sure if it's relevant, but the same error was showing up in a fork of Proxmox at one point: pimox/pimox7#45

[justinclift]

Interestingly, the above mentioned error in the other fork seems relevant here too.

If I comment out that same die "SSH public key validation error\n" if $@; line (line 1726 with modern Proxmox), then the ssh key is accepted and things seem happy:

# vi /usr/share/perl5/PVE/Tools.pm
# systemctl restart pvedaemon   <-- so the edited Tools.pm gets loaded

---

Although the ssh key is accepted, and shows up in the cloud-init section for the VM, it's not actually happy. The key itself doesn't appear to have been loaded into the user in question.

Checked by looking through the VM disk with a rescue system, and the authorized_keys file it should have been loaded into is 0 bytes.

So, maybe something in the Go code really is mucking up that string somehow.

[justinclift]

Hmmm, the server might actually be wanting the actual text of the ssh key, rather than a path.

However, I'm not having much luck in figuring out what PVE calls "url encoding" as it doesn't seem to be any of the common URL encoding calls in Go. 😦

Example:

{Name: "sshkeys", Value: url.PathEscape("ssh-rsa AAAAB ...")}

Result:

2024/04/30 06:34:41 bad request: 400 Parameter verification failed. - {"sshkeys":"invalid format - invalid urlencoded string: ssh-rsa%20AAAAB ...

Also tried base64.URLEncoding.EncodeToString(), base64.RawURLEncoding.EncodeToString(), base64.StdEncoding.EncodeToString() and url.QueryEscape() without any improvement.

It feels like PVE may have it's own ideas about url encoding, and there might need to be a special purpose encoder created just for this one interaction with it. 😉

[justinclift]

Ahhh yep, got it somewhat figured out. That sshkeys does need the whole ssh key as the value, and the problem does seem to be in how it's presently getting quoted.

Someone on the Proxmox forums had a similar issue (not via Go though) a Proxmox staff member gave a Python solution for doing the quoting:

https://forum.proxmox.com/threads/how-to-use-pvesh-set-vms-sshkeys.52570/#post-243381

urllib.quote(key, safe='')

That's a Python 2 version of things. As Python 3 is where things are at these days, it's now:

$ python3
>>> import urllib.parse
>>> urllib.parse.quote(key, safe='')

Using that Python 3 snippet I was able to manually url encode an ssh key such that Proxmox accepts it:

$ python3
>>> import urllib.parse
>>> urllib.parse.quote("ssh-rsa AAAAB[...]", safe='')
'ssh-rsa%20AAAAB[...]

Copying that string into the Go vmProps and just passing it directly works, with the VM being created and the ssh key name showing up in the webUI the same way it does for qm:

{Name: "sshkeys", Value: "ssh-rsa%20AAAAB[...]}

[luthermonson]

yup and the docs say you can do multilple keys in one config just separate with a newline. glad you figured it out!

[justinclift]

Any idea if there's a Go function call for doing the "url encoding" that Proxmox wants?

[luthermonson]

try this: https://pkg.go.dev/net/url#QueryEscape

[justinclift]

Heh, that's literally one of my above examples of something that doesn't work. 😉

Tried it again now, just in case... and nope, it's definitely not a winner:

 {"sshkeys":"invalid format - invalid urlencoded string: ssh-rsa+AAAAB (etc)

That's from calling it this way in my Go code:

{Name: "sshkeys", Value: url.QueryEscape("ssh-rsa AAAAB (etc)

[luthermonson]

https://github.com/proxmox/pve-common/blob/1a6005ad2377b6586e084b3840ac622752b666b8/src/PVE/JSONSchema.pm#L187

it appears to be failing this regex, check your output from your funcs to escape and find the right combintation

[justinclift]

Thanks. Saw that and might investigate that later on. 😄

[LewsTherinSedai]

Facing this same issue - if Proxmox is going to even come close to VMWare on an enterprise level, or even SMB, this kind of stuff can't exist. How has no one fixed this blatant UI issue - I have tried multiple forms of SSH keys generated from PuttyGen (including removing the Comment line as documentation says it isn't supported) and I cannot get this to work.

I'm not going to go editing my Proxmox - this should work out of the box - or at a minimum there should be a clear guide on how it should work (i.e. if the PuttyGen style SSH key doesn't work, then what does.)

[justinclift]

Ahhh. Sorry for not investigating this further. After finding the lack of disk import functionality (#145) I've given up on using the Proxmox API until basic required functionality (aka "being able to create a new VM") is present.