SSH communicator fails when ssh_password contains "#"

[digitalqualms]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

When using the proxmox-iso, I was encountering SSH handshake error. It turned out that because my ssh_password value I was using contained the pound/hash symbol "#". It looks like this value isn't escaped properly.

Reproduction Steps

Create an HCL template that uses the ssh communicator. Set the ssh_password value to something that contain "#". Build fails with:

Packer version

1.9.4

Simplified Packer Template

source "proxmox-iso" "ubuntu22-04" {
  // Proxmox config
  iso_file       = "local:iso/ubuntu-22.04.2-live-server-amd64.iso"
  iso_checksum   = "none"
  proxmox_url    = "https:/proxmox:8006/api2/json"
  username       = ${var.pve_username
  token          = var.pve_token
  node           = "pve1"
  http_directory = "config"

  onboot             = true
  memory             = 8192
  ballooning_minimum = 4096
  cores              = 2
  cpu_type           = "host"
  os                 = "l26"
  bios               = "ovmf"
  scsi_controller    = "virtio-scsi-single"
  efi_config {
    efi_storage_pool  = "local-thin"
    efi_type          = "4m"
    pre_enrolled_keys = true
  }
  disks {
    disk_size    = "50G"
    storage_pool = "local-thin"
    type         = "scsi"
  }
  network_adapters {
    bridge   = "vmbr0"
    model    = "virtio"
    firewall = true
  }
  boot_command = [
    "c",
    "linux /casper/vmlinuz --- pv6.disable=1 autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/'",
    "<enter><wait>",
    "initrd /casper/initrd<enter><wait>",
    "boot<enter>"
  ]
  boot_wait            = "10s"
  ssh_username         = "myuser"
  ssh_password         = "abc-def#"
  ssh_timeout          = "120m"
  ssh_handshake_attempts = 10
  template_description = "Ubuntu 22.04, generated on ${timestamp()}"
  template_name        = "ubuntu-22.04"
  unmount_iso          = true
}

build {
  sources = ["source.proxmox-iso.ubuntu22-04"]
  provisioner "shell" {
    inline = [
      "while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done"
    ]
  }
}

Operating system and Environment details

Packer run from macOS 14.4.1 (23E224), arm64
Proxmox plugin: v1.1.7_x5.0_darwin_arm64
Connecting to Ubuntu 22.04

Log Fragments and crash.log files

02 19:50:12 [INFO] Attempting SSH connection to 192.168.0.97:22...
2024/04/02 19:50:12 packer-plugin-proxmox_v1.1.7_x5.0_darwin_arm64 plugin: 2024/04/02 19:50:12 [DEBUG] reconnecting to TCP connection for SSH
2024/04/02 19:50:12 packer-plugin-proxmox_v1.1.7_x5.0_darwin_arm64 plugin: 2024/04/02 19:50:12 [DEBUG] handshaking with SSH
2024/04/02 19:50:14 packer-plugin-proxmox_v1.1.7_x5.0_darwin_arm64 plugin: 2024/04/02 19:50:14 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
2024/04/02 19:50:14 packer-plugin-proxmox_v1.1.7_x5.0_darwin_arm64 plugin: 2024/04/02 19:50:14 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/04/02 19:50:21 packer-plugin-proxmox_v1.1.7_x5.0_darwin_arm64 plugin: 2024/04/
==> proxmox-iso.ubuntu22-04: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

[lbajolet-hashicorp]

Hi @digitalqualms,

Looking into this one, this looks like a bug, but I'm not yet certain Packer is to blame here.

I'll explain myself: I've not tested Proxmox yet (and likely won't as I don't have a working cluster to play with, nor am I an expert with the platform), but I've been playing with Qemu and a handcrafted preseed.txt on a Debian 12.5 VM.

With a preseed statement like this: d-i passwd/user-password password debian#, the account does get created, with the expected password, as the line in /etc/shadow shows: debian:$y$j9T$LTmLYDDC/4.TIyQPVFeAP0$jelmzGWdfOZnVt8m7ehMOsquC..Ebb0ZtstFudXepjB:19822:0:99999:7:::.
The shadow line points to a yescrypt entry with LTmLYDDC/4.TIyQPVFeAP0 as salt, which we can validate with a bit of perl:

$ perl -e 'print crypt "debian#", "\$y\$j9T\$LTmLYDDC/4.TIyQPVFeAP0\$"'
$y$j9T$LTmLYDDC/4.TIyQPVFeAP0$jelmzGWdfOZnVt8m7ehMOsquC..Ebb0ZtstFudXepjB

In this scenario, the hash matches the expected password debian#, and Packer is able to connect to the VM without any issue, so this rules out a bug on our side, at least for the Qemu plugin. That doesn't yet absolve the Proxmox plugin that being said.

I did try as well to escape in the preseed file, and the quotes were part of the password in this case, but I'm not a preseed spec expert and I don't know what's the normal way to escape characters like these (if at all necessary).

Looking at your template, I'm not sure how the VM's user accounts are initialised, but I suspect there may be something here, especially given that # is a quasi-universal oneline comment start token.

To help narrow-down the problem and help troubleshoot this, I suggest running Packer with a --on-error=ask option (this will let you connect to the VM and will not automatically kill/cleanup on error), and when the VM is ready to receive connections, take a look at your /etc/shadow to see what it looks like (this implies either sudo is available or root access), and try to play with that perl script I shared to compare the hashes. This will let us know if the password is registered as expected or not.
If this is not registered as abd-def#, there likely is a bug while the VM is initialised, and this is where we should focus our efforts.
Otherwise this may point to a SSH problem, which could be either us in the SDK (or maybe the Proxmox plugin), or even in golang.org/x/crypto, but it's hard to pinpoint what's the error as of now.

Hope that helps and we can narrow down the source of the problem!

[github-actions]

This issue has been migrated to hashicorp/packer-plugin-qemu#170 due to the Packer Plugin split.

Please follow the new issue for updates.

[github-actions]

This issue has been migrated to hashicorp/packer-plugin-proxmox#258 due to the Packer Plugin split.

Please follow the new issue for updates.

[digitalqualms]

Hi @lbajolet-hashicorp,

Well, I thought I had a good reproducible issue but everything is working now. Not sure what changed but I'm not able to reproduce it.

I'll reopen/refile if I find it again.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.