Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

post-processor vsphere: Handle prompts from ovftool when using custom CA #297

Closed
1 task done
goldstar611 opened this issue Aug 1, 2023 · 19 comments · Fixed by #413
Closed
1 task done

post-processor vsphere: Handle prompts from ovftool when using custom CA #297

goldstar611 opened this issue Aug 1, 2023 · 19 comments · Fixed by #413

Comments

@goldstar611
Copy link

goldstar611 commented Aug 1, 2023

Overview of the Issue

hashicorp/packer#7314 closed hashicorp/packer#7234, for error message: Write 'yes' or 'no'.

I recently upgraded my packer version and started using the plugins and this "write yes or no" message appears endlessly in the packer log.

Adding --noSSLVerify to ovftool parameters resolves the endless, interactive prompts.

Reproduction Steps

  1. Set up vcenter with an ESXi host OR set up nginx/apache server with a self signed SSL Cert (reproduction only) the real issue is when a custom CA bundle is being used.
  2. Create a packer template that uses the vshpere post-processor and targets the above server
  3. Wait for packer to finish building the VM and upload to the vcenter endpoint
  4. Endless interactive prompt Write 'yes' or 'no' will be displayed until the packer process is killed (Ctrl+C)

Packer Version

1.9.2

Plugin Version and Builders

Please provide the plugin version.

vsphere 1.0
vmware 1.0

Please select the builder.

  • vsphere-iso

VMware vSphere Version

7.0

Guest Operating System

Windows 10

Simplified Packer Buildfile

Packer Buildfile
  packer {
    required_plugins {
      vmware = {
        source  = "github.com/hashicorp/vmware"
        version = "~> 1"
      }
      vsphere = {
        source  = "github.com/hashicorp/vsphere"
        version = "~> 1"
      }
    }
  }
  
  variable "iso_checksum" {
    type    = string
    default = "send_via_command_line"
  }
  
  variable "iso_url" {
    type    = string
    default = "send_via_command_line"
  }
  
  source "vmware-iso" "autogenerated_1" {
    boot_command         = []
    boot_wait            = "15m"
    communicator         = "winrm"
    cpus                 = "2"
    disk_adapter_type    = "lsisas1068"
    disk_size            = "61440"
    disk_type_id         = "1"
    floppy_files         = ["./windows/10/answer_files/", "./windows/10/scripts/"]
    guest_os_type        = "windows9-64"
    headless             = "false"
    http_directory       = "./windows/10/tools/"
    iso_checksum         = "${var.iso_checksum}"
    iso_url              = "${var.iso_url}"
    memory               = "4096"
    network              = "nat"
    network_adapter_type = "E1000E"
    shutdown_command     = "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\""
    shutdown_timeout     = "45m"
    version              = 14
    vm_name              = "temp_windows10x64_packer"
    vmx_data = {
      "RemoteDisplay.vnc.enabled"    = "false"
      "RemoteDisplay.vnc.port"       = "5900"
      "isolation.tools.hgfs.disable" = "true"
      "mks.enable3d"                 = "true"
      "tools.upgrade.policy"         = "manual"
    }
    vmx_remove_ethernet_interfaces = false
    vnc_port_max                   = 5980
    vnc_port_min                   = 5900
    winrm_password                 = "winrm_password"
    winrm_timeout                  = "60m"
    winrm_username                 = "winrm_username"
  }
  
  build {
    sources = ["source.vmware-iso.autogenerated_1"]
  
    post-processor "vsphere" {
      cluster    = "blah"
      datacenter = "Network"
      datastore  = "datastore"
      host       = "vcenter"
      overwrite  = true
      password   = "PASSWORD"
      username   = "user"
      vm_folder  = "vmFolder"
      vm_name    = "temp_windows10x64_packer"
      vm_network = "network"
    }
  }

Operating System and Environment Details

Debian 12

Log Fragments and crash.log Files

ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Calling OVFtool to upload vmESC[0m

2023/08/01 15:29:58 packer-plugin-vsphere_v1.2.1_x5.0_linux_amd64 plugin: 2023/08/01 15:29:58 [INFO] (vsphere): starting ovftool command: ovftool --acceptAllEulas --name=windows10x64_packer --datastore=datastore --diskMode=thick --vmFolder=vmFolder --network=network --overwrite output-autogenerated_1/temp_windows10x64_packer.vmx vi://user%40vcenter:PASSWORD@vcenter/Network/host/blah

2023/08/01 15:29:58 packer-plugin-vsphere_v1.2.1_x5.0_linux_amd64 plugin: 2023/08/01 15:29:58 [INFO] (shell-local communicator): Executing local shell command [ovftool --acceptAllEulas --name=windows10x64_packer --datastore=datastore --diskMode=thick --vmFolder=vmFolder --network=network --overwrite output-autogenerated_1/temp_windows10x64_packer.vmx vi://user%40vcenter:PASSWORD@vcenter/Network/host/blah]
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Opening VMX source: output-autogenerated_1/temp_windows10x64_packer.vmxESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Accept SSL fingerprint (AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB) for host vcenter as target type.ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Fingerprint will be added to the known host fileESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m
ESC[0;32m    vmware-iso.autogenerated_1 (vsphere): Write 'yes' or 'no'ESC[0m

...forever
@goldstar611 goldstar611 added the bug label Aug 1, 2023
@goldstar611
Copy link
Author

in lieu of disabling SSL checking by default, I think a viable option is to handle the interactive prompt gracefully

@goldstar611
Copy link
Author

I should mention that it's probably not necessary to set up a complete vcenter/vshpere instance. I'm sure that targeting a nginx or apache server with a self signed cert would be sufficient to hit the issue described.

@tenthirtyam
Copy link
Collaborator

Appears that the issue is in the post-processor:

func (p *PostProcessor) ValidateOvfTool(args []string, ofvtool string) error {
	args = append([]string{"--verifyOnly"}, args...)
	var out bytes.Buffer
	cmdCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	cmd := exec.CommandContext(cmdCtx, ovftool, args...)
	cmd.Stdout = &out

Ref: https://github.com/hashicorp/packer-plugin-vsphere/blob/56a37102c278fae91807dfd343d54eede06d313c/post-processor/vsphere/post-processor.go#L220C1-L226

May be fixed with:

func (p *PostProcessor) ValidateOvfTool(args []string, ofvtool string) error {
	args = append([]string{"--noSSLVerify", "--verifyOnly"}, args...)
	var out bytes.Buffer
	cmdCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	cmd := exec.CommandContext(cmdCtx, ovftool, args...)
	cmd.Stdout = &out

Additional reference from the packer-plugin-vmware change. https://github.com/hashicorp/packer/pull/7314/files

cc @nywilken

@tenthirtyam tenthirtyam changed the title Regression in ovftool parameters: missing --noSSLVerify ovftool parameters: missing --noSSLVerify Aug 7, 2023
@rgl
Copy link
Contributor

rgl commented Aug 7, 2023

Please do not unconditionally disable TLS verification, as that defeats its purpose; instead, somehow, use the value from https://developer.hashicorp.com/packer/plugins/builders/vsphere/vsphere-iso#insecure_connection to decide.

@goldstar611
Copy link
Author

@goldstar611 goldstar611 changed the title ovftool parameters: missing --noSSLVerify Handle prompts from ovftool during untrusted SSL connection attempt Aug 7, 2023
@goldstar611
Copy link
Author

goldstar611 commented Aug 7, 2023

I've updated the title to propose better handling of the prompts since disabling ovftool's SSL check can simply be added to the options parameter currently:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    options    = ["--noSSLVerify"]
  }
}

@goldstar611
Copy link
Author

Any idea when using own certificate authority will be working again?

@goldstar611 goldstar611 changed the title Handle prompts from ovftool during untrusted SSL connection attempt Handle prompts from ovftool when using custom CA bundle Oct 10, 2023
@tenthirtyam
Copy link
Collaborator

I agree, but you should know it's already hardcoded in at least 2 places.

https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/step_export.go#L42

https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/driver_esx5.go#L373

These references are from a seperate plugin.

@goldstar611
Copy link
Author

I agree, but you should know it's already hardcoded in at least 2 places.
https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/step_export.go#L42
https://github.com/hashicorp/packer-plugin-vmware/blob/17fb61b1c2f949f8bf41299d11814c6944087a2a/builder/vmware/common/driver_esx5.go#L373

These references are from a seperate plugin.

correct, that is just auxiliary information for rgl who doesn't want ssl disabled by default. He should know it already is disabled by default in other vmware-owned plugins

@tenthirtyam
Copy link
Collaborator

in other vmware-owned plugins

HashiCorp maintains these plugins, but we (VMware) do collaborate on them. 😄

@tenthirtyam
Copy link
Collaborator

tenthirtyam commented Nov 8, 2023

Taking a look at this the following should be possible currently:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    options = ["--TargetSSLThumbprint AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB"]
  }
}
build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    options = ["--TargetPEM /usr/lib/vmware-ovftool/certs/cacert.pem"]
  }
}

As an alternative, the following could be explicitly added:

import (
  ...
  "os"
  ...
)

type Config struct {
  common.PackerConfig `mapstructure:",squash"`
  ..
  TargetSSLThumbprint string `mapstructure:"target_ssl_thumbprint"`
  TargetPEM string `mapstructure:"target_pem"`
  ...
}

...

func (p *PostProcessor) BuildArgs(source, ovftool_uri string) ([]string, error) {
  args := []string{
    "--acceptAllEulas",
    fmt.Sprintf(`--name=%s`, p.config.VMName),
    fmt.Sprintf(`--datastore=%s`, p.config.Datastore),
  }

  if p.config.Insecure {
    args = append(args, fmt.Sprintf(`--noSSLVerify=%t`, p.config.Insecure))
  }

  if p.config.TargetSSLThumbprint != "" {
    args = append(args, fmt.Sprintf("--targetSSLThumbprint=%s", p.config.TargetSSLThumbprint))
  }

  if p.config.TargetPEM != "" {
    pemBytes, err := os.ReadFile(p.config.TargetPEM)
    if err != nil {
      return nil, fmt.Errorf("failed to read PEM file: %w", err)
    }
    args = append(args, fmt.Sprintf("--targetPEM=%s", string(pemBytes)))
  }

  ...

  if len(p.config.Options) > 0 {
    args = append(args, p.config.Options...)
  }

  args = append(args, source)
  args = append(args, ovftool_uri)

  return args, nil
}

Then you could use:

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    target_ssl_thumbprint = "AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB:CC:DD:EE:FF:AA:BB"
  }
}

or

build {
  sources = ["source.vmware-iso.autogenerated_1"]
  post-processor "vsphere" {
   ...
    target_pem = "/usr/lib/vmware-ovftool/certs/cacert.pem"
  }
}

Let me know your thoughts.

cc @nywilken

Ryan Johnson
Senior Staff Solutions Architect | Product Engineering @ VMware, Inc.

@tenthirtyam tenthirtyam changed the title Handle prompts from ovftool when using custom CA bundle Add support to use and accept custom certificates Nov 8, 2023
@tenthirtyam tenthirtyam changed the title Add support to use and accept custom certificates post-processor vsphere: Add support to use and accept custom certificates Nov 8, 2023
@goldstar611
Copy link
Author

Hi Ryan,
Thanks for spending some time on this again.

Let me know your thoughts.

CA/Browser forum requires that TLS certificates have a max validity of about 1 year (398 days) so if we used a finger/thumb print, then that would need to be updated once a year. It's better than disabling SSL/TLS altogether security wise but not as good as verifying to a trusted certificate authority.

If --TargetPEM is available today under options, I'm all in and this would resolve my issue. I will try this out and verify it works as intended.

@tenthirtyam
Copy link
Collaborator

Great - let me know. May be ideal to add an example to the docs if that's good for you.

@tenthirtyam tenthirtyam self-assigned this Nov 10, 2023
@tenthirtyam
Copy link
Collaborator

Any update @goldstar611?

@goldstar611
Copy link
Author

goldstar611 commented Nov 17, 2023

Hey Ryan,

Thanks for the ping. I made the corresponding changes to my .pkr.hcl file to remove
--noSSLVerify
and add
--TargetPEM /usr/lib/vmware-ovftool/certs/cacert.pem
but ovftool failed twice. Once for unknown option, then once for unexpected parameter. The correct case and an equal sign is needed, like
--targetPEM=/usr/lib/vmware-ovftool/certs/cacert.pem

With the params sent correctly, it's now a fight with ovftool to accept my certificate file. I've tried several base64 encoded certificates (the standard PEM format) but each invocation returns:

Error: Invalid PEM file: <file location>

Not even the file at /usr/lib/vmware-ovftool/certs/cacert.pem works.

[Edit] This targetPEM option in ovftool seems to be broken in my limited experience.
[Edit2] removed
[Edit3] ovftool seems to only accept a leaf certificates (not root CA certificates) so this option has no advantage over the SSL thumbprint option since it will be changing each year.

@goldstar611 goldstar611 changed the title post-processor vsphere: Add support to use and accept custom certificates post-processor vsphere: Handle prompts from ovftool when using custom CA Nov 17, 2023
@goldstar611
Copy link
Author

My last message was a bit of a mess so let me summarize here.

  • vsphere post-processor does not handle ovftool prompts (stemming from use of a custom CA)
  • ovftool doesn't accept CA certs in the --targetPEM parameter

I've changed the title back to Handle prompts from ovftool when using custom CA because, as I see it, that would resolve my biggest concern of an automated process never returning. The 2nd bullet point is a limitation in ovftool, so outside the scope of this issue I think.

@tenthirtyam tenthirtyam removed their assignment Dec 20, 2023
@tenthirtyam tenthirtyam self-assigned this Mar 8, 2024
@tenthirtyam tenthirtyam added this to the v1.2.8 milestone Apr 12, 2024
@tenthirtyam
Copy link
Collaborator

I'm working on this one.

@tenthirtyam
Copy link
Collaborator

tenthirtyam commented Apr 25, 2024

Hi @goldstar611 - I've tested the following changes that does allow an option and mre graceful failure.

func (p *PostProcessor) PostProcess(ctx context.Context, ui packersdk.Ui, artifact packersdk.Artifact) (packersdk.Artifact, bool, bool, error) {
        ...	

        ... Existing
	ui.Message("Validating username and password...")
	err = p.ValidateOvfTool(args, ovftool, ui)
	if err != nil {
		return nil, false, false, err
	}
	... Existing

        ...
}

func (p *PostProcessor) ValidateOvfTool(args []string, ofvtool string, ui packersdk.Ui) error {
    args = append([]string{"--verifyOnly"}, args...)
    if p.config.Insecure {
        args = append(args, "--noSSLVerify")
		ui.Message("Skipping SSL thumbprint verification; insecure flag set to true...")
    }
    var out bytes.Buffer
	cmdCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	cmd := exec.CommandContext(cmdCtx, ovftool, args...)
	cmd.Stdout = &out

	// Need to manually close stdin or else the ofvtool call will hang
	// forever in a situation where the user has provided an invalid
	// password or username
	stdin, err := cmd.StdinPipe()
	if err != nil {
		return err
	}
	defer stdin.Close()

	if err := cmd.Run(); err != nil {
		outString := out.String()
		if strings.Contains(outString, "Enter login information for source") {
			err = fmt.Errorf("error running ovftool with --verifyOnly; the username " +
				"or password you provided may be incorrect")
			return err
		} else if strings.Contains(outString, "Accept SSL fingerprint") {
			err = fmt.Errorf("error running ovftool with --verifyOnly; the ssl thumbprint " +
				"returned by the server is not trusted. manually accept the thumbprint " +
				"with ovftool, or set the insecure flag to true for this post-processor.")
			return err
        }
	}
	return nil
}

TL;DR -

  1. If insecure is not provided or is set to false explicitly for the post-processor and the certificate is not trusted, it will fail with an error by capturing "Accept SSL fingerprint" in the stdout. Unfortunately, I'm not able to get it to pause and allow for user input with a fmt.Scanln(&response). (I've tried endlessly to get that to work. ¯\_(ツ)_/¯ )

  2. If insecure is set to true, it will proceed and present a message.

=> vsphere-iso.linux-photon: Running post-processor:  (type vsphere)
    vsphere-iso.linux-photon (vsphere): Uploading /Users/ryan/Library/Mobile Documents/com~apple~CloudDocs/Code/Personal/<sensitive>-examples-for-vsphere12/artifacts/linux-photon-5.0-develop/linux-photon-5.0-develop.ovf to m01-vc01.rainpole.io
    vsphere-iso.linux-photon (vsphere): Validating username and password...
    vsphere-iso.linux-photon (vsphere): Skipping SSL thumbprint verification; insecure flag set to true...
    vsphere-iso.linux-photon (vsphere): Uploading virtual machine...
    vsphere-iso.linux-photon (vsphere): Opening OVF source: /Users/ryan/Library/Mobile Documents/com~apple~CloudDocs/Code/Personal/<sensitive>-examples-for-vsphere12/artifacts/linux-photon-5.0-develop/linux-photon-5.0-develop.ovf
    vsphere-iso.linux-photon (vsphere): The manifest validates

I'll put in a pull request tomorrow for these changes to help mitigate the issue of interacting with ovftool's options. I hope this will help.

cc: @nywilken @lbajolet-hashicorp

Ryan

@tenthirtyam
Copy link
Collaborator

Please note that I'm also trying to track down the issue seen in #279 after the completion, but I'm mostly convinced this is a packersdk issue.

vsphere-iso.linux-photon (vsphere): Deploying to VI: vi://administrator%[email protected]:443/m01-dc01/host/m01-cl01
vsphere-iso.linux-photon (vsphere): Transfer Completed
vsphere-iso.linux-photon (vsphere): Completed successfully
Build 'vsphere-iso.linux-photon' errored after 5 minutes 38 seconds: 1 error(s) occurred:

* Error destroying builder artifact: reading body msgpack decode error [pos 1556]: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error; bad artifact: []string(nil)

==> Wait completed after 5 minutes 38 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso.linux-photon: 1 error(s) occurred:

* Error destroying builder artifact: reading body msgpack decode error [pos 1556]: reflect.Set: value of type map[interface {}]interface {} is not assignable to type error; bad artifact: []string(nil)

==> Builds finished but no artifacts were created.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants