Packer and global plugins don't behave predictable/properly/logically.

[ladar]

Overview of the Issue

The packer docs state you can install plugins in the same location as the binary and everything should work. Since we already install packer in /usr/local/bin/ on shared build servers, it made sense (although it wasn't ideal), to also install the packer plugins there.

Reproduction Steps

Steps in install packer:

curl --location --output packer_1.9.4_linux_amd64.zip https://releases.hashicorp.com/packer/1.9.4/packer_1.9.4_linux_amd64.zip
unzip packer_1.9.4_linux_amd64.zip
install -m755 packer /usr/local/bin/
chown root:root /usr/local/bin/packer
chcon unconfined_u:object_r:bin_t:s0 /usr/local/bin/packer

Steps to install the plugins:

alias packer='/usr/local/bin/packer'
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/qemu
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/hyperv
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/docker
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/vmware
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/vagrant
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/parallels
PACKER_PLUGIN_PATH=/usr/local/bin/ packer plugins install github.com/hashicorp/virtualbox

Note the alias above only needed if the system tool packer appears in the path before the packer binary we just installed:. On this system, that happens with the root user, but not with regular users, like the vagrant user we use to test with below.

Confirmation

As a test you can verify that the when packer is run as a human, aka the vagrant user,. packer works, and the plugins show up as being installed (again, you only need an alias if /usr/local/bin/ is after /usr/sbin/... you can also use the full path isntead of an alias, for the same results):

[vagrant@alma8 robox]$ packer plugins installed
/usr/local/bin/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.0.3_x5.0_linux_amd64
/usr/local/bin/github.com/hashicorp/virtualbox/packer-plugin-virtualbox_v1.0.5_x5.0_linux_amd64
/usr/local/bin/github.com/hashicorp/docker/packer-plugin-docker_v1.0.8_x5.0_linux_amd64
/usr/local/bin/github.com/hashicorp/qemu/packer-plugin-qemu_v1.0.9_x5.0_linux_amd64
/usr/local/bin/github.com/hashicorp/hyperv/packer-plugin-hyperv_v1.1.1_x5.0_linux_amd64
/usr/local/bin/github.com/hashicorp/parallels/packer-plugin-parallels_v1.1.5_x5.0_linux_amd64
[vagrant@alma8 robox]$ 

Problems

As you can see the plugins are installed, but if you try and run validate on the generic templates linked below, you will get a cornucopia of errors which are both confusing and annoying .

[vagrant@alma8 robox]$ packer validate generic-hyperv.json 
Warning: Bundled plugins used

This template relies on the use of plugins bundled into the Packer binary.
The practice of bundling external plugins into Packer will be removed in an
upcoming version.

To remove this warning and ensure builds keep working you can install these
external plugins with the 'packer plugins install' command

* packer plugins install github.com/hashicorp/vagrant

Alternatively, if you upgrade your templates to HCL2, you can use 'packer init'
with a 'required_plugins' block to automatically install external plugins.

You can try HCL2 by running 'packer hcl2_upgrade
/home/vagrant/robox/generic-hyperv.json'


Error: Failed to initialize build "generic-alma8-hyperv"

error initializing builder 'hyperv-iso': Unknown builder hyperv-iso

Error: Failed to initialize build "generic-alma9-hyperv"

error initializing builder 'hyperv-iso': Unknown builder hyperv-iso

Error: Failed to initialize build "generic-alpine310-hyperv"

error initializing builder 'hyperv-iso': Unknown builder hyperv-iso

Error: Failed to initialize build "generic-alpine311-hyperv"

error initializing builder 'hyperv-iso': Unknown builder hyperv-iso

Error: Failed to initialize build "generic-alpine312-hyperv"

The errors repeat. You can try other templates, and you will get various different errors depending on the plugins involved, etc. While I troubleshooting I tried converting the template to HCL. It didn't help. And to make things more confusing, if I ran init/required on the template, it will give you more incorrect results.

Workarounds

If you run:

[vagrant@alma8 robox]$ PACKER_PLUGIN_PATH=/usr/local/bin/ packer validate generic-hyperv.json 
The configuration is valid.

Everything works. Alternatively, can also run:

[vagrant@alma8 robox]$ sudo mv `find /usr/local/bin/github.com/ -type f` /usr/local/bin/
[vagrant@alma8 robox]$ sudo rm -rf /usr/local/bin/github.com/
[vagrant@alma8 robox]$ packer validate generic-hyperv.json
The configuration is valid.

Which will move the plugins to /usr/local/bin/ .... once you do everything works as intended. I suspect this is the setup the docs are referring to. Unfortunately there doesn't appear to be a way to download plugins into a single folder (rather than creating the URL based path),

We've gone with running the find command above to move the plugins into /usr/local/bin/ as a semi-permanent fix:

As you can see from the behavior above, there is some mucked logic at work. Running other packer functions like build, etc will give you different results as well.

Please looking into a proper fix for this issue, for those of us with system-wide packer installs that also want to manage a system-wide collection of packer plugins.
In the short term this means making it easier to run packer plugins install and control where the files go. so they can be installed into /usr/local/bin/. Long term probably means adding more options for admins to exploit which allow us to define/control how packer loads plugins.

Packer version

.19.4

Simplified Packer Template(s):

generic-hyperv.json
generic-docker.json
generic-libvirt.json

Operating system and Environment details

Alma v8,,8 on x64.

Debug Logs

A more detailed debug log is attached.
packer_log.txt

[nywilken]

Hi @ladar thanks for opening up this issue. We have some work lined up to improve the plugin installation documentation, which I hope will clarify things for you and other users. That said, I do want to call out that support for loading plugins from the packer executable directory is currently supported for historical reasons and does not support the filesystem layout used by packer init and packer plugins install. By this I mean that the plugin binaries must be adjacent to the packer binary and not within the directory structure github.com/org/name/

We recommend installing plugins within the default Packer configuration directory for your environment or using the PACKER_PLUGIN_PATH environment variable for changing the default plugins installation path.

By default Packer looks for plugins in PACKER_CONFIG_DIR/plugins. If PACKER_CONFIG_DIR is either not set or empty, a default equal to $HOME/.config/packer/plugins on UNIX, or %APPDATA%\packer.d\plugins for Windows, will be used.

As you demoed above you can change the installation location for Packer plugins by setting PACKER_PLUGIN_PATH but that environment variable must always be set otherwise Packer will use the default location. Which seems to be the problem you are running into.

Since you are using a shared location you could have users of the system set PACKER_PLUGIN_PATH to the system wide location to ensure Packer always looks in the appropriate location. Keep in mind that users may not be able to install new plugins if they don't have write access.

Internally we are discussing the idea of a rc file that could be used for setting the Packer plugin path globally but we have not fully vetted that idea. Your best option at the moment is to have all users set PACKER_PLUGIN_PATH and use the same environment variable when executing packer plugins install.

[ladar]

@nywilken thank you for responding. I think part of the problem is that running:

sudo PACKER_PLUGIN_PATH=/usr/local/bin/ /usr/local/bin/packer plugins install github.com/hashicorp/qemu
sudo PACKER_PLUGIN_PATH=/usr/local/bin/ /usr/local/bin/packer plugins install github.com/hashicorp/vagrant

Followed by:

 /usr/local/bin/packer plugins installed

Gives you:

/usr/local/bin/github.com/hashicorp/qemu/packer-plugin-qemu_v1.0.10_x5.0_linux_amd64
/usr/local/bin/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.1.1_x5.0_linux_amd64

But if you run:

git clone https://github.com/lavabit/robox && cd robox
/usr/local/bin/packer validate generic-libvirt-x64.json

You still get the warning:

Warning: Bundled plugins used

This template relies on the use of plugins bundled into the Packer binary.
The practice of bundling external plugins into Packer will be removed in an
upcoming version.

To remove this warning and ensure builds keep working you can install these
external plugins with the 'packer plugins install' command

* packer plugins install github.com/hashicorp/vagrant
* packer plugins install github.com/hashicorp/qemu

Alternatively, if you upgrade your templates to HCL2, you can use 'packer init'
with a 'required_plugins' block to automatically install external plugins.

You can try HCL2 by running 'packer hcl2_upgrade

Which is indistinguishable from:

sudo rm -rf /usr/local/bin/github.com/
/usr/local/bin/packer validate generic-libvirt-x64.json

Which produces the exact same output. You are correct though:

sudo PACKER_PLUGIN_PATH=/usr/local/bin/ /usr/local/bin/packer plugins install github.com/hashicorp/qemu
sudo PACKER_PLUGIN_PATH=/usr/local/bin/ /usr/local/bin/packer plugins install github.com/hashicorp/vagrant
PACKER_PLUGIN_PATH=/usr/local/bin/ /usr/local/bin/packer validate generic-libvirt-x64.json

Will silence the warning.

As for your suggestions:

sudo mkdir /usr/local/bin/packer_plugins/
sudo cp /usr/local/bin/github.com/hashicorp/qemu/packer-plugin-qemu_v1.0.10_x5.0_linux_amd64 /usr/local/bin/packer_plugins/ 
sudo cp /usr/local/bin/github.com/hashicorp/vagrant/packer-plugin-vagrant_v1.1.1_x5.0_linux_amd64 /usr/local/bin/packer_plugins/
PACKER_PLUGIN_PATH=/usr/local/bin/packer_plugins/ packer plugins installed

Says zero plugins as being installed. In other words, the only way to install all of the plugins in the same place is to have them alongside the packer binary...

I do like your idea of a global config file to control things. If/when you go this route, might I suggest adding the ability to set a precedence order... so a global plugin location would go first, followed by the user directory on my system.

If an admin didn't want user installed plugins being used, they could omit the user directory....

Likewise, it should be possible to control which directory takes precedence when the same plugin is installed in both places. Some admins might not want internal, or otherwise sensitive plugins being overwritten.

You could achieve this with a "search path" scheme, like the PATH (or LD_LIBRARY_PATH) environment variables, where they are searched in order, using a colon separator. Then in documentation make it clear how to use that scheme to achieve the different scenarios (user dir (default), global + user dir, user + global dir) ...

For the security rules work properly though, you'd need to also have flags that control whether the global config values can be overridden by PACKER_PLUGIN_PATH / PACKER_CONFIG_PATH.

When you implement the global config, please also add the the ability to control the PACKER_CACHE_DIR variable as well. I like having the ISOs cached alongside the configs using the historical packer_cache/ directory rather than "out of sight," which makes it easy to manage/purge. I've had build robots run into disk space issues because ISOs were getting tucked away someplace else, and I didn't realize it (I now set the variable during login so that builds which don't go through the robox.sh script use the same dir).

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.