Plugin Discovery in Packer

[lbajolet-hashicorp]

Packer, as a tool for creating machine images across a variety of platforms, consists of two critical components: the build runner, referred to as Packer core, and its plugins for interfacing with the APIs of multiple platforms during the build process. Plugins have historically been bundled into the main Packer binary to help Packer users get up and running quickly.

In Packer v1.7.0, we began introducing new sub-commands to manage the lifecycle (e.g.: installation and updating) of Packer plugins: packer init and packer plugins. This approach gives users more options for loading the plugins installed by those tools, while maintaining compatibility with plugins that were previously installed.

Although maintaining backwards-compatibility made sense when we introduced it, we received community feedback that plugin discovery is confusing and inconsistent between legacy JSON and HCL2 templates. We take this feedback seriously and have been considering it as we continue our work to remove bundled plugins from the Packer binary (refer to our blog post for more information).

The remainder of this issue focuses on reported problems users encounter when using HCL2 templates with the required_plugins blocks, or legacy JSON templates with manually downloaded and installed plugins due to environment restrictions or company policies.

Issues reported by users of Packer requesting support for non-GitHub plugin repositories or rate-limiting build failures are still being discussed internally and will not be addressed in this proposal. We will publish another issue for community feedback when we are ready to share proposals for those issues.

Plugin installation and discovery today

We have two ways to manage plugins:

Automated Plugin Installation

• packer init: if you're using HCL2 for your templates, you can specify the plugins you're using in a required_plugins block, then invoke packer init to install the plugins automatically from GitHub into the correct plugin filesystem directory structure.

• packer plugins install: not dependent on template types, packer plugins install lets you install a plugin by specifying its URI and an optional version on the command line. The plugins subcommand, like packer init, will install the plugins from GitHub into the correct plugin filesystem directory structure.

Manual Plugin Installation

• An installation process where users download a version of a plugin and install it into a well-known plugin folder for Packer to load it. Typically this would be Packer's config directory, but other options are available, such as the directory from which Packer is invoked.

Re: manual installation

Packer's loading logic is complex and involves some heuristics when inferring the loading order and location of installed plugins. This issue is not the right place to delve into details, but we encourage referring to our documentation for more insights on plugin loading. However, this logic to infer plugin location and order contributes to the issues reported so far.

In addition to this, plugins required through the required_plugins block bypass manually installed plugins, unless they adhere to the correct plugin filesystem directory structure enforced by the packer init and packer plugins install commands. This is an error-prone process and is not well documented at the moment.

Options explored for Packer 1.10.0

• packer plugins install --path: This option allows packer plugins install to install a plugin from a binary either downloaded or built locally.

• packer build --plugin-override option: This option is implemented for both packer build and packer validate and forces Packer to consider only the provided binary as the executable for a plugin. This applies only to the process that is created when invoking Packer and therefore has no impact on the locally installed plugins.

• Adding a path attribute: Finally, this option introduces a path attribute to the required_plugins block. This is a solution that only applies to HCL2 templates, not to legacy JSON. This option modifies the template itself making it less reusable since executions in other requirements will require the same filesystem layout to execute reliably.

Among those options, we are leaning towards the first option, packer plugins install --path to support installing a plugin locally from a binary, as it solves most of the confusion regarding plugins described in required_plugins and how they're loaded. At least that's what we think.

We want to hear from you!

To help steer the direction of the next iteration of plugin discovery for Packer we're opening this issue to gather feedback from the community on a few proposed approaches. For the proposals that follow, we are interested in your answers to the following questions:

• What do you think about the proposal?
• Does the proposed solution make sense to you as a user?
• Do you have alternative approaches that would better suit your individual or team use case?
• Are there any gaps in the proposed solutions that we should consider?

The (farther) future

During this work, we recognized that the current workflow is confusing, and with the extra layer of required plugins, the plugin loading experience deteriorates further. As we embark on this journey for v1.10.0 and above we aim to simplify how plugins are loaded by potentially removing the legacy loading scheme, in favor of only supporting loading plugins from a specific filesystem structure that is compatible with required_plugins and consistent for users who must rely on manually installed plugins.

Stay tuned and thanks for the continued support!

[nywilken]

This work is ongoing. Please see the lastest Packer changelog for details on the upcoming changes in 1.11.0.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.