Support for pnpm

[willsoto]

Seems to be a fairly popular alternative to npm and yarn. Apologies if this has already been suggested but I couldn't find an existing issue. It would be great to be able to volta pin pnpm.

https://pnpm.js.org/en/

[charlespierce]

Agreed, this is definitely something we’ve discussed on Discord, I’m surprised there wasn’t an issue already either. Thanks!

[arxpoetica]

What would need to take place for this to happen? Point me to the right place in the code, and I'll see if I can get around to a PR.

[charlespierce]

Hi @arxpoetica, we would definitely welcome that! Adding support for pnpm—or really any unsupported package manager—is a bit involved, though hopefully not too difficult. There are 3 main areas that need to be updated:

• Fetching / installing / pinning the version, which is all contained within the tool module in crates/volta-core (https://github.com/volta-cli/volta/tree/main/crates/volta-core/src/tool)
  • The changes here can mostly be based on what is done for npm, the only caveat I'm aware of is that instead of editing the launcher files, we need to create them (so that the bare command pnpm works as expected).
• Reading / writing pnpm settings, contained in the platform, project, and toolchain modules.
  • The changes here should mostly be adding the pnpm key alongside npm and yarn
• Running pnpm through the shim, contained in the run module.
  • Here the logic should be almost identical to the yarn logic.

That's a very high-level overview of the needed changes, but I'll take a look at write some more this week about any corner cases / difficulties I can predict. I'm also happy to pair on this for a while if that would help.

[arxpoetica]

@charlespierce this ended up being something I just haven't had time to get around to yet. It's definitely still on my radar, but if someone feels like giving it a head start (or even attacking totally)...

...I think you'll find it happening more quickly. Sorry for my lack of involvement (re: time).

[charlespierce]

@arxpoetica No worries at all, thanks for updating!

[willsoto]

@charlespierce slightly related to this, has any thought been given to pinning arbitrary dependencies that would useful to have available on the command-line? For example, it might be useful to have mocha or tsc installed and available directly on the command line at a specific version without telling users to volta install it themselves. Another example is something like lerna or rush which are the primary entry-points for repos that are managed by those tools.

Under the hood, maybe something specific happens when pnpm is installed but in some ways it is just another command line tool being managed by Volta.

Just wondering if you consider this out of scope since the documentation for pin has something specifically about this.

[charlespierce]

Hi @willsoto, are you talking about something like #861? In that issue there's some discussion about the complications, however that is definitely something we'd like to support. Or do you mean a setting in package.json that says "Use Volta to manage running lerna"?

If the latter, there's a couple reasons that's not really in scope for Volta:

1. Technically speaking, Volta operates with shims in a shell-agnostic way, there's no way for us to shim tools that we don't know about ahead of time (if a user runs lerna and the shell says "Command not found", that never goes through anything Volta controls). Individual shells may have a way to control that behavior, but it's unlikely to be universally applicable across OS and shell combinations.
2. Philosophically, we decided against creating automatic shims for everything that had been installed, because it caused issues and raised some security concerns (see remove autoshimming #235 for the discussions). We ultimately decided that the end user should have direct control over which tools we do or do not manage, so it would go against that to allow packages to take that control away from the user.

[willsoto]

Of course I didn't see #861 when I went looking for existing issues 🤦

After reading through it a couple of times, it does seem like that would also solve my problem, just in a different way.

Or do you mean a setting in package.json that says "Use Volta to manage running lerna"?

Yeah, something like this is what I had in mind:

// package.json
{
  "volta": {
    "lerna": "2.0.0"
  }
}

I totally understand if this is considered out of scope for Volta, but I thought since pnpm is something that gets installed via npm (I don't think they have an alternative way to download?) then you could really get any tool in this way. I'll subscribe to #861 and see where that goes so I don't clutter this issue up. Thanks

[zkochan]

@muuvmuuv suggested us to document Volta as the preferred tool with pnpm. Unlike Yarn, pnpm currently doesn't have the ability to seamlessly switch between different versions of pnpm. So if Volta was to support pnpm, I think we could just recommend using the pin feature of Volta.

Also, @arcanis might be interested in this discussion, he created corepack

[zkochan]

I think we can offer a $50 bounty from our opencollective funds (if this is enough, I don't know how much work it involves).

[charlespierce]

@zkochan I took a look at it before our 1.0 release late last year, and it actually was somewhat involved, as our models and abstractions around the package managers aren't as clean as they could be (some room for tech debt cleanup). However, it was mostly straightforward. The main sticking point was around the global package directory: As I recall, there isn't a clean way to redirect both the installation and bin directories to a custom location in pnpm using only environment variables.

Volta does that redirection when installing global packages so that we can sandbox them to a specific Node version, and I recall struggling to get that working. I think I could do it with a command-line flag, but that gets a little tricky if there's already one specified and so I didn't dig in too deeply. There also appeared to be a "layout version" type directory that was always appended, which we would need to take into account, but I think I had worked out how we can handle that on our side.

I determined most of that with some experimentation and poking around the source of pnpm, so if there's something I missed about how to define the global directories, that would be much appreciated!

So in all, I think it's probably a larger task than a $50 bounty, however if someone was interested and wanted to collect that on top of the experience, I'd be more than happy to mentor them through it. Working on Volta is also part of my day-to-day, so if I can get it onto the roadmap that would be the other way.

[zkochan]

Is there something we could implement on pnpm's side to simplify the task?

You may set the NPM_CONFIG_GLOBAL_DIR env variable to change the global directory used by pnpm. The global bin directory will currently be selected automatically from one of the directories in PATH. pnpm has to have write access to the directory and the directory should have npm, pnpm, nodejs, or node in the path

https://github.com/pnpm/pnpm/blob/9c41e11b0bfa3f4ece2b7d4d67690496f1c9b9c0/packages/global-bin-dir/src/index.ts#L35-L40

[charlespierce]

Is there something we could implement on pnpm's side to simplify the task?

I don't think so, only because those changes would necessarily be in only newer versions of pnpm, while we would ideally want to support a larger range of versions.

You may set the NPM_CONFIG_GLOBAL_DIR env variable to change the global directory used by pnpm. The global bin directory will currently be selected automatically from one of the directories in PATH. pnpm has to have write access to the directory and the directory should have npm, pnpm, nodejs, or node in the path

https://github.com/pnpm/pnpm/blob/9c41e11b0bfa3f4ece2b7d4d67690496f1c9b9c0/packages/global-bin-dir/src/index.ts#L35-L40

Interesting, that is jogging my memory a bit. I think I may have been able to get that to work by modifying the PATH, but I don't recall the specific issues I was hitting any more (and I appear to have failed to write them down anywhere). I don't think any of them were insurmountable, just that they were decent amount of work to fit into everything, taking into account our current model.

[muuvmuuv]

I would add another 30€.

Just saying, but I'm already using pnpm with Volta since three months without any problems but would also love to see a "clean" way of implementation for future package managers/binaries so Volta will stay on top of the other managers.

[cjk]

I'm using pnpm a lot, so I'll wait until proper pnpm-support is available before switching to Volta

[Kinrany]

I don't think so, only because those changes would necessarily be in only newer versions of pnpm, while we would ideally want to support a larger range of versions.

Is it possible to provide support for newer versions and just warn the user that older versions are not supported?

I think most people using pnpm are OK with upgrading once even if they want to pin their version and not upgrade regularly.

[shiftgeist]

I think most people using pnpm are OK with upgrading once even if they want to pin their version and not upgrade regularly.

pnpm warns about outdated versions whenever you use it. This should reduce the amount of outdated versions. At least for desktop use.

[charlespierce]

@Kinrany @shiftgeist Those are good points, if we need to we can likely work to get changes upstreamed to pnpm and then only support that version or later. Ideally we'd make it work without that, but as a last resort that makes sense, especially if pnpm itself is vocal about pushing you to upgrade.

[kaelonR]

As I would also like to see pnpm officially supported as a pinnable package manager by Volta, I will be adding a $110 bounty for getting this supported. Together with zkochan's $50 bounty offer, this brings the total collectable bounty to $160. (+30EUR/35 USD if muuvmuuv is actually offering another 30EUR, but not sure whether he's offering €30 or was suggesting to raise the bounty by €30).

Hopefully this is enough to stimulate integration of pnpm :)

[arxpoetica]

I'll add $50 USD to that. This is one of my most watched issues.

[muuvmuuv]

Imagine they just pokering to get more people putting $$$ into the issue 😆

[kaelonR]

Imagine they just pokering to get more people putting $$$ into the issue 😆

The goal is to entice someone who's knowledgable enough to pick up this issue and integrate pnpm. The collectable bounty for implementing this is now $210. Surely someone must be interested in making a quick buck?

[muuvmuuv]

This was meant as a joke...

[shiftgeist]

Is there an issuehunt or similar project?

[ekil1100]

Is there an issuehunt or similar project?

I don't think so...can't find the link.

[shiftgeist]

Is there an issuehunt or similar project?

I don't think so...can't find the link.

Might be useful for collecting the funds (doesn't need to be issuehunt tho). I imagine multiple paypal transaction being a little sketchy.

[charlespierce]

Hey everyone! Sorry about the delayed response, I've been out for a week or so. Super exciting to see so much interest for pnpm, I'd definitely be happy to help facilitate the bounty if someone wants to take it on (and help guide people through the code for that section). My time has been pretty limited lately, as I just started a new job, but I'll also try to carve some out to push this forward.

[haosmos]

I just opened #1394 to add a feature flag environment variable to ensure we maintain backwards compatibility for users who currently have pnpm as a global package. Once that is approved / merged, we should be able to release Volta 1.1.1 with pnpm support included. The feature flag will mean that anyone who wants to try this behavior will need to set VOLTA_FEATURE_PNPM to 1 in their environment variables or profile script, for example:

export VOLTA_HOME="$HOME/.volta"
export VOLTA_FEATURE_PNPM=1
export PATH="$VOLTA_HOME/bin:$PATH"

@charlespierce, hi! 😃

Any news? When should we expect the long-awaited release with pnpm support? Is there any chance it will be released this year?

[charlespierce]

Sorry for the delay on this, I've been swamped with other work! Volta 1.1.1 was just released and it includes the experimental pnpm support (HUGE thanks to @chawyehsu for the implementation)! I also added a section to the advanced docs about the pnpm Support, noting the known limitations and how to activate it!

[arxpoetica]

@chawyehsu sent my sponsorship!

[BleedingDev]

Great, thanks! I am already trying it.

I would also add to limitations that it is not possible to pin pnpm. :)

[chawyehsu]

@pegak pnpm pinning is supported in 1.1.1. Perhaps you are encountering some issues with it. Please feel free to open a new issue to let us know.

[BleedingDev]

Oh, didn't have the update, sorry. Trying now!

[j]

@chawyehsu

~/Code/example main* ❯ volta --version
1.1.1

~/Code/example main* ❯ pnpm --version
7.27.1

~/Code/example main* ❯ volta pin [email protected]
error: Only node and yarn can be pinned in a project

Use `npm install` or `yarn add` to select a version of pnpm for this project.

[j]

Also, pnpm run commands use global node:

 WARN  Unsupported engine: wanted: {"node":">=19.0.0"} (current: {"node":"v18.13.0","pnpm":"7.27.1"})

[chawyehsu]

@j and for future comments here, you CANNOT pin pnpm if you do not enable the native pnpm feature for Volta while it's experimental. Follow the doc and enable native pnpm support, and notice the limitations including the migration note.

$ volta --version
1.1.1


# VOLTA_FEATURE_PNPM=0
#
# You are using the packaged pnpm which will fail to pin
$ volta pin [email protected]
error: Only node and yarn can be pinned in a project

Use `npm install` or `yarn add` to select a version of pnpm for this project.


# VOLTA_FEATURE_PNPM=1
#
# Enable the _native_ pnpm feature in order to pin
$ volta pin node [email protected]
success: pinned [email protected] (with [email protected]) in package.json
success: pinned [email protected] in package.json

[j]

I just came across that doc. Sort of a pain as things seem so disconnected when using pnpm. pnpm is great and recommended by turborepo, but pnpm commands use global node, etc, etc. Fun!

[chawyehsu]

Sorry for the inconvenience @j

Also, pnpm run commands use global node:

 WARN  Unsupported engine: wanted: {"node":">=19.0.0"} (current: {"node":"v18.13.0","pnpm":"7.27.1"})

but pnpm commands use global node, etc, etc

This warning is from pnpm and not relevant to the current implementation of pnpm support for Volta. In the implementation, we only intercept a basic set of pnpm commands to work with Volta in pinning, etc. And most of the workflows are directly passed to pnpm transparently. Hence there may be things unlinked when using pnpm at present because pnpm may work outside of Volta's control.

Talking about the warning message, that's because you specified the node version via engines in package.json for your project. Something like this:

{
  "engines": {
    "node": ">=19"
  }
}

And pnpm will print out the warning message when you are not using the corresponding version of node.¹ It's not about the global node being used, it's about that you haven't pinned the node via Volta for your project.

IIRC, Volta does not support reading the engines property and pinning node as per its specified version yet. There have been tracking issues about supporting engines²³. And as one of the users of Volta, I wish it will be eventually supported, too.

For now, to avoid pnpm's warning message, you can pin a node version that is identical to the one of engines by volta pin [email protected], or delete the engines property.

+  "volta": {
+    "node": "18.13.0",
+    "pnpm": "7.27.1"
+  },
- "engines": {
-   "node": ">=19"
-  }

Anyways, doing some searches on the issue trackers will help.

Footnotes

1. https://github.com/pnpm/pnpm/issues/3673 ↩

2. https://github.com/volta-cli/volta/issues/1422 ↩

3. https://github.com/volta-cli/volta/issues/355 ↩

[michaelhays]

Is there somewhere we're tracking work/requirements on the known limitations of pnpm support?

I'd love to help on those -- from my understanding, we just have the global installations and automatic migrations from the Volta pnpm global left until this can go from experimental to stable.

(FWIW, I've been using pnpm with VOLTA_FEATURE_PNPM set (see #1394) for a couple of months now and it's been working perfectly)

[lolmaus]

I was running into this error:

❯ volta install pnpm
error: Executable 'pnpm' is already installed by corepack

I resolved it by using this command instead:

corepack prepare pnpm@latest --activate

[dwiyatci]

@chawyehsu Seems like I got into this awkward state in a project where:

"volta": {
  "node": "18.18.2"
},

But then when I ran: pnpm, it threw: "Volta error: No pnpm version found in this project.".

I've got volta v1.1.1 and export VOLTA_FEATURE_PNPM=0 set in my .zshrc.

I've read this bit from https://docs.volta.sh/advanced/pnpm#migrating, but I don't quite get it:

Once you switch to the native pnpm support, you may not remove the isolated old pnpm package by calling the same command because lack of uninstall implementation as of now.

Running which pnpm outside of the project results in /Users/Glenn/Library/pnpm/pnpm.
But running which pnpm from the project results in /Users/Glenn/.volta/bin/pnpm.

My intention is to keep using globally-installed pnpm without volta gets in the way. How can I achieve that back? Thank you 🙏

[chawyehsu]

keep using globally-installed pnpm without volta gets in the way.

@dwiyatci There is nothing VOLTA_FEATURE_PNPM can do, Volta can't deal with pnpm that's not installed with Volta. You want to use pnpm outside of Volta then you may need to remove Volta's pnpm shims from PATH.

[dwiyatci]

you may need to remove Volta's pnpm shims from PATH.

Thanks for the pointer, @chawyehsu! I did the following and seems like I managed to switch back using global pnpm properly:

• Remove pnpm entry from /Users/Glenn/.volta/tools/user/platform.json
• Remove /Users/Glenn/.volta/bin/pnpm
• (Re-)install pnpm (globally): curl -fsSL https://get.pnpm.io/install.sh | sh -

Have a nice weekend! :)🎉

[ghost]

It's almost four years since this issue was raised, but why is it still supported as an experiment and not made official?

Please let us know if there are any unresolved issues.
Volta won't support pnpm forever, so I'm starting to avoid it.

[ShravanSunder]

i'm also curious about the experimental tag and what's outstanding

[chawyehsu]

The project is fully community driven now. It's ok that you going to avoid this avoid that, your business is your business. However that does not help at all just complaining and pressuring in this thread. Read the source code, know how it work and what is unresolved, and submit PRs to contribute, that is the way! And that's how I did and added the existing pnpm support.

[smblee]

FYI I use https://mise.jdx.dev/ now instead of volta.

[BleedingDev]

And now I switched to Proto. :) Would be good to see some comparison.

https://moonrepo.dev/proto

[TheBrenny]

Read the source code, know how it work and what is unresolved, and submit PRs to contribute, that is the way!

Totally agree! However, there's 4 years of lessons learnt and large amounts of lore in this thread that people jumping in aren't tracking. Yes they can read the thread, but then there's also things stuck in brains that we don't have access to (such as why certain decisions were made).

I don't know Rust, and my patience for it is incredibly low 🙁, but in the interest of the community, are people able to brain-dump what we all know about this issue, it's challenges, opportunities, and the decisions made throughout the dev process? Perhaps @willsoto could then add these points to the original post he made?

It's ok that you going to avoid this avoid that, your business is your business.

Fwiw, I did move to moonrepo, but it didn't really feel as nice as Volta, and so I've moved back.

[RoyRao2333]

Fwiw, I did move to moonrepo, but it didn't really feel as nice as Volta, and so I've moved back.

They are both great tools. But the reason I chose Proto over Volta is not only because it supports pnpm, but also because of its positive attitude towards fixing issues and the high frequency of updates and patches. It also has an active Discord community and enthusiastic maintainers.

Not gonna make any personal comments about this repo and its maintainers, because it's a great library all the same, and I admire the people who have maintained it for so long.

But for those people who are just reading this thread, I hope you go ahead and try moonrepo/proto.

[chriskrycho]

We definitely don’t take it personally – if another tool works better, by all means go for it! And we’re all painfully aware of how little activity the repo has. Moonrepo and proto has actual direct funding to work on it, so more power to them! Volta does not have any ongoing funding, and indeed has only had a few months of active funding since it got to the “solves all of the primary corporate sponsor’s internal needs” point back in late 2019 (and those couple months of work were for solving another of those internal needs).

This is, alas, an all-too-familiar tradeoff of corporate-sponsored OSS: it is really hard to justify further work on it when it solves the problems folks have in whatever corporate context. Insofar as you see work happening on Volta at all, it’s on our personal time—which, in my case for example, is also competing against family time, other non-software projects 🎼, and hobbies besides my job 🏃🏻‍♂️🚴🏼‍♂️!

We are doing what we can even so, but again, we’re not offended if people find the velocity and community engagement of other actively funded tools more to their taste: that’s perfectly reasonable and understandable!

[chawyehsu]

However, there's 4 years of lessons learnt and large amounts of lore in this thread that people jumping in aren't tracking ... but then there's also things stuck in brains that we don't have access to (such as why certain decisions were made).

If you are concerned about what's unresolved of pnpm support, I believe #1273 and #1394 should be the place to learn instead of this thread, comments in there will help to understand.

But the reason I chose Proto over Volta is not only because it supports pnpm, but also because of its positive attitude towards fixing issues and the high frequency of updates and patches.

No offense, but taking a YC funded project to be compared with a fully community driven project that has no funding, that makes no sense imho. It's good for projects like moonrepo/proto to receive ongoing and healthy funding, especially for open source projects (and I want to say big thanks to those who have sponsored me last year, that made the current state of pnpm support possible). VC funding is a double-edged sword though, the team have to be under the pressure of generating revenue.

For end users, people take advantage of it if they find something meets their needs. That's definitely reasonable, one however should know the funding problem in OSS has been there for decades...

This does not mean demand a sponsorship. But when you don't have funds to sponsor for things you demand, and if you could implement it then contributing code can be the other way to give back to the community. I spent my personal time to contribute to Volta (when I did #1761 that was 11PM), and I still believe that's the positive thing to the project/community.

other non-software projects 🎼

Holy that IS awesome!