Volta does not use all certificates in the system CA certificate store, making corporate use difficult

[gabrielbauman]

Hello!

Volta does not appear to use all certificates installed in the system CA certificate store.

This is a problem for corporate developers who are required to trust a company CA to allow their administrators to see all encrypted traffic. For example, our company uses Cloudflare Zero Trust and their CA root is trusted by my system (ubuntu).

Here's what this looks like:

$ volta install node
error: Could not download Node version registry
from https://nodejs.org/dist/index.json

Please verify your internet connection.
Error details written to /home/user/.volta/log/volta-error-2023-12-28_18_08_10.834.log

The error log shows:

Error cause: Io Error: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

However other apps that DO use all certs in the system CA certificate store work just fine:

$ wget https://nodejs.org/dist/index.json
--2023-12-28 18:09:25--  https://nodejs.org/dist/index.json
Resolving nodejs.org (nodejs.org)... 2606:4700:10::6814:172e, 2606:4700:10::6814:162e, 104.20.23.46, ...
Connecting to nodejs.org (nodejs.org)|2606:4700:10::6814:172e|:443... connected.
HTTP request sent, awaiting response... 200 OK

Many users, particularly those who've had their laptops set up by an IT department, will be completely unaware that a company CA has been installed or that their SSL traffic is man-in-the-middled by corporate. However, uninstalling the CA or circumventing it are often grounds for disciplinary action.

Could volta please be adjusted to use all CA certificates installed in the system for outbound HTTPS connections?

[chriskrycho]

Thank for flagging this up, @gabrielbauman – it certainly seems reasonable to use the system CA cert store! @charlespierce can likely comment further and possibly point in the right direction for where our implementation internals will need to change to pull that off, but as you might expect he (and most of the rest of us!) are largely on vacation this week.

[rwjblue]

I think this is likely related to the swap from linking with OpenSSL to using RustTLS statically. Based on a quick search, I think we might need to take a look at using the tls-rustls-native-roots feature added in attohttpc v0.24.0 (which uses https://github.com/rustls/rustls-native-certs).

References:

• Switching to a Statically-Linked TLS implementation rfcs#47
• Statically link to Rustls instead of using OpenSSL #1214
• https://github.com/rustls/rustls-native-certs?tab=readme-ov-file#should-i-use-this-or-webpki-roots
• https://github.com/sbstp/attohttpc#features

[charlespierce]

Hi @gabrielbauman, which version of Volta are you on? In the 1.1.1 Release of Volta, we updated our network handling to use the native certificate store (using the tls-rustls-native-roots feature that @rwjblue mentioned above. If you're using a version below that, does updating fix the issue?

[gabrielbauman]

volta --version says 1.1.1. This is on Ubuntu Jammy (LTS). The issue is present in the current release.

Sorry, should have included version in the original issue. Happy New Year!

[charlespierce]

Hi @gabrielbauman, thanks for verifying the version. That's very odd, since as I mentioned, we are currently set up to use the system root certificate store. I wonder if this is a case that the specific certificate is in a format that our underlying HTTPS implentation (attohttpc) doesn't recognize.

That's very much a stab in the dark, however, and I'm not at all sure how to test that hypothesis.

[gabrielbauman]

It's a bog-standard cert in PEM format. I would be really surprised if that were the issue, especially since wget and other command-line apps work fine.

I've now tested this on another laptop running latest fedora - same issue. Have you tried adding a self-signed cert to your system?

Let me know if there's anything I can test for you.

[charlespierce]

@gabrielbauman Thanks for confirming. That's not surprising, though it makes it harder to really understand what's going on. Out of curiosity, do you know how / where the certificate is installed onto the system? Full disclosure, I'm not an expert in the underpinnings of the certificates, but from reading through the code I'm wondering if it comes down to this issue in one of our transitive dependencies: rustls/rustls-native-certs#9

It looks like on Linux, if the certificate is installed using the "hash directory", where it's put into a specific directory and given a specific name (the hash of the certificate with .0 appended), then we aren't currently loading it.

[gabrielbauman]

Ubuntu wants .pem files, but renamed to .crt. So here's how I install the cert:

# curl https://developers.cloudflare.com/cloudflare-one/static/documentation/connections/Cloudflare_CA.pem > /usr/local/share/ca-certificates/Cloudflare_CA.crt
# update-ca-certificates

This creates two symlinks in /etc/ssl/certs:

# cd /etc/ssl/certs
# ls -la | grep Cloud
lrwxrwxrwx 1 root root     17 Feb  2 12:27 c38b367a.0 -> Cloudflare_CA.pem
lrwxrwxrwx 1 root root     50 Feb  2 12:27 Cloudflare_CA.pem -> /usr/local/share/ca-certificates/Cloudflare_CA.crt

As you can see, if a symlink to a root certificate is present in /etc/ssl/certs, Ubuntu uses it. At this point curl, wget and all other system utilities see and accept the certificate; Volta does not.

You can see that the "hash" link is created, but there's also a simple Cloudflare_CA.pem link that the rust crate should pick up.

[charlespierce]

@gabrielbauman Thanks, I think that helps narrow it down! It looks like update-ca-certificates should be concatenating the certificate information into /etc/ssl/certs/ca-certificates.crt, so it's definitely one of the files that is searched. My suspicion is that the root cause is the same as rustls/rustls-native-certs#28, namely that there is some other, higher-precedence certificate file that is found first, so the rustls-native-certs check doesn't find the appropriate file.

I'm reaching out to the projects involved (rustls-native-certs and openssl-probe) to see if there's something we can do to make sure the behavior matches that of apps such as wget and curl, because in theory the approach should yield the same set of root certificates.

As a quick check & potential workaround, does that file (/etc/ssl/certs/ca-certificates.crt) exist on your system? If so, does running export SSL_CERT_FILE="/etc/ssl/certs/ca-certificates.crt" in your terminal allow Volta to work?

[gabrielbauman]

Yes, /etc/ssl/certs/ca-certificates.crt exists. It contains all of the active CA certs concatenated together. The Cloudflare CA I've been trying to use is the last one in the list.

$ export SSL_CERT_FILE="/etc/ssl/certs/ca-certificates.crt"
$ volta fetch node
error: Could not download Node version registry
from https://nodejs.org/dist/index.json

Please verify your internet connection.
Error details written to /home/user/.volta/log/volta-error-2024-02-02_14_32_45.974.log

$ tail /home/user/.volta/log/volta-error-2024-02-02_14_32_45.974.log
"volta" "fetch" "node"
Volta v1.1.1

Could not download Node version registry
from https://nodejs.org/dist/index.json

Please verify your internet connection.

Error cause: Io Error: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

So no, no change with the environment variable set. I did see that node has a similar env var set:

declare -x NODE_EXTRA_CA_CERTS="/etc/ssl/certs/ca-certificates.crt"

[charlespierce]

Interesting! That's a major clue, though also a big setback. It seems like it's not related to not finding the certificate, rather it's something else going on that's causing the problem. Setting SSL_CERT_FILE as far as I can tell will definitely force that file to be read, and as you said the relevant certificate is in that file.

It seems like it may be something related to the certificate itself. I'll do some deeper digging on the error messages, maybe there's a set of failure conditions we can narrow down that would cause it.

Thank you again for the iteration and debugging, it's super helpful!

[gabrielbauman]

Happy to help, thanks for taking the time to look at what's definitely an edge case that doesn't affect most users.

We know the cert is read by the Volta/the crate (regardless of the env variable being set) because the error message in the log complains that it's self-issued: Error cause: Io Error: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

We know that the certificate is in a valid format (of course it is, other apps can also see and use it, and volta is happily parsing it before deciding to ignore it)

Something in Volta or a dep is rejecting the cert because it has an unknown issuer - but that's true of all CA roots! They are the ultimate authority. There's no one to issue a CA root. And most CA roots on my system are seen and used by volta.

Looking at the certificate, the sig is SHA512 with ECDSA and it is properly marked as a CA (you can download for yourself here) Maybe the error is misleading and there's something about the cert that your dependency can't handle.

Let m know if there's anything else I can do to help.

[charlespierce]

Ah ha, I think I found the issue! It looks like ring—the Rust-based cryptography library that is used by default in rustls—itself doesn't support signature verification for ECDSA + SHA512 at the moment. See:

• Support for ecdsaWithSHA512 signature algorithm rustls/rustls#1367
• Add support for verifying P-521 signatures with SHA-512 and SHA-384 briansmith/ring#1771

It appears there's some active work (or at least active as of a couple weeks ago) in ring to add that behavior, which would then need to percolate through the rest of the ecosystem. Once it gets added to rustls, I'm happy to create a PR to attohttpc to bump the version there—I've submitted changes to them before. So I think this may be a case of having to wait until that work is completed; I know I don't have anywhere near the necessary skill set to help, unfortunately.

If that works stalls, there's also the possibility of using a different crypto provider—rustls has a feature flag to use aws-lc-rs, however it looks like that comes with additional compile-time dependencies that might not be trivial to work out from our level. We would also need to figure out the best way to propagate that change, whether it's something that attohttpc accepts as yet another configuration on their end, or if it's something we silently patch in directly within Volta.

[jay-steyn]

Has any progress been made on this issue? We have been using Volta for some time until they rolled out WARP and now we have had to revert back to NVM.

[chriskrycho]

Unfortunately, no. The maintainers have all been swamped, and it looks like at least some of what we would need is still blocked upstream, per the comment before this one.

[tenoriojuann]

I had to downgrade to v1.08 to be able to use volta again. We have a self-signed certificate along with a proxy and the changes in v1.1 seem to have some breaking changes.

[israel2176]

I did this,
And that solved the problem for me.
sudo rm -rf /usr/lib/ssl/cert.pem
sudo cp /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem