Decoder: pkcs11-uri in pem

[0pq76r]

This is an attempt to implement the feature "Support key in PEM file" #222.

The PEM file consists out of two fields an ASN1_OBJECT called "type" containing an oId to identify the object and an ASN1_UTF8STRING called "uri". The URI is parsed and created using the existing p11prov_parse_uri and p11prov_key_to_uri functions.

There are still some points left to improve:

• Flag to disable this non-standardized feature
• Improve variable/file naming (keypair_ref, active_object, ...)
• Find a more appropriate oId (Currently, I'm using "2.5.4.83" == {joint-iso-itu-t(2) ds(5) attributeType(4) uri(83)})
• Add tests

[0pq76r]

There are still some points left to improve:

* Flag to disable this non-standardized feature

The encoder has to be enabled by setting "pkcs11-module-encode-key-uri-to-pem = true".

* Improve variable/file naming (keypair_ref, active_object, ...)

Slightly improved but I'm terrible at naming stuff. Suggestions are welcome.

• Find a more appropriate oId (Currently, I'm using "2.5.4.83" == {joint-iso-itu-t(2) ds(5) attributeType(4) uri(83)})

This still needs to be addressed. Dropping the oId and just encoding an UTF8String could also be an option.

* Add tests

Tests showed that I need to loop over all tokens in the decoder. A similar loop is used in the store_fetch function, it could make sense to re-use that code.

I added libtasn1 as dependency for generating pem files in the tests. Originally I planed to use ossl asn1parse but it is broken in 3.2.0 (openssl/openssl#22992)

[simo5]

Do we really need this whole ASN.1 dependency when all we are doing is the simplest ASN.1 thing ever?
Pretty sure we can basically hardcode a generator if it is for testing only, given the only variable here is the length of the UTF8String and it's content.

[0pq76r]

I removed the dependency, what else needs to be changed? Do you have any opinion about the oId? Should it be removed completely?

[simo5]

Does it need to be an OID?
What if we have a regular string as the first member of the asn.1 structure and we put in a value like: "PKCS#11 Provider URI v1.0" ?

[simo5]

If OpenSSL expects OIDs (I have not checked) I can provide one from the Red Hat pool, let me know and I can assign.

[0pq76r]

It's not necessary to have an OID. I update the code to have a ASN1_VISIBLESTRING instead.
Though, I'm starting to wonder if the decoder is the right mechanism. My gut tells me that there should be a way to return the URI to the store and have it do the actual key lookup. But I have not figured out how that would work (or if it is possible at all).

[simo5]

I do not think there is a callback mechanisms that can change the nature of the load.

This is basically store(file) -> load(pem) -> decode(URI?) ...

And the decode is supposed to return a pkey type, so the only way is to decode(URI?) -> store(URI)

This is why this is a "hack" in the end and I was uncertain it would be a good idea.
OTOH it can work and it can make it easier for some software to work unmodified ...

Given we can avoid the use of OIDs and still make it very clear it is a pkcs#11 provider custom format I think I am happy enough.

[simo5]

I realized that perhaps you should just call the store api directly via OSSL_STORE_open_ex() with the uri and the provider private libctx and avoid re-implementing a bunch of store functionality.
Or have you already tried that and found a showstopper?

[0pq76r]

I realized that perhaps you should just call the store api directly via OSSL_STORE_open_ex() with the uri and the provider private libctx and avoid re-implementing a bunch of store functionality. Or have you already tried that and found a showstopper?

The OSSL_STORE returns an EVP_PKEY but the decoder is supposed to call the object_cb with an OSSL_PARAM array. The EVP_PKEY_todata calls the export method which for obvious reasons does not work for private keys.

I think calling the p11prov_store_*() is the way to go.

[simo5]

I realized that perhaps you should just call the store api directly via OSSL_STORE_open_ex() with the uri and the provider private libctx and avoid re-implementing a bunch of store functionality. Or have you already tried that and found a showstopper?

The OSSL_STORE returns an EVP_PKEY but the decoder is supposed to call the object_cb with an OSSL_PARAM array. The EVP_PKEY_todata calls the export method which for obvious reasons does not work for private keys.

I think calling the p11prov_store_*() is the way to go.

Ah, that's right!

[simo5]

I think we are very close, I like the changes you made, and I really only found nits.

[0pq76r]

I gave a build of this PR to some other devs. They found out that I forgot to add the encoder/decoder for Edwards curves. I added a commit to fix that.

[simo5]

I think we have all that is needed here, I am just confused about the patch to handle certificates.
My understanding is that this feature exclusively deals with private key pem<->uri ?

[simo5]

I guess the only way to make CI pass on debian, is to rebase on master, I have merged your PR for the t64 madness

[0pq76r]

I think we have all that is needed here, I am just confused about the patch to handle certificates. My understanding is that this feature exclusively deals with private key pem<->uri ?

The encoder is only implemented for private keys. In the decoder, the support for public keys comes for free. Explicitly filtering for private keys seems like an arbitrary restriction.

Unfortunately, the ossl_store_handle_load_result (https://github.com/openssl/openssl/blob/master/crypto/store/store_result.c#L85) only calls a second decoder for the keys (der->evp_pkey) . To have a similar decoding behaviour for the certificates as for the keys, the certificate needs to be returned while processing the pem file.

[simo5]

are you thinking of a certificate stored in the token, and a uri only be present in the PEM file ?

So basically this means you can use PEM files for any object in the token ?

[0pq76r]

Well, ideally the application would accept a pkcs11-uri for both key and certificate in the configuration.
Having a PEM file that supports any token object is more of a means to an end.

In the end, the goal is to only have to touch the token when doing a key/cert renewal. When an application only uses references to these objects the token can be used to distribute the new key/cert pairs.

[simo5]

Note, the code currently only handles OSSL_STORE_INFO_PKEY in expect, so it will have to be adjusted, but we can do as a followup to this PR, I think we should merge this one once good enough, to avoid dragging it forever

[simo5]

Sounds like Fedora address sanitizer is really mad, I kicked a rerun but see it it was not a fluke, unfortunately the logs are not very enlightening of where the problem lies ...

[numinit]

How do I export a key for this? I've been trying something like this with my Yubikey:

./result/bin/openssl pkeyutl -propquery provider=pkcs11 -in /dev/null -inkey 'pkcs11:serial=12345;object=Private key for Digital Signature' -out pk.pem

But get:

Enter pass phrase for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Public Key operation error
40A790B9707F0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2080:key type = id-ecPublicKey
40A790B9707F0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2080:key type = id-ecPublicKey
Segmentation fault (core dumped)

[numinit]

OK, I kind of got it working, but am still having trouble signing with the previous error, and get a segfault at the end...

./result/bin/openssl storeutl -keys -text 'pkcs11:serial=6460026;object-type=private'
Enter pass phrase for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
0: Pkey
PKCS11 EC Private Key (384 bits)
[Can't export and print private key data]
URI pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20(www.yubico.com);serial=12345;token=YubiKey%20PIV%20%2312345;id=%02;object=Private%20key%20for%20Digital%20Signature;type=private
-----BEGIN PKCS#11 PROVIDER URI-----
(redacted)
-----END PKCS#11 PROVIDER URI-----
1: Pkey
PKCS11 RSA Private Key (2048 bits)
[Can't export and print private key data]
URI pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20(www.yubico.com);serial=12345;token=YubiKey%20PIV%20%2312345;id=%19;object=Private%20key%20for%20PIV%20Attestation;type=private
-----BEGIN PKCS#11 PROVIDER URI-----
(redacted)
-----END PKCS#11 PROVIDER URI-----
Total found: 2
Segmentation fault (core dumped)

[0pq76r]

This does not export or wrap a key.
You can use openssl asn1parse to look at the content of the

-----BEGIN PKCS#11 PROVIDER URI-----
(redacted)
-----END PKCS#11 PROVIDER URI-----

You should find that it just contains the URI to the key.

About the segfault...
There is a no-deinit quirk, did you enable it?
It is necessary for some p11 libraries. They don't always like the OSSL cleanup being called in the atexit handler, especially those written in C++ that create global singletons. I don't know if it is necessary for the YubiKey.
If the no-deinit quirk is enabled and you still get a segfault or if the no-deinit was not necessary without this change, could you please share a stack trace?

[numinit]

The no-deinit quirk was the solution, thanks.

To be clear: for some reason, it appears to be having trouble exporting the public key during sign(!) operations, even when I'm not using the pem exported key reference, which is strange:

./result/bin/openssl pkeyutl -sign -inkey provider.pem -keyform pem -rawin -digest sha256 -in default.nix -out default.sig
Enter pass phrase for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Public Key operation error
4047A0C25C7F0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2080:key type = id-ecPublicKey
4047A0C25C7F0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2080:key type = id-ecPublicKey

./result/bin/openssl pkeyutl -sign -inkey 'pkcs11:model=YubiKey%20YK4;manufacturer=Yubico%20(www.yubico.com);serial=12345;token=YubiKey%20PIV%20%2312345;id=%02;object=Private%20key%20for%20Digital%20Signature;type=private' -rawin -digest sha256 -in default.nix -out default.sig
Enter pass phrase for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Public Key operation error
4007B944FE7E0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2080:key type = id-ecPublicKey
4007B944FE7E0000:error:030000CD:digital envelope routines:evp_pkey_copy_downgraded:keymgmt export failure:crypto/evp/p_lib.c:2080:key type = id-ecPublicKey

Additionally, this occurs on the 0.3 branch, leaving me to believe it isn't related to this changeset. So I'll keep further commentary off this ticket.

[numinit]

Got it working. Problem was I was doing shenanigans with a PKCS#11 engine, too. Confirmed (mostly) working:

$ ./result/bin/openssl pkeyutl -sign -inkey provider.pem -keyform pem -rawin -digest sha256 -in default.nix -out default.sig
Enter pass phrase for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):

$ hexdump -C default.sig
00000000  30 64 02 30 70 df 9f 72  f7 2a af f4 b9 f4 0b 64  |0d.0p..r.*.....d|
00000010  d4 89 c2 e0 f8 c2 de 97  bd b6 9e ef 5d 6c 46 9d  |............]lF.|

Then, in Ruby, which only supports loading PEM files:

key = OpenSSL::PKey.read(File.read('provider.pem'))
Enter PEM pass phrase:
Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
=> #<OpenSSL::PKey::EC:0x00007f9949099fb8 oid=id-ecPublicKey>
irb(main):003> sig = key.sign 'SHA256', 'hello world'
(irb):3: [BUG] Segmentation fault at 0x00000000000000281

Oops, a ruby bug :-)

[simo5]

Thanks for this contribution, looking forward to use it.

[numinit]

This is awesome, well done.

[numinit]

Node:

const crypto = require('node:crypto');
const fs = require('node:fs');
const privkey = crypto.createPrivateKey(fs.readFileSync('provider.pem').toString('ascii'))
// Enter PIN for PKCS#11 Token (Slot 2 - Yubico YubiKey OTP+FIDO+CCID 02 00):
const sign = crypto.createSign('SHA256'); sign.update('hello, node!'); sign.end();
const signature = sign.sign(privkey);
// <Buffer 30 65 02 31 00 99 8f e7 43 1a a4 90 4d cd 0c 15 bf 98 db 41 dd a3 a0 e5 5c 95 19 fe 4d 2d 33 d1 32 6a 5b 7b c4 05 f5 22 a1 ea 67 65 6a e0 0f ef 2f 13 ... 53 more bytes>