missing BIO_FLAGS_BASE64_NO_NL flag in est_client.c/b64_decode_cacerts function

[tomaszpr]

Hi,

When I try to get CA chain from OpenXPKI by est_client_get_cacerts function I get error:

***EST [INFO][est_io_get_response_internal:1589]--> Read 4075 bytes of HTTP data
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Date:Mon, 29 Mar 2021 11:49:27 GMT
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Server:Apache/2.4.38 (Debian)
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Transfer-Encoding:base64
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Length:3748
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Strict-Transport-Security:max-age=31536000
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> X-Frame-Options:deny
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> X-XSS-Protection:1; mode=block;
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Connection:close
***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Type:application/pkcs7-mime; smime-type=certs-only
***EST [INFO][parse_http_headers:903]--> Found 9 HTTP headers

***EST [INFO][est_io_get_response_internal:1610]--> HTTP status 200 received
***EST [INFO][est_io_get_response_internal:1767]--> HTTP Content len=3748

***EST [ERROR][create_PKCS7:472]--> Unable to read in PKCS7 based certificate buffer
***EST [WARNING][ossl_dump_ssl_errors:322]--> OSSL error: 3046806624:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:crypto/asn1/asn1_lib.c:101:

***EST [ERROR][verify_cacert_resp:587]--> Failed to build PKCS7 structure from received buffer
***EST [ERROR][est_client_send_cacerts_request:3121]--> Returned CACerts chain was invalid

But when I add

    in = BIO_push(b64, in);

    decoded_buf = malloc(*cacerts_len);
    if (decoded_buf == NULL) {
        EST_LOG_ERR("Unable to allocate CA cert buffer for decode");
        BIO_free_all(in);........
        return (EST_ERR_MALLOC);........
    }
+++BIO_set_flags(in, BIO_FLAGS_BASE64_NO_NL);
decoded_buf_len = BIO_read(in, decoded_buf, *cacerts_len);

everything works correct.

I have tested it on buildroot and Zynq CPU.
libest_client-3.2.0p.so

Best regards,
Tomasz Przybysz

[GuidoKiener]

Hi Tomasz,

I can confirm your observation. Setting the flag BIO_FLAGS_BASE64_NO_NL only helps in your situation, but not for other EST servers that are using Base64 code with NL. There are functions that could solve the problem like est_client_get_pkcs7_from_buf(..) in est_client.c or est_base64_decode(...) in est.c. I don't know why verify_cacert_resp(..) doesn't use it.
Maybe Pete can tell us whether the OpenXPKI server is just using the wrong format or the EST client needs a bugfix.

BTW Did you succeed to get a certificate from OpenXPKI?

Regards,

Guido

[tomaszpr]

Hi Guido,

I succeeded in getting CA chain and end-certificate from OpenXPKI.
I modified est_client.c source file:

1st modification: as I wrote before, because of base64 decoding error

static EST_ERROR b64_decode_cacerts (unsigned char *cacerts, int *cacerts_len,
                                     unsigned char **cacerts_decoded,
                                     int *cacerts_decoded_len)
...

    in = BIO_push(b64, in);    
    decoded_buf = malloc(*cacerts_len);
    if (decoded_buf == NULL) {
        EST_LOG_ERR("Unable to allocate CA cert buffer for decode");
        BIO_free_all(in);        
        return (EST_ERR_MALLOC);        
    }

    BIO_set_flags(in, BIO_FLAGS_BASE64_NO_NL); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    decoded_buf_len = BIO_read(in, decoded_buf, *cacerts_len);
...

2nd modification: after removing CRL from certificate, output length was 0

static EST_ERROR est_client_remove_crls (EST_CTX *ctx, unsigned char *cacerts,
                                         int *cacerts_len, PKCS7 *p7)

...
        p7bio_out = BIO_new(BIO_s_mem());
        if (p7bio_out == NULL) {
            EST_LOG_ERR("Unable to access the CA cert buffer");
            ossl_dump_ssl_errors();
            return(EST_ERR_MALLOC);
        }
        p7bio_out = BIO_push(b64_enc, p7bio_out);
        BIO_set_flags(b64_enc, BIO_FLAGS_BASE64_NO_NL); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
        
        memzero_s(cacerts, *cacerts_len);
        
        count = i2d_PKCS7_bio(p7bio_out, p7);
...

Now I get:

est_client_get_cacerts:
emb_certverify: [EST] [GETCA] service 'dm-tls' get CA request send, url: https://xxxxxxxxxx
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][tcw_direct_connect:168]--> getaddrinfo(xxxxx, 443)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][tcw_direct_connect:204]--> connect(xxx.xxx.49.123 port 443)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][tcw_connect:592]--> Successfully connected to xxxx:443
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][cert_verify_cb:671]--> entering: Cert passed up from OpenSSL. error = 0 (ok)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][cert_verify_cb:671]--> entering: Cert passed up from OpenSSL. error = 0 (ok)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][cert_verify_cb:671]--> entering: Cert passed up from OpenSSL. error = 0 (ok)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_verifyhost:2543]--> Found 1 SubjectAlternateNames
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_verifyhost:2560]--> Checking FQDN against SAN xxx
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_verifyhost:2621]--> subjectAltName: xxx matched
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_send_cacerts_request:3009]--> TLS wrote 153 bytes, attempted 153 bytes
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_io_get_response_internal:1589]--> Read 4075 bytes of HTTP data
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Date:Thu, 08 Apr 2021 13:38:45 GMT
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Server:Apache/2.4.38 (Debian)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Transfer-Encoding:base64
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Length:3748
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Strict-Transport-Security:max-age=31536000
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> X-Frame-Options:deny
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> X-XSS-Protection:1; mode=block;
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Connection:close
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Type:application/pkcs7-mime; smime-type=certs-only
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:903]--> Found 9 HTTP headers
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_io_get_response_internal:1610]--> HTTP status 200 received
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_io_get_response_internal:1767]--> HTTP Content len=3748
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][verify_cacert_resp:571]--> Adding cert to trusted store (H
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][verify_cacert_resp:605]--> Adding cert to store (
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_cacert_verify_cb:230]--> enter function: ok=1 cert_error=0
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_cacert_verify_cb:230]--> enter function: ok=1 cert_error=0
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][verify_cacert_resp:605]--> Adding cert to store (H
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_cacert_verify_cb:230]--> enter function: ok=1 cert_error=0
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_remove_crls:308]--> CRL(s) attached with the CA Certificates.  Removing CRL(s)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_send_cacerts_request:3077]--> CACerts buf: MIIK9AYJKoZIhvcNAQcCoIIK5TCCCuECAQExADALBgkqhkiG9w0BBwGgggrJMIIFmTCCA4GgAwIBAgIUQ8Je+ALqMLvZ0lD44aPUoyWrxOgwDQYJKoZIhvcNAQELBQAwHTEbMBkGA1UEAwwST3BlblhQS0kgUm9vdCBDQSAxMB4XDTIwMDYyNjA1MzUxMloXDTI1MDYyODA1MzUxMlowUzELMAkGA1UEBhMCREUxETAPBgNVBAoMCE9wZW5YUEtJMQwwCgYDVQQLDANQS0kxIzAhBgNVBAMMGk9wZW5YUEtJIERlbW8gSXNzdWluZyBDQSAxMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEArG/b1hupG4bpYpm4lKwlLeocMbxnNWVa3/ZeXSk404UrtfWnNVePVAuXnGR3O0wdJV6znoz0XSG
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_send_cacerts_request:3078]--> CACerts length: 3744
emb_certverify: [EST] [GETCA] resolved server ip:xxx.xxx.xxx.xxx
emb_certverify: [EST] [DBG][GETCA] service 'dm-tls' est_client_copy_cacerts: result: 0, base64_len: 3744
emb_certverify: [EST] [DBG][est_response_ca_ra_cert_get] b64_decode: result: 0, cacerts_decoded_len: 2808
emb_certverify: [EST] [GETCA] CA certs request: received 2 cert(s)
emb_certverify: [EST] [GETCA] service 'dm-tls' sig cert update: /C=DE/O=OpenXPKI/OU=PKI/CN=OpenXPKI Demo Issuing CA 1, kusage: 0086
emb_certverify: [EST] [GETCA] service 'dm-tls' CA Root cert change detected. Do not update!: /CN=OpenXPKI Root CA 1, kusage: 0086

est_client_enroll_csr:
emb_certverify: [EST] [ENROLL] service 'dm-tls' cert enroll request check, url: https://xxx
emb_certverify [ENROLL] service 'dm-tls' state set to: 15
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][tcw_direct_connect:168]--> getaddrinfo(xxx, 443)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][tcw_direct_connect:204]--> connect(xxx.xxx.49.123 port 443)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][tcw_connect:592]--> Successfully connected to xxx:443
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_verifyhost:2543]--> Found 1 SubjectAlternateNames
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_verifyhost:2560]--> Checking FQDN against SAN xxx
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_verifyhost:2621]--> subjectAltName: xxx matched
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_add_auth_hdr:1228]--> No HTTP auth mode set, sending anonymous request
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_send_enroll_request_internal:1633]--> TLS wrote 1292 bytes, attempted 1292 bytes
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_io_get_response_internal:1589]--> Read 2611 bytes of HTTP data
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Date:Thu, 08 Apr 2021 13:42:51 GMT
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Server:Apache/2.4.38 (Debian)
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Transfer-Encoding:base64
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Length:2284
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Strict-Transport-Security:max-age=31536000
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> X-Frame-Options:deny
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> X-XSS-Protection:1; mode=block;
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Connection:close
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:893]--> Found HTTP header -> Content-Type:application/pkcs7-mime; smime-type=certs-only
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][parse_http_headers:903]--> Found 9 HTTP headers
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_io_get_response_internal:1610]--> HTTP status 200 received
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_io_get_response_internal:1767]--> HTTP Content len=2284
emb_certverify: [EST] [DBG] openssl: ***EST [INFO][est_client_enroll_req:2086]--> Newly Enrolled Client certificate:
emb_certverify: [EST] [ENROLL] resolved server ip:xxx.xxx.xxx.xxx
emb_certverify [ENROLL] service 'dm-tls' state set to: 17
emb_certverify: [EST] [DBG][ENROLL] service 'dm-tls' est_client_copy_enrolled_cert: result: 0, base64_len: 2284
emb_certverify: [EST] [DBG][est_response_cert_get] b64_decode: result: 0, cacerts_decoded_len: 1711
emb_certverify: [EST] [DBG][est_response_cert_get] OBJ_obj2nid: 22
emb_certverify: [EST] [ENROLL] certs request: received 1 cert(s)
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            f5:ff:2b:cd:3e:dc:a3:7a:b1:b3
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=DE, O=OpenXPKI, OU=PKI, CN=OpenXPKI Demo Issuing CA 1
        Validity
            Not Before: Apr  8 13:42:44 2021 GMT
            Not After : Oct  8 13:42:44 2021 GMT
        Subject: DC=org, DC=OpenXPKI, DC=Test Deployment, CN=13410612
        Subject Public Key Info:

Best regards,
Tomasz

[tomaszpr]

Hi,
Is there any progress in this issue ?

I have tested it with OpenXPKI server where there are 2 certificates returned on est_client_get_cacerts request.
The testrfc7030.com server returns only one certificate.
This is the 1st difference.

The 2nd difference is that
OpenXPKI returns all data in one text line, testrfc7030.com divides return data into multiple line,

Can library support both of this cases ?
Is this project still alive ?

[GuidoKiener]

Hi Tomasz,

You are right. It seems to me that nobody cares about this project anymore. It's just a demo, and you have to fix the code to allow reading base64 data w/ and w/out newlines. It's more or less a strange OpenSSL feature that you have to set the flag BIO_FLAGS_BASE64_NO_NL for reading base64 data without newlines.
If you like GO then you also can try this client/server https://github.com/globalsign/est. It has some more features but doesn't support POP (Prove of Possession).

-Guido

[Jmennius]

Same issue here.
But globalsign/est is very nice (@GuidoKiener thanks for the reference!).