api: add client authentication (BROKEN)

[yaauie]

When adding client auth, I get weird behaviour that I am having a hard time chasing down.

First, I have generated one.jks, which is attached in a targz.

First, I generated a new self-signed CA ca.key.pem + ca.crt.pem
Then I generated a new key one.key.pem, and used it to create a CSR.
I then used the CSR with my CA to create a signed key one.crt.pem.

I concatenated my one.crt.pem and my ca.crt.pem into one.chain.pem, and exported it along-side one.key.pem into a PKCS12-formatted container with password 12345678 named one.p12, and then imported that into a new JKS-formatted keystore one.jks with same password 12345678.

shared-ca.tar.gz

Using the above with the base of this PR, I can configure Logstash's API to have SSL enabled.

api.ssl.enabled: true
api.ssl.keystore.path: "${KEYSTORES}/one.jks"
api.ssl.keystore.password: "12345678"

After starting Logstash, I can curl the API providing my generated CA, and everything works:

╭─{ yaauie@limbo-2:~/src/elastic/ls/ (✘ secure-api-client-authentication-hangs) }
╰─● curl --verbose --cacert ca.crt.pem https://127.0.0.1:9600
* Rebuilt URL to: https://127.0.0.1:9600/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9600 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: ca.crt.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=WA; O=Elastic; OU=Logstash; CN=one.test.logstash.elastic.co
*  start date: Oct 12 20:24:27 2021 GMT
*  expire date: Oct 10 20:24:27 2031 GMT
*  subjectAltName: host "127.0.0.1" matched cert's IP address!
*  issuer: C=US; ST=WA; O=Elastic; OU=Logstash; CN=test.logstash.elastic.co
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: 127.0.0.1:9600
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Content-Length: 284
<
* Connection #0 to host 127.0.0.1 left intact
{"host":"limbo-2.lan","version":"8.0.0","http_address":"127.0.0.1:9600","id":"881920b9-fc6b-4369-b113-c8d318be679e","name":"limbo-2.lan","ephemeral_id":"4f651489-1e95-494a-8a75-d2da9c184b4b","status":"green","snapshot":null,"pipeline":{"workers":16,"batch_size":125,"batch_delay":50}}%                                                                                                                                                                                 [success]

Then I configured Logstash to enable SSL and use required for the new api.ssl.client_authentication setting.

When api.ssl.client_authentication is required and I fail to provide a peer, my connection is correctly terminated without a response:

╭─{ yaauie@limbo-2:~/src/elastic/ls/ (✘ secure-api-client-authentication-hangs) }
╰─● curl --verbose --cacert ca.crt.pem https://127.0.0.1:9600
* Rebuilt URL to: https://127.0.0.1:9600/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9600 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: ca.crt.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9600
* stopped the pause stream!
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:9600
[error: 35]

But when api.ssl.client_authentication is required (minissl: force_peer) and I do provide a valid cert (one.crt.pem) and key (one.key.pem), or when api.ssl.client_authentication is optional (minissl: peer), the connection establishes and then hangs without a response causing curl to timeout 30s later:

╭─{ yaauie@limbo-2:~/src/elastic/ls/ (✘ secure-api-client-authentication-hangs) }
╰─● curl --verbose --cacert ca.crt.pem --cert one.crt.pem --key one.key.pem https://127.0.0.1:9600
* Rebuilt URL to: https://127.0.0.1:9600/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9600 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: ca.crt.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=WA; O=Elastic; OU=Logstash; CN=one.test.logstash.elastic.co
*  start date: Oct 12 20:24:27 2021 GMT
*  expire date: Oct 10 20:24:27 2031 GMT
*  subjectAltName: host "127.0.0.1" matched cert's IP address!
*  issuer: C=US; ST=WA; O=Elastic; OU=Logstash; CN=test.logstash.elastic.co
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: 127.0.0.1:9600
> User-Agent: curl/7.54.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server
[error: 52 (30.000s)]