Deploying Timestamp authority in Airgap mode

[Mukuls77]

Question
We want to deploy Sigstore Timestamp authority in Airgap mode in which it will have no access to internet and external could providers. The query is that from available documentation Sigstore TSA can be deployed in production mode with cloud KMS as Key store. In that list we could see hashi corp vault is also an option provided. so can we deploy a local instance of hashi corp vault and integrate it with Sigstore TSA.
can you pls provide some more details on how we can integrate Sigstore TSA with locally deployed hashicorp vault.

[haydentherapper]

I am not familiar with Vault deployments, but if the vault and key is accessible locally, you can specify the key starting with hashivault://. You'll also need to disable ntp monitoring.

[Mukuls77]

I created Cert chain (Root->intermediate->leaf) using opnessl.
I used the file based signer of the TSA and used the leaf key and the cert chain created as input for TSA. It worked. I generated the tsq, tsr and verified it and it worked for file based signer.

Now i imported the Leaf key crated before to hashicorp.

openssl pkcs8 -topk8 -inform PEM -outform DER -in leafCA.key -out leafCA.pkcs8 -nocrypt

base64 -i leafCA.pkcs8 >leafCA.base64

vault transit import transit/keys/tsaLeafkey @leafCA.base64 type=rsa-2048

I now configured the TSA to use the KMS as signer.

./timestamp-server serve --port=3000 --timestamp-signer=kms --kms-key-resource=hashivault://tsaLeafkey --certificate-chain-path=./TSAcertchain.crt --disable-ntp-monitoring=true
2023-07-06T12:22:40.712+0530 INFO app/serve.go:52 starting timestamp-server @ {
"gitVersion": "v1.1.1-29-g76c4368",
"gitCommit": "76c436891c620782c5f8cb8169a39e2bab58ab20",
"gitTreeState": "clean",
"buildDate": "2023-06-07T12:07:29Z",
"goVersion": "go1.20.4",
"compiler": "gc",
"platform": "linux/amd64"
}
2023-07-06T12:22:40.712+0530 DEBUG app/serve.go:68 pprof enabled: false
2023-07-06T12:22:40.712+0530 INFO app/serve.go:83 ntp monitoring disabled
2023-07-06T12:22:40.718+0530 INFO restapi/server.go:228 Serving timestamp server at http://127.0.0.1:3000

i now followed the steps to create a TSQ, generate the TSR and verify it.

====
/home/mukul/tsa>curl http://localhost:3000/api/v1/timestamp/certchain > ts_chain.pem
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4287 0 4287 0 0 4186k 0 --:--:-- --:--:-- --:--:-- 4186k
/home/mukul/tsa>csplit -s -f tmpcert- ts_chain.pem '/-----BEGIN CERTIFICATE-----/' '{}'
/home/mukul/tsa>rm tmpcert-00
/home/mukul/tsa>mv $(ls tmpcert- | tail -1) root.crt.pem
/home/mukul/tsa>cat tmpcert-* > chain.crts.pem
/home/mukul/tsa>rm tmpcert-*
/home/mukul/tsa>echo "myblob" > myblob
/home/mukul/tsa>openssl ts -query -data myblob -cert -sha256 -out request.tsq
Using configuration from /usr/lib/ssl/openssl.cnf
/home/mukul/tsa>curl -sSH "Content-Type: application/timestamp-query" --data-binary @request.tsq http://localhost:3000/api/v1/timestamp -o response.tsr
/home/mukul/tsa>openssl ts -verify -in response.tsr -data "myblob" -CAfile root.crt.pem -untrusted chain.crts.pem
Using configuration from /usr/lib/ssl/openssl.cnf
Verification: FAILED
140489272837440:error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:66:
140489272837440:error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:587:
140489272837440:error:21071069:PKCS7 routines:PKCS7_signatureVerify:signature failure:../crypto/pkcs7/pk7_doit.c:1041:
140489272837440:error:2F06A06D:time stamp routines:TS_RESP_verify_signature:signature failure:../crypto/ts/ts_rsp_verify.c:143:

====
So here the Verification of signature is failing if i import the generated key in the Vault and use that as the source for Key. can you pls guide what procedure we need to use to store the key in the hashicorp vault case.

[haydentherapper]

Everything you’ve done seems correct. Are you able to inspect the tsq and tsr with OpenSSL?

[Mukuls77]

I was able to print the tsq and tsr in human readable from using openssl
/home/mukul/tsa/kmssigner>openssl ts -reply -in response.tsr -text
Using configuration from /usr/lib/ssl/openssl.cnf
Status info:
Status: Granted.
Status description: unspecified
Failure info: unspecified

TST info:
Version: 1
Policy OID: 1.3.6.1.4.1.57264.2
Hash Algorithm: sha256
Message data:
0000 - e4 9a 89 7a ac 4c 98 19-21 27 0a 3f 79 e4 68 85 ...z.L..!'.?y.h.
0010 - c9 95 2f c5 ba 1a b4 c0-cd ef 38 0b e7 73 74 81 ../.......8..st.
Serial number: 0x7A078BC2BA5CFB1870683CD85CFC60DCEAA99A1E
Time stamp: Jul 7 04:36:27 2023
Accuracy: 0x01 seconds, unspecified millis, unspecified micros
Ordering: no
Nonce: 0x818BAD99F817CEA0
TSA: DirName:/C=IN/ST=DEL/L=DEL/O=NOKIA/OU=IT/CN=TSA/emailAddress=[email protected]
Extensions:
/home/mukul/tsa/kmssigner>openssl ts -query -in request.tsq -text
Using configuration from /usr/lib/ssl/openssl.cnf
Version: 1
Hash Algorithm: sha256
Message data:
0000 - e4 9a 89 7a ac 4c 98 19-21 27 0a 3f 79 e4 68 85 ...z.L..!'.?y.h.
0010 - c9 95 2f c5 ba 1a b4 c0-cd ef 38 0b e7 73 74 81 ../.......8..st.
Policy OID: unspecified
Nonce: 0x818BAD99F817CEA0
Certificate required: yes
Extensions:
/home/mukul/tsa/kmssigner>./timestamp-cli inspect --timestamp response.tsr --format json
crypto/rsa: verification error

but when i tried to use the timestamp-cli to inspect the tsr it failed with validation error. i am attaching the logs and details for working filesigner case and not working kms signer case

TSAlogs.tar.gz

[Mukuls77]

@haydentherapper did you find any error in the procedure i used for KMS based signer

[haydentherapper]

What's the hashing algorithm used with the key? One guess is if it's not sha256, then it'll fail.

Everything you've specified seems correct, so you might need to step through verification and see at what point it's failing.

[Mukuls77]

The hashing algo used in sha256 only. As per the logs the verification is failing in the crypto library functions.
Afile root.crt.pem -untrusted chain.crts.pem
Using configuration from /usr/lib/ssl/openssl.cnf
Verification: FAILED
140343681848640:error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:66:
140343681848640:error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:587:
140343681848640:error:21071069:PKCS7 routines:PKCS7_signatureVerify:signature failure:../crypto/pkcs7/pk7_doit.c:1041:
140343681848640:error:2F06A06D:time stamp routines:TS_RESP_verify_signature:signature failure:../crypto/ts/ts_rsp_verify.c:143:

i am not sure but what is see is that if we use the hashivault:// option the TSA try to get the key from the transit secret engine of the Hashicorp.
now to store the key in transit engine we need to convert it first to DER encoding.
openssl pkcs8 -topk8 -inform PEM -outform DER -in leafCA.key -out kmsTSAkey.pkcs8 -nocrypt
than so the base 64 encoding
base64 -i kmsTSAkey.pkcs8 >kmsTSAkey.base64
and than import the cert in the transit engine
vault transit import transit/keys/kmsTSAkey @kmsTSAkey.base64 type=rsa-2048

now in the cert chain the public key is stored in the PEM format only.
so not sure if due to this mismatch we are facing this issue.
As if i use the same private key as PEM format (using the filebased signer) every thing works fine.
I hope TSA should support using the KV secret engine of hashicorp in which we can directly store the PEM key mapping to a key and give the path of this key as the source of the key if we allow this i think this problem will get solved.

[ahmedwarsama]

We are also looking for a way to deploy this locally with Vault. Looking forward to seeing if you can solve this @Mukuls77.

[berkitamas]

Hi,

I am running into the same issue. Looks like the signature algorithm is wrong. In https://github.com/sigstore/sigstore/blob/11bb41a30564029ec477dda566bba110d3a7b14b/pkg/signature/kms/hashivault/client.go#L278-L282 there is no signature algorithm specified, so Vault will use the default one, which is pss.
By modifying it like this:

	signResult, err := client.Write(fmt.Sprintf("/%s/sign/%s%s", h.transitSecretEnginePath, h.keyPath, hashString(alg)), map[string]interface{}{
		"input":       base64.StdEncoding.Strict().EncodeToString(digest),
		"prehashed":   alg != crypto.Hash(0),
		"key_version": keyVersion,
+               "signature_algorithm": "pkcs1v15",
	})

I could make it work. However, with lacking the full understanding the code (especially since it is used by other packages too), I am not sure if this would break other things.

Maybe somebody who is more fluent with it can help resolving this.

EDIT: I observed similar behaviour with Fulcio and Rekor (Hopefully this problem will be resolved with sigstore/sigstore#1735 ).