[META] gnokms - a TM2 Key Management System

[zivkovicmilos]

Overview

This issue outlines the proposal for gnokms, a TM2-based key management system, used primarily for secure key management of gno.land (TM2) nodes.

The proposal outlines different services and suggestions for potential implementation, and should be utilized as a master issue.

The implementation of gnokms should live directly in the gnolang/gno monorepo, under contribs. The idea is to encapsulate the functionality mentioned below in a single tool that is able to be orchestrated easily.

I'm calling the Gno DevOps family to give thoughts and share opinions.
@sw360cab @mazzy89 @r3v4s @D4ryl00 @albttx @n0izn0iz @moul @n2p5 @aeddi

What is a KMS?

A Key Management System (KMS) is a specialized tool or system used to securely manage and control cryptographic keys, such as those used by blockchain validator nodes (for signing consensus messages, or p2p communication encryption).

The primary purpose of a KMS is to securely store sensitive key-value pairs, such as validator-related secrets:

• the private validator key (required)
• the node p2p key (less important)

Tight Access Control

Access to the KMS (and by relation secrets) is limited by high-level access lists and secured by the platform. This means that the scope of entities that have access to the validator machine cannot interfere or make unauthorized copies of the validator secrets, just by having read access to the machine.

Rapid Node Deployments

Integrating a KMS into a blockchain system enables scalable and rapid node deployments. For example, a failover cluster can easily be instantiated when the primary validator node failure is detected — all without the need to copy or transfer keys beforehand. Validator secrets no longer need to be duplicated over machines in the failover cluster, but are stored securely in a central (or distributed!) place, completely decoupled from the TM2 blockchain client.

What we have now

Tendermint2 already has a secrets management system in place, but it’s not very robust.

A gno.land node keeps track of 3 secret types:

• the private validator key - used for signing validator consensus messages. There is a heavy loss if this key is compromised in any way.
• the node p2p key - used for encrypting p2p communication. This secret can be regenerated and replaced in case of being compromised in any way, and the risk for losing it is low.
• the private validator state - a TM consensus optimization for keeping track of the latest consensus signed state, to avoid double signing consensus messages. The risk for it being compromised is medium, because it can incur a double sign penalty.

The gno.land node utilizes the gnoland secrets command suite to generate and manage these 3 secrets.

The biggest issue with the current approach is that the secrets are stored locally on disk, in plaintext, for example:

// priv_validator_key.json
{
	"address": "g1ga09luuxhpwlkykvvtrj7qpjs7ndr5e9ge3df3",
	"pub_key": {
		"@type": "/tm.PubKeyEd25519",
		"value": "6ut+J16TQrdEqcFzcwz9AJeWF2sCAfCBHyCufJYs2lg="
	},
	"priv_key": {
		"@type": "/tm.PrivKeyEd25519",
		"value": "K0BLKdTe1/AvRCcI3spOQQkgebn6kU0KR41xg6lwapjq634nXpNCt0SpwXNzDP0Al5YXawIB8IEfIK58lizaWA=="
	}
}

// node_key.json
{
	"priv_key": {
		"@type": "/tm.PrivKeyEd25519",
		"value": "+JnhtuerUoCIQY8Z+coxq6RQwtKnWDcynm4CNkhiPY+6R0n2Wm1IYxnshdJmfguqteN7QHwz9vR7MDfQAGhBCw=="
	}
}

Upon the startup of the node, they are loaded into memory, and utilized by different TM2 modules.

Scope of gnokms

gnokms has one goal - provide a secure system for the storage and utilization of node secrets (keys).

It accomplishes this through a mechanism called the Remote Signer.

A remote signer is a secondary process that is separate from the running node. Its purpose is to sign data with a specific key. This means that the (secret) key does not live on the (TM2) node itself, but within the confines of the remote signing service. It increases security because the validator node which is connected to the remote signer can be swapped without missing any important checkpoints, like signing blocks.

gnokey

gnokey is the main CLI keychain used within the gno.land ecosystem.

It has the ability to generate, manage and use secp256k1 private keys.

gnokey stores private key data encrypted on disk, with the encryption passphrase being unique to each individual key.

There should be a Remote signer support added for gnokey, that would allow for remote signing using gnokey on the same machine, or on completely different machines from the validator node process.

• [ops] Add gnokey support as a TM2 remote signer #3231

Cloud-based solutions

There are many cloud-based solutions for storing node secrets currently used in production blockchain systems. The proposal will outline a few that have seen most usage by the community, and provided acceptable latency when it comes to remote key storage.

AWS Secrets Manager

AWS Secrets Manager is a managed AWS service for securely storing, managing, and retrieving secrets. It enables automatic secret rotation, encryption with AWS KMS, and fine-grained access control via IAM.

The Go client we should utilize for interacting with the AWS Secrets Manager service is aws-sdk-go-v2.

It works as a simple key-value storage, with additional options such as region customization, encryption, and overwrite protection.

• [ops] Add AWS Secrets Manager support as a TM2 remote signer #3232

GCP Secret Manager

GCP Secret Manager is a fully managed service by Google Cloud that securely stores, manages, and accesses sensitive information. It offers strong encryption, with secrets encrypted at rest using Google-managed keys or customer-managed keys in Cloud KMS. Secret Manager supports fine-grained access control with IAM and provides audit logs for monitoring access.

The Go client we should utilize for interacting with the GCP Secret Manager service is cloud.google.com/go/secretmanager/apiv1.

Exactly like AWS Secrets Manager, it works as a simple key-value store. The nice feature here is that GCP Secret Manager also support replication configuration.

• [ops] Add GCP Secret Manager support as a TM2 remote signer #3233

Hashicorp Vault

HashiCorp Vault is a powerful, open-source tool designed for securely storing, managing, and controlling access to secrets. It provides a centralized solution for secret management, offering features like dynamic secrets, automated key rotation, and robust access policies using fine-grained role-based access control. Vault encrypts data at rest and in transit, ensuring a high level of security.

The go client we should utilize for interacting with the Hashicorp Vault service is github.com/hashicorp/vault/api.

Hashicorp Vault is probably one of the most popular Cloud-based secrets managers among node operators.

• [ops] Add Hashicorp Vault support as a TM2 remote signer #3234

HSM-based solutions

The Hardware Security Module (HSM)-based solutions are one of the most secure in terms of validator secret management. For this section, only 2 of the most popular ones are outlined and discussed.

All of the HSM-based solutions, as opposed to the cloud-based solutions, can only manage validator private keys, and not other secrets such as node p2p keys.

Ledger

Utilizing a Ledger device for validator key management provides a secure and efficient way to safeguard the private keys essential for signing in TM2 networks. Ledger hardware wallets store private keys in a secure, tamper-resistant environment, isolating them from potential threats. By integrating a Ledger device, validators can sign transactions and participate in consensus without exposing their keys to online systems, significantly reducing the risk of key compromise.

The exact implementation details should be researched, but the standard Cosmos ledger adapter in Go should be utilized: ledger-go.

• [ops] Add Ledger device support as a TM2 remote signer #3235

YubiHSM2

The YubiHSM 2 from Yubico is a relatively low-cost solution for online key storage featuring support for random key generation, encrypted backup and export, and audit logging.

The exact implementation details should be researched. The yubihsm-go module can be utilized. It is worth nothing that this package has not been actively maintained for some time, and could require more flexibility (~2yrs).

• [ops] Add YubiHSM2 device support as a TM2 remote signer #3236

Related proposals

The usage of the Remote Signer mechanism in TM2 is nothing new.

The gno team plans to actively develop, or sponsor development, on gnocrux - a threshold TM2 signer that utilizes TM2’s remote signing mechanism.

[D4ryl00]

You wrote for Gnokey:
There should be a Remote signer support added for gnokey, that would allow for remote signing using gnokey on the same machine, or on completely different machines from the validator node process.

If gnokey implements the remote signer, what would be the difference with gnokms?

[zivkovicmilos]

You wrote for Gnokey: There should be a Remote signer support added for gnokey, that would allow for remote signing using gnokey on the same machine, or on completely different machines from the validator node process.

If gnokey implements the remote signer, what would be the difference with gnokms?

gnokey should be an option in gnokms, but gnokey on its own (as a standalone binary) does many more things than just signing data