Skip to content

Commit

Permalink
fixed MsgOptOut to not have a delay and added different way to opt in
Browse files Browse the repository at this point in the history
  • Loading branch information
@insumity
insumity committed Jan 23, 2024
1 parent 4a418c8 commit 3bdb04b
Showing 1 changed file with 63 additions and 53 deletions.
116 changes: 63 additions & 53 deletions docs/docs/adrs/adr-015-partial-set-security.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Proposed
## Context
Currently, in _Replicated Security_, the **whole** validator set of the Cosmos Hub _provider chain_ is used to secure consumer chains. There are at least three concerns with this approach. First, a big chunk of validators might be forced to validate consumer chains they are not interested in securing. Second, it is costly for small validators to secure additional chains. This concern is only partially addressed through [soft opt-out](https://github.com/cosmos/interchain-security/blob/main/docs/docs/adrs/adr-009-soft-opt-out.md) where the bottom (based on voting power) 5% validator can opt out from validating consumer chains. Third and for the above reasons, it is challenging for a new consumer chain to join Replicated Security.

As a solution, we present _Partial Set Security_ (PSS). As the name suggests, PSS allows for only a subset of the validator set of Cosmos to secure a consumer chain. In what follows we propose the exact steps we need to take to implement PSS. This is a first iteration of Partial Set Security, and therefore we present the most minimal solution that achieves PSS possible.
As a solution, we present _Partial Set Security_ (PSS). As the name suggests, PSS allows for only a subset of the validator set of the Cosmos Hub to secure a consumer chain. In what follows we propose the exact steps we need to take to implement PSS. This is a first iteration of Partial Set Security, and therefore we present the most minimal solution that achieves PSS possible.

## Decision
In Replicated Security, 95% (see soft opt-out) of the top Cosmos Hub validators have to secure a consumer chain. In Partial Set Security (PSS) we introduce a parameter `N` that is associated with a consumer chain and require that the top `N%` of the Cosmos Hub validators have to secure this consumer chain. Additionally, we allow validators that do not belong to the top `N%` of the Cosmos Hub validators to dynamically opt in if they want to validate on the consumer chain or not.
Expand All @@ -30,23 +30,27 @@ Consumer chains join Partial Set Security the same way chains now join Replicate
`bool is_top_N`: flag that captures whether the consumer chain joins under the Top N case.
`int8 top_N`: value that captures the percentage of the Top N% validators that should validate this consumer chain. This is only set if `is_top_N` is `true`.

Note that `top_N` can be between `[34, 95]`. `top_N` cannot be less than 34 (~⅓) because in the case of `Top N` we want to give the ability to the top 33% validators to veto a proposal so that they are not forced by the remaining validators to secure a consumer chain they do not want to secure. `top_N` can be up to 95% to capture the current case Replicated Security where we allow the bottom 5% of validators to opt out. There is no reason for `top_N` to be higher than 95%. Smaller chains that belong in the bottom 5% validators can choose to opt in if they want to validate.
Note that `top_N` resides between `[50, 95]`.
Assuming that at most `1/3` of validators could be malicious, by having 50% as the minimum value for `top_N` we guarantee that we cannot have a successful incorrect-execution attack on a Top N consumer chain. This is because, a Top N consumer chain with `N >= 50%` would have at least `1/3` correct validators that would be able to censor any incorrect-execution attack.
Additionally, by having a `N >= 50%` and hence `N > 33%` we provide the ability for validators to `Veto` a `ConsumerAdditionProposal` if they do not desire to validate a consumer chain.

`top_N` can be up to 95% to capture the current case of Replicated Security where we allow the bottom 5% of validators to opt out. There is no reason for `top_N` to be higher than 95%. Smaller chains that belong in the bottom 5% validators can choose to opt in if they want to validate.

So, the fields can take the following values:
| `is_top_N` | `top_N`|
| :--- | :---|
| `true` | `[34, 95]` |
| `true` | `[50, 95]` |
| `false` | not set |


If a proposal has those arguments wrongly set, it should get rejected in [ValidateBasic](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/provider/types/proposal.go#L86).

Note that we could easily distinguish whether a chain is _Top N_ or _Opt In_ by only having the single field `top_N` and if `top_N == 0`, then we’re in the _Opt In_ case, although for clarity, we introduce both fields.

In a future version of PSS, we intend to introduce a way to modify the parameters of a consumer chain, e.g, a chain that is _Opt In_ to become _Top N—, etc.
In a future version of PSS, we intend to introduce a way to modify the parameters of a consumer chain, e.g, a chain that is _Opt In_ to become _Top N_, etc.

#### State & Query
We have to augment the provider module’s state to keep track of the `is_top_N` and `top_N` values for each consumer chain. They key to store this information would be:
We augment the provider module’s state to keep track of the `is_top_N` and `top_N` values for each consumer chain. They key to store this information would be:

```
ChainIdWithLenKey(IsTopNBytePrefix, chainID)
Expand All @@ -68,10 +72,10 @@ In a future version of PSS, we intend to extend the `gaiad query provider list-c
A validator can opt in by sending a new type of message that we introduce in [tx.proto](https://github.com/cosmos/interchain-security/blob/v3.3.0/proto/interchain_security/ccv/provider/v1/tx.proto#L1).
```
message MsgOptIn {
// the chain id of the consumer chain to opt in to
string chain_id = 1;
// the provider address of the validator on Cosmos Hub
string provider_addr = 2;
// the chain id of the consumer chain to opt in to
string chain_id = 1;
// the provider address of the validator on Cosmos Hub
string provider_addr = 2;
}
```
Note that in a Top N consumer chain, the top N% Cosmos Hub validators are forced by default to validate the consumer chain. Nevertheless, validators in the bottom 100 - N% can opt in to validate as well.
Expand All @@ -84,12 +88,12 @@ Additionally, a validator that opts in can then use `MsgAssignConsumerKey` to ch
#### State & Query
We store the list of validators that have opted in under the key:
```
ChainIdAndConsAddrKey(OptedInBytePrefix, chainID)
ChainIdWithLenKey(OptedInByteKey, chainID)
```

And for simplicity for each validator we store a boolean on whether this validator has opted in on this chain or not.
```
ChainIdAndConsAddrKey(ValidatorOptedInBytePrefix, chainID, addr)
ChainIdAndConsAddrKey(OptedInByteKey, chainID, addr)
```

We augment the state of the provider to keep track for each chain all the validators that have opted in and if a specific validator has opted in.
Expand All @@ -106,52 +110,31 @@ gaiad query provider optedInValidators $chainId
```

#### When do validators opt in?
We could technically opt in validators automatically if they voted Yes during the `ConsumerAdditionProposal`. We do not follow this approach and all validators that opt in should send a `MsgOptIn` message. This is because, opting in validators automatically would obfuscate the semantics of proposals. We can envision cases where a validator votes `Yes` for a proposal but might not want to validate or is not ready to validate the consumer chain. Note that if a validator opts in on a consumer chain, the validator cannot opt out immediately as is described later on. Also, because we provide a `MsgOptIn` message for validators that opt in after a `ConsumerAdditionProposal` has passed, it is reasonable to enforce the same interface for all validators that opt in.
Validators can not only opt in by sending a `MsgOptIn` message, but can also opt in automatically if they voted `Yes`
during the `ConsumerAdditionProposal` that introduced a consumer chain. This simplifies validators operations because
they do not have to send an additional message to opt in.

Because the `Tally` method [deletes the votes](https://github.com/cosmos/cosmos-sdk/blob/v0.47.7/x/gov/keeper/tally.go#L71) after reading them, we cannot check the votes of the validators after the votes have been tallied. To circumvent this, we introduce
a hook for [`AfterProposalVote`](https://github.com/cosmos/cosmos-sdk/blob/v0.47.7/x/gov/keeper/vote.go#L35) and keep track
of all the votes cast by a validator. If a validator votes more than once, we only consider the latest vote.
Finally, we only consider a validator has opted in if it casts a 100% `Yes` vote in case of a [weighted vote](https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-037-gov-split-vote.md).


### How do validators opt out?
Validators that have opted in on a chain can opt out as well. The opt out is not instantaneous and the validator only opts out after a tunable delay of `X` weeks (e.g., 5 weeks). We introduce the delay in order to inform consumer chains about a certain upcoming validator set change in case they want to use this information to take action (e.g., all validators in an opt in chain decide to opt out).
Validators that have opted in on a chain can opt out as well by sending the following message:

A validator has send this message to opt out:
```
message MsgInitiateOptOut {
message MsgOptOut {
// the chain id of the consumer chain to opt out from
string chain_id = 1;
// the provider address of the validator on Cosmos Hub
string provider_addr = 2;
}
```
Note the `Initiate` in the name of the message that points to the fact that opting out is not instantaneous. The above should only be called when a consumer chain is actually running. We want to prevent cases where you opt in and then opt out before the chain has even started running.

Another approach for opting out could be the following. A validator just opts out by stopping validating on the consumer chain. If we notice this, then we could opt out the validator if the validator is not in the Top N case. We do not follow this approach because it would slow consumer chains (opted out validators would still become proposers in consensus rounds) and additionally it would make it harder for the consumer chain to know which validators are currently securing it – Is a validator just down or did it fully stop validating it?

Another benefit of introducing a time delay on when the actual opt out takes place is that we prevent a validator from joining and then rejoining multiple times in short bursts. This way we simplify the reward distribution because we know that when we distribute the rewards on the provider chain that a validator that is currently there has actually contributed for some time and deserves rewards.

Finally, a validator could technically opt out automatically by having all the tokens undelegated or redelegated. This is not an issue per se because: i) the validator remains opted in if it joins later on (during the time delay it takes for the opt out to take place), and ii) unbonding tokens need 21 days to unbond and be reused and similarly the number of redelegations is bounded so an adversary with multiple tokens cannot be joining and leaving Opt In chains ad infinitum.
Validators can only opt out after a consumer chain has started and hence the above message returns an error if the chain with `chain_id` is not running.

#### State & Query
We store information for each the list of validators that have initiated opt out and update the list accordingly at the beginning of each block.
```
ChainIdAndConsAddrKey(OptedOutBytePrefix, chainID)
```

And for simplicity for each validator we store a boolean on whether this validator has opted out on this chain or not. This also includes information on the exact date the opt out was initiated and when it is going to complete.

```
ChainIdAndConsAddrKey(ValidatorOptedOutBytePrefix, chainID, addr.ToSdkConsAddr())
```

We augment the state of the provider to keep track for each chain all the validators that have opted in.
```
// returns all the validators that have opted out on chain `chainID`
func (k Keeper) GetOptedOutValidators(ctx sdk.Context, chainID string) []validators
```

We also update the state of the opted-in validators when a validator has opted out. We do this check in every `BeginBlock`.

We introduce a query to retrieve the validators that are about to be opted out. The query would operate as follows.
```
gaiad query provider optedOutValidators $chainId
```
We also update the state of the opted-in validators when a validator has opted out.

### When does a consumer chain start?
A Top N consumer chain starts at the specified date (`spawn_time`) if the [`ConsumerAdditionProposal`](https://github.com/cosmos/interchain-security/blob/v3.3.0/proto/interchain_security/ccv/provider/v1/provider.proto#L27) has passed. An Opt In consumer chain starts if at least one validator has opted in. We check this in [BeginBlockInit](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/provider/keeper/proposal.go#L372):
Expand Down Expand Up @@ -226,18 +209,44 @@ func (k Keeper) QueueVSCPackets(ctx sdk.Context) {
Note that we use `0` to remove validators that are not opted in as described [here](https://docs.cometbft.com/v0.37/spec/abci/abci++_methods#endblock) and as is done for the [assignment of consumer keys](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/provider/keeper/key_assignment.go#L525).

### How do we distribute rewards?
Currently, rewards are distributed as follows: The consumer [periodically sends rewards](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/consumer/keeper/distribution.go#L148) on the provider `ConsumerRewardsPool` address. The provider then [transfers those rewards to the fee collector address](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/provider/keeper/distribution.go#L77) and those transferred rewards are distributed to validators and delegators.
Currently, rewards are distributed as follows: The consumer [periodically sends rewards](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/consumer/keeper/distribution.go#L148) on the provider `ConsumerRewardsPool` address.
The provider then [transfers those rewards to the fee collector address](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/provider/keeper/distribution.go#L77) and those transferred rewards are distributed to validators and delegators.

In PSS, we distribute rewards only to validators that actually validated the consumer chain. To do this we generate a slice with bogus `VoteInfo` that contains the opted in (or Top N) validators and call [AllocateTokens](https://github.com/cosmos/cosmos-sdk/blob/v0.47.7/x/distribution/keeper/allocation.go#L14).
In PSS, we distribute rewards only to validators that actually validate the consumer chain. To do this, we have a pool associated with each consumer chain and consumers IBC transfer the rewards to this pool.
Then, because we do not use the canonical fee pool and hence rewards are not automatically distributed, we will create a new and modified version of [AllocateTokens](https://github.com/cosmos/cosmos-sdk/blob/v0.47.7/x/distribution/keeper/allocation.go#L14) as follows:

```
totalPower := 0
var []abci.VoteInfo votes
for _, val := range validators {
votes = append(votes, []abci.VoteInfo{{Validator: val, SignedLastBlock: true}})
func (k Keeper) AllocateTokensFromPool(ctx sdk.Context, poolAddress string, totalPreviousPower int64, bondedVotes []abci.VoteInfo) {
rewardsCollectedInt := k.bankKeeper.GetAllBalances(ctx, poolAddress)
// rest in the same that `AllocateTokens` operates
...
```

In [TransferRewardsToFeeCollector](https://github.com/cosmos/interchain-security/blob/v3.3.0/x/ccv/provider/keeper/distribution.go#L61) we then do:

```
for _, chain := range consumerChains {
...
totalPower := 0
var []abci.VoteInfo votes
for _, val := range (validators that opted in or Top N) {
votes = append(votes, []abci.VoteInfo{{Validator: val, SignedLastBlock: true}})
}
k.AllocateTokens(ctx, pool associated with this chain, totalPower, votes)
...
}
```

k.AllocateTokens(ctx, totalPower, votes)
instead of calling:
```
// 4. Transfer the balance to the fee collector address
err := k.bankKeeper.SendCoinsFromModuleToModule(
ctx,
types.ConsumerRewardsPool,
k.feeCollectorName,
sdk.NewCoins(balance),
)
```

### Misbehaviour
Expand All @@ -246,7 +255,8 @@ k.AllocateTokens(ctx, totalPower, votes)
For fraud votes, we add a new type `FraudVoteProposal` similar to [EquivocationProposal](https://github.com/cosmos/interchain-security/pull/703). The time to detect incorrect execution evidence is 7 days (the same time as that of detecting a light client attack on a client with a `trustingPeriod` of 14 days); We have 7 days to detect an incorrect execution and then 14 days for the proposal to be accepted.

##### When are fraud votes eligible?
Because fraud votes allow the slashing of a validator through a governance proposal, we are cautious on when a fraud vote is eligible and try to restrict their eligibility as much as possible to avoid an adversary proposing a bogus fraud vote. For this, we only consider validators that have opted in in an Opt In chain. Fraud votes are also **only** intended to be used for **malicious** behaviour, for example, in a scenario similar to what [happened to Neutron](https://blog.neutron.org/neutron-halt-post-mortem-927dbe4540c8), validators should vote against slashing those validators.
Because fraud votes allow the slashing of a validator through a governance proposal, we are cautious on when a fraud vote is eligible and try to restrict their eligibility as much as possible to avoid an adversary proposing a bogus fraud vote.
As described earlier, because in Top N chains we consider that `N > 50%` we only consider validators that have opted in an Opt In chain. Fraud votes are also **only** intended to be used for **malicious** behaviour, for example, in a scenario similar to what [happened to Neutron](https://blog.neutron.org/neutron-halt-post-mortem-927dbe4540c8), validators should vote against slashing those validators.

#### Message
```
Expand Down

0 comments on commit 3bdb04b

Please sign in to comment.