feat!: throttle with retries provider changes #1230
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1937,50 +1937,41 @@ func (tr TestRun) assignConsumerPubKey(action assignConsumerPubKeyAction, verbos | |
| tr.waitBlocks(chainID("provi"), 2, 30*time.Second) | ||
| } | ||
|
|
||
| // slashMeterReplenishmentAction polls the slash meter on provider until value is achieved | ||
| type slashMeterReplenishmentAction struct { | ||
| targetValue int64 | ||
| // panic if timeout is exceeded | ||
| timeout time.Duration | ||
| } | ||
|
|
||
| func (tr TestRun) waitForSlashMeterReplenishment( | ||
| action slashMeterReplenishmentAction, | ||
| verbose bool, | ||
| ) { | ||
| timeout := time.Now().Add(action.timeout) | ||
| initialSlashMeter := tr.getSlashMeter() | ||
|
|
||
| if initialSlashMeter >= 0 { | ||
| panic(fmt.Sprintf("No need to wait for slash meter replenishment, current value: %d", initialSlashMeter)) | ||
|
There was a problem hiding this comment. Is this a bad idea to panic on? It makes the tests flakey, because if they happen to run slowly, the slash meter might be already-full, right? There was a problem hiding this comment. Hmm I don't see a good way to decouple the logic of the slash meter away from time. Any test for throttling must have a pretty good and mostly deterministic control over chain time |
||
| } | ||
|
|
||
| for { | ||
| slashMeter := tr.getSlashMeter() | ||
| if verbose { | ||
| fmt.Printf("waiting for slash meter to be replenished, current value: %d\n", slashMeter) | ||
| } | ||
|
|
||
| // check if meter has reached target value | ||
| if slashMeter >= action.targetValue { | ||
| break | ||
| } | ||
|
|
||
| if time.Now().After(timeout) { | ||
| panic(fmt.Sprintf("\n\nwaitForSlashMeterReplenishment has timed out after: %s\n\n", action.timeout)) | ||
| } | ||
|
|
||
| time.Sleep(500 * time.Millisecond) | ||
|
There was a problem hiding this comment. Can we replace this with tr.waitBlocks/tr.waitTime? There was a problem hiding this comment. I think it should be waitTime from my understanding of the slashMeter (it fills by timestamp, not by blockcount?) There was a problem hiding this comment. Nice catch 53c02ea |
||
| } | ||
| } | ||
|
|
||
| // GetPathNameForGorelayer returns the name of the path between two given chains used by Gorelayer. | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -278,7 +278,7 @@ func stepsThrottledDowntime(consumerName string) []Step { | |
| validator: validatorID("bob"), | ||
| }, | ||
| state: State{ | ||
| // slash packet queued for bob on consumer, but powers not affected on either chain yet | ||
| chainID("provi"): ChainState{ | ||
| ValPowers: &map[validatorID]uint{ | ||
| validatorID("alice"): 511, | ||
|
|
@@ -312,11 +312,6 @@ func stepsThrottledDowntime(consumerName string) []Step { | |
| validatorID("bob"): 0, // bob is jailed | ||
| validatorID("carol"): 500, | ||
| }, | ||
| }, | ||
| chainID(consumerName): ChainState{ | ||
| // VSC packet applying jailing is not yet relayed to consumer | ||
|
|
@@ -328,13 +323,13 @@ func stepsThrottledDowntime(consumerName string) []Step { | |
| }, | ||
| }, | ||
| }, | ||
| // Invoke carol downtime slash on consumer | ||
| { | ||
| action: downtimeSlashAction{ | ||
| chain: chainID(consumerName), | ||
| validator: validatorID("carol"), | ||
| }, | ||
| state: State{ | ||
| chainID("provi"): ChainState{ | ||
| ValPowers: &map[validatorID]uint{ | ||
| validatorID("alice"): 511, | ||
|
|
@@ -343,10 +338,9 @@ func stepsThrottledDowntime(consumerName string) []Step { | |
| }, | ||
| }, | ||
| chainID(consumerName): ChainState{ | ||
| ValPowers: &map[validatorID]uint{ | ||
| validatorID("alice"): 511, | ||
| validatorID("bob"): 500, // VSC packet applying bob jailing is not yet relayed to consumer | ||
shaspitz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| validatorID("carol"): 500, | ||
| }, | ||
| }, | ||
|
|
@@ -364,42 +358,35 @@ func stepsThrottledDowntime(consumerName string) []Step { | |
| ValPowers: &map[validatorID]uint{ | ||
| validatorID("alice"): 511, | ||
| validatorID("bob"): 0, | ||
| validatorID("carol"): 500, // slash packet for carol recv by provider, carol not slashed due to throttling | ||
| }, | ||
| }, | ||
| chainID(consumerName): ChainState{ | ||
| ValPowers: &map[validatorID]uint{ | ||
| validatorID("alice"): 511, | ||
| validatorID("bob"): 0, // VSC packet applying bob jailing is also relayed and recv by consumer | ||
| validatorID("carol"): 500, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| // TODO(Shawn): Improve this test to have the consumer retry it's downtime slash, and to assert queue size on consumer. | ||
|
There was a problem hiding this comment. Note this TODO, the e2e test ends without retrying the slash packet. This will be addressed in a separate PR There was a problem hiding this comment. Just to confirm, we are removing the Queue size from the chain state, despite this TODO, right? It reads to me as if that would reintroduce it, but maybe I'm missing context There was a problem hiding this comment. Good question, we are removing the queues from the provider in this PR. The queues exist on the consumers as of #1024, so incoming e2e test will assert queue size on consumer |
||
| // See https://github.com/cosmos/interchain-security/issues/1103 and https://github.com/cosmos/interchain-security/issues/1233 | ||
| { | ||
| action: slashMeterReplenishmentAction{ | ||
| targetValue: 0, // We just want slash meter to be non-negative | ||
|
|
||
| // Slash meter replenish fraction is set to 10%, replenish period is 20 seconds, see config.go | ||
| // Meter is initially at 10%, decremented to -23% from bob being jailed. It'll then take three replenishments | ||
| // for meter to become positive again. 3*20 = 60 seconds + buffer = 100 seconds | ||
| timeout: 100 * time.Second, | ||
|
There was a problem hiding this comment. Just curious, was there a reason the buffer needs to be increased? There was a problem hiding this comment. I think this is an artifact of #1304 and trying to get this to work with cometmock |
||
| }, | ||
| state: State{ | ||
| chainID("provi"): ChainState{ | ||
| ValPowers: &map[validatorID]uint{ | ||
| validatorID("alice"): 511, | ||
| validatorID("bob"): 0, | ||
| validatorID("carol"): 500, // Carol still not slashed, packet must be retried | ||
| }, | ||
| }, | ||
| chainID(consumerName): ChainState{ | ||
|
|
@@ -412,36 +399,5 @@ func stepsThrottledDowntime(consumerName string) []Step { | |
| }, | ||
| }, | ||
| }, | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The slash packet queue size of the provider is no longer relevant, but the slash meter value is relevant. I've replaced this action to wait on slash meter being replenished