EPIC: Improve throttling mechanism with retries

[shaspitz]

Problem

Currently the throttling mechanism is designed so that provider logic (slash meter, etc.) dictates how many slash packets can be handled over time. Throttled slash packets are persisted on the provider, leading to issues such as #594. We can improve the throttling mechanism to instead queue/persist relevant data on each consumer, and have consumers retry slash requests as needed. CC @jtremback who came up with this idea

Closing criteria

Relevant data is persisted on each consumer with retry system implemented to enforce throttling.

Problem details

• Throttling logic will stay on provider
• Queuing/data persisting will now be on consumer
• Consumer will retry slash requests when they cannot be immediately handled by the provider. IBC acknowledgements are used to implement such an idea
• VSC matured packet delivery will likely need to be retried as well, as these packets are throttled in some cases

Task list

Must have

Preview Give feedback

1. ADR for throttling with retries #1023
   scope: docs +1
   
2. Improve storage of PendingPackets on consumer #1033
   +0
   
3. Consumer changes for throttling v2 #1101
   +0
   
4. Revert changes from fix: limit vsc matured packets handled per endblocker #1004 as they now should be implicitly handled
5. Error-prone storage of throttled packet data #609
   S: ImprovingThings type: feature-request type: refactoring +3
6. Slash packet throttling needs to consider exporting/rebuilding genesis state #564
   S: NewThings type: bug type: feature-request type: tech-debt +4
   
7. Provider changes for throttle v2 #1102
   S: NewThings +1
   
8. Replace provider throttle state query with a consumer query #1233
   S: ImprovingThings +1
   
9. E2e tests for throttle v2 #1103
   S: NewThings +1
   
10. Throttling v2 provider migration #1032
    S: NewThings question +2
    
11. PR for final feature branch

Nice to have

Preview Give feedback

1. Potentially delay consumer removal until slash packet queue is empty #566
   S: NewThings question spec +3
2. Slashing/jailing naming refactors are needed #706
   S: Productivity type: refactoring type: tech-debt +3

[shaspitz]

Note: Implementing this idea would close #594. We also need to consider the importance of this issue in relation to more general "untrusted consumer" features

[mpoke]

This could considerably increase the cost of relaying.