[Epic]: Refactor e2e tests to increase coverage and reduce complexity

[mpoke]

Problem

Currently, the e2e tests consists of the following steps (see tests/e2e/README.md for more details):

• start a docker container
• run a defined sequence of actions with expected states
• after each action, query the system state and compare against the expected state

Also, there are a set of predefined test cases (more than 20) that can be executed directly. Run go run ./tests/e2e/... --help for details.

Although at the high level this e2e framework is good, there are several issues that need to be addressed:

• Some of the tests are non-deterministic as they depend on timing assumptions. As a result, tests are sometimes failing and need to be rerun.
• Most of the existing test cases have unnecessary actions resulting in tests with a high number of steps (e.g., the happy-path has 39 steps).
  • This makes the testing time long. As a result, the non-deterministic aspect has an even higher impact, e.g., if step 38 in the happy-path test case is randomly failing, the entire test case needs to be retried.
  • This also makes the tests complex and hard to modify and debug, e.g., there is no reason to start consumer chains or create IBC channels to test the consumer lifecycle on the provider.
• Many of the test cases are not part of the CI or the nightly tests, which means that are rarely run.

Closing criteria

The tasks below are completed.

Task list

Must have

Preview Give feedback

1. Create simpler actions, i.e., one task performed per action (e.g., start only the provider).
2. Create EPIC with test cases that require both the provider and a consumer (e.g., CCV channel handshake)
3. Refactor e2e testing to run provider actions only on provider #2315
   scope: testing +1
   
4. Create EPIC with test cases that require only the consumer (there might be none)
5. Replace long test cases with shorter ones that address a single concern
6. Enable multiple test cases to run in parallel (one failing test case should not affect others)
7. Remove test cases that are checking protocol logic (the scope of e2e testing is making sure the API works)

Questions

Preview Give feedback

1. Is the interchain testing framework better?
2. How can we reduce non-determinism in certain actions?
3. What are the pain points with compatibility tests?

Nice to have

Preview Give feedback

1. De-couple actions from TestConfig #1325
   good first issue scope: testing +2
2. Create a user-friendly interface for debugging e2e tests #1487
   scope: testing +1

Others (to triage)

Preview Give feedback

1. e2e tests: support version specific actions for compatbility tests #2198
   scope: testing +1
   
2. e2e tests with gaia fail #1568
   scope: testing type: bug +2
   
3. Automate the base case e2e infrastructure creation #1135
   scope: testing type: tech-debt +2
   
4. make e2e it's own module (de-couple from ICS lib and versioning scheme) #1132
   scope: testing type: tech-debt +2
5. EPIC: Refactor existing e2e testing framework #877
   7 of 9
   admin: epic scope: testing type: tech-debt +3
6. E2e tests for chain upgrade #1296
   scope: testing +1