Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create documentation guide on how to deal with coverage errors on PRs porting legacy tests #1153

Open
marioevz opened this issue Jan 29, 2025 · 0 comments

Comments

@marioevz
Copy link
Member

We should add a small page to the documentation on how to port legacy tests and also how to deal with expected coverage failures from the CI runs.

Expected coverage decrements usually come from the distinct approaches to test case writing that we have in eest versus the tests written in ethereum/tests:

EEST favors dynamic contract creation for each different test vector, whereas ethereum/tests favors creating a single static contract with jumps to different test vectors depending on the transaction input data.

For this reason, there will be normally some lost coverage on JUMP opcodes and also jumpdest analysis because we don't rely on those features (unless of course that is the purpose of the test).

We could link to an example failing action in the documentation such as: https://github.com/ethereum/execution-spec-tests/actions/runs/13037332959/job/36370897481

And then explain how to navigate the resulting coverage artifacts, visualize them and also how to bring the obtained information into the PR discussion in order to justify merging the PR with the failing coverage CI.

Also highlight that there could be cases where the coverage contains legitimate drops in coverage along with the expected drops, so we should be aware of this and pay attention.

@marioevz marioevz changed the title Create documentation guide on how to deal with coverage errors on legacy tests porting PRs Create documentation guide on how to deal with coverage errors on PRs porting legacy tests Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant