From ab46fff2fe69f33fa3e6fd14502492b039bc969b Mon Sep 17 00:00:00 2001 From: Mario Vega Date: Fri, 7 Jun 2024 10:15:27 -0600 Subject: [PATCH 1/2] feat(fw): Add `with_all_contract_creating_tx_types` marker (#602) * fix(evm_transition_tool): Allow formatting to="" for some tool types * fix(fw,types): Make Transaction more permissible to always create a valid tx * feat(forks): Add `contract_creating_tx_types` class method to forks * feat(plugins): create "with_all_contract_creating_tx_types" marker * docs: Add test markers page * fix(fw): remove auto-fill blob_versioned_hashes * chengelog * Update src/evm_transition_tool/evmone.py Co-authored-by: spencer * Update src/evm_transition_tool/transition_tool.py Co-authored-by: spencer * Update src/evm_transition_tool/transition_tool.py Co-authored-by: spencer * docs: lint --------- Co-authored-by: spencer --- docs/CHANGELOG.md | 1 + docs/navigation.md | 1 + docs/writing_tests/test_markers.md | 111 +++++++++++++++++++++ src/ethereum_test_forks/base_fork.py | 8 ++ src/ethereum_test_forks/forks/forks.py | 21 ++++ src/ethereum_test_tools/common/types.py | 32 +++--- src/evm_transition_tool/evmone.py | 8 ++ src/evm_transition_tool/transition_tool.py | 25 ++++- src/pytest_plugins/forks/forks.py | 7 ++ 9 files changed, 196 insertions(+), 18 deletions(-) create mode 100644 docs/writing_tests/test_markers.md diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index fc7eb345a3..0096413587 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -37,6 +37,7 @@ Test fixtures for use by clients are available for each release on the [Github r - ✨ Add more Transaction and Block exceptions from existing ethereum/tests repo ([#572](https://github.com/ethereum/execution-spec-tests/pull/572)). - ✨ Add "description" and "url" fields containing test case documentation and a source code permalink to fixtures during `fill` and use them in `consume`-generated Hive test reports ([#579](https://github.com/ethereum/execution-spec-tests/pull/579)). - ✨ Add git workflow evmone coverage script for any new lines mentioned in converted_ethereum_tests.txt ([#503](https://github.com/ethereum/execution-spec-tests/pull/503)). +- ✨ Add a new covariant marker `with_all_contract_creating_tx_types` that allows automatic parametrization of a test with all contract-creating transaction types at the current executing fork ([#602](https://github.com/ethereum/execution-spec-tests/pull/602)). ### 🔧 EVM Tools diff --git a/docs/navigation.md b/docs/navigation.md index 6b5a3d3e23..05f5a320c3 100644 --- a/docs/navigation.md +++ b/docs/navigation.md @@ -13,6 +13,7 @@ * [Types of Test](writing_tests/types_of_tests.md) * [Adding a New Test](writing_tests/adding_a_new_test.md) * [Writing a New Test](writing_tests/writing_a_new_test.md) + * [Test Markers](writing_tests/test_markers.md) * [Referencing an EIP Spec Version](writing_tests/reference_specification.md) * [Verifying Changes Locally](writing_tests/verifying_changes.md) * [Exception Tests](writing_tests/exception_tests.md) diff --git a/docs/writing_tests/test_markers.md b/docs/writing_tests/test_markers.md new file mode 100644 index 0000000000..ad2a0fb884 --- /dev/null +++ b/docs/writing_tests/test_markers.md @@ -0,0 +1,111 @@ +# Test Markers + +Test markers are used to categorize tests and to run specific subsets of tests. They are defined in the test files using the `pytest.mark` decorator. + +## Fork Markers + +These markers are used to specify the forks for which a test is valid. + +### pytest.mark.valid_from("FORK_NAME") + +This marker is used to specify the fork from which the test is valid. The test will not be filled for forks before the specified fork. + +```python +import pytest + +@pytest.mark.valid_from("London") +def test_something_only_valid_after_london(): + pass +``` + +In this example, the test will only be filled for the London fork and after, e.g. London, Paris, Shanghai, Cancun, etc. + +### pytest.mark.valid_until("FORK_NAME") + +This marker is used to specify the fork until which the test is valid. The test will not be filled for forks after the specified fork. + +```python +import pytest + +@pytest.mark.valid_until("London") +def test_something_only_valid_until_london(): + pass +``` + +In this example, the test will only be filled for the London fork and before, e.g. London, Berlin, Istanbul, etc. + +### pytest.mark.valid_at_transition_to("FORK_NAME") + +This marker is used to specify that a test is only meant to be filled at the transition to the specified fork. + +The test usually starts at the fork prior to the specified fork at genesis and at block 5 (for pre-merge forks) or at timestamp 15,000 (for post-merge forks) the fork transition occurs. + +## Fork Covariant Markers + +These markers are used in conjunction with the fork markers to automatically parameterize tests with values that are valid for the fork being tested. + +### pytest.mark.with_all_tx_types + +This marker is used to automatically parameterize a test with all transaction types that are valid for the fork being tested. + +```python +import pytest + +@pytest.mark.with_all_tx_types +@pytest.mark.valid_from("Berlin") +def test_something_with_all_tx_types(tx_type): + pass +``` + +In this example, the test will be parameterized for parameter `tx_type` with values `[0, 1]` for fork Berlin, but with values `[0, 1, 2]` for fork London (because of EIP-1559). + +### pytest.mark.with_all_contract_creating_tx_types + +This marker is used to automatically parameterize a test with all contract creating transaction types that are valid for the fork being tested. + +This marker only differs from `pytest.mark.with_all_tx_types` in that it does not include transaction type 3 (Blob Transaction type) on fork Cancun and after. + +### pytest.mark.with_all_precompiles + +This marker is used to automatically parameterize a test with all precompiles that are valid for the fork being tested. + +```python +import pytest + +@pytest.mark.with_all_precompiles +@pytest.mark.valid_from("Shanghai") +def test_something_with_all_precompiles(precompile): + pass +``` + +In this example, the test will be parameterized for parameter `precompile` with values `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]` for fork Shanghai, but with values `[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]` for fork Cancun (because of EIP-4844). + +## Other Markers + +### pytest.mark.slow + +This marker is used to mark tests that are slow to run. These tests are not run during tox testing, and are only run when a release is being prepared. + +### pytest.mark.skip("reason") + +This marker is used to skip a test with a reason. + +```python +import pytest + +@pytest.mark.skip("Not implemented") +def test_something(): + pass +``` + +### pytest.mark.xfail("reason") + +This marker is used to mark a test as expected to fail. + +```python +import pytest + +@pytest.mark.xfail("EVM binary doesn't support this opcode") +def test_something(): + pass +``` diff --git a/src/ethereum_test_forks/base_fork.py b/src/ethereum_test_forks/base_fork.py index 4126498e28..2e845caa4a 100644 --- a/src/ethereum_test_forks/base_fork.py +++ b/src/ethereum_test_forks/base_fork.py @@ -181,6 +181,14 @@ def tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: """ pass + @classmethod + @abstractmethod + def contract_creating_tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: + """ + Returns a list of the transaction types supported by the fork that can create contracts + """ + pass + @classmethod @abstractmethod def precompiles(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: diff --git a/src/ethereum_test_forks/forks/forks.py b/src/ethereum_test_forks/forks/forks.py index 4338e6e9d0..19a616eda8 100644 --- a/src/ethereum_test_forks/forks/forks.py +++ b/src/ethereum_test_forks/forks/forks.py @@ -156,6 +156,13 @@ def tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: """ return [0] + @classmethod + def contract_creating_tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: + """ + At Genesis, only legacy transactions are allowed + """ + return [0] + @classmethod def precompiles(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: """ @@ -274,6 +281,13 @@ def tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: """ return [1] + super(Berlin, cls).tx_types(block_number, timestamp) + @classmethod + def contract_creating_tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: + """ + At Berlin, access list transactions are introduced + """ + return [1] + super(Berlin, cls).contract_creating_tx_types(block_number, timestamp) + class London(Berlin): """ @@ -294,6 +308,13 @@ def tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: """ return [2] + super(London, cls).tx_types(block_number, timestamp) + @classmethod + def contract_creating_tx_types(cls, block_number: int = 0, timestamp: int = 0) -> List[int]: + """ + At London, dynamic fee transactions are introduced + """ + return [2] + super(London, cls).contract_creating_tx_types(block_number, timestamp) + # Glacier forks skipped, unless explicitly specified class ArrowGlacier(London, solc_name="london", ignore=True): diff --git a/src/ethereum_test_tools/common/types.py b/src/ethereum_test_tools/common/types.py index 293130ee69..a2e6789722 100644 --- a/src/ethereum_test_tools/common/types.py +++ b/src/ethereum_test_tools/common/types.py @@ -937,39 +937,37 @@ def model_post_init(self, __context): or self.max_fee_per_blob_gas is not None ): raise Transaction.InvalidFeePayment() - - if ( - self.gas_price is None - and self.max_fee_per_gas is None - and self.max_priority_fee_per_gas is None - and self.max_fee_per_blob_gas is None - ): - self.gas_price = 10 - - if self.v is not None and self.secret_key is not None: - raise Transaction.InvalidSignaturePrivateKey() - - if self.v is None and self.secret_key is None: - self.secret_key = Hash(TestPrivateKey) - if "ty" not in self.model_fields_set: # Try to deduce transaction type from included fields - if self.max_fee_per_blob_gas is not None: + if self.max_fee_per_blob_gas is not None or self.blob_kzg_commitments is not None: self.ty = 3 - elif self.max_fee_per_gas is not None: + elif self.max_fee_per_gas is not None or self.max_priority_fee_per_gas is not None: self.ty = 2 elif self.access_list is not None: self.ty = 1 else: self.ty = 0 + if self.v is not None and self.secret_key is not None: + raise Transaction.InvalidSignaturePrivateKey() + + if self.v is None and self.secret_key is None: + self.secret_key = Hash(TestPrivateKey) + # Set default values for fields that are required for certain tx types + if self.ty <= 1 and self.gas_price is None: + self.gas_price = 10 if self.ty >= 1 and self.access_list is None: self.access_list = [] + if self.ty >= 2 and self.max_fee_per_gas is None: + self.max_fee_per_gas = 7 if self.ty >= 2 and self.max_priority_fee_per_gas is None: self.max_priority_fee_per_gas = 0 + if self.ty == 3 and self.max_fee_per_blob_gas is None: + self.max_fee_per_blob_gas = 1 + def with_error( self, error: List[TransactionException] | TransactionException ) -> "Transaction": diff --git a/src/evm_transition_tool/evmone.py b/src/evm_transition_tool/evmone.py index 568c2d48cc..535e5c7e64 100644 --- a/src/evm_transition_tool/evmone.py +++ b/src/evm_transition_tool/evmone.py @@ -37,3 +37,11 @@ def is_fork_supported(self, fork: Fork) -> bool: Currently, evmone-t8n provides no way to determine supported forks. """ return True + + @classmethod + def empty_string_to(cls) -> bool: + """ + Evmone requires an empty string within the `to` field for + contract-creating transactions. + """ + return True diff --git a/src/evm_transition_tool/transition_tool.py b/src/evm_transition_tool/transition_tool.py index 9b718106f8..db611bca2a 100644 --- a/src/evm_transition_tool/transition_tool.py +++ b/src/evm_transition_tool/transition_tool.py @@ -225,6 +225,14 @@ def detect_binary(cls, binary_output: str) -> bool: return cls.detect_binary_pattern.match(binary_output) is not None + @classmethod + def empty_string_to(cls) -> bool: + """ + Returns True if the tool requires an empty string `to` field + for contract creating transactions. + """ + return False + def version(self) -> str: """ Return name and version of tool used to state transition @@ -311,6 +319,15 @@ class TransitionToolData: fork_name: str chain_id: int = field(default=1) reward: int = field(default=0) + empty_string_to: bool = field(default=False) + + def __post_init__(self): + """ + Ensure that the `to` field of transactions is not None + """ + if self.empty_string_to: + for tx in self.txs: + tx["to"] = tx.get("to") or "" def _evaluate_filesystem( self, @@ -571,7 +588,13 @@ def evaluate( if int(env["currentNumber"], 0) == 0: reward = -1 t8n_data = TransitionTool.TransitionToolData( - alloc=alloc, txs=txs, env=env, fork_name=fork_name, chain_id=chain_id, reward=reward + alloc=alloc, + txs=txs, + env=env, + fork_name=fork_name, + chain_id=chain_id, + reward=reward, + empty_string_to=self.empty_string_to(), ) if self.t8n_use_stream: diff --git a/src/pytest_plugins/forks/forks.py b/src/pytest_plugins/forks/forks.py index a040079ab4..fe05e13b9f 100644 --- a/src/pytest_plugins/forks/forks.py +++ b/src/pytest_plugins/forks/forks.py @@ -139,6 +139,13 @@ def add_values(self, metafunc: Metafunc, fork_parametrizer: ForkParametrizer) -> fork_attribute_name="tx_types", parameter_name="tx_type", ), + CovariantDescriptor( + marker_name="with_all_contract_creating_tx_types", + description="marks a test to be parametrized for all tx types that can create a contract" + " at parameter named tx_type of type int", + fork_attribute_name="contract_creating_tx_types", + parameter_name="tx_type", + ), CovariantDescriptor( marker_name="with_all_precompiles", description="marks a test to be parametrized for all precompiles at parameter named" From 9786d5a28bc079195f72b748935a552289b14947 Mon Sep 17 00:00:00 2001 From: spencer Date: Sat, 8 Jun 2024 06:13:17 +0700 Subject: [PATCH 2/2] feat(ci): use solc-select for tox (#604) * feat(ci): use solc-select for tox. * chore(docs): update changelog for solc-select with tox. --- .github/workflows/tox_verify.yaml | 12 ------------ docs/CHANGELOG.md | 1 + setup.cfg | 1 + tox.ini | 3 +++ 4 files changed, 5 insertions(+), 12 deletions(-) diff --git a/.github/workflows/tox_verify.yaml b/.github/workflows/tox_verify.yaml index 7014d1833d..553654d4bc 100644 --- a/.github/workflows/tox_verify.yaml +++ b/.github/workflows/tox_verify.yaml @@ -10,28 +10,23 @@ jobs: include: - os: ubuntu-latest python: '3.10' - solc: '0.8.20' evm-type: 'stable' tox-cmd: 'tox run-parallel --parallel-no-spinner' - os: ubuntu-latest python: '3.12' - solc: '0.8.23' evm-type: 'stable' tox-cmd: 'tox run-parallel --parallel-no-spinner' - os: ubuntu-latest python: '3.11' - solc: '0.8.21' evm-type: 'develop' tox-cmd: 'tox -e tests-develop' # Disabled to not be gated by evmone implementation # - os: ubuntu-latest # python: '3.11' - # solc: '0.8.21' # evm-type: 'eip7692' # tox-cmd: 'tox -e tests-eip7692' - os: macos-latest python: '3.11' - solc: '0.8.22' evm-type: 'stable' tox-cmd: 'tox run-parallel --parallel-no-spinner' steps: @@ -47,13 +42,6 @@ jobs: with: python-version: ${{ matrix.python }} allow-prereleases: true - - name: Install solc compiler - run: | - if [ "$RUNNER_OS" == "Linux" ]; then PLATFORM="linux-amd64"; else PLATFORM="macosx-amd64"; fi - RELEASE_NAME=$(curl https://binaries.soliditylang.org/${PLATFORM}/list.json | jq -r --arg SOLC_VERSION "${{ matrix.solc }}" '.releases[$SOLC_VERSION]') - wget -O $GITHUB_WORKSPACE/bin/solc https://binaries.soliditylang.org/${PLATFORM}/$RELEASE_NAME - chmod a+x $GITHUB_WORKSPACE/bin/solc - echo $GITHUB_WORKSPACE/bin >> $GITHUB_PATH - name: Setup Tools/Dependencies Ubuntu if: runner.os == 'Linux' run: | diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 0096413587..602c226246 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -52,6 +52,7 @@ Test fixtures for use by clients are available for each release on the [Github r - 🐞 Relax minor and patch dependency requirements to avoid conflicting package dependencies ([#510](https://github.com/ethereum/execution-spec-tests/pull/510)). - 🔀 Update all CI actions to use their respective Node.js 20 versions, ahead of their Node.js 16 version deprecations ([#527](https://github.com/ethereum/execution-spec-tests/pull/527)). - ✨ Releases now contain a `fixtures_eip7692.tar.gz` which contains all EOF fixtures ([#573](https://github.com/ethereum/execution-spec-tests/pull/573)). +- ✨ Use `solc-select` for tox when running locally and within CI ([#604](https://github.com/ethereum/execution-spec-tests/pull/604)). ### 💥 Breaking Change diff --git a/setup.cfg b/setup.cfg index a296760b6e..5728e4bc8a 100644 --- a/setup.cfg +++ b/setup.cfg @@ -44,6 +44,7 @@ install_requires = click>=8.0.0,<9 pydantic>=2.6.3,<3 rich>=13.7.0,<14 + solc-select>=1.0.4 [options.package_data] ethereum_test_tools = diff --git a/tox.ini b/tox.ini index fe0e6e4723..6d6d6dbdec 100644 --- a/tox.ini +++ b/tox.ini @@ -11,6 +11,7 @@ eip7692 = CancunEIP7692 [testenv] package = wheel wheel_build_env = .pkg +commands_pre = solc-select use 0.8.24 --always-install [testenv:framework] description = Run checks on helper libraries and test framework @@ -83,6 +84,8 @@ extras = lint docs +commands_pre = # Override commands pre to not run solc-select install + setenv = SPEC_TESTS_AUTO_GENERATE_FILES = true # Required for `cairosvg` so tox can find `libcairo-2`.