Skip to content

Commit

Permalink
Update documentation (#24)
Browse files Browse the repository at this point in the history
* Update documentation

Signed-off-by: Fabio Di Fabio <[email protected]>

* minor edits

---------

Signed-off-by: Fabio Di Fabio <[email protected]>
Co-authored-by: Sally MacFarlane <[email protected]>
  • Loading branch information
fab-10 and macfarla authored Jun 24, 2024
1 parent e48caeb commit 0bf5d57
Show file tree
Hide file tree
Showing 3 changed files with 98 additions and 202 deletions.
174 changes: 85 additions & 89 deletions PLUGINS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,132 +5,128 @@
The profitability calculator is a shared component, that is used to check if a tx is profitable.
It is applied, with different configuration to:
1. `linea_estimateGas` endpoint
2. Tx validation for the txpool
2. Tx validation for the txpool (if tx profitability check is enabled)
3. Tx selection during block creation

#### CLI Options

| Option Name | Default Value | Command Line Argument |
|--------------------------|---------------|-------------------------------------------|
| L1_VERIFICATION_GAS_COST | 1_200_000 | `--plugin-linea-verification-gas-cost` |
| L1_VERIFICATION_CAPACITY | 90_000 | `--plugin-linea-verification-capacity` |
| L1_L2_GAS_PRICE_RATIO | 15 | `--plugin-linea-gas-price-ratio` |
| L2_GAS_PRICE_ADJUSTMENT | 0 wei | `--plugin-linea-gas-price-adjustment` |
| MIN_MARGIN | 1.0 | `--plugin-linea-min-margin` |
| ESTIMATE_GAS_MIN_MARGIN | 1.0 | `--plugin-linea-estimate-gas-min-margin` |
| TX_POOL_MIN_MARGIN | 0.5 | `--plugin-linea-tx-pool-min-margin` |
| UNPROFITABLE_CACHE_SIZE | 100_000 | `--plugin-linea-unprofitable-cache-size` |
| UNPROFITABLE_RETRY_LIMIT | 10 | `--plugin-linea-unprofitable-retry-limit` |
| TX_POOL_ENABLE_CHECK_API | true | `--plugin-linea-tx-pool-profitability-check-api-enabled` |
| TX_POOL_ENABLE_CHECK_P2P | false | `--plugin-linea-tx-pool-profitability-check-p2p-enabled` |
| Command Line Argument | Default Value |
|----------------------------------------------------------|---------------|
| `--plugin-linea-fixed-gas-cost-wei` | 0 |
| `--plugin-linea-variable-gas-cost-wei` | 1_000_000_000 |
| `--plugin-linea-extra-data-pricing-enabled` | false |
| `--plugin-linea-min-margin` | 1.0 |
| `--plugin-linea-estimate-gas-min-margin` | 1.0 |
| `--plugin-linea-tx-pool-min-margin` | 0.5 |

### Module line count validator
The Module line count validator is a shared component, that is used to check if a tx exceeds any of the configured line count limits.
It is used in:
1. `linea_estimateGas` endpoint
2. Tx validation for the txpool (if tx simulation is enabled)
3. Tx selection during block creation

#### CLI Options

| Command Line Argument | Default Value |
|-------------------------------------------------------|----------------------|
| `--plugin-linea-module-limit-file-path` | moduleLimitFile.toml |
| `--plugin-linea-over-line-count-limit-cache-size` | 10_000 |


### L1 L2 Bridge
These values are just passed to the ZkTracer

#### CLI Options

| Option Name | Default Value | Command Line Argument |
|------------------------------|---------------|---------------------------------------------|
| L1L2_BRIDGE_CONTRACT_ADDRESS | | `--plugin-linea-l1l2-bridge-contract` |
| L1L2_BRIDGE_LOG_TOPIC | | `--plugin-linea-l1l2-bridge-topic` |
| Command Line Argument | Default Value |
|----------------------------------------|---------------|
| `--plugin-linea-l1l2-bridge-contract` | |
| `--plugin-linea-l1l2-bridge-topic` | |


## Sequencer
### Transaction Selection - LineaTransactionSelectorPlugin

### Transaction Selection - LineaTransactionSelectorPlugin
This plugin extends the standard transaction selection protocols employed by Besu for block creation.
It leverages the TransactionSelectionService to manage and customize the process of transaction selection.
It leverages the `TransactionSelectionService` to manage and customize the process of transaction selection.
This includes setting limits such as `TraceLineLimit`, `maxBlockGas`, and `maxCallData`, and check the profitability
of a transaction.

The selectors are in the package `net.consensys.linea.sequencer.txselection.selectors`.

#### CLI Options

| Option Name | Default Value | Command Line Argument |
|-------------------------------------|----------------------|-------------------------------------------------------|
| MAX_BLOCK_CALLDATA_SIZE | 70000 | `--plugin-linea-max-block-calldata-size` |
| MODULE_LIMIT_FILE_PATH | moduleLimitFile.toml | `--plugin-linea-module-limit-file-path` |
| OVER_LINE_COUNT_LIMIT_CACHE_SIZE | 10_000 | `--plugin-linea-over-line-count-limit-cache-size` |
| MAX_GAS_PER_BLOCK | 30_000_000L | `--plugin-linea-max-block-gas` |
| TX_POOL_ENABLE_SIMULATION_CHECK_API | true | `--plugin-linea-tx-pool-simulation-check-api-enabled` |
| TX_POOL_ENABLE_SIMULATION_CHECK_P2P | false | `--plugin-linea-tx-pool-simulation-check-p2p-enabled` |
| Command Line Argument | Default Value |
|--------------------------------------------------------|----------------------|
| `--plugin-linea-max-block-calldata-size` | 70000 |
| `--plugin-linea-module-limit-file-path` | moduleLimitFile.toml |
| `--plugin-linea-over-line-count-limit-cache-size` | 10_000 |
| `--plugin-linea-max-block-gas` | 30_000_000L |
| `--plugin-linea-unprofitable-cache-size` | 100_000 |
| `--plugin-linea-unprofitable-retry-limit` | 10 |


### Transaction validation - LineaTransactionValidatorPlugin
### Transaction Validation - LineaTransactionValidatorPlugin

This plugin extends the default transaction validation rules for adding transactions to the
transaction pool. It leverages the PluginTransactionValidatorService to manage and customize the
process of transaction validation. This includes, for example, setting a deny list of addresses
that are not allowed to add transactions to the pool.
transaction pool. It leverages the `PluginTransactionValidatorService` to manage and customize the
process of transaction validation.
This includes setting limits such as `TraceLineLimit`, `maxTxGasLimit`, and `maxTxCallData`, and checking the profitability
of a transaction.
The validators are in the package `net.consensys.linea.sequencer.txpoolvalidation.validators`.

#### CLI Options

| Option Name | Default Value | Command Line Argument |
|-------------------------|-------------------|---------------------------------------|
| DENY_LIST_PATH | lineaDenyList.txt | `--plugin-linea-deny-list-path` |
| MAX_TX_GAS_LIMIT_OPTION | 30_000_000 | `--plugin-linea-max-tx-gas-limit` |
| MAX_TX_CALLDATA_SIZE | 60_000 | `--plugin-linea-max-tx-calldata-size` |
| Command Line Argument | Default Value |
|----------------------------------------------------------|-------------------|
| `--plugin-linea-deny-list-path` | lineaDenyList.txt |
| `--plugin-linea-max-tx-gas-limit` | 30_000_000 |
| `--plugin-linea-max-tx-calldata-size` | 60_000 |
| `--plugin-linea-tx-pool-simulation-check-api-enabled` | false |
| `--plugin-linea-tx-pool-simulation-check-p2p-enabled` | false |
| `--plugin-linea-tx-pool-profitability-check-api-enabled` | true |
| `--plugin-linea-tx-pool-profitability-check-p2p-enabled` | false |


## RPC
## RPC Methods

### Linea Estimate Gas
#### `linea_estimateGas`

This endpoint simulates a transaction and returns the estimated gas used ( as the standard `eth_estimateGas`) plus the estimated gas price to be used when submitting the tx.
This endpoint simulates a transaction, including line count limit validation, and returns the estimated gas used ( as the standard `eth_estimateGas`) plus the estimated gas price to be used when submitting the tx.

#### Parameters

same as `eth_estimateGas`

### Counters - CountersEndpointServicePlugin
#### `rollup_getTracesCountersByBlockNumberV0`

The CountersEndpointServicePlugin registers an RPC endpoint named `getTracesCountersByBlockNumberV0`
under the `rollup` namespace. When this endpoint is called, returns trace counters based on the provided request parameters.

#### Parameters

- `blockNumber`: _string_ - The block number

- `tracerVersion`: _string_ - The tracer version. It will return an error if the
requested version is different from the tracer runtime
#### Result
```json
{
"jsonrpc": "2.0",
"id": 53,
"result": {
"gasLimit": "0x5208",
"baseFeePerGas": "0x7",
"priorityFeePerGas": "0x123456"
}
}
```

### Trace generation - TracesEndpointServicePlugin
### Linea Set Extra Data
#### `linea_setExtraData`

This plugin registers an RPC endpoint named `generateConflatedTracesToFileV0` under the `rollup` namespace.
The endpoint generates conflated file traces.
This endpoint is used to configure the extra data based pricing, and it only makes sense to call it on the sequencer.
Internally it sets runtime pricing configuration and then calls, via the in-process RPC service, `miner_setExtraData`
and `miner_setMinGasPrice` to update internal Besu configuration, and add the extra data pricing to the future built blocks.

#### Parameters

- `fromBlock`: _string_ - the fromBlock number
- `toBlock`: _string_ - The toBlock number
- `tracerVersion`: _string_ - The tracer version. It will return an error if the
requested version is different from the tracer runtime

## Continuous Tracing

The continuous tracing plugin allows to trace every newly imported block and use Corset to check if the constraints are
valid. In case of an error a message will be sent to the configured Slack channel.

### Usage

The continuous tracing plugin is disabled by default. To enable it, use the `--plugin-linea-continuous-tracing-enabled`
flag. If the plugin is enabled it is mandatory to specify the location of `zkevm.bin` using
the `--plugin-linea-continuous-tracing-zkevm-bin` flag. The user with which the node is running needs to have the
appropriate permissions to access `zkevm.bin`.

In order to send a message to Slack a webhook URL needs to be specified by setting the `SLACK_SHADOW_NODE_WEBHOOK_URL`
environment variable. An environment variable was chosen instead of a command line flag to avoid leaking the webhook URL
in the process list.

The environment variable can be set via systemd using the following command:

```shell
Environment=SLACK_SHADOW_NODE_WEBHOOK_URL=https://hooks.slack.com/services/SECRET_VALUES
same as `miner_setExtraData` with the added constraint that the number of bytes must be 32

#### Result
```json
{
"jsonrpc": "2.0",
"id": 53,
"result": "true"
}
```

### Invalid trace handling

In the success case the trace file will simply be deleted.

In case of an error the trace file will be renamed to `trace_$BLOCK_NUMBER_$BLOCK_HASH.lt` and moved
to `$BESU_USER_HOME/invalid-traces`. The output of Corset will be saved in the same directory in a file
named `corset_output_$BLOCK_NUMBER_$BLOCK_HASH.txt`. After that an error message will be sent to Slack.
126 changes: 13 additions & 113 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,32 @@
# Besu Plugins related to tracer and sequencer functionality

A Linea tracing implementation for [Hyperledger Besu](https://github.com/hyperledger/besu) based on
an [existing implementation in Go](https://github.com/Consensys/zk-evm/).
A set of Linea plugins for the sequencer and RPC nodes.

## Quickstart - Running Besu with Linea Plugins
## Quickstart - Running [Linea Besu](https://github.com/ConsenSys/linea-besu) with plugins

- compile linea-plugins `gradlew installDist`
- copy jar file to besu runtime plugins/ directory (where you will run besu from, not where you're building besu)
- add `ROLLUP` to besu config to enable the plugin RPC methods
- rpc-http-api=\["ADMIN","ETH","NET","WEB3","ROLLUP"\]
- add `LINEA` to besu config to enable the plugin RPC methods
- rpc-http-api=\["ADMIN","ETH","NET","WEB3","LINEA"\]
- start besu (command line or from IDE) and you should see plugins registered at startup
- call the RPC endpoint eg
- call the RPC endpoint eg:

```shell
curl --location --request POST 'http://localhost:8545' --data-raw '{
"jsonrpc": "2.0",
"method": "rollup_generateConflatedTracesToFileV0",
"params": [0, 0, "6.16.0"],
"method": "linea_estimateGas",
"params": [
"from": "0x73b2e0E54510239E22cC936F0b4a6dE1acf0AbdE",
"to": "0xBb977B2EE8a111D788B3477D242078d0B837E72b",
"value": "0x123"
],
"id": 1
}'
```

## Development Setup

### Install Java 17

```
brew install openjdk@17
```
### Install Java 21

### Native Lib Prerequisites

Expand All @@ -40,98 +39,18 @@ Windows

On release native libs are built for all the supported platforms,
if you want to test this process locally run `./gradlew -PreleaseNativeLibs jar`,
jar is generated in `arithmetization/build/libs`.

### Install Rust

```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Use local git executable to fetch from repos (needed for private repos)
echo "net.git-fetch-with-cli=true" >> .cargo/config.toml
```

### Install Corset

```shell
cargo install --git ssh://[email protected]/Consensys/corset
```

### Update Constraints [Submodule](https://github.com/Consensys/zkevm-constraints/)

```shell
git submodule update --init --recursive
```

### Install [pre-commit](https://pre-commit.com/)

```shell
pip install --user pre-commit

# For macOS users.
brew install pre-commit
```

Then run `pre-commit install` to set up git hook scripts.
Used hooks can be found [here](.pre-commit-config.yaml).

______________________________________________________________________

NOTE

> `pre-commit` aids in running checks (end of file fixing,
> markdown linting, linting, runs tests, json validation, etc.)
> before you perform your git commits.
______________________________________________________________________
jar is generated in `sequencer/build/libs`.

### Run tests

```shell
# Run all tests
./gradlew clean test

# Run only unit tests
./gradlew clean unitTests

# Run only acceptance tests
./gradlew clean acceptanceTests

# Run EVM test suite BlockchainTests
./gradlew clean referenceBlockchainTests

# Run EVM test suite GeneralStateTests
./gradlew clean referenceGeneralStateTests

# Run all EVM test suite reference tests
./gradlew clean referenceTests

# Run single reference test via gradle, e.g for net.consensys.linea.generated.blockchain.BlockchainReferenceTest_583
./gradlew :reference-tests:referenceTests --tests "net.consensys.linea.generated.blockchain.BlockchainReferenceTest_583"
```

______________________________________________________________________

NOTE

> Please be aware if the reference test code generation tasks `blockchainReferenceTests` and
> `generalStateReferenceTests` do not generate any java code, then probably you are missing the Ethereum tests"
> submodule which you can clone via `git submodule update --init --recursive`.
______________________________________________________________________

### Capturing a replay

For debugging and inspection purposes, it is possible to capture a _replay_, _i.e._ all the minimal information required to replay a series of blocks as they played on the blockchain, which is done with `scripts/capture.pl`.

A typical invocation would be:

```
scripts/capture.pl --start 1300923
```

which would capture a replay of block #1300923 and store it in `arithmetization/src/test/resources/replays`. More options are available, refer to `scripts/capture.pl -h`.

## IntelliJ IDEA Setup

### Enable Annotation Processing
Expand Down Expand Up @@ -160,30 +79,11 @@ NOTE
______________________________________________________________________

### Set Up IDE Code Re-formatting

- Install [Checkstyle](https://plugins.jetbrains.com/plugin/1065-checkstyle-idea) plugin and set IDE code
reformatting to comply with the project's Checkstyle configuration:

- Go to `Settings | Editor | Code Style | Java | <hamburger menu> | Import Scheme | Checkstyle configuration`:

![idea_checkstyle_reformat.png](images/idea_checkstyle_reformat.png)

and select `<project_root>/config/checkstyle.xml`.

### Install Optional Plugins

- Install [Spotless Gradle](https://plugins.jetbrains.com/plugin/18321-spotless-gradle) plugin to re-format through
the IDE according to spotless configuration.

## Debugging Traces

- JSON files can be debugged with the following command:

```shell
corset check -T <JSON_FILE> -v zkevm-constraints/zkevm.bin
```

## Plugins

Plugins are documented [here](PLUGINS.md).
Expand Down
Binary file removed images/idea_checkstyle_reformat.png
Binary file not shown.

0 comments on commit 0bf5d57

Please sign in to comment.