Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/devel' into devel
Browse files Browse the repository at this point in the history
  • Loading branch information
mattmalec committed Mar 13, 2024
2 parents 4770e1a + 90e7a44 commit 47c2699
Show file tree
Hide file tree
Showing 720 changed files with 138,064 additions and 32,195 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ jobs:
- name: Install golangci-lint
if: runner.os == 'Linux'
uses: golangci/golangci-lint-action@v3
uses: golangci/golangci-lint-action@v4
with:
version: v1.55.2
version: v1.56.1
skip-build-cache: true
args: --help

Expand Down
78 changes: 78 additions & 0 deletions .github/workflows/qa-clean-exit-block-downloading.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: QA - Clean exit (block downloading)

on:
push:
branches:
- 'release/**'
pull_request:
branches:
- devel
- 'release/**'
types:
- ready_for_review

jobs:
long-running-test:
runs-on: self-hosted
env:
ERIGON_REFERENCE_DATA_DIR: /opt/erigon-release/datadir
ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir
WORKING_TIME_SECONDS: 600

steps:
- name: Check out repository
uses: actions/checkout@v2

- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.21'

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.x'

#- name: Install dependencies
# run: |
# sudo apt-get update
# sudo apt-get install -y build-essential make gcc

- name: Restore Erigon Testbed Data Directory
run: |
rm -rf $ERIGON_TESTBED_DATA_DIR/chaindata
rsync -a --delete $ERIGON_REFERENCE_DATA_DIR/ $ERIGON_TESTBED_DATA_DIR/
- name: Clean Erigon Build Directory
run: |
make clean
- name: Build Erigon
run: |
make erigon
working-directory: ${{ github.workspace }}

- name: Run Erigon, send ctrl-c and check for clean exiting
run: |
# Run Erigon, send ctrl-c and check logs
python3 ${{ github.workspace }}/../../../../erigon-qa/test_system/qa-tests/clean-exit/run_and_check_clean_exit.py ${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $WORKING_TIME_SECONDS
# Capture monitoring script exit status
monitoring_exit_status=$?

# Clean up Erigon process if it's still running
if kill -0 $ERIGON_PID 2> /dev/null; then
echo "Terminating Erigon"
kill $ERIGON_PID
wait $ERIGON_PID
else
echo "Erigon has already terminated"
fi

# Check monitoring script exit status
if [ $monitoring_exit_status -eq 0 ]; then
echo "Monitoring completed successfully"
else
echo "Error detected in Erigon logs or monitoring script exited unexpectedly"
exit 1
fi
4 changes: 0 additions & 4 deletions .github/workflows/qa-clean-exit-snapshot-downloading.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,6 @@ jobs:
make erigon
working-directory: ${{ github.workspace }}

#- name: Download Python Script for Logs Checking
# run: |
# curl -o check_erigon_exit.py 'https://gist.githubusercontent.com/mriccobene/8db4030a745de34d527f136f2caa104f/raw/3c1a860cb87d61075e78ce399e17f0ab157cacc6/check_erigon_exit.py'

- name: Run Erigon, send ctrl-c and check for clean exiting
run: |
# Run Erigon, send ctrl-c and check logs
Expand Down
101 changes: 101 additions & 0 deletions .github/workflows/qa-tip-tracking.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
name: QA - Tip tracking

on:
schedule:
- cron: '0 0 * * *' # Run every day at 00:00 AM UTC
push:
branches:
- workflow_tip_tracking

jobs:
long-running-test:
runs-on: self-hosted
env:
ERIGON_REFERENCE_DATA_DIR: /opt/erigon-release/datadir
ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir
TRACKING_TIME_SECONDS: 14400 # 4 hours
TOTAL_TIME_SECONDS: 28800 # 8 hours

steps:
- name: Check out repository
uses: actions/checkout@v4

- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.21'

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'

- name: Restore Erigon Testbed Data Directory
run: |
rm -rf $ERIGON_TESTBED_DATA_DIR/chaindata
rsync -a --delete $ERIGON_REFERENCE_DATA_DIR/ $ERIGON_TESTBED_DATA_DIR/
- name: Clean Erigon Build Directory
run: |
make clean
- name: Build Erigon
run: |
make erigon
working-directory: ${{ github.workspace }}

- name: Pause the Erigon instance dedicated to db maintenance
run: |
curl -X POST -H "Accept: application/json" -d '{"status": "paused"}' http://localhost:8080/production/default/status || true
- name: Run Erigon, wait sync and check ability to maintain sync
id: test_step
run: |
set +e # Disable exit on error
# 1. Launch the testbed Erigon instance
# 2. Allow time for the Erigon to achieve synchronization
# 3. Begin timing the duration that Erigon maintains synchronization
python3 ${{ github.workspace }}/../../../../erigon-qa/test_system/qa-tests/tip-tracking/run_and_check_tip-tracking.py ${{ github.workspace }}/build/bin $ERIGON_TESTBED_DATA_DIR $TRACKING_TIME_SECONDS $TOTAL_TIME_SECONDS
# Capture monitoring script exit status
test_exit_status=$?

# Clean up Erigon process if it's still running
if kill -0 $ERIGON_PID 2> /dev/null; then
echo "Terminating Erigon"
kill $ERIGON_PID
wait $ERIGON_PID
else
echo "Erigon has already terminated"
fi

# Check test runner script exit status
if [ test_exit_status -eq 0 ]; then
echo "Tests completed successfully"
echo "TEST_RESULT=success" >> "$GITHUB_OUTPUT"
else
echo "Error detected during tests"
echo "TEST_RESULT=failure" >> "$GITHUB_OUTPUT"
fi

- name: Resume the Erigon instance dedicated to db maintenance
run: |
curl -X POST -H "Accept: application/json" -d '{"status": "running"}' http://localhost:8080/production/default/status || true
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results
path: ${{ github.workspace }}/../../../../erigon-qa/test_system/qa-tests/tip-tracking/result.json

- name: Action for Success
if: steps.test_step.outputs.TEST_RESULT == 'success'
run: echo "::notice::Tests completed successfully"

- name: Action for Not Success
if: steps.test_step.outputs.TEST_RESULT != 'success'
run: |
echo "::error::Error detected during tests"
exit 1
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
id: prepare
run: |
TAG=${GITHUB_REF#refs/tags/}
echo ::set-output name=tag_name::${TAG}
echo "tag_name=${TAG}" >> $GITHUB_OUTPUT
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
Expand Down
5 changes: 2 additions & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax = docker/dockerfile:1.2
FROM docker.io/library/golang:1.20-alpine3.17 AS builder
FROM docker.io/library/golang:1.21-alpine3.17 AS builder

RUN apk --no-cache add build-base linux-headers git bash ca-certificates libstdc++

Expand All @@ -18,7 +18,7 @@ RUN --mount=type=cache,target=/root/.cache \
make BUILD_TAGS=nosqlite,noboltdb,nosilkworm all


FROM docker.io/library/golang:1.20-alpine3.17 AS tools-builder
FROM docker.io/library/golang:1.21-alpine3.17 AS tools-builder
RUN apk --no-cache add build-base linux-headers git bash ca-certificates libstdc++
WORKDIR /app

Expand Down Expand Up @@ -79,7 +79,6 @@ COPY --from=builder /app/build/bin/state /usr/local/bin/state
COPY --from=builder /app/build/bin/txpool /usr/local/bin/txpool
COPY --from=builder /app/build/bin/verkle /usr/local/bin/verkle
COPY --from=builder /app/build/bin/caplin /usr/local/bin/caplin
COPY --from=builder /app/build/bin/caplin-regression /usr/local/bin/caplin-regression


EXPOSE 8545 \
Expand Down
5 changes: 2 additions & 3 deletions Dockerfile.debian
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax = docker/dockerfile:1.2
FROM docker.io/library/golang:1.20-bullseye AS builder
FROM docker.io/library/golang:1.21-bullseye AS builder

RUN apt update
RUN apt install -y build-essential git bash ca-certificates libstdc++6
Expand All @@ -17,7 +17,7 @@ RUN --mount=type=cache,target=/root/.cache \
make all


FROM docker.io/library/golang:1.20-alpine3.17 AS tools-builder
FROM docker.io/library/golang:1.21-alpine3.17 AS tools-builder

RUN apk --no-cache add build-base linux-headers git bash ca-certificates libstdc++
WORKDIR /app
Expand Down Expand Up @@ -75,7 +75,6 @@ COPY --from=builder /app/build/bin/state /usr/local/bin/state
COPY --from=builder /app/build/bin/txpool /usr/local/bin/txpool
COPY --from=builder /app/build/bin/verkle /usr/local/bin/verkle
COPY --from=builder /app/build/bin/caplin /usr/local/bin/caplin
COPY --from=builder /app/build/bin/caplin-regression /usr/local/bin/caplin-regression

EXPOSE 8545 \
8551 \
Expand Down
4 changes: 1 addition & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,6 @@ COMMANDS += verkle
COMMANDS += evm
COMMANDS += sentinel
COMMANDS += caplin
COMMANDS += caplin-regression
COMMANDS += tooling
COMMANDS += snapshots


Expand Down Expand Up @@ -247,7 +245,7 @@ install:
@ls -al "$(DIST)"

PACKAGE_NAME := github.com/ledgerwatch/erigon
GOLANG_CROSS_VERSION ?= v1.20.7
GOLANG_CROSS_VERSION ?= v1.21.6

.PHONY: release-dry-run
release-dry-run: git-submodules
Expand Down
39 changes: 38 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ Erigon is an implementation of Ethereum (execution layer with embeddable consens
frontier. [Archive Node](https://ethereum.org/en/developers/docs/nodes-and-clients/archive-nodes/#what-is-an-archive-node)
by default.

An accessible and complete version of the documentation is available at **[erigon.gitbook.io](https://erigon.gitbook.io)**.
<br>

![Build status](https://github.com/ledgerwatch/erigon/actions/workflows/ci.yml/badge.svg)

![Coverage](https://gist.githubusercontent.com/revitteth/ee38e9beb22353eef6b88f2ad6ed7aa9/raw/badge.svg)
Expand All @@ -20,13 +23,15 @@ by default.
+ [GoDoc](https://godoc.org/github.com/ledgerwatch/erigon)
+ [Beacon Chain](#beacon-chain-consensus-layer)
+ [Dev Chain](#dev-chain)
+ [Caplin (Internal Consensus Layer)](#caplin)

- [Key features](#key-features)
+ [More Efficient State Storage](#more-efficient-state-storage)
+ [Faster Initial Sync](#faster-initial-sync)
+ [JSON-RPC daemon](#json-rpc-daemon)
+ [Run all components by docker-compose](#run-all-components-by-docker-compose)
+ [Grafana dashboard](#grafana-dashboard)
+ [Internal Consensus Layer](#caplin)
- [Documentation](#documentation)
- [FAQ](#faq)
- [Getting in touch](#getting-in-touch)
Expand Down Expand Up @@ -105,7 +110,7 @@ Use `--datadir` to choose where to store data.
Use `--chain=gnosis` for [Gnosis Chain](https://www.gnosis.io/), `--chain=bor-mainnet` for Polygon Mainnet,
`--chain=mumbai` for Polygon Mumbai and `--chain=amoy` for Polygon Amoy.
For Gnosis Chain you need a [Consensus Layer](#beacon-chain-consensus-layer) client alongside
Erigon (https://docs.gnosischain.com/node/guide/beacon).
Erigon (https://docs.gnosischain.com/node/manual/beacon).

Running `make help` will list and describe the convenience commands available in the [Makefile](./Makefile).

Expand Down Expand Up @@ -303,6 +308,26 @@ Once Erigon is running, you need to point your CL client to `<erigon address>:85
where `<erigon address>` is either `localhost` or the IP address of the device running Erigon, and also point to the JWT
secret path created by Erigon.

### Caplin

Caplin is a full-fledged validating Consensus Client like Prysm, Lighthouse, Teku, Nimbus and Lodestar. Its goal is:

* provide better stability
* Validation of the chain
* Stay in sync
* keep the execution of blocks on chain tip
* serve the Beacon API using a fast and compact data model alongside low CPU and memory usage.

The main reason why developed a new Consensus Layer is to experiment with the possible benefits that could come with it. For example, The Engine API does not work well with Erigon. The Engine API sends data one block at a time, which does not suit how Erigon works. Erigon is designed to handle many blocks simultaneously and needs to sort and process data efficiently. Therefore, it would be better for Erigon to handle the blocks independently instead of relying on the Engine API.

#### Caplin's Usage.

Caplin can be enabled through the `--internalcl` flag. from that point on, an external Consensus Layer will not be need anymore.

Caplin also has an archivial mode for historical states and blocks. it can be enabled through the `--caplin.archive` flag.
In order to enable the caplin's Beacon API, the flag `--beacon.api=<namespaces>` must be added.
e.g: `--beacon.api=beacon,builder,config,debug,node,validator,lighthouse` will enable all endpoints. **NOTE: Caplin is not staking-ready so aggregation endpoints are still to be implemented. Additionally enabling the Beacon API will lead to a 6 GB higher RAM usage.

### Multiple Instances / One Machine

Define 6 flags to avoid conflicts: `--datadir --port --http.port --authrpc.port --torrent.port --private.api.addr`.
Expand Down Expand Up @@ -603,6 +628,18 @@ RFC 922, Section 7
Same
in [IpTables syntax](https://ethereum.stackexchange.com/questions/6386/how-to-prevent-being-blacklisted-for-running-an-ethereum-client/13068#13068)

### How to run erigon as a separate user? (e.g. as a `systemd` daemon)

Running erigon from `build/bin` as a separate user might produce an error:

error while loading shared libraries: libsilkworm_capi.so: cannot open shared object file: No such file or directory

The library needs to be *installed* for another user using `make DIST=<path> install`. You could use `$HOME/erigon` or `/opt/erigon` as the installation path, for example:

make DIST=/opt/erigon install

and then run `/opt/erigon/erigon`.

### How to get diagnostic for bug report?

- Get stack trace: `kill -SIGUSR1 <pid>`, get trace and stop: `kill -6 <pid>`
Expand Down
Loading

0 comments on commit 47c2699

Please sign in to comment.