Add Gnosis #925

kaladinlight · 2023-05-12T19:53:33Z

No geth support, the execution node is nethermind which is written in C#. This resulted in the need for hand rolling the Header type and associated marshal/rlp handlers. I did start to think about if it is even necessary to use the Hash() function on the Header to get the block hash instead of just getting hash directly off the Block. Was this an intentional decision for some edge case that I am not thinking about?
Start the consensus node first so it creates the jwt and then start the execution node.
Nethermind is super quick to sync a full node to start testing (~10hrs), archive nodes still takes the better part of a week I believe

kaladinlight · 2023-06-07T17:19:27Z

@martinboehm ready for review whenever you have free cycles :)

matyushkins · 2023-07-17T10:50:06Z

@kaladinlight Hi. Archive node.
websocket: read limit exceeded

E0717 10:45:03.174752       1 ethrpc.go:652] debug_traceBlockByHash block 0xdc31a7341d59c76c89e2a20bccfa726b15ef165cf2cb57cf67af92cd46417133, error websocket: read limit exceeded
E0717 10:45:03.174769       1 ethrpc.go:652] debug_traceBlockByHash block 0xcdc64212c941d29f0a9b9905a858222de498b0377524cc42a90b76836e0910dc, error websocket: read limit exceeded
E0717 10:45:03.174774       1 ethrpc.go:652] debug_traceBlockByHash block 0x5dfc70726803b8b6a2db378c9e1d1686e8fb2f2a07a7f10dac398478b4f6b5b6, error websocket: read limit exceeded
E0717 10:45:03.174759       1 ethrpc.go:652] debug_traceBlockByHash block 0xc64f390b6556bf77708b5d11a8161c5ce15f3dbe2c066a02dde1f4e0b0490d95, error websocket: read limit exceeded

kaladinlight · 2023-07-17T16:31:56Z

@kaladinlight Hi. Archive node. websocket: read limit exceeded

E0717 10:45:03.174752       1 ethrpc.go:652] debug_traceBlockByHash block 0xdc31a7341d59c76c89e2a20bccfa726b15ef165cf2cb57cf67af92cd46417133, error websocket: read limit exceeded
E0717 10:45:03.174769       1 ethrpc.go:652] debug_traceBlockByHash block 0xcdc64212c941d29f0a9b9905a858222de498b0377524cc42a90b76836e0910dc, error websocket: read limit exceeded
E0717 10:45:03.174774       1 ethrpc.go:652] debug_traceBlockByHash block 0x5dfc70726803b8b6a2db378c9e1d1686e8fb2f2a07a7f10dac398478b4f6b5b6, error websocket: read limit exceeded
E0717 10:45:03.174759       1 ethrpc.go:652] debug_traceBlockByHash block 0xc64f390b6556bf77708b5d11a8161c5ce15f3dbe2c066a02dde1f4e0b0490d95, error websocket: read limit exceeded

Did you build blockbook using the Makefile? There is a hack in place to manually increase the websocket message size limit that should fix this issue unless the gnosis block traces are actually larger than the current 80mb setting.

prepare-sources:
	@ [ -n "`ls /src 2> /dev/null`" ] || (echo "/src doesn't exist or is empty" 1>&2 && exit 1)
	rm -rf $(BLOCKBOOK_SRC)
	mkdir -p $(BLOCKBOOK_BASE)
	cp -r /src $(BLOCKBOOK_SRC)
	cd $(BLOCKBOOK_SRC) && go mod download
	sed -i 's/wsMessageSizeLimit\ =\ 15\ \*\ 1024\ \*\ 1024/wsMessageSizeLimit = 80 * 1024 * 1024/g' $(GOPATH)/pkg/mod/github.com/ethereum/go-ethereum*/rpc/websocket.go
	sed -i 's/wsMessageSizeLimit\ =\ 15\ \*\ 1024\ \*\ 1024/wsMessageSizeLimit = 80 * 1024 * 1024/g' $(GOPATH)/pkg/mod/github.com/ava-labs/coreth*/rpc/websocket.go

matyushkins · 2023-07-17T16:53:23Z

@kaladinlight Hi. Archive node. websocket: read limit exceeded

E0717 10:45:03.174752       1 ethrpc.go:652] debug_traceBlockByHash block 0xdc31a7341d59c76c89e2a20bccfa726b15ef165cf2cb57cf67af92cd46417133, error websocket: read limit exceeded
E0717 10:45:03.174769       1 ethrpc.go:652] debug_traceBlockByHash block 0xcdc64212c941d29f0a9b9905a858222de498b0377524cc42a90b76836e0910dc, error websocket: read limit exceeded
E0717 10:45:03.174774       1 ethrpc.go:652] debug_traceBlockByHash block 0x5dfc70726803b8b6a2db378c9e1d1686e8fb2f2a07a7f10dac398478b4f6b5b6, error websocket: read limit exceeded
E0717 10:45:03.174759       1 ethrpc.go:652] debug_traceBlockByHash block 0xc64f390b6556bf77708b5d11a8161c5ce15f3dbe2c066a02dde1f4e0b0490d95, error websocket: read limit exceeded

Did you build blockbook using the Makefile? There is a hack in place to manually increase the websocket message size limit that should fix this issue unless the gnosis block traces are actually larger than the current 80mb setting.

prepare-sources:
	@ [ -n "`ls /src 2> /dev/null`" ] || (echo "/src doesn't exist or is empty" 1>&2 && exit 1)
	rm -rf $(BLOCKBOOK_SRC)
	mkdir -p $(BLOCKBOOK_BASE)
	cp -r /src $(BLOCKBOOK_SRC)
	cd $(BLOCKBOOK_SRC) && go mod download
	sed -i 's/wsMessageSizeLimit\ =\ 15\ \*\ 1024\ \*\ 1024/wsMessageSizeLimit = 80 * 1024 * 1024/g' $(GOPATH)/pkg/mod/github.com/ethereum/go-ethereum*/rpc/websocket.go
	sed -i 's/wsMessageSizeLimit\ =\ 15\ \*\ 1024\ \*\ 1024/wsMessageSizeLimit = 80 * 1024 * 1024/g' $(GOPATH)/pkg/mod/github.com/ava-labs/coreth*/rpc/websocket.go

Yes. I run make command with BASE_IMAGE=ubuntu:22.04.
make BASE_IMAGE=ubuntu:22.04

kaladinlight · 2023-07-17T17:19:57Z

I can push a change to bump that limit if you want to rebuild the image to test. I don't have an archive node to point at currently to test locally...

matyushkins · 2023-07-17T21:17:29Z

{
       "jsonrpc": "2.0",
       "method": "debug_traceBlockByHash",
       "params": ["0x9bccfe5644bc698235f904c76375e3ed882d037451b5dc8bd66a09c592b3902d", {"tracer": "callTracer"}],
       "id": 1
}

Size json file 205 MB

kaladinlight · 2023-07-17T21:20:36Z

{
       "jsonrpc": "2.0",
       "method": "debug_traceBlockByHash",
       "params": ["0x9bccfe5644bc698235f904c76375e3ed882d037451b5dc8bd66a09c592b3902d", {"tracer": "callTracer"}],
       "id": 1
}

Size json file 205 MB

Good lord... I will bump to 256 for temp fix. I would like to look into adding a logic fork to leverage the parity trace module instead of geth debug to reduce these payload sizes and improve perf long term.

matyushkins · 2023-07-17T22:38:38Z

@kaladinlight I raised the limit to 2048 because the page size is 1800MB. And rpc timeout to 300 sec.

{
       "jsonrpc": "2.0",
       "method": "debug_traceBlockByHash",
       "params": ["0xcd3cd8cf7540380c376d61799adc5990a707ac02c34fa9f4c802bf891de4679b", {"tracer": "callTracer"}],
       "id": 1
}

5 million blocks. The flight is normal.

matyushkins · 2023-07-18T13:03:07Z

~2200 MB

{
       "jsonrpc": "2.0",
       "method": "debug_traceBlockByHash",
       "params": ["0x7eb96747d3b5356abfef48cd3729e69394140315e6945254e3256e902ec1db72", {"tracer": "callTracer"}],
       "id": 1
}

kaladinlight · 2023-07-18T15:45:34Z

~2200 MB

{
       "jsonrpc": "2.0",
       "method": "debug_traceBlockByHash",
       "params": ["0x7eb96747d3b5356abfef48cd3729e69394140315e6945254e3256e902ec1db72", {"tracer": "callTracer"}],
       "id": 1
}

that is crazy... at this point I would say raise it as you need to for your purposes and when I have time I will work on supporting the trace module for gnosis which will hopefully help things

kaladinlight · 2024-01-18T23:08:17Z

@matyushkins not sure if you are still running this, but I finally got around to adding the parity trace debug for gnosis if you wanted to give that a shot. it should be significantly lighter to run with this change. let me know!

kaladinlight · 2024-01-18T23:10:18Z

bchain/coins/eth/ethrpc.go

+	Parser                  *EthereumParser
+	PushHandler             func(bchain.NotificationType)
+	OpenRPC                 func(string) (bchain.EVMRPCClient, bchain.EVMClient, error)
+	GetInternalDataForBlock func(blockHash string, blockHeight uint32, transactions []bchain.RpcTransaction) ([]bchain.EthereumInternalData, []bchain.ContractInfo, error)


support overridable GetInternalDataForBlock to support the use of the parity trace module for debug. This is only used by gnosis at this time, but I suppose I could also add that logic directly to ethrpc.go and leverage a DebugType flag if you would like that better.

kaladinlight · 2024-01-18T23:10:36Z

bchain/coins/eth/ethrpc.go

@@ -592,7 +595,8 @@ type rpcTraceResult struct {
 	Result rpcCallTrace `json:"result"`
 }

-func (b *EthereumRPC) getCreationContractInfo(contract string, height uint32) *bchain.ContractInfo {
+// GetCreationContractInfo retrieves the info for a contract address and sets the creation block height
+func (b *EthereumRPC) GetCreationContractInfo(contract string, height uint32) *bchain.ContractInfo {


expose for use in GnosisRPC

kaladinlight · 2024-01-18T23:11:32Z

bchain/coins/gnosis/gnosisrpc.go

+}
+
+// getInternalDataForBlock extracts internal transfers and creation or destruction of contracts using the parity trace module
+func (b *GnosisRPC) getInternalDataForBlock(blockHash string, blockHeight uint32, transactions []bchain.RpcTransaction) ([]bchain.EthereumInternalData, []bchain.ContractInfo, error) {


parity trace logic for internal data

kaladinlight · 2024-01-18T23:12:55Z

@martinboehm another one that would be nice to get upstreamed if you have any spare cycles. thanks!

kaladinlight · 2024-05-06T20:07:08Z

bchain/coins/eth/ethaddrcontracts.proto

+
+option go_package = "github.com/trezor/blockbook/bchain/coins/eth";
+
+message ProtoAddrContracts {


update addr contracts to proto type to fix performance impact of manual pack/unpack related to huge multi addr contracts on chain

seemed to help a bit, but sync has since crawled to a stop...

blockbook.go

kaladinlight · 2024-05-31T18:04:52Z

@martinboehm happy to talk through this guy as well, but quick tldr:

due to massive amounts of nft on gnosis, the sync speed crawled to a halt
after profiling, the bottleneck was pack/unpack -> read/write to db for address contracts
I went ahead and upgraded the address contracts to protobufs for more efficient encode/decode along with a configurable address cache size in an effort to improve performance
there did seem to be improvements, but fresh sync is slowing down drastically again resulting in my perf improvement attempts not being enough

Would love to brainstorm with you if you have the time and are up for it, but at this point gnosis is not really production ready in the current state of blockbook :(

kaladinlight force-pushed the add-gnosis branch from 9c25fc0 to 6a3cad8 Compare June 21, 2023 16:50

kaladinlight force-pushed the add-gnosis branch from c4f0ec4 to 31d16be Compare July 17, 2023 21:21

kaladinlight commented Jan 18, 2024

View reviewed changes

kaladinlight marked this pull request as draft April 12, 2024 21:29

kaladinlight force-pushed the add-gnosis branch from 0975b18 to df9fb37 Compare May 6, 2024 20:04

kaladinlight commented May 6, 2024

View reviewed changes

blockbook.go Outdated Show resolved Hide resolved

kaladinlight force-pushed the add-gnosis branch 3 times, most recently from 55f1bba to f22919d Compare May 7, 2024 20:39

kaladinlight marked this pull request as ready for review May 7, 2024 20:40

kaladinlight force-pushed the add-gnosis branch from f22919d to 6d09e34 Compare May 9, 2024 22:04

add gnosis support

a7d8a7f

kaladinlight force-pushed the add-gnosis branch from 6d09e34 to a7d8a7f Compare July 12, 2024 18:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Gnosis #925

Add Gnosis #925

kaladinlight commented May 12, 2023

kaladinlight commented Jun 7, 2023

matyushkins commented Jul 17, 2023 •

edited

Loading

kaladinlight commented Jul 17, 2023

matyushkins commented Jul 17, 2023 •

edited

Loading

kaladinlight commented Jul 17, 2023

matyushkins commented Jul 17, 2023

kaladinlight commented Jul 17, 2023

matyushkins commented Jul 17, 2023 •

edited

Loading

matyushkins commented Jul 18, 2023

kaladinlight commented Jul 18, 2023

kaladinlight commented Jan 18, 2024

kaladinlight Jan 18, 2024

kaladinlight Jan 18, 2024

kaladinlight Jan 18, 2024

kaladinlight commented Jan 18, 2024

kaladinlight May 6, 2024

kaladinlight Jul 11, 2024

kaladinlight commented May 31, 2024


		option go_package = "github.com/trezor/blockbook/bchain/coins/eth";

		message ProtoAddrContracts {

Add Gnosis #925

Are you sure you want to change the base?

Add Gnosis #925

Conversation

kaladinlight commented May 12, 2023

kaladinlight commented Jun 7, 2023

matyushkins commented Jul 17, 2023 • edited Loading

kaladinlight commented Jul 17, 2023

matyushkins commented Jul 17, 2023 • edited Loading

kaladinlight commented Jul 17, 2023

matyushkins commented Jul 17, 2023

kaladinlight commented Jul 17, 2023

matyushkins commented Jul 17, 2023 • edited Loading

matyushkins commented Jul 18, 2023

kaladinlight commented Jul 18, 2023

kaladinlight commented Jan 18, 2024

kaladinlight Jan 18, 2024

Choose a reason for hiding this comment

kaladinlight Jan 18, 2024

Choose a reason for hiding this comment

kaladinlight Jan 18, 2024

Choose a reason for hiding this comment

kaladinlight commented Jan 18, 2024

kaladinlight May 6, 2024

Choose a reason for hiding this comment

kaladinlight Jul 11, 2024

Choose a reason for hiding this comment

kaladinlight commented May 31, 2024

matyushkins commented Jul 17, 2023 •

edited

Loading

matyushkins commented Jul 17, 2023 •

edited

Loading

matyushkins commented Jul 17, 2023 •

edited

Loading