Retrieve all transaction receipts for a block in one request #6646

Wetitpig · 2024-02-29T13:54:06Z

Thanks for sending a pull request! Have you done the following?

Checked out our contribution guidelines?
Considered documentation and added the doc-change-required label to this PR if updates are required.
Considered the changelog and included an update if required.
For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Most advanced CI tests are deferred until PR approval, but you could:

locally run all unit tests via: ./gradlew build
locally run all acceptance tests via: ./gradlew acceptanceTest
locally run all integration tests via: ./gradlew integrationTest
locally run all reference tests via: ./gradlew ethereum:referenceTests:referenceTests

PR description

Retrieve all transaction receipts in a block in 1 call to blockchain query instead of a call for each transaction in a block
Calculate transaction fee reward for miner based on effectivePriorityFeePerGas instead of effectiveGasPrice because baseFee would be burnt.

Wetitpig · 2024-02-29T17:16:39Z

My node on snap sync shows static block reward as 0 even at genesis block of ETH mainnet.
Not sure if the protocol specification was purged as part of the state or simply a bug.

macfarla

I think the changes make sense. Would like a test for the new method on BlockchainQueries, think other changes are covered already.

ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/query/BlockchainQueries.java

macfarla · 2024-03-21T01:15:34Z

My node on snap sync shows static block reward as 0 even at genesis block of ETH mainnet. Not sure if the protocol specification was purged as part of the state or simply a bug.

is this comment related to this change or a separate issue?

Wetitpig · 2024-03-21T02:32:23Z

My node on snap sync shows static block reward as 0 even at genesis block of ETH mainnet. Not sure if the protocol specification was purged as part of the state or simply a bug.

is this comment related to this change or a separate issue?

Separate issue. Incidentally discovered when testing the RPC method.

macfarla · 2024-03-21T04:07:56Z

My node on snap sync shows static block reward as 0 even at genesis block of ETH mainnet. Not sure if the protocol specification was purged as part of the state or simply a bug.

is this comment related to this change or a separate issue?

Separate issue. Incidentally discovered when testing the RPC method.

ok - can you open an issue with more info if we need to look into it? thanks!

macfarla · 2024-03-21T04:08:33Z

@Wetitpig this PR needs updating with latest main - I don't have permission to update your branch

macfarla

@Wetitpig can you add a test for BlockchainQueries. transactionReceiptsByBlockHash

jframe · 2024-06-10T21:44:51Z

@ahamlat Would you mind having a look at this changes and see if they make sense?

ahamlat · 2024-07-02T18:24:51Z

Yes, these changes make sens here. I suggested this modification in this discord thread : https://discord.com/channels/905194001349627914/905205502940696607/1156154204981821500
@Wetitpig Could you give a clear description of the PR, then I can review the changes ?

Wetitpig · 2024-07-03T06:12:59Z

Yes, these changes make sens here. I suggested this modification in this discord thread : https://discord.com/channels/905194001349627914/905205502940696607/1156154204981821500
@Wetitpig Could you give a clear description of the PR, then I can review the changes ?

Thanks.
I was trying to reorganize the step in which transaction receipts are retrieved, such that transaction receipts in one single block are retrieved in a batch if all of them are required.
Currently they are retrieved one by one with block height and transaction index, even if the transactions originate from the same block.

ahamlat · 2024-07-03T07:14:57Z

...ain/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/EthGetBlockReceipts.java

  private BlockReceiptsResult getBlockReceiptsResult(final Hash blockHash) {
    final List<TransactionReceiptResult> receiptList =
        blockchainQueries
            .get()
-            .blockByHash(blockHash)
+            .transactionReceiptsByBlockHash(blockHash, protocolSchedule)


Is there any reason why you didn't use DefaultBlockchain.getTxReceipts here ?

...a/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/EthGetMinerDataByBlockHash.java

ahamlat · 2024-07-03T07:17:53Z

With DefaultBlockchain.getTxReceipts, the block receipts are retrieved in one go. Also, keep in mind that DefaultBlockchain.getTxReceipts uses the cache to retrieved the receipts if --cache-last-blocks=n is enabled. With this flag, Besu caches blockchain data while importing blocks.

macfarla · 2024-12-03T00:42:33Z

@Wetitpig are you still engaged on this PR? it's marked as stale and will be closed in 2 weeks if no activity

Wetitpig · 2024-12-07T15:44:31Z

Tests have been written for transactionReceiptsByBlockHash, but they have not been run to see if they would pass.

The BlockchainQueries.getblockchain() did not get replaced, because the transactionReceiptsByBlockHash method is required for calculating the effective priority gas used.

the test I requested has been added

ahamlat

Nice work @Wetitpig
I confirmed that :

eth_getBlockReceipts is much faster especially when using --cache-last-blocks=512 (from ~500ms to ~50 ms)
eth_getMinerDataByBlockHash is much faster as well with -cache-last-blocks=512 (from ~500ms to ~50 ms)
eth_getMinerDataByBlockHash netBlockReward returns the block reward without the burned fees (see example below)

eth_getMinerDataByBlockHash for block 21363910
Before this PR
netBlockReward = 0.198194985313316095

{"jsonrpc":"2.0","id":1,"result":{"netBlockReward":"0x00000000000000000000000000000000000000000000000002c02149ffdd2cff","staticBlockReward":"0x0000000000000000000000000000000000000000000000000000000000000000","transactionFee":"0x00000000000000000000000000000000000000000000000002c02149ffdd2cff","uncleInclusionReward":"0x0000000000000000000000000000000000000000000000000000000000000000","uncleRewards":[],"coinbase":"0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97","extraData":"0x546974616e2028746974616e6275696c6465722e78797a29","difficulty":"0x0000000000000000000000000000000000000000000000000000000000000000","totalDifficulty":"0x000000000000000000000000000000000000000000000c70d815d562d3cfa955"}}

After this PR
netBlockReward = 0.026228768668364131

{"jsonrpc":"2.0","id":1,"result":{"netBlockReward":"0x000000000000000000000000000000000000000000000000005d2eed175fad63","staticBlockReward":"0x0000000000000000000000000000000000000000000000000000000000000000","transactionFee":"0x000000000000000000000000000000000000000000000000005d2eed175fad63","uncleInclusionReward":"0x0000000000000000000000000000000000000000000000000000000000000000","uncleRewards":[],"coinbase":"0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97","extraData":"0x546974616e2028746974616e6275696c6465722e78797a29","difficulty":"0x0000000000000000000000000000000000000000000000000000000000000000","totalDifficulty":"0x000000000000000000000000000000000000000000000c70d815d562d3cfa955"}}

Signed-off-by: Wetitpig <[email protected]>

Exclude burnt fees Signed-off-by: Wetitpig <[email protected]>

Signed-off-by: Wetitpig <[email protected]>

macfarla · 2024-12-09T23:46:33Z

@Wetitpig there are some unit and other tests failing - compile errors. you should be able to run these tests locally

Signed-off-by: Wetitpig <[email protected]>

macfarla · 2024-12-10T07:25:42Z

CHANGELOG.md

@@ -67,6 +67,7 @@
 - Add histogram to Prometheus metrics system [#7944](https://github.com/hyperledger/besu/pull/7944)
 - Improve newPayload and FCU logs [#7961](https://github.com/hyperledger/besu/pull/7961)
 - Proper support for `pending` block tag when calling `eth_estimateGas` and `eth_createAccessList` [#7951](https://github.com/hyperledger/besu/pull/7951)
+- Retrieve all transaction receipts for a block in one request [#6646](https://github.com/hyperledger/besu/pull/6646)


this should move up to the "Unreleased" section

Signed-off-by: Wetitpig <[email protected]>

…dger#6646) Signed-off-by: Sally MacFarlane <[email protected]> Signed-off-by: Daniel Lehrner <[email protected]>

Wetitpig force-pushed the receiptLoop branch from d89f198 to ae09fcb Compare February 29, 2024 13:57

Wetitpig force-pushed the receiptLoop branch 3 times, most recently from 89ea57c to 4341667 Compare March 1, 2024 10:42

Wetitpig marked this pull request as ready for review March 2, 2024 02:25

Wetitpig force-pushed the receiptLoop branch 4 times, most recently from 8d61d3e to ef71f57 Compare March 6, 2024 03:35

Wetitpig force-pushed the receiptLoop branch 2 times, most recently from 6c1a93d to 027d056 Compare March 19, 2024 10:24

macfarla reviewed Mar 21, 2024

View reviewed changes

ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/query/BlockchainQueries.java Show resolved Hide resolved

Wetitpig force-pushed the receiptLoop branch 2 times, most recently from 2844cfa to 0a67874 Compare March 22, 2024 07:32

macfarla previously requested changes Mar 24, 2024

View reviewed changes

ahamlat requested changes Jul 3, 2024

View reviewed changes

macfarla added the Stale label Dec 3, 2024

Wetitpig force-pushed the receiptLoop branch from 0a67874 to d0a68d8 Compare December 7, 2024 14:22

Wetitpig requested a review from ahamlat December 7, 2024 14:27

Wetitpig force-pushed the receiptLoop branch from 0d3fa32 to 39a0d92 Compare December 7, 2024 14:28

Wetitpig force-pushed the receiptLoop branch 3 times, most recently from dfa9af0 to 5e8e86a Compare December 7, 2024 15:40

Wetitpig requested a review from macfarla December 7, 2024 15:42

Wetitpig force-pushed the receiptLoop branch from 5e8e86a to 2cfaa1c Compare December 7, 2024 16:07

macfarla assigned ahamlat Dec 8, 2024

ahamlat approved these changes Dec 9, 2024

View reviewed changes

Wetitpig added 5 commits December 9, 2024 23:08

Retrieve all receipts in block in one call

75dbee5

Signed-off-by: Wetitpig <[email protected]>

Fix eth_getMinerDataByBlockHash transaction fees

bee2553

Exclude burnt fees Signed-off-by: Wetitpig <[email protected]>

Collapse txReceipts

8b32417

Signed-off-by: Wetitpig <[email protected]>

Bypass for TransactionAdapter

f880401

Signed-off-by: Wetitpig <[email protected]>

Skip receipts on pending tx

87f9e73

Signed-off-by: Wetitpig <[email protected]>

Wetitpig force-pushed the receiptLoop branch 2 times, most recently from 47ad5a5 to 535de72 Compare December 9, 2024 15:54

ahamlat assigned Wetitpig and unassigned ahamlat Dec 9, 2024

Wetitpig added 2 commits December 10, 2024 08:16

Add tests to transactionReceiptsByBlockHash

ec55aa1

Signed-off-by: Wetitpig <[email protected]>

Add docs to methods

d08319c

Signed-off-by: Wetitpig <[email protected]>

Wetitpig force-pushed the receiptLoop branch from 535de72 to a07b1d7 Compare December 10, 2024 00:16

macfarla reviewed Dec 10, 2024

View reviewed changes

Wetitpig force-pushed the receiptLoop branch from a07b1d7 to 02b56ec Compare December 10, 2024 07:53

Update CHANGELOG.md

d00a9e4

Signed-off-by: Wetitpig <[email protected]>

Wetitpig force-pushed the receiptLoop branch from 02b56ec to d00a9e4 Compare December 10, 2024 09:41

macfarla merged commit 4435f75 into hyperledger:main Dec 10, 2024
43 checks passed

Wetitpig deleted the receiptLoop branch December 10, 2024 13:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retrieve all transaction receipts for a block in one request #6646

Retrieve all transaction receipts for a block in one request #6646

Wetitpig commented Feb 29, 2024 •

edited

Loading

Wetitpig commented Feb 29, 2024

macfarla left a comment

macfarla commented Mar 21, 2024

Wetitpig commented Mar 21, 2024

macfarla commented Mar 21, 2024

macfarla commented Mar 21, 2024

macfarla left a comment

jframe commented Jun 10, 2024

ahamlat commented Jul 2, 2024

Wetitpig commented Jul 3, 2024

ahamlat Jul 3, 2024

ahamlat commented Jul 3, 2024 •

edited

Loading

macfarla commented Dec 3, 2024

Wetitpig commented Dec 7, 2024 •

edited

Loading

ahamlat left a comment •

edited

Loading

macfarla commented Dec 9, 2024

macfarla Dec 10, 2024

Retrieve all transaction receipts for a block in one request #6646

Retrieve all transaction receipts for a block in one request #6646

Conversation

Wetitpig commented Feb 29, 2024 • edited Loading

Thanks for sending a pull request! Have you done the following?

Most advanced CI tests are deferred until PR approval, but you could:

PR description

Wetitpig commented Feb 29, 2024

macfarla left a comment

Choose a reason for hiding this comment

macfarla commented Mar 21, 2024

Wetitpig commented Mar 21, 2024

macfarla commented Mar 21, 2024

macfarla commented Mar 21, 2024

macfarla left a comment

Choose a reason for hiding this comment

jframe commented Jun 10, 2024

ahamlat commented Jul 2, 2024

Wetitpig commented Jul 3, 2024

ahamlat Jul 3, 2024

Choose a reason for hiding this comment

ahamlat commented Jul 3, 2024 • edited Loading

macfarla commented Dec 3, 2024

Wetitpig commented Dec 7, 2024 • edited Loading

ahamlat left a comment • edited Loading

Choose a reason for hiding this comment

macfarla commented Dec 9, 2024

macfarla Dec 10, 2024

Choose a reason for hiding this comment

Wetitpig commented Feb 29, 2024 •

edited

Loading

ahamlat commented Jul 3, 2024 •

edited

Loading

Wetitpig commented Dec 7, 2024 •

edited

Loading

ahamlat left a comment •

edited

Loading