Upgrade cometbft to v0.38.10

.changelog/v0.38.10/bug-fixes/3002-invalid-txs-results.md

@@ -0,0 +1,3 @@
+- `[rpc]` Fix an issue where a legacy ABCI response, created on `v0.37` or before, is not returned properly in `v0.38` and up
+  on the `/block_results` RPC endpoint.
+  ([\#3002](https://github.com/cometbft/cometbft/issues/3002))

.changelog/v0.38.10/bug-fixes/3406-blocksync-dont-stall-if-blocking-chain.md

@@ -0,0 +1,3 @@
+- `[blocksync]` Do not stay in blocksync if the node's validator voting power
+  is high enough to block the chain while it is not online
+  ([\#3406](https://github.com/cometbft/cometbft/pull/3406))

.changelog/v0.38.10/bug-fixes/486-p2p-max-outbound.md

@@ -0,0 +1,3 @@
+- `[p2p]` Node respects configured `max_num_outbound_peers` limit when dialing
+  peers provided by a seed node
+  ([\#486](https://github.com/cometbft/cometbft/issues/486))

.changelog/v0.38.10/improvements/3382-single-send-monitor-per-packet.md

@@ -0,0 +1,2 @@
+- `[p2p/conn]` Update send monitor, used for sending rate limiting, once per batch of packets sent
+  ([\#3382](https://github.com/cometbft/cometbft/pull/3382))

.changelog/v0.38.10/improvements/3401-allow-dash-in-event-tags.md

@@ -0,0 +1,2 @@
+- `[libs/pubsub]` Allow dash (`-`) in event tags
+  ([\#3401](https://github.com/cometbft/cometbft/issues/3401))

.changelog/v0.38.10/improvements/3403-remove-pool-buffer-usage-in-secretconn.md

@@ -0,0 +1,2 @@
+- `[p2p/conn]` Remove the usage of a synchronous pool of buffers in secret connection, storing instead the buffer in the connection struct. This reduces the synchronization primitive usage, speeding up the code.
+  ([\#3403](https://github.com/cometbft/cometbft/issues/3403))

.changelog/v0.38.10/summary.md

@@ -0,0 +1,5 @@
+*July 16, 2024*
+
+This release fixes a bug in `v0.38.x` that prevented ABCI responses from being
+correctly read when upgrading from `v0.37.x` or below. It also includes a few other
+bug fixes and performance improvements.

.changelog/v0.38.6/bug-fixes/2047-privval-retry-accepting-conn.md

@@ -0,0 +1 @@
+- `[privval]` Retry accepting a connection ([\#2047](https://github.com/cometbft/cometbft/pull/2047))

.changelog/v0.38.6/bug-fixes/2136-fix-state-rollback.md

@@ -0,0 +1,2 @@
+- `[state]` Fix rollback to a specific height
+  ([\#2136](https://github.com/cometbft/cometbft/pull/2136))

.changelog/v0.38.6/features/2362-e2e-block-max-bytes.md

@@ -0,0 +1,2 @@
+- `[e2e]` Add `block_max_bytes` option to the manifest file.
+  ([\#2362](https://github.com/cometbft/cometbft/pull/2362))

.changelog/v0.38.6/improvements/2016-blocksync-avoid-double-calling-block-from-proto.md

@@ -0,0 +1,2 @@
+- `[blocksync]` Avoid double-calling `types.BlockFromProto` for performance
+  reasons ([\#2016](https://github.com/cometbft/cometbft/pull/2016))

.changelog/v0.38.6/improvements/2094-e2e-load-max-txs.md

@@ -0,0 +1,2 @@
+- `[e2e]` Add manifest option `load_max_txs` to limit the number of transactions generated by the
+  `load` command. ([\#2094](https://github.com/cometbft/cometbft/pull/2094))

.changelog/v0.38.6/improvements/2434-jsonrpc-websocket-basic-auth.md

@@ -0,0 +1 @@
+- `[jsonrpc]` enable HTTP basic auth in websocket client ([#2434](https://github.com/cometbft/cometbft/pull/2434))

.changelog/v0.38.6/improvements/2467-decrease-n-of-requested-blocks.md

@@ -0,0 +1,3 @@
+- `[blocksync]` make the max number of downloaded blocks dynamic.
+  Previously it was a const 600. Now it's `peersCount * maxPendingRequestsPerPeer (20)`
+  [\#2467](https://github.com/cometbft/cometbft/pull/2467)

.changelog/v0.38.6/improvements/2475-blocksync-2nd-request.md

@@ -0,0 +1,3 @@
+- `[blocksync]` Request a block from peer B if we are approaching pool's height
+  (less than 50 blocks) and the current peer A is slow in sending us the
+  block [\#2475](https://github.com/cometbft/cometbft/pull/2475)

.changelog/v0.38.6/improvements/2475-blocksync-no-block-response.md

@@ -0,0 +1,3 @@
+- `[blocksync]` Request the block N from peer B immediately after getting
+  `NoBlockResponse` from peer A
+  [\#2475](https://github.com/cometbft/cometbft/pull/2475)

.changelog/v0.38.6/improvements/2475-blocksync-sort-peers.md

@@ -0,0 +1,2 @@
+- `[blocksync]` Sort peers by download rate (the fastest peer is picked first)
+  [\#2475](https://github.com/cometbft/cometbft/pull/2475)

.changelog/v0.38.6/summary.md

@@ -0,0 +1,5 @@
+*March 12, 2024*
+
+This release fixes a security bug in the light client. It also introduces many
+improvements to the block sync in collaboration with the
+[Osmosis](https://osmosis.zone/) team.

.changelog/v0.38.7/bug-fixes/2225-fix-checktx-request-returns-error.md

@@ -0,0 +1,2 @@
+- [`mempool`] Panic when a CheckTx request to the app returns an error
+  ([\#2225](https://github.com/cometbft/cometbft/pull/2225))

.changelog/v0.38.7/bug-fixes/2774-bitarray-unmarshal-json.md

@@ -0,0 +1,2 @@
+- [`bits`] prevent `BitArray.UnmarshalJSON` from crashing on 0 bits
+  ([\#2774](https://github.com/cometbft/cometbft/pull/2774))

.changelog/v0.38.7/features/2793-boostrap.md

@@ -0,0 +1,2 @@
+- [`node`] Add `BootstrapStateWithGenProvider` to boostrap state using a custom
+  genesis doc provider ([\#2793](https://github.com/cometbft/cometbft/pull/2793))

.changelog/v0.38.7/improvements/2839-tx_index-lower-heap-allocation.md

@@ -0,0 +1,2 @@
+- `[state/indexer]` Lower the heap allocation of transaction searches
+  ([\#2839](https://github.com/cometbft/cometbft/pull/2839))

.changelog/v0.38.7/improvements/2841-speedup-bits-pick-random.md

@@ -0,0 +1,2 @@
+- `[internal/bits]` 10x speedup and remove heap overhead of bitArray.PickRandom (used extensively in consensus gossip)
+  ([\#2841](https://github.com/cometbft/cometbft/pull/2841)).

.changelog/v0.38.7/improvements/2846-speedup-json-encoding.md

@@ -0,0 +1,2 @@
+- `[libs/json]` Lower the memory overhead of JSON encoding by using JSON encoders internally
+  ([\#2846](https://github.com/cometbft/cometbft/pull/2846)).

.changelog/v0.38.7/summary.md

@@ -0,0 +1,3 @@
+*April 26, 2024*
+
+This release contains a few bug fixes and performance improvements.

.changelog/v0.38.8/breaking-changes/3314-mempool-preupdate.md

@@ -0,0 +1,4 @@
+- `[mempool]` Add to the `Mempool` interface a new method `PreUpdate()`. This method should be
+  called before acquiring the mempool lock, to signal that a new update is coming. Also add to
+  `ErrMempoolIsFull` a new field `RecheckFull`.
+  ([\#3314](https://github.com/cometbft/cometbft/pull/3314))

.changelog/v0.38.8/bug-fixes/14-abc.md

@@ -0,0 +1,4 @@
+- `[blockstore]` Added peer banning in blockstore
+  ([\#ABC-0013](https://github.com/cometbft/cometbft/security/advisories/GHSA-hg58-rf2h-6rr7))
+- `[blockstore]` Send correct error message when vote extensions do not align with received packet
+  ([\#ABC-0014](https://github.com/cometbft/cometbft/security/advisories/GHSA-hg58-rf2h-6rr7))

.changelog/v0.38.8/bug-fixes/1827-fix-recheck-async.md

@@ -0,0 +1,2 @@
+- [`mempool`] Fix data race when rechecking with async ABCI client
+  ([\#1827](https://github.com/cometbft/cometbft/issues/1827))

.changelog/v0.38.8/bug-fixes/3092-consensus-timeout-ticker-data-race.md

@@ -0,0 +1,2 @@
+- `[consensus]` Fix a race condition in the consensus timeout ticker. Race is caused by two timeouts being scheduled at the same time.
+  ([\#3092](https://github.com/cometbft/cometbft/pull/2136))

.changelog/v0.38.8/bug-fixes/3195-batch-verification.md

@@ -0,0 +1,2 @@
+- `[types]` Do not batch verify a commit if the validator set keys have different
+  types. ([\#3195](https://github.com/cometbft/cometbft/issues/3195)

.changelog/v0.38.8/improvements/1827-config-mempool-recheck-timeout.md

@@ -0,0 +1,3 @@
+- `[config]` Added `recheck_timeout` mempool parameter to set how much time to wait for recheck
+  responses from the app (only applies to non-local ABCI clients).
+  ([\#1827](https://github.com/cometbft/cometbft/issues/1827/))

.changelog/v0.38.8/improvements/2867-rpc-batch-size-config.md

@@ -0,0 +1,2 @@
+- `[rpc]` Add a configurable maximum batch size for RPC requests.
+  ([\#2867](https://github.com/cometbft/cometbft/pull/2867)).

.changelog/v0.38.8/improvements/2911-remove-event-bus-debug-logs.md

@@ -0,0 +1,2 @@
+- `[event-bus]` Remove the debug logs in PublishEventTx, which were noticed production slowdowns.
+  ([\#2911](https://github.com/cometbft/cometbft/pull/2911))

.changelog/v0.38.8/improvements/2924-consensus-cache-block-hash.md

@@ -0,0 +1,2 @@
+- `[state/execution]` Cache the block hash computation inside of the Block Type, so we only compute it once.
+  ([\#2924](https://github.com/cometbft/cometbft/pull/2924))

.changelog/v0.38.8/improvements/2928-remove-redundant-verifyblock-call-in-finalize-commit.md

@@ -0,0 +1,2 @@
+- `[consensus/state]` Remove a redundant `VerifyBlock` call in `FinalizeCommit`
+  ([\#2928](https://github.com/cometbft/cometbft/pull/2928))

.changelog/v0.38.8/improvements/2949-reduce-protoio-writer-creation-time.md

@@ -0,0 +1,2 @@
+- `[p2p/channel]` Speedup `ProtoIO` writer creation time, and thereby speedup channel writing by 5%.
+  ([\#2949](https://github.com/cometbft/cometbft/pull/2949))

.changelog/v0.38.8/improvements/2952-lower-next-packet-msg-time.md

@@ -0,0 +1,2 @@
+- `[p2p/conn]` Minor speedup (3%) to connection.WritePacketMsgTo, by removing MinInt calls.
+  ([\#2952](https://github.com/cometbft/cometbft/pull/2952))

.changelog/v0.38.8/improvements/2959-speedup-initialized-bitarray-construction.md

@@ -0,0 +1,2 @@
+- `[internal/bits]` 10x speedup creating initialized bitArrays, which speedsup extendedCommit.BitArray(). This is used in consensus vote gossip.
+  ([\#2959](https://github.com/cometbft/cometbft/pull/2841)).

.changelog/v0.38.8/improvements/2964-skip-revalidation-of-blockstore-LoadBlockFromMeta-.md

@@ -0,0 +1,2 @@
+- `[blockstore]` Remove a redundant `Header.ValidateBasic` call in `LoadBlockMeta`, 75% reducing this time.
+  ([\#2964](https://github.com/cometbft/cometbft/pull/2964))

.changelog/v0.38.8/improvements/2986-lower-memory-allocation-in-packet-writing.md

@@ -0,0 +1,2 @@
+- `[p2p/conn]` Speedup connection.WritePacketMsgTo, by reusing internal buffers rather than re-allocating.
+  ([\#2986](https://github.com/cometbft/cometbft/pull/2986))

.changelog/v0.38.8/improvements/3003-use-lru-caches-in-blockstore.md

@@ -0,0 +1,2 @@
+- [`blockstore`] Use LRU caches in blockstore, significiantly improving consensus gossip routine performance
+  ([\#3003](https://github.com/cometbft/cometbft/issues/3003)

.changelog/v0.38.8/improvements/3017-speedup-consensus-metrics.md

@@ -0,0 +1,2 @@
+- [`consensus`] Improve performance of consensus metrics by lowering string operations
+  ([\#3017](https://github.com/cometbft/cometbft/issues/3017)

.changelog/v0.38.8/improvements/3019-reduce-allocations-in-packet-reads.md

@@ -0,0 +1,3 @@
+- [`protoio`] Remove one allocation and new object call from `ReadMsg`,
+  leading to a 4% p2p message reading performance gain.
+  ([\#3018](https://github.com/cometbft/cometbft/issues/3018)

.changelog/v0.38.8/improvements/3314-mempool-update-consider-full-when-rechecking.md

@@ -0,0 +1,3 @@
+- `[mempool]` Before updating the mempool, consider it as full if rechecking is still in progress.
+  This will stop accepting transactions in the mempool if the node can't keep up with re-CheckTx.
+  ([\#3314](https://github.com/cometbft/cometbft/pull/3314))

.changelog/v0.38.8/summary.md

@@ -0,0 +1,3 @@
+*June 27, 2024*
+
+This release contains a few bug fixes and performance improvements.

.changelog/v0.38.9/breaking-changes/3361-mempool-preupdate.md

@@ -0,0 +1,4 @@
+- `[mempool]` Revert adding the method `PreUpdate()` to the `Mempool` interface, recently introduced
+  in the previous patch release (v0.38.8). Its logic is now moved into the `Lock` method. With this change,
+  the `Mempool` interface is the same as in v0.38.7.
+  ([\#3361](https://github.com/cometbft/cometbft/pull/3361))

.changelog/v0.38.9/bug-fixes/3352-nil-pointer-tx-search.md

@@ -0,0 +1,2 @@
+- `[rpc]` Fix nil pointer error in `/tx` and `/tx_search` when block is
+  absent ([\#3352](https://github.com/cometbft/cometbft/issues/3352))

.changelog/v0.38.9/summary.md

@@ -0,0 +1,5 @@
+*July 1, 2024*
+
+This release reverts the API-breaking change to the Mempool interface introduced in the last patch
+release (v0.38.8) while still keeping the performance improvement added to the mempool. It also
+includes a minor fix to the RPC endpoints /tx and /tx_search.

.github/workflows/check-generated.yml

@@ -13,7 +13,7 @@ permissions:
   contents: read
 
 jobs:
-  check-mocks:
+  check-mocks-metrics:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/setup-go@v5
@@ -22,17 +22,18 @@ jobs:
 
       - uses: actions/checkout@v4
 
-      - name: "Check generated mocks"
+      - name: "Check generated mocks and metrics"
         run: |
           set -euo pipefail
 
-          make mockery
+          make mockery metrics
 
-          if ! git diff --stat --exit-code ; then
+          git add .
+          if ! git diff HEAD --stat --exit-code ; then
             echo ">> ERROR:"
             echo ">>"
-            echo ">> Generated mocks require update (either Mockery or source files may have changed)."
-            echo ">> Ensure your tools are up-to-date, re-run 'make mockery' and update this PR."
+            echo ">> Generated mocks and/or metrics require update (either Mockery or source files may have changed)."
+            echo ">> Ensure your tools are up-to-date, re-run 'make mockery metrics' and update this PR."
             echo ">>"
             git diff
             exit 1
@@ -55,7 +56,8 @@ jobs:
 
           make proto-gen
 
-          if ! git diff --stat --exit-code ; then
+          git add .
+          if ! git diff HEAD --stat --exit-code ; then
             echo ">> ERROR:"
             echo ">>"
             echo ">> Protobuf generated code requires update (either tools or .proto files may have changed)."

.github/workflows/cometbft-docker.yml

@@ -41,17 +41,17 @@ jobs:
           platforms: all
 
       - name: Set up Docker Build
-        uses: docker/setup-buildx-action@v3.0.0
+        uses: docker/setup-buildx-action@v3.4.0
 
       - name: Login to DockerHub
         if: ${{ github.event_name != 'pull_request' }}
-        uses: docker/login-action@v3.0.0
+        uses: docker/login-action@v3.2.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Publish to Docker Hub
-        uses: docker/build-push-action@v5.1.0
+        uses: docker/build-push-action@v6.3.0
         with:
           context: .
           file: ./DOCKER/Dockerfile

.github/workflows/e2e-manual-multiversion.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        group: ['00', '01', '02', '03', '04']
+        group: ['00', '01', '02', '03', '04', '05']
     runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
@@ -27,12 +27,19 @@ jobs:
         run: make -j2 docker generator runner tests
 
       - name: Generate testnets
+        if: matrix.group != 5
         working-directory: test/e2e
         # When changing -g, also change the matrix groups above
         # Generate multi-version tests with double the quantity of E2E nodes
         # based on the current branch as compared to the latest version.
         run: ./build/generator -g 5 -m "latest:1,local:2" -d networks/nightly/ -p
 
-      - name: Run ${{ matrix.p2p }} p2p testnets
+      - name: Run p2p testnets (${{ matrix.group }})
+        if: matrix.group != 5
         working-directory: test/e2e
         run: ./run-multiple.sh networks/nightly/*-group${{ matrix.group }}-*.toml
+
+      - name: Run p2p testnets (regression)
+        if: matrix.group == 5
+        working-directory: test/e2e
+        run: ./run-multiple.sh networks_regressions/*.toml

.github/workflows/e2e-manual.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        group: ['00', '01', '02', '03', '04']
+        group: ['00', '01', '02', '03', '04', '05']
     runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
@@ -27,10 +27,17 @@ jobs:
         run: make -j2 docker generator runner tests
 
       - name: Generate testnets
+        if: matrix.group != 5
         working-directory: test/e2e
         # When changing -g, also change the matrix groups above
         run: ./build/generator -g 5 -d networks/nightly/ -p
 
-      - name: Run ${{ matrix.p2p }} p2p testnets
+      - name: Run p2p testnets (${{ matrix.group }})
+        if: matrix.group != 5
         working-directory: test/e2e
         run: ./run-multiple.sh networks/nightly/*-group${{ matrix.group }}-*.toml
+
+      - name: Run p2p testnets (regression)
+        if: matrix.group == 5
+        working-directory: test/e2e
+        run: ./run-multiple.sh networks_regressions/*.toml

.github/workflows/e2e-nightly-34x.yml

@@ -55,7 +55,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Notify Slack on failure
-        uses: slackapi/slack-github-action@v1.24.0
+        uses: slackapi/slack-github-action@v1.26.0
         env:
           SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
           SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

.github/workflows/e2e-nightly-37x.yml

@@ -55,7 +55,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Notify Slack on failure
-        uses: slackapi/slack-github-action@v1.24.0
+        uses: slackapi/slack-github-action@v1.26.0
         env:
           SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
           SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

.github/workflows/e2e-nightly-main.yml

@@ -16,7 +16,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        group: ['00', '01', '02', '03', "04"]
+        group: ['00', '01', '02', '03', '04', '05']
     runs-on: ubuntu-latest
     timeout-minutes: 60
     steps:
@@ -32,21 +32,28 @@ jobs:
         run: make -j2 docker generator runner tests
 
       - name: Generate testnets
+        if: matrix.group != 5
         working-directory: test/e2e
         # When changing -g, also change the matrix groups above
         run: ./build/generator -g 5 -d networks/nightly/ -p
 
-      - name: Run ${{ matrix.p2p }} p2p testnets
+      - name: Run p2p testnets (${{ matrix.group }})
+        if: matrix.group != 5
         working-directory: test/e2e
         run: ./run-multiple.sh networks/nightly/*-group${{ matrix.group }}-*.toml
 
+      - name: Run p2p testnets (regression)
+        if: matrix.group == 5
+        working-directory: test/e2e
+        run: ./run-multiple.sh networks_regressions/*.toml
+
   e2e-nightly-fail:
     needs: e2e-nightly-test
     if: ${{ failure() }}
     runs-on: ubuntu-latest
     steps:
       - name: Notify Slack on failure
-        uses: slackapi/slack-github-action@v1.24.0
+        uses: slackapi/slack-github-action@v1.26.0
         env:
           SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
           SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

.github/workflows/fuzz-nightly.yml

@@ -77,7 +77,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Notify Slack on failure
-        uses: slackapi/slack-github-action@v1.24.0
+        uses: slackapi/slack-github-action@v1.26.0
         env:
           SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
           SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

.github/workflows/janitor.yml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 3
     steps:
-      - uses: styfle/[email protected].0
+      - uses: styfle/[email protected].1
         with:
           workflow_id: 1041851,1401230,2837803
           access_token: ${{ github.token }}

.github/workflows/lint.yml

@@ -29,7 +29,7 @@ jobs:
             **/**.go
             go.mod
             go.sum
-      - uses: golangci/golangci-lint-action@v3
+      - uses: golangci/golangci-lint-action@v6
         with:
           version: latest
           args: --timeout 10m