[How to use snap sync on sepolia] <snap sync on sepolia full node running forever>

[karta020500]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

I have been running Sepolia snap sync for a week. I expected the snap sync to be much faster, taking only a couple of days.

Protocol Description

op-geth version: v1.101315.0
op-node version: v1.7.5
network name: sepolia

Node Logs

op-node:

t=2024-06-26T07:54:36+0000 lvl=info msg="Received signed execution payload from p2p" id=0x1a2095125b6d085ba84857a0931a329daccf93bdfdc7b500198b0916688119fb:13792968 peer=16Uiu2HAmPoBKuQkThkP8kMR7iRayNbvHNVRD34iCJCUYaY9z7h3j
t=2024-06-26T07:54:36+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0x1a2095125b6d085ba84857a0931a329daccf93bdfdc7b500198b0916688119fb:13792968
t=2024-06-26T07:54:36+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer while in EL sync" l2_finalized=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_pending_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_unsafe=0x1a2095125b6d085ba84857a0931a329daccf93bdfdc7b500198b0916688119fb:13792968 l2_time=1719388476 l1_derived=0x0000000000000000000000000000000000000000000000000000000000000000:0
t=2024-06-26T07:54:39+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmANSn75bRzVGaaFk8kHedn1WAN1D8QwKKLM9QSvHaEXYD
t=2024-06-26T07:54:39+0000 lvl=info msg="Received signed execution payload from p2p" id=0x61d1c2b86ab4f116435116a9ed497878c04c610975977a3ef490e0736e12ce69:13792969 peer=16Uiu2HAmLRJHcXJ1E8Va1EQ5VQWwav2eLHDb61xFcRzd43kDnpix
t=2024-06-26T07:54:39+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0x61d1c2b86ab4f116435116a9ed497878c04c610975977a3ef490e0736e12ce69:13792969
t=2024-06-26T07:54:39+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer while in EL sync" l2_finalized=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_pending_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_unsafe=0x61d1c2b86ab4f116435116a9ed497878c04c610975977a3ef490e0736e12ce69:13792969 l2_time=1719388478 l1_derived=0x0000000000000000000000000000000000000000000000000000000000000000:0
t=2024-06-26T07:54:41+0000 lvl=info msg="Received signed execution payload from p2p" id=0xdef82a96613c2dfff7e26a6c1dccc3a95865409040057e43df05a4c32e8d5633:13792970 peer=16Uiu2HAkxiSrQAm1F9W4CbAw3gc35thh2JXnxX7ZvFyjXymATKnX
t=2024-06-26T07:54:41+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0xdef82a96613c2dfff7e26a6c1dccc3a95865409040057e43df05a4c32e8d5633:13792970
t=2024-06-26T07:54:41+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer while in EL sync" l2_finalized=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_pending_safe=0x00000000000000000000

op-geth:
INFO [06-26|07:55:20.590] Forkchoice requested sync to new head number=13,792,990 hash=96a489..ec0c8a INFO [06-26|07:55:22.588] Forkchoice requested sync to new head number=13,792,991 hash=9934e9..5e886c INFO [06-26|07:55:24.899] Forkchoice requested sync to new head number=13,792,992 hash=e51606..63a460 INFO [06-26|07:55:26.587] Forkchoice requested sync to new head number=13,792,993 hash=61920f..dc2a48 INFO [06-26|07:55:28.590] Forkchoice requested sync to new head number=13,792,994 hash=1eaffb..8aef84 INFO [06-26|07:55:30.821] Forkchoice requested sync to new head number=13,792,995 hash=5cadf5..3cc379 INFO [06-26|07:55:32.821] Forkchoice requested sync to new head number=13,792,996 hash=c9a932..03247a INFO [06-26|07:55:34.818] Forkchoice requested sync to new head number=13,792,997 hash=a68c36..3be108

Additional Information

op-geth running script:

cd /root/op-geth

nohup ./build/bin/geth \
  --datadir=./datadir \
  --http \
  --http.port=8545 \
  --http.addr=0.0.0.0 \
  --authrpc.vhosts="*" \
  --authrpc.addr=localhost \
  --authrpc.port=8551 \
  --authrpc.jwtsecret=./jwt.txt \
  --verbosity=3 \
  --rollup.sequencerhttp=$SEQUENCER_URL \
  --maxpeers=0 \
  --http.api=web3,debug,eth,txpool,net,engine \
  --ws \
  --ws.addr=0.0.0.0 \
  --ws.port=8546 \
  --ws.origins="*" \
  --ws.api=debug,eth,txpool,net,engine \
  --rpc.batch-request-limit=2000 \
  --nodiscover \
  --txpool.accountqueue=64 \
  --txpool.accountslots=16 \
  --txpool.globalqueue=1024 \
  --txpool.globalslots=5120 \
  --op-network=op-sepolia >> /root/logs/geth.log 2>&1 &

op-node running script:

 nohup ./bin/op-node \
  --l1=$L1_RPC_URL \
  --l1.rpckind=$L1_RPC_KIND \
  --l1.beacon=$L1_BEACON_URL \
  --l2=ws://localhost:8551 \
  --l2.jwt-secret=./jwt.txt \
  --network=op-sepolia \
  --syncmode=execution-layer >> /root/logs/node.log 2>&1 &

Feedback

No response

[opfocus]

try to remove --nodiscover \ --maxpeers=0 \

[karta020500]

ok! i will keep waiting!

[karta020500]

still looking for peers seems like no progress....

[karta020500]

any sepolia snapshot i can use?

[opfocus]

INFO [06-28|01:55:55.301] Looking for peers peercount=0 tried=87 static=0

This confuses me a bit, I've run the op-sepolia node exactly according to the documentation before, and it's not difficult to find the peers. https://docs.optimism.io/builders/node-operators/tutorials/testnet Can you rerun it following the documentation

[karta020500]

after i changed the op-geth version to v1.101315.2 and op-node version to v1.7.7, snap sync now works!

[dev-jb]

I have same error log on non-sequencer op-node. But on op-geth, the sync hung up to 99.98%. The sync restarted after long l2 re-org due to op-batcher out of gas issue. I am having trouble to get it fully synced. Any solution to get non sequencer node to fully sync?

Logs:
op-geth:

INFO [07-26|06:00:22.730] Syncing: chain download in progress synced=99.98% chain=154.86MiB headers=389,[email protected] bodies=389,[email protected] receipts=389,[email protected] eta=487.091ms
WARN [07-26|06:00:22.789] Ignoring payload while snap syncing number=389,194 hash=a0efc2..f35c09 reason="chain gapped, head: 389192, newHead: 389194"
INFO [07-26|06:00:22.793] Forkchoice requested sync to new head number=389,194 hash=a0efc2..f35c09
INFO [07-26|06:00:22.793] Restarting sync cycle reason="chain gapped, head: 389192, newHead: 389194"
WARN [07-26|06:00:22.800] Ignoring payload while snap syncing number=389,195 hash=6be786..be31ef reason="beacon syncer reorging"
INFO [07-26|06:00:22.802] Forkchoice requested sync to new head number=389,195 hash=6be786..be31ef
WARN [07-26|06:00:22.802] Served engine_forkchoiceUpdatedV3 conn=172.19.0.10:53662 reqid=2188 duration="300.704µs" err="beacon syncer reorging"
ERROR[07-26|06:00:23.736] Beacon backfilling failed err="syncing canceled (requested)"
ERROR[07-26|06:00:23.767] Cleaning spurious beacon sync leftovers stale_head_1=389,192 stale_tail_1=389,108 stale_next_1=eb40ff..dc2ff9
INFO [07-26|06:00:23.771] Syncing beacon headers downloaded=393,861 left=0 eta=0s
INFO [07-26|06:00:23.779] Disabling direct-ancient mode origin=389,107 ancient=298,117
WARN [07-26|06:00:24.303] Ignoring payload while snap syncing number=389,196 hash=b7a2c4..626017 reason="chain gapped, head: 389194, newHead: 389196"
INFO [07-26|06:00:24.307] Forkchoice requested sync to new head number=389,196 hash=b7a2c4..626017
INFO [07-26|06:00:24.307] Restarting sync cycle reason="chain gapped, head: 389194, newHead: 389196"
ERROR[07-26|06:00:24.822] Beacon backfilling failed err="syncing canceled (requested)"
ERROR[07-26|06:00:24.852] Cleaning spurious beacon sync leftovers stale_head_1=389,194 stale_tail_1=389,130 stale_next_1=9e2159..6ee6df
INFO [07-26|06:00:24.856] Syncing beacon headers downloaded=393,927 left=0 eta=0s

op-node:

t=2024-07-26T05:58:32+0000 lvl=info msg="Received signed execution payload from p2p" id=0xf54ebda6acca37a7f8a76a39bbd7623abe64fc8357cb9e7b087ddd2074125071:389140 peer=16Uiu2HAm3jCJmCPobN4Gqy827gDKqV67wne2EhxB5PdCHoYxRvZc
t=2024-07-26T05:58:32+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0xf54ebda6acca37a7f8a76a39bbd7623abe64fc8357cb9e7b087ddd2074125071:389140
t=2024-07-26T05:58:32+0000 lvl=info msg="Received signed execution payload from p2p" id=0xfb60a7621c8de2b513a55a1dbde8546eca19c8db51c05bc4efda43b1aa99c869:389139 peer=16Uiu2HAm3jCJmCPobN4Gqy827gDKqV67wne2EhxB5PdCHoYxRvZc
t=2024-07-26T05:58:32+0000 lvl=info msg="Sync progress" reason="new chain head block" l2_finalized=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_pending_safe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_unsafe=0xf54ebda6acca37a7f8a76a39bbd7623abe64fc8357cb9e7b087ddd2074125071:389140 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1721973512
t=2024-07-26T05:58:34+0000 lvl=info msg="Received signed execution payload from p2p" id=0x7b2db440dec6136e1a610736e7a200088c34bcad824b4537657c94234f96e00a:389141 peer=16Uiu2HAm3jCJmCPobN4Gqy827gDKqV67wne2EhxB5PdCHoYxRvZc
t=2024-07-26T05:58:34+0000 lvl=info msg="Optimistically inserting unsafe L2 execution payload to drive EL sync" id=0x7b2db440dec6136e1a610736e7a200088c34bcad824b4537657c94234f96e00a:389141

[opfocus]

You run an op stack testnet and then add a replica node. Your replica node cannot be fully synchronized with the sequencer node, right?

[opfocus]

If your batcher is interrupted for more than 12 hours, serious problems will occur with L2 and the replica nodes will not be fully synchronized.

[opfocus]

In that case, I suggest you stop your chain, delete datadir, regenerate getting-start.json, then regenerate genesis.json, rollup.json. Re-initialize op-geth. This will reuse the deployed L1 contracts to deploy a new chain. This is my opinion

[dev-jb]

In that case, I assume all the data on the chain will be lost?

[opfocus]

The sequencer node was on re-org state where logs showed payload is old on op-node. Have you performed an Ecotone upgrade on your op stack testnet?

[opfocus]

Sorry, I'm not sure if what I said is right. This is a bit of a waste of your time. And I can't find the right person to help you.
Debugging itself is not an easy task either