OP-Stack node falling behind frequently for a short duration

[abbaahmad]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

We run Lisk mainnet nodes and both nodes frequently lag at almost the same time for a short duration at a time, multiple times a day. We run Lisk Sepolia nodes as well as several other Op-stack based nodes and we're not seeing similar behavior we them either and we're not seeing similar behavior. Lisk dev team asked we direct such a question here.

Steps to Reproduce

Deploy an op-stack node with the following arguments:
op-geth service-args:

--ws --ws.addr=0.0.0.0 --ws.origins="*" --http --http.addr=0.0.0.0 --http.vhosts="*" --http.corsdomain="*" --authrpc.addr=localhost --authrpc.jwtsecret=/home/op/jwtsecret --authrpc.port=8551 --authrpc.vhosts="*" --datadir=/home/op/.datadir --rollup.sequencerhttp=https://rpc.api.lisk.com --rollup.halt=major --rollup.superchain-upgrades --gcmode=archive --syncmode=full --http.api=eth,net,web3,debug --http.port=8545 --ws.port=8546 --snapshot=true --cache=4096 --metrics --metrics.influxdb --metrics.influxdb.database=op-stack --metrics.influxdb.username=op-stack --metrics.influxdb.password=<influxdb-password> --metrics.influxdb.endpoint=<influxdb-ip-addr:port> --maxpeers=50 --rollup.disabletxpoolgossip=true

op-node service-args:

--l2.jwt-secret=/home/op/jwtsecret --l2=ws://localhost:8551 --rpc.addr=0.0.0.0 --metrics.enabled --metrics.addr=0.0.0.0 --metrics.port=7300 --l1="<l1-rpc-ip-addr:port>" --l1.beacon="<l1-beacon-rpc-addr:port>" --rollup.config=/home/op/rollup.json --rpc.port=9545

genesis-file: https://raw.githubusercontent.com/LiskHQ/lisk-node/main/mainnet/genesis.json
rollup-config: https://raw.githubusercontent.com/LiskHQ/lisk-node/main/mainnet/rollup.json
op-geth-version: v1.101408.0
op-node-version: v1.9.1

Which component is affected?

• op-geth
• op-node
• Other

Expected vs. Actual Behavior

Expected behavior: Node to keep up with chain head
Actual behavior: Node frequently lags before catching up to chain head
Y-axis:- number of blocks behind rpc.api.lisk.com

Environment Details

Ubuntu: 22.04
Hardware: 8 CPUs, 32 GB RAM, >1TB Storage

Protocol Information

op-geth-version: v1.101408.0
op-node-version: v1.9.1
network-name: Lisk
chain ID: 1135
L1 chain: Ethereum

Node Logs

No response

Troubleshooting Attempts

No response

Additional Information

No response

Feedback

No response

[hash1go]

Hello.
We have escalated this issue to the OP team, so please wait for a while.