Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: avoid duplicated batch updates #1029

Merged

Conversation

colinlyguo
Copy link
Member

@colinlyguo colinlyguo commented Dec 8, 2023

Purpose or design rationale of this PR

Currently, the batch index information is not updated repeatedly within the buffer until the arrival of the next message. In situations where messages are sparse, this could significantly reduce the speed of the L1 fetcher.

An example (in block 4259 to block 4345 there is no message):

...
INFO [12-08|14:18:42.315] update batch info of L2 withdrawals      index=483 start=4259 end=4263
INFO [12-08|14:18:42.317] update batch info of L2 withdrawals      index=484 start=4264 end=4271
INFO [12-08|14:18:42.319] update batch info of L2 withdrawals      index=485 start=4272 end=4276
INFO [12-08|14:18:42.321] update batch info of L2 withdrawals      index=486 start=4277 end=4282
INFO [12-08|14:18:42.323] update batch info of L2 withdrawals      index=487 start=4283 end=4287
INFO [12-08|14:18:42.325] update batch info of L2 withdrawals      index=488 start=4288 end=4292
INFO [12-08|14:18:42.327] update batch info of L2 withdrawals      index=489 start=4293 end=4297
INFO [12-08|14:18:42.329] update batch info of L2 withdrawals      index=490 start=4298 end=4305
INFO [12-08|14:18:42.331] update batch info of L2 withdrawals      index=491 start=4306 end=4310
INFO [12-08|14:18:42.333] update batch info of L2 withdrawals      index=492 start=4311 end=4315
INFO [12-08|14:18:42.335] update batch info of L2 withdrawals      index=493 start=4316 end=4320
INFO [12-08|14:18:42.337] update batch info of L2 withdrawals      index=494 start=4321 end=4325
INFO [12-08|14:18:42.339] update batch info of L2 withdrawals      index=495 start=4326 end=4330
INFO [12-08|14:18:42.341] update batch info of L2 withdrawals      index=496 start=4331 end=4335
INFO [12-08|14:18:42.343] update batch info of L2 withdrawals      index=497 start=4336 end=4340
INFO [12-08|14:18:42.345] update batch info of L2 withdrawals      index=498 start=4341 end=4345
INFO [12-08|14:18:42.347] fetch and save L1 events                 from=4,060,146 to=4,060,175

PR title

Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:

  • fix: A bug fix

Deployment tag versioning

Has tag in common/version.go been updated or have you added bump-version label to this PR?

  • No, this PR doesn't involve a new deployment, git tag, docker image tag

Breaking change label

Does this PR have the breaking-change label?

  • No, this PR is not a breaking change

@colinlyguo colinlyguo requested a review from georgehao December 8, 2023 08:05
Copy link

codecov bot commented Dec 8, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ No coverage uploaded for pull request base (fix-bridge-history-api-write-db@2f1abc0). Click here to learn what that means.

Additional details and impacted files
@@                        Coverage Diff                         @@
##             fix-bridge-history-api-write-db    #1029   +/-   ##
==================================================================
  Coverage                                   ?   56.27%           
==================================================================
  Files                                      ?       88           
  Lines                                      ?     9000           
  Branches                                   ?        0           
==================================================================
  Hits                                       ?     5065           
  Misses                                     ?     3516           
  Partials                                   ?      419           
Flag Coverage Δ
bridge-history-api 78.83% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bridge-history-api/Makefile Outdated Show resolved Hide resolved
bridge-history-api/README.md Outdated Show resolved Hide resolved
bridge-history-api/README.md Outdated Show resolved Hide resolved
bridge-history-api/internal/logic/history_logic.go Outdated Show resolved Hide resolved
Comment on lines 304 to 314
// Perform a final check to confirm the existence of the key to ensure consistency between ZCard and ZRange data reads.
exists, err := h.redis.Exists(ctx, cacheKey).Result()
if err != nil {
log.Error("failed to check if key exists", "error", err)
return nil, 0, false, err
}

// If the key does not exist, we consider it a cache miss and return accordingly.
if exists == 0 {
return nil, 0, false, nil
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is no necessary

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's necessary because if the key does not exist, it will return nil value, and cannot differentiate:

  1. cache key exists but the value is nil.
  2. cache key does not exist.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think just check total or value whether empty

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. but if the txs under the address are empty, it would be treated as "key not found", too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed in 1d325cb

@georgehao georgehao merged commit 3676368 into fix-bridge-history-api-write-db Dec 11, 2023
10 checks passed
@georgehao georgehao deleted the feat-reduce-duplicated-batches-update branch December 11, 2023 06:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants