moving backfilled oracle blocking start work to background thread #1201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
patrickhuie19
requested review from
mateusz-sekara,
makramkd,
dimkouv and
matYang
dimkouv
approved these changes
Jul 19, 2024
| @@ -175,7 +180,6 @@ func (r *ChainAgnosticBackFilledOracle) Start(ctx context.Context) error { | |||
| } | |||
| if err := ctx.Err(); err != nil { | |||
There was a problem hiding this comment.
Not 100% related to this PR but we could do here:
eg := new(errgroup.Group)
// Replay in parallel if both requested.
eg.Go(func() error {
s := time.Now()
if err := r.srcProvider.Start(ctx); err != nil {
return err
}
r.lggr.Infow("finished replaying src chain", "time", time.Since(s))
return nil
})
eg.Go(func() error {
s := time.Now()
if err := r.dstProvider.Start(ctx); err != nil {
return err
}
r.lggr.Infow("finished replaying dst chain", "time", time.Since(s))
return nil
})
if err := eg.Wait(); err != nil {
r.lggr.Criticalw("unexpected error replaying, continuing plugin boot without all the logs backfilled", "err", err)
}
| @@ -141,7 +141,12 @@ type ChainAgnosticBackFilledOracle struct { | |||
| } | |||
|
|
|||
| func (r *ChainAgnosticBackFilledOracle) Start(ctx context.Context) error { | |||
| ctx, cancelFn := context.WithCancel(context.Background()) | |||
| go r.run(ctx) | |||
There was a problem hiding this comment.
We need to make sure that the caller of Start() does not expect to have something ready.
e.g. The caller might expect that the blocks are replayed before moving on.
| @@ -141,7 +141,12 @@ type ChainAgnosticBackFilledOracle struct { | |||
| } | |||
|
|
|||
| func (r *ChainAgnosticBackFilledOracle) Start(ctx context.Context) error { | |||
There was a problem hiding this comment.
Are we 100% sure that we can keep the caller's ctx while the process is running in the background?
Maybe we want to do ctx, cancelFn := context.WithCancel(context.Background()), maybe caller's context is cancelled after Start responds.
mateusz-sekara
approved these changes
Jul 24, 2024
|
mateusz-sekara
added a commit
that referenced
this pull request
Jul 24, 2024
## Solution It adds the missing part to this PR #1201 We can't pass the foreground routine context to the task run in the background and could take minutes or hours to finish. Instead of that, we create a background context and expose cancelFn. This is exactly how it used to work before introducing loopp layer
asoliman92
pushed a commit
that referenced
this pull request
Jul 31, 2024
) ## Motivation [Old impl](https://github.com/smartcontractkit/ccip/blob/7d2cae3660bac29ba9dd48b03d88a761c154d6ae/core/services/ocr2/plugins/ccip/internal/oraclelib/backfilled_oracle.go#L41) of backfilled oracle would run log poller in a non-blocking fashion on job creation. The new implementation has it blocking on job creation, which is impacting load testing. ## Solution Move back to the old background thread pattern.
asoliman92
pushed a commit
that referenced
this pull request
Jul 31, 2024
## Solution It adds the missing part to this PR #1201 We can't pass the foreground routine context to the task run in the background and could take minutes or hours to finish. Instead of that, we create a background context and expose cancelFn. This is exactly how it used to work before introducing loopp layer
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Old impl of backfilled oracle would run log poller in a non-blocking fashion on job creation. The new implementation has it blocking on job creation, which is impacting load testing.
Solution
Move back to the old background thread pattern.