Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(dedupe): run dedupe only for repositories found at startup #1844

Merged

Conversation

eusebiu-constantin-petu-dbk
Copy link
Collaborator

no need to run dedupe/restore blobs for images being pushed or synced while running dedupe task, they are already deduped/restored inline.

What type of PR is this?
bug

Which issue does this PR fix:
#1843

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@eusebiu-constantin-petu-dbk eusebiu-constantin-petu-dbk force-pushed the startup_dedupe branch 3 times, most recently from 7da603e to 61fbbbe Compare September 22, 2023 13:41
andaaron
andaaron previously approved these changes Sep 22, 2023
@codecov
Copy link

codecov bot commented Sep 22, 2023

Codecov Report

Merging #1844 (41e9702) into main (92e382c) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1844      +/-   ##
==========================================
- Coverage   92.03%   92.03%   -0.01%     
==========================================
  Files         146      146              
  Lines       26465    26492      +27     
==========================================
+ Hits        24358    24381      +23     
- Misses       1556     1559       +3     
- Partials      551      552       +1     
Files Coverage Δ
pkg/storage/common/common.go 92.90% <100.00%> (-0.60%) ⬇️
pkg/storage/gc/gc.go 90.92% <100.00%> (-0.89%) ⬇️
pkg/storage/imagestore/imagestore.go 87.00% <100.00%> (+0.08%) ⬆️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@rchincha
Copy link
Contributor

Pls fix the conflict.

@rchincha
Copy link
Contributor

Pls fix conflicts.

no need to run dedupe/restore blobs for images being pushed or synced while
running dedupe task, they are already deduped/restored inline.

Signed-off-by: Petu Eusebiu <[email protected]>
/* at first run get from storage currently found repositories so that we skip the ones that gets synced/uploaded
while this generator runs, there are deduped/restored inline, no need to run dedupe/restore again */
if len(gen.repos) == 0 {
gen.repos, err = gen.ImgStore.GetRepositories()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This list could be very large potentially

Copy link
Contributor

@rchincha rchincha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@rchincha rchincha merged commit c3801dc into project-zot:main Sep 27, 2023
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Make sure the background dedupe generator runs only on the repos present when zot starts
3 participants