Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd:chunked refactoring for early review #2503

Merged
merged 6 commits into from
Aug 6, 2024

Conversation

mtrmac
Copy link
Collaborator

@mtrmac mtrmac commented Aug 5, 2024

This is a subset of #2487 , ready for review; it doesn’t make that much sense separately, but I’m filing it early to allow earlier review / merging, and to decrease the size of the later review. See #2487 for more on how this is intended to be used.

This is:

  • One mostly unrelated bug fix
  • A set of refactoring of the BlobInfoCache implementations, so that compression matching decisions are centralized in a single place.
  • A change to correctly use BaseVariantName instead of hard-coding a zstd:chunked exception.

@giuseppe PTAL.

mtrmac added a commit to mtrmac/libpod that referenced this pull request Aug 5, 2024
@mtrmac
Copy link
Collaborator Author

mtrmac commented Aug 5, 2024

Podman test PR: containers/podman#23506

if v2Options == nil {
return true, types.PreserveOriginal, nil // Anything goes. The (compressionOp, compressionAlgo) values are not used.
return &CandidateTemplate{ // Anything goes. The CompressionOperation, CompressionAlgorithm values are not used.
Copy link
Contributor

@kolyshkin kolyshkin Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. If compression{Type,Algorithm} fields are not used, it does not make much sense to initialize them explicitly -- we can just omit those and rely on default values assigned by Go (those seems to be nil for algo and types.PreserveOriginal for op).

  2. nit: s/Compression/compression/.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thanks!

@mtrmac mtrmac force-pushed the chunked-bic2-partial branch from 351bb81 to 5bb9639 Compare August 5, 2024 17:26
Comment on lines 115 to 117
Location: types.BICLocationReference{Opaque: ""},
},
LastSeen: time.Time{},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: these initializations may be omitted.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer being explicit here — especially about the fact that Time is not set.

// digest just because some remote author claims so (e.g. because a manifest says so);
// RecordDigestCompressorData records data for the blob with the specified digest.
// WARNING: Only call this with LOCALLY VERIFIED data:
// - don’t record a compressor for a digest just because some remote author claims so
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if you want this to be rendered as a list item in godoc, it needs to be indented. If you don't, maybe do not make it look like a list.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This is going to be a two-item list with #2487 .)

Godoc doesn’t show internal symbols, and seems not to format documentation of interface methods at all.

Anyway, approximately 0% of current doc strings are currently formatted to benefit from “the new” syntax, and we have to start somewhere. Updated.

Copy link
Contributor

@kolyshkin kolyshkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall, left a few minor nits (can be ignored).

mtrmac added 6 commits August 5, 2024 20:46
…er match

The rules expect us to set manifest editing updates.

Signed-off-by: Miloslav Trmač <[email protected]>
... and add CandidateWithLocation and CandidateWithUnknownLocation ,
so that the BIC implementations only need to deal with one value
instead of carrying around three; we will want to add one more,
and encapsulating them all into a single template will make it
transparent to the cache implementations.

Should not change behavior.

Signed-off-by: Miloslav Trmač <[email protected]>
... just because we now can, and to nudge all future caches to be designed
around CandidateTemplateWithCompression.

Should not change behavior.

Signed-off-by: Miloslav Trmač <[email protected]>
We will add more logic to the default case, so sharing the
CandidateCompressionMatchesReuseConditions call is not going to be
as easy. Split the two code paths.

Should not change behavior.

Signed-off-by: Miloslav Trmač <[email protected]>
We will want to record more than a single alghoritm name. For now,
just introduce the structure and modify users, we'll add the new fields
later.

Should not change behavior.

Signed-off-by: Miloslav Trmač <[email protected]>
… blobs

... because we don't trust the TOC data, if any.

This allows us to remove the zstd:chunked hack; we, at least,
now record those blobs as zstd.

Signed-off-by: Miloslav Trmač <[email protected]>
@mtrmac mtrmac force-pushed the chunked-bic2-partial branch from 5bb9639 to babdac8 Compare August 5, 2024 18:48
Copy link
Member

@giuseppe giuseppe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@giuseppe giuseppe merged commit e3e9287 into containers:main Aug 6, 2024
10 checks passed
@mtrmac mtrmac deleted the chunked-bic2-partial branch August 6, 2024 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants