Skip to content

Commit

Permalink
docs(fingerprint): Track each use of Metadata separately
Browse files Browse the repository at this point in the history
  • Loading branch information
@epage
epage committed Nov 21, 2024
1 parent 34331a8 commit a5cf2ff
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 46 deletions.
36 changes: 20 additions & 16 deletions src/cargo/core/compiler/build_runner/compilation_files.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,24 +41,25 @@ impl fmt::Debug for UnitHash {
}
}

/// The `Metadata` is a hash used to make unique file names for each unit in a
/// build. It is also used for symbol mangling.
/// [`Metadata`] tracks several [`UnitHash`]s, including
/// [`Metadata::unit_id`], [`Metadata::c_metadata`], and [`Metadata::c_extra_filename`].
///
/// For example:
/// We use a hash because it is an easy way to guarantee
/// that all the inputs can be converted to a valid path.
///
/// [`Metadata::unit_id`] is used to uniquely identify a unit in the build graph.
/// This serves as a similar role as [`Metadata::c_extra_filename`] in that it uniquely identifies output
/// on the filesystem except that its always present.
///
/// [`Metadata::c_extra_filename`] is needed for cases like:
/// - A project may depend on crate `A` and crate `B`, so the package name must be in the file name.
/// - Similarly a project may depend on two versions of `A`, so the version must be in the file name.
///
/// In general this must include all things that need to be distinguished in different parts of
/// This also acts as the main layer of caching provided by Cargo
/// so this must include all things that need to be distinguished in different parts of
/// the same build. This is absolutely required or we override things before
/// we get chance to use them.
///
/// It is also used for symbol mangling, because if you have two versions of
/// the same crate linked together, their symbols need to be differentiated.
///
/// We use a hash because it is an easy way to guarantee
/// that all the inputs can be converted to a valid path.
///
/// This also acts as the main layer of caching provided by Cargo.
/// For example, we want to cache `cargo build` and `cargo doc` separately, so that running one
/// does not invalidate the artifacts for the other. We do this by including [`CompileMode`] in the
/// hash, thus the artifacts go in different folders and do not override each other.
Expand All @@ -73,17 +74,20 @@ impl fmt::Debug for UnitHash {
/// more space than needed. This makes not including something in `Metadata`
/// a form of cache invalidation.
///
/// You should also avoid anything that would interfere with reproducible
/// Note that the `Fingerprint` is in charge of tracking everything needed to determine if a
/// rebuild is needed.
///
/// [`Metadata::c_metadata`] is used for symbol mangling, because if you have two versions of
/// the same crate linked together, their symbols need to be differentiated.
///
/// You should avoid anything that would interfere with reproducible
/// builds. For example, *any* absolute path should be avoided. This is one
/// reason that `RUSTFLAGS` is not in `Metadata`, because it often has
/// reason that `RUSTFLAGS` is not in [`Metadata::c_metadata`], because it often has
/// absolute paths (like `--remap-path-prefix` which is fundamentally used for
/// reproducible builds and has absolute paths in it). Also, in some cases the
/// mangled symbols need to be stable between different builds with different
/// settings. For example, profile-guided optimizations need to swap
/// `RUSTFLAGS` between runs, but needs to keep the same symbol names.
///
/// Note that the `Fingerprint` is in charge of tracking everything needed to determine if a
/// rebuild is needed.
#[derive(Copy, Clone, Debug)]
pub struct Metadata {
unit_id: UnitHash,
Expand Down
66 changes: 36 additions & 30 deletions src/cargo/core/compiler/fingerprint/mod.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,46 +47,48 @@
//! reliable and reproducible builds at the cost of being complex, slow, and
//! platform-dependent.
//!
//! ## Fingerprints and Metadata
//! ## Fingerprints and [`UnitHash`]s
//!
//! [`Metadata`] tracks several [`UnitHash`]s, including
//! [`Metadata::unit_id`], [`Metadata::c_metadata`], and [`Metadata::c_extra_filename`].
//! See its documentation for more details.
//!
//! The [`Metadata`] hash is a hash added to the output filenames to isolate
//! each unit. See its documentationfor more details.
//! NOTE: Not all output files are isolated via filename hashes (like dylibs).
//! The fingerprint directory uses a hash, but sometimes units share the same
//! fingerprint directory (when they don't have Metadata) so care should be
//! taken to handle this!
//!
//! Fingerprints and Metadata are similar, and track some of the same things.
//! The Metadata contains information that is required to keep Units separate.
//! Fingerprints and [`UnitHash`]s are similar, and track some of the same things.
//! [`UnitHash`]s contains information that is required to keep Units separate.
//! The Fingerprint includes additional information that should cause a
//! recompile, but it is desired to reuse the same filenames. A comparison
//! of what is tracked:
//!
//! Value | Fingerprint | Metadata
//! -------------------------------------------|-------------|----------
//! rustc | ✓ | ✓
//! [`Profile`] | ✓ | ✓
//! `cargo rustc` extra args | ✓ | ✓
//! [`CompileMode`] | ✓ | ✓
//! Target Name | ✓ | ✓
//! `TargetKind` (bin/lib/etc.) | ✓ | ✓
//! Enabled Features | ✓ | ✓
//! Declared Features | ✓ |
//! Immediate dependency’s hashes | ✓[^1] | ✓
//! [`CompileKind`] (host/target) | ✓ | ✓
//! `__CARGO_DEFAULT_LIB_METADATA`[^4] | | ✓
//! `package_id` | | ✓
//! authors, description, homepage, repo | ✓ |
//! Target src path relative to ws | ✓ |
//! Target flags (test/bench/for_host/edition) | ✓ |
//! -C incremental=… flag | ✓ |
//! mtime of sources | ✓[^3] |
//! RUSTFLAGS/RUSTDOCFLAGS | ✓ |
//! [`Lto`] flags | ✓ | ✓
//! config settings[^5] | ✓ |
//! `is_std` | | ✓
//! `[lints]` table[^6] | ✓ |
//! `[lints.rust.unexpected_cfgs.check-cfg]` | ✓ |
//! Value | Fingerprint | `Metadata::unit_id` | `Metadata::c_metadata` | `Metadata::c_extra_filename`
//! -------------------------------------------|-------------|---------------------|------------------------|----------
//! rustc | ✓ | ✓ | ✓ | ✓
//! [`Profile`] | ✓ | ✓ | ✓ | ✓
//! `cargo rustc` extra args | ✓ | ✓ | ✓ | ✓
//! [`CompileMode`] | ✓ | ✓ | ✓ | ✓
//! Target Name | ✓ | ✓ | ✓ | ✓
//! `TargetKind` (bin/lib/etc.) | ✓ | ✓ | ✓ | ✓
//! Enabled Features | ✓ | ✓ | ✓ | ✓
//! Declared Features | ✓ | | |
//! Immediate dependency’s hashes | ✓[^1] | ✓ | ✓ | ✓
//! [`CompileKind`] (host/target) | ✓ | ✓ | ✓ | ✓
//! `__CARGO_DEFAULT_LIB_METADATA`[^4] | | ✓ | ✓ | ✓
//! `package_id` | | ✓ | ✓ | ✓
//! authors, description, homepage, repo | ✓ | | |
//! Target src path relative to ws | ✓ | | |
//! Target flags (test/bench/for_host/edition) | ✓ | | |
//! -C incremental=… flag | ✓ | | |
//! mtime of sources | ✓[^3] | | |
//! RUSTFLAGS/RUSTDOCFLAGS | ✓ | | |
//! [`Lto`] flags | ✓ | ✓ | ✓ | ✓
//! config settings[^5] | ✓ | | |
//! `is_std` | | ✓ | ✓ | ✓
//! `[lints]` table[^6] | ✓ | | |
//! `[lints.rust.unexpected_cfgs.check-cfg]` | ✓ | | |
//!
//! [^1]: Build script and bin dependencies are not included.
//!
Expand Down Expand Up @@ -348,6 +350,10 @@
//!
//! [`check_filesystem`]: Fingerprint::check_filesystem
//! [`Metadata`]: crate::core::compiler::Metadata
//! [`Metadata::unit_id`]: crate::core::compiler::Metadata::unit_id
//! [`Metadata::c_metadata`]: crate::core::compiler::Metadata::c_metadata
//! [`Metadata::c_extra_filename`]: crate::core::compiler::Metadata::c_extra_filename
//! [`UnitHash`]: crate::core::compiler::UnitHash
//! [`Profile`]: crate::core::profiles::Profile
//! [`CompileMode`]: crate::core::compiler::CompileMode
//! [`Lto`]: crate::core::compiler::Lto
Expand Down

0 comments on commit a5cf2ff

Please sign in to comment.