Tracking Issue for Packages as (optional) namespaces

[oli-obk]

This is a tracking issue for the RFC "3243" (rust-lang/rfcs#3243).

• crates.io tracking issue: Tracking Issue for Packages as (optional) namespaces crates.io#8292
• cargo tracking issue: Tracking Issue for Packages as (optional) namespaces cargo#13576
• docs.rs tracking issue: Tracking Issue for Packages as (optional) namespaces  docs.rs#2457

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement the RFC (can anyone write up mentoring
  instructions?)
• Adjust documentation (see instructions on rustc-dev-guide)
• Formatting for new syntax has been added to the Style Guide (nightly-style-procedure)
• Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

• How exactly should the Cargo.toml lib.name key work in this world, and how does that integrate with --extern and -L and sysroots?
• Decide on and document precise name conflict rules for the compiler (RFC: Packages as (optional) namespaces rfcs#3243 (comment))
• Should we allow renames like "foo::bar" = { package = "foo_bar", version = "1.0" } in Cargo.toml?
• How precisely should this be represented in the index trie?
• How we should name the .crate file / download URL

Third-parties, like Linux distributions, will need to decide how to encode
cargo package names in their distribution package names according to their
individual rules.
Compared to existing ecosystems with namespaces that they package, the only new
wrinkle is that there can be 0-1 namespace levels.

Implementation history

[epage]

Started a zulip thread exploring if there is any UX work we can do for rustdoc / docs.rs to help users.

[epage]

Basic cargo support is in as of rust-lang/cargo#13591 (submodule update: #122575)

I created a cargo::issue_13576 lib and then next part to resolve in the implementation appears to be in rustc:

cargo-features = ["open-namespaces"]

[package]
name = "cargo::issue_13576"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

    Finished dev [unoptimized + debuginfo] target(s) in 0.07s
     Running `/home/epage/src/personal/cargo/target/debug/cargo -Zscript check -vvv`
    Checking cargo::issue_13576 v0.1.0 (/home/epage/src/personal/dump/cargo-13576)
     Running `CARGO=/home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME='cargo::issue_13576' CARGO_MANIFEST_DIR=/home/epage/src/personal/dump/cargo-13576 CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME='cargo::issue_13576' CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 CARGO_RUSTC_CURRENT_DIR=/home/epage/src/personal/dump/cargo-13576 LD_LIBRARY_PATH='/home/epage/src/personal/dump/cargo-13576/target/debug/deps:/home/epage/src/personal/cargo/target/debug/build/curl-sys-2b2243103868cb3e/out/build:/home/epage/src/personal/cargo/target/debug/build/libgit2-sys-71ef3ff041cafba4/out/build:/home/epage/src/personal/cargo/target/debug/build/libnghttp2-sys-5d0ed877a94b784a/out/i/lib:/home/epage/src/personal/cargo/target/debug/build/libsqlite3-sys-5134432fb6771f3a/out:/home/epage/src/personal/cargo/target/debug/build/libssh2-sys-f4299e26a5b2bcb1/out/build:/home/epage/src/personal/cargo/target/debug/deps:/home/epage/src/personal/cargo/target/debug:/home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' /home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name 'cargo::issue_13576' --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 -C metadata=5d93f16ea53919c5 -C extra-filename=-5d93f16ea53919c5 --out-dir /home/epage/src/personal/dump/cargo-13576/target/debug/deps -C incremental=/home/epage/src/personal/dump/cargo-13576/target/debug/incremental -L dependency=/home/epage/src/personal/dump/cargo-13576/target/debug/deps`
error: invalid character `':'` in crate name: `cargo::issue_13576`
  |
  = help: you can either pass `--crate-name` on the command line or add `#![crate_name="…"]` to set the crate name

error: could not compile `cargo::issue_13576` (lib) due to 1 previous error

Caused by:
  process didn't exit successfully: `CARGO=/home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME='cargo::issue_13576' CARGO_MANIFEST_DIR=/home/epage/src/personal/dump/cargo-13576 CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME='cargo::issue_13576' CARGO_PKG_README='' CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 CARGO_RUSTC_CURRENT_DIR=/home/epage/src/personal/dump/cargo-13576 LD_LIBRARY_PATH='/home/epage/src/personal/dump/cargo-13576/target/debug/deps:/home/epage/src/personal/cargo/target/debug/build/curl-sys-2b2243103868cb3e/out/build:/home/epage/src/personal/cargo/target/debug/build/libgit2-sys-71ef3ff041cafba4/out/build:/home/epage/src/personal/cargo/target/debug/build/libnghttp2-sys-5d0ed877a94b784a/out/i/lib:/home/epage/src/personal/cargo/target/debug/build/libsqlite3-sys-5134432fb6771f3a/out:/home/epage/src/personal/cargo/target/debug/build/libssh2-sys-f4299e26a5b2bcb1/out/build:/home/epage/src/personal/cargo/target/debug/deps:/home/epage/src/personal/cargo/target/debug:/home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' /home/epage/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name 'cargo::issue_13576' --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 -C metadata=5d93f16ea53919c5 -C extra-filename=-5d93f16ea53919c5 --out-dir /home/epage/src/personal/dump/cargo-13576/target/debug/deps -C incremental=/home/epage/src/personal/dump/cargo-13576/target/debug/incremental -L dependency=/home/epage/src/personal/dump/cargo-13576/target/debug/deps` (exit status: 1)

The implementation will need to watch for any files/directories that are created that include :: in the name as that is unsupported on Windows.

[epage]

@Nemo157

My understanding from the RFC was that the library name was meant to just be bar, that would avoid the rustc issue you noted

The autogenerated lib.name key for such a crate will just be bar, the leaf crate name, and the expectation is that to use such crates one must use --extern foo::bar=bar.rlib syntax. There may be some better things possible here, perhaps foo_bar can be used here.
—https://rust-lang.github.io/rfcs/3243-packages-as-optional-namespaces.html#reference-level-explanation

(I'd assume this also applies to the autogenerated bin.name for src/main.rs)

(centralizing the conversation)

[epage]

That gets complicated as cargo assumes the crate-name and extern-name are the same (and both pull from <target>.name) and I suspect that runs deeper than that in both programs.

[Nemo157]

Deeper than rustc's assumption that a crate-name is an identifier? (IIRC pre-1.0 - actually made it into the crate name, so there was extern crate "foo-bar" as foo_bar; syntax, but that was removed from rustc and normalization to an identifier was moved to cargo).

[epage]

I can't speak to that or most of the rustc side. If someone familiar enough with both wants to put forward a working proposal for how both sides can work, I'd love to look over it and see what the trade offs are.

[epage]

Thinking about this some more, some routes we can go

• Allow :: in crate names, breaking rustc's assumption that its an identifier
• Translate before <target>.name but this breaks cargo's assumption that <target>.name is the name callers use to access the item
• Translate from <target>.name (with ::) to an identifier when doing --extern calls

Some things we need to keep in mind

• Rustdoc tests should be accessed with the :: name. I've seen rustdoc do enough weird stuff that I hold no assumptions that its behaving like a normal compilation process.
• Ideally, user errors would be in terms of the name they set, not a name used internally for bookkeeping. If the crate name doesn't show up much in errors, maybe this isn't a big deal.

[hydroperfox]

Thinking about this some more, some routes we can go

• Allow :: in crate names, breaking rustc's assumption that its an identifier
• Translate before <target>.name but this breaks cargo's assumption that <target>.name is the name callers use to access the item
• Translate from <target>.name (with ::) to an identifier when doing --extern calls

Some things we need to keep in mind

• Rustdoc tests should be accessed with the :: name. I've seen rustdoc do enough weird stuff that I hold no assumptions that its behaving like a normal compilation process.
• Ideally, user errors would be in terms of the name they set, not a name used internally for bookkeeping. If the crate name doesn't show up much in errors, maybe this isn't a big deal.

Can you intercept file system calls to a crate's filename (whether for the crate's download or for the crate's target artifacts) on file open? (Where, in Windows, :: could translate to -- as suggested earlier)

[Manishearth]

Can you intercept file system calls to a crate's filename (whether for the crate's download or for the crate's target artifacts) on file open?

I'd prefer we don't use hacks like this.