Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Packages as (optional) namespaces #3243

Merged
merged 36 commits into from
Mar 11, 2024
Merged
Changes from 33 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
e590c72
Copy over RFC text from https://github.com/Manishearth/namespacing-rfc/
Manishearth Mar 9, 2022
5e6ad7c
Fix template
Manishearth Mar 9, 2022
d9d1556
Add basic FAQ
Manishearth Mar 9, 2022
e29aa55
Add more separator options
Manishearth Mar 9, 2022
3a0b099
leaf crate names
Manishearth Mar 10, 2022
4422adf
sep choice
Manishearth Mar 10, 2022
24db227
Clarify motivation on projects vs organizations
Manishearth Mar 10, 2022
1986c3d
Switch to colons
Manishearth Mar 25, 2022
42046f8
fixes
Manishearth Mar 26, 2022
2fff101
Document Python prior art
epage Mar 26, 2022
8e667ce
Expand on the guide-level explanation
epage Mar 26, 2022
01d37b2
Document feature flags drawbacks
epage Mar 26, 2022
4501047
Merge pull request #4 from epage/guide
Manishearth Mar 27, 2022
1f62468
Merge pull request #3 from epage/prior
Manishearth Mar 27, 2022
1ca3e62
Merge pull request #5 from epage/flags
Manishearth Mar 27, 2022
f7be349
Fix a couple of typos
epage Mar 28, 2022
9a144fb
Updaet motivation for `::` semantics
epage Mar 28, 2022
a578a67
Merge pull request #7 from epage/motivation
Manishearth Mar 28, 2022
3396a2e
Merge pull request #6 from epage/typos
Manishearth Mar 28, 2022
11c6354
Update summary to focus on `::` semantics
epage Mar 29, 2022
cc5872d
Merge pull request #8 from epage/summary
Manishearth Mar 29, 2022
65f7083
add renames as unresolved q
Manishearth Mar 29, 2022
5c497c5
fix syntax
Manishearth May 24, 2023
67ef913
fixes
Manishearth May 24, 2023
56fae2e
trie-unresolved
Manishearth May 24, 2023
482fae8
Update 0000-packages-as-optional-namespaces.md
Manishearth Jul 23, 2023
4c14f9c
Update text/0000-packages-as-optional-namespaces.md
Manishearth Nov 4, 2023
d9a3c90
Update text/0000-packages-as-optional-namespaces.md
Manishearth Nov 4, 2023
8bd7fbc
Add .crate file as unresolved
epage Nov 13, 2023
abde93f
Be explicit that open questions are deferred out
epage Nov 13, 2023
7d77485
Call out distributions
epage Nov 13, 2023
75dd867
Merge pull request #9 from epage/unresolved
Manishearth Nov 13, 2023
0db096f
Update text/0000-packages-as-optional-namespaces.md
Manishearth Mar 11, 2024
6cc8886
Rename 0000-packages-as-optional-namespaces.md to 3243-packages-as-op…
oli-obk Mar 11, 2024
a93b1e4
Add RFC PR link
oli-obk Mar 11, 2024
656203f
Link to rust tracking issue
oli-obk Mar 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 241 additions & 0 deletions text/0000-packages-as-optional-namespaces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,241 @@
- Feature Name: `packages_as_namespaces`
- Start Date: (fill me in with today's date, 2022-03-09)
- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000)
- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000)

# Summary

Languages like C++ have open namespaces where anyone can write code in any namespace. In C++'s case, this includes the `std` namespace and is only limited by convention. In contrast, Rust has closed namespaces which can only include code from the original namespace definition (the crate).
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a pedantic nit not particularly relevant to Rust, but C++ declares the declaration of names in the std namespace to be UB (or at least Ill-formed, no diagnostic required; I couldn't find a normative reference offhand), so it's not just convention that limits the std namespace.

Every other namespace is only managed by convention in C++, though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I'm being blasé with the standard but that sounds like "by convention" to me in that there is no enforcement of that at compile time but instead a document just says "don't do it"

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, it's a pedantic nit that doesn't matter much, but I do think there's a difference between convention and mandate (even unenforced). You can turn off Rust's naming conventions and suffer nothing but funny looks, but if you violate the C++ mandate that user code shall not define names in the std namespace you're actively courting disaster.

But I don't quite know where the cutoff point is. E.g. saying that safe Rust code is only safe by convention sounds quite wrong, but nothing is factually incorrect about that statement; the Rust compiler won't prevent you from writing a safe function that just calls an unsafe one if you've decided to do so.


This proposal extends Rust to have partially open namespaces by allowing crate owners to create crates like `parent::foo` that will be available as part of the crate `parent`'s namespace. To protect the use of open namespaces, the owners of `parent` has exclusive access to publishing crates in that namespace.

# Motivation

While Rust crates are practically unlimited in size, it is a common pattern for organizations to split their projects into many crates, especially if they expect users to only need a fraction of their crates or they have different backwards compatibility guarantees.

For example, [unic](https://crates.io/search?page=1&per_page=10&q=unic-), [tokio](https://crates.io/search?page=1&per_page=10&q=tokio-), [async-std](https://crates.io/search?page=1&per_page=10&q=async-), [rusoto](https://crates.io/search?q=rusoto) all do something like this, with lots of `projectname-foo` crates. At the moment, it is not necessarily true that a crate named `projectname-foo` is maintained by `projectname`, and in some cases that is even desired! E.g. `serde` has many third party "plugin" crates like [serde-xml-rs](https://github.com/RReverser/serde-xml-rs). Similarly, [async-tls](https://crates.io/crates/async-tls) is a general crate not specific to the async-std ecosystem.

Regardless, it is nice to have a way to signify "these are all crates belonging to a single project, and you may trust them the same" and discover these related crates. When starting up [ICU4X](https://github.com/unicode-org/icu4x/), we came up against this problem: We wanted to be able to publish ICU4X as an extremely modular system of `icu-foo` or `icu4x-foo` crates, but it would be confusing to users if third-party crates could also exist there (or take names we wanted to use).

It's worth spending a bit of time talking about "projects" and "organizations", as nebulous as those terms are. This feature is *primarily* motivated by the needs of "projects". By this I mean a _single_ Rust API developed as multiple crates, for example `serde` and `serde::derive`, or `icu` and `icu::provider`, or `servo::script` and `servo::layout`. One would expect "projects" like this to live under a single Git repository according to the norms of project organization; they are logically a single project and API even if they are multiple crates.

The feature suggested here is unlikely to be used by "organizations" as this would put independent concerns in the same Rust API. By "organizations", I mean a group of people who are coming together to build likely related crates, under the same "brand", likely developed in multiple repos under a GitHub organization.


The motivation here is distinct from the general problem of squatting -- with general squatting, someone else might come up with a cool crate name before you do. However, with `projectname-foo` crates, it's more of a case of third parties "muscling in" on a name you have already chosen and are using.

# Guide-level explanation

The owners of the `foo` crate may provide other crates under the `foo` namespace, like `foo::bar`. For users, this makes its official status clearer and makes it easier to discover.

Users import these crates in Cargo.toml as normal:

```toml
[dependencies]
"foo" = "1.0.42"
"foo::bar" = "3.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any special considerations for rustdoc we should keep in mind with this feature?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think rustdoc may need some code changes for this to work; but overall I think that it ought to be fine? it might choose to render these slightly differently.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(sorry for the necro post)

Locally, we are treating them as one package but they won't find that by browsing rustdoc, locally or via docs.rs with a naiive solution. My concern is that this would be like features (without autocfg) where it can be weird correlating everything which would impact understandability of a code base / example code.

Random ideas:

  • When building locally, show foo::bar as a ``mod barinfoo` with a visual tag that this is the `foo::bar` dependency
  • More docs.rs, should have a way to link these together for discovery of foo::bar from foo or to browse back to foo
    • As a cheap idea, what if we could do #[cfg_attr(docsrs, rustdoc(namespaced))] mod bar {} and then rustdoc would make that mod a link.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to figure out how namespaced names are written to the filesystem for rustdoc, and the mechanism that rustdoc will use to find them.

Currently, rustdoc hunts for a directory with the same name as the library. However, we can't use foo::bar as a directory name (Windows), and we can't use just bar (it is not sufficiently unique). Other normalizations might have problems (like foo_bar could also collide with a real foo_bar). The directories could be nested, but I don't know how complicated that will be for ExternalCrate::location.

Cargo could also start using something like --extern-html-root-url, but unfortunately that is hung up on doing more design work (see rust-lang/cargo#10241 (comment) and rust-lang/cargo#8296).

```

They will then access this through a facade made of `foo` and all `foo::*` crates, for example:

```rs
let baz = foo::bar::Baz::new();
foo::render(baz);
```

Some reasons for `foo`s owner to consider using namespaces:
- Avoid name conflicts with third-party authors (since they are reserved)
- Improve discoverability of official crates
- As an alternative to feature flags for optional subsystems
- When different parts of your API might have different compatibility guarantees

When considering this, keep in mind:
- Does it makes sense for this new crate to be presented in the `foo` facade?
- How likely is a crate to move into or out of the namespace?
- Moving the crate in or out of a namespace is a breaking change though it can be worked around by having the old crate re-export the new crate but that does add extra friction to the process.
- There is not currently a mechanism to raise awareness with users that a crate has migrated into or out of a namespace and you might end up leaving users behind.
- If users import both `foo` and `foo::bar` but `foo` also has a `bar` item in its API that isn't just `foo::bar` re-exported, then rustc will error.
Copy link
Contributor

@pitaj pitaj Mar 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to include the workaround for this case:

In a case like this, either crate may be renamed in Cargo.toml to bypass the ambiguity.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the foo::bar module is a semver-incompatible version of the foo::bar crate?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing mod foo::bar in favor of crate foo::bar is a compatiblity-breakage.

It seems like the opposite will be true as well.

Having a cargo command to detect crate-level breakages like this would be a big help rather than adding to our endless of pile of semver compatibility documentation that is too overwhelming for people to dig through :)


Only the owners of `foo` may _create_ the `foo::bar` crate (and all owners of `foo` are implicitly owners of `foo::bar`). After the `foo::bar` crate is created, additional per-crate publishers may be added who will be able to publish subsequent versions as usual.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like server-side enforcement.

Should there be any client-side enforcement as well?

Example: foo is in crates.io, foo::bar is from a third-party registry and might be from a different user. Should cargo limit foo::bar to be from the same source as foo? If so, what does that mean for patching?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to explicitly specify the registry in the dependency specification if you use a third-party registry, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think client side enforcement seems reasonable but not required here (i.e. I don't think we need to make that call in this RFC)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That wouldn't allow using "foo::bar" = { git = "https://github.com/foo/bar" } during development of it as each git repository is treated as an independent registry.


# Reference-level explanation

_This section will maintain a distinction between "package" (a crates.io package) and "crate" (the actual rust library). The rest of the RFC does not attempt to make this distinction_

`::` is now considered valid inside package names on Crates.io. For now, we will restrict package names to having a single `::` in them, not at the beginning or end of the name, but this can be changed in the future.

When publishing a package `foo::bar`, if the package does not exist, the following must be true:

- `foo` must exist
- The user publishing the package must be an owner of `foo`

For the package `foo::bar`, all owners of `foo` are always considered owners of `foo::bar`, however additional owners may be added. People removed from ownership of `foo` will also lose access to `foo::bar` unless they were explicitly added as owners to `foo::bar`.

Crates.io displays `foo::bar` packages with the name `foo::bar`, though it may stylistically make the `foo` part link to the `foo` package.

The [registry index trie](https://doc.rust-lang.org/nightly/cargo/reference/registries.html#index-format) may represent subpackages by placing `foo::bar` as just `foo::bar`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows forbids : character in path names. The git index with a file named foo::bar will not be possible to check out on Windows.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got any suggestions for alternatives?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick look, it appears the sensible options are .+~@=^. []{} may also be options but are kinda ugly.

My top two choices would be == and @@.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A single @ seems cool; it is sufficiently clear about the intent (foo@bar)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't @ already used to refer to package@version?

If . works that seems interesting as it has precedent with other packaging systems.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't work for mapping these names to package names for distros. Those can't contain slashes or colons.

Copy link

@ericlagergren ericlagergren Nov 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's true. It would help the original concern, though:

Windows forbids : character in path names. The git index with a file named foo::bar will not be possible to check out on Windows.

But, ISTM this RFC probably shouldn't define a file name encoding.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This RFC or the implementation must define a file name encoding, both for how to lookup the crate in the index and how to map the resolved crate to a download URL, these are public APIs that have multiple implementations on both the client and server side https://doc.rust-lang.org/nightly/cargo/reference/registry-index.html

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, I see two separate things here:

  • How to represent namespaces in the registry index.
  • A general-purpose encoding format for package names. E.g.:

[...] and distribution packagers who need to represent namespaced crate names as distro-specific packages.

I think it would be a good idea for this RFC to stipulate a filename-friendly encoding for namespaced package names which can be consistently used for all of these use cases, rather than each use case inventing its own encoding.

Obviously this RFC must specify the former, but I'm not sure it needs to do the latter.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Continuing this discussion in the crates.io tracing issue: rust-lang/crates.io#8292 (comment)


`rustc` will need some changes. When `--extern foo::bar=crate.rlib` is passed in, `rustc` will include this crate during resolution as if it were a module `bar` living under crate `foo`. If crate `foo` is _also_ in scope, this will not automatically trigger any errors unless `foo::bar` is referenced, `foo` has a module `bar`, and that module is not just a reexport of crate `foo::bar`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having foo::bar appear as if it exists within foo seems confusing if there is also a dependency on foo. Would it be possible to disambiguate somehow?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering if this also requires a dependency on foo. Or what if this crate is foo?


The autogenerated `lib.name` key for such a crate will just be `bar`, the leaf crate name, and the expectation is that to use such crates one _must_ use `--extern foo::bar=bar.rlib` syntax. There may be some better things possible here, perhaps `foo_bar` can be used here.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming the following will be legal

"foo::bar" = { package = "foo_bar", version = "1.0" }

Should we let this remain legal or add a check to disallow this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ambivalent

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pitaj brought up a reason to allow this

For example, if someone were to add evil-serde::derive = { package = "serde::derive" }, would this now be importable as serde::derive?

It would be the opposite:

"serde::derive" = { package = "evil-serde::derive", version = 1 }

I think it's important to allow this so people can use git dependencies, local paths, and drop-in replacements in place of any crate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think git dependencies and local paths are an absolute requirement. Allowing drop-in replacements maybe not so much, but it would be kinda weird to forbid just that unless we have a good reason.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this to unresolved questions. I'm overall okay with this being banged out as a part of the experimentation process.

Copy link
Contributor

@kornelski kornelski Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are very few crates with __ in their name. Double underscore could realistically be reserved in crate names, and then function as a namespace separator that is compatible with all file systems and distro packages. _-_ is not used in any crate names.



# Drawbacks


## Namespace root taken
Not all existing projects can transition to using namespaces here. For example, the `unicode` crate is reserved, so `unicode-rs` cannot use it as a namespace despite owning most of the `unicode-foo` crates. In other cases, the "namespace root" `foo` may be owned by a different set of people than the `foo-bar` crates, and folks may need to negotiate (`async-std` has this problem, it manages `async-foo` crates but the root `async` crate is taken by someone else). Nobody is forced to switch to using namespaces, of course, so the damage here is limited, but it would be _nice_ for everyone to be able to transition.


## Slow migration

Existing projects wishing to use this may need to manually migrate. For example, `unic-langid` may become `unic::langid`, with the `unic` project maintaining `unic-langid` as a reexport crate with the same version number. Getting people to migrate might be a bit of work, and furthermore maintaining a reexport crate during the (potentially long) transition period will also be some work. Of course, there is no obligation to maintain a transition crate, but users will stop getting updates if you don't.

A possible path forward is to enable people to register aliases, i.e. `unic-langid` is an alias for `unic::langid`.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we expand a bit more on this? Aliasing crates as a feature can be independent of this RFC and work could be started on it without affecting this RFC.

From time to time, I keep wishing for such a feature on NPM whenever I have to deal with migrated packages.



## Requires rustc changes

There are alternate solutions below that don't require the _language_ getting more complex and can be done purely at the Cargo level. Unfortunately they have other drawbacks.


# Rationale and alternatives

This change solves the ownership problem in a way that can be slowly transitioned to for most projects.

## Slash as a separator

**For discussions about separator choice, please discuss them in [this issue](https://github.com/Manishearth/namespacing-rfc/issues/1) to avoid overwhelming the main RFC thread.**

A previous version of the RFC had `/` as a separator. It would translate it to `foo_bar` in source, and disambiguated in feature syntax with `foo/bar/` vs `foo/bar`. It had the following drawbacks:


### Slashes
So far slashes as a "separator" have not existed in Rust. There may be dissonance with having another non-identifier character allowed on crates.io but not in Rust code. Dashes are already confusing for new users. Some of this can be remediated with appropriate diagnostics on when `/` is encountered at the head of a path.


Furthermore, slashes are ambiguous in feature specifiers (though a solution has been proposed above for this):
epage marked this conversation as resolved.
Show resolved Hide resolved

```toml
[dependencies]
"foo" = "1"
"foo/std" = { version = "1", optional = true }

[features]
# Does this enable crate "foo/std", or feature "std" of crate "foo"?
default = ["foo/std"]
```

### Dash typosquatting

This proposal does not prevent anyone from taking `foo-bar` after you publish `foo/bar`. Given that the Rust crate import syntax for `foo/bar` is `foo_bar`, same as `foo-bar`, it's totally possible for a user to accidentally type `foo-bar` in `Cargo.toml` instead of `foo/bar`, and pull in the wrong, squatted, crate.

We currently prevent `foo-bar` and `foo_bar` from existing at the same time. We _could_ do this here as well, but it would only go in one direction: if `foo/bar` exists, neither `foo-bar` nor `foo_bar` will be allowed to be published. However, if `foo-bar` or `foo_bar` exist, we would choose to allow `foo/bar` to be published, because we don't want to limit the use of names within a crate namespace due to crates outside the namespace existing. This limits the "damage" to cases where someone pre-squats `foo-bar` before you publish `foo/bar`, and the damage can be mitigated by checking to see if such a clashing crate exists when publishing, if you actually care about this attack vector. There are some tradeoffs there that we would have to explore.

One thing that could mitigate `foo/bar` mapping to the potentially ambiguous `foo_bar` is using something like `foo::crate::bar` or `~foo::bar` or `foo::/bar` in the import syntax.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved



### Using identical syntax in Cargo.toml and Rust source

The `/` proposal does not require changes to Rust compiler to allow slash syntax (or whatever) to parse as a Rust path. Such changes could be made (though not with slash syntax due to parsing ambiguity, see [below](#separator-choice) for more options); this RFC is attempting to be minimal in its effects on rustc.

However, the divergence between Cargo.toml and rustc syntax does indeed have a complexity cost, and may be confusing to some users. Furthermore, it increases the chances of [Dash typosquatting](#dash-typosquatting) being effective.

Some potential mappings for `foo/bar` could be:

- `foo::bar`
- `foo::crate::bar`
- `foo::/bar`
- `~foo::bar`

and the like.

## Whole crate name vs leaf crate name in Rust source


**For discussions about separator choice, please discuss them in [this issue](https://github.com/Manishearth/namespacing-rfc/issues/1) to avoid overwhelming the main RFC thread.**

It may be potentially better to use just the leaf crate name in Rust source. For example, when using crate `foo/bar` from Cargo.toml, the Rust code would simply use `bar::`. Cargo already supports [renaming dependencies](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#renaming-dependencies-in-cargotoml) which can be used to deal with any potential ambiguities here. This also has the added benefit of not having to worry about the separator not parsing as valid Rust.

A major drawback to this approach is that while it addresses the "the namespace is an organization" use case quite well (e.g. `unicode/segmentation` vs `unicode/line-break` and `rust-lang/libc` vs `rust-lang/lazy-static`, etc), this is rather less amenable to the "the namespace is a _project_" case (e.g. `serde` vs `serde/derive`, `icu/datetime` vs `icu/provider`, etc), where the crates are related not just by provenance. In such cases, users may wish to rename the crates to avoid confusion in the code. This may be an acceptable cost.

## Separator choice


**For discussions about separator choice, please discuss them in [this issue](https://github.com/Manishearth/namespacing-rfc/issues/1) to avoid overwhelming the main RFC thread.**

A different separator might make more sense. See the [previous section](#slash-as-a-separator) for more on the original proposal of `/` as a separator.

We could continue to use `/` but also use `@`, i.e. have crates named `@foo/bar`. This is roughly what npm does and it seems to work. The `@` would not show up in source code, but would adequately disambiguate crates and features in Cargo.toml and in URLs.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am rather partial to this approach; it has a bunch of benefits:

  • It has a clear mapping to crates.io/docs.rs URLs
  • There's no ambiguity in feature specs
  • We could use the same syntax directly in source (@ <ident> / triggers parsing as a namespaced path) though this might be too large a syntax change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think it would be ideal for the same syntax to be used both in Cargo.toml and in source code. I'd like to suggest the usage of ::: but I think it has the same issue that led to ... becoming ..=.

Can you expand on why @foo::bar might be too large a syntax change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it changes what counts as an identifier in Rust: it's a change at the token level, which in turn affects proc macros and such.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(it doesn't have to be at the token level tbh, but that does still complicate how paths work)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to have @rust-lang/lang input on this as a syntax change

(if this RFC starts going down this route I'll of course add them as an approving team)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does @ mean in npm's context?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does @ mean in npm's context?

The way npm does package names in a package.json is like @scope/package@version.
Where scope is a existing organization on npm and @version a published version.

Though they also allow you to use Github repos as shown in the image
image

Copy link

@nbittich nbittich Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please don't introduce new keywords. I like the idea of namespace, but please take a simple path, or don't implement it at all until you find one. rust is already complex enough, I'm affraid we have to write things like this in the future:

from crate<'a,T> import T::foo::<'a> where T: Pin<Box<dyn Crate>>> + 'a Namespace + Copy + Debug as @foo

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a character that is currently not possible within a crate name does avoid issues like a bunch of crates already being published with a name that want as a root. For example, if your org uses something like foo_bar and foo_baz but someone else already published foo, you can't use those names. But you could for something like @foo/bar.

That said, I see @ehuss's comment below that cargo has already closed on support for ::. Does that effectively mean this RFC is decided?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its been approved by 2 of the 3 related teams and is just waiting on enough votes from the last team.


We could perhaps have `foo-*` get autoreserved if you publish `foo`, as outlined in https://internals.rust-lang.org/t/pre-rfc-hyper-minimalist-namespaces-on-crates-io/13041 . I find that this can lead to unfortunate situations where a namespace traditionally used by one project (e.g. `async-*`) is suddenly given over to a different project (the `async` crate). Furthermore, users cannot trust `foo-bar` to be owned by `foo` because the vast number of grandfathered crates we will have.

Triple colons could work. People might find it confusing, but `foo:::bar` evokes Rust paths without being ambiguous.

We could use `~` which enables Rust code to directly name namespaced packages (as `~` is no longer used in any valid Rust syntax). It looks extremely weird, however.
Manishearth marked this conversation as resolved.
Show resolved Hide resolved

We could use dots (`foo.bar`). This does evoke some similarity with Rust syntax, however there are ambiguities: `foo.bar` in Rust code could either mean "the field `bar` of local/static `foo`" or it may mean "the crate `foo.bar`".

Note that unquoted dots have semantic meaning in TOML, and allowing for unquoted dots would freeze the list of dependency subfields allowed (to `version`, `git`, `branch`, `features`, etc).


We could reverse the order and use `@`, i.e. `foo/bar` becomes `bar@foo`. This might be a tad confusing, and it's unclear how best to surface this in the source.


## User / org namespaces

Another way to handle namespacing is to rely on usernames and GitHub orgs as namespace roots. This ties `crates.io` strongly to Github -- currently while GitHub is the only login method, there is nothing preventing others from being added.

Furthermore, usernames are not immutable, and that can lead to a whole host of issues.

The primary goal of this RFC is for _project_ ownership, not _org_ ownership, so it doesn't map cleanly anyway.

## Feature Flags

This proposal allows for optional subsystems. This can be created today with feature flags by adding a dependency as optional and re-exporting it.

Draw backs to feature flags
- Solutions for documenting feature flags are limited
- Feature flags can be cumbersome to work with for users
- A semver breakage in the optional-subsystem crate is a semver breakage in the namespace crate
- The optional-subsystem crate cannot depend on the namespace crate
- There is limited tooling for crate authors to test feature combinations especially in workspaces with feature unification and its slow (re-running all tests even if they aren't relevant)

# Prior art
Copy link
Member

@scottmcm scottmcm Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional prior art: Microsoft had to add https://learn.microsoft.com/en-us/nuget/nuget-org/id-prefix-reservation to NuGet, since otherwise a ton of people were publishing things under Microsoft. and System..

One interesting thing it allows is delegation of subnamespaces. If applied here, that could mean that if serde- were reserved then they could delegate serde-json to a particular team, if they wanted to curate-but-not-own some of the sub-packages.

EDIT: Not intending to say that's the model this should use—I haven't formed a position yet—just think it's good prior art for the RFC.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a bad argument since C# works pretty differently for packages, with the difference between top-level packages and modules not being as clear. Systems which use domain-style namespaces are a different story, since for example it's not clear whether Category.Thing is a single dependency by itself, or if Thing is part of the Category dependency.

In Rust, the top-level name is the dependency, period. So, you can be confident that serde::json is a json module within the serde crate, and serde_json is a crate by itself. Reserving the serde_ prefix for all crates feels like it's solving a problem that the solution created, since the only notion of ownership is for the root serde crate itself.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is that in rust a crate can't be splitted while still indicating verifiable common ownership of the resulting cluster of crates, and that it will also stay that way.


This proposal is basically the same as https://internals.rust-lang.org/t/pre-rfc-packages-as-namespaces/8628 and https://internals.rust-lang.org/t/pre-rfc-idea-cratespaces-crates-as-namespace-take-2-or-3/11320 .

Namespacing has been discussed in https://internals.rust-lang.org/t/namespacing-on-crates-io/8571 , https://internals.rust-lang.org/t/pre-rfc-domains-as-namespaces/8688, https://internals.rust-lang.org/t/pre-rfc-user-namespaces-on-crates-io/12851 , https://internals.rust-lang.org/t/pre-rfc-hyper-minimalist-namespaces-on-crates-io/13041 , https://internals.rust-lang.org/t/blog-post-no-namespaces-in-rust-is-a-feature/13040/4 , https://internals.rust-lang.org/t/crates-io-package-policies/1041/37, https://internals.rust-lang.org/t/crates-io-squatting/8031, and many others.

Python has a similar coupling of top-level namespaces and modules with the filesystem. Users coming from other packaging systems, like Perl, wanted to be able to split up a package under a common namespace. A hook to support this was added in Python 2.3 (see [PEP 402](https://peps.python.org/pep-0402/#the-problem)). In [PEP 420](https://peps.python.org/pep-0420/) they formalized a convention for packages to opt-in to sharing a namespace. Differences:
- Python does not have a coupling between package names and top-level namespaces so there is no need for extending the package name format or ability to extend their registry for permissions support.
- In Python, nothing can be in the namespace package while this RFC allows the namespace package to also provide an API.

# Unresolved questions

Deferred to tracking issue to be resolved pre-stabilization:
- How exactly should the Cargo.toml `lib.name` key work in this world, and how does that integrate with `--extern` and `-L` and sysroots?
- Should we allow renames like `"foo::bar" = { package = "foo_bar", version = "1.0" }` in Cargo.toml?
- How precisely should this be represented in the index trie?
- How we should name the `.crate` file / download URL

Third-parties, like Linux distributions, will need to decide how to encode
cargo package names in their distribution package names according to their
individual rules.
Compared to existing ecosystems with namespaces that they package, the only new
wrinkle is that there can be 0-1 namespace levels.

# Future possibilities

We can allow multiple layers of nesting if people want it.

# FAQ

## What if I don't want to publish my crate under a namespace?

You don't have to, namespaces are completely optional when publishing.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't everything be grandfathered into a default namespace like crates::serde?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, because the current set of crates is the top level namespace


## Does this stop people from squatting on `coolcratename`?

No, this proposal does not intend to address the general problem of squatting (See [crates.io's policy](https://crates.io/policies#squatting), a lot of this has been discussed many times before). Instead, it allows people who own an existing crate to publish sub-crates under the same namespace. In other words, if you own `coolcratename`, it stops people from squatting `coolcratename::derive`.