What are the validity requirements of wide pointers/references with dyn Trait tail?

[RalfJung]

This is the last remaining part of #166, now that #510 has been resolved.
Note that the safety invariant for both of these is pretty much set already since casts from *const dyn Trait to *const dyn Supertrait are safe -- that means the vtable pointer must be valid for the given trait, even on raw pointers.

For the validity of the metadata in references, there seems to be general consensus that it is the same as the safety invariant: the vtable must be a vtable for the right trait (and an arbitrary type).

So the open question is what to do with the validity of metadata in raw pointers. The two contradicting goals pursued by different people here are:

• avoid the confusion that can arise when validity and safety invariant diverge
• avoid the pitfall of a NULL vtable pointer (or some other sentinel value) being insta-UB

This was discussed in Zulip somewhat recently. My proposal was to take a maximally liberal validity invariant, and allow any vtable pointer, but several people had concerns that it would be confusing to allow allow a null (or general, unsafe) vtable to exist but then make it UB to cast that pointer or use it as a dyn receiver, even though that operation is safe. One possible compromise was to explicitly allow only null and then make that actually safe by adding checks on casts and dyn calls -- but that would go against the idea of using raw pointers in cases where performance matters and safe code has too many checks.

[RalfJung]

One interesting sub-question to consider here is the exact definition of "the right trait". Currently in Miri, "the right trait" is defined as "has the same principal trait", where the "principal" trait is the trait that is left after removing all auto traits.

So for instance, transmutes between dyn Debug and dyn Debug+Send are allowed, but transmutes between dyn Debug and dyn Display are not. dyn Debug + Send vs dyn Send is also not allowed, because the former has Debug as the principal trait while the latter does not have a principal trait.

It seems to me that the most consistent and conservative option right now would be to require full equality of the trait list, i.e. to also rule out transmutes between dyn Debug and dyn Debug+Send. Though that does raise the question whether order should matter... arguably, it should not.

[celinval]

I'm curious to know if there will be any validity restriction regarding the size and the alignment values in the vtable. Is it valid for size to be greater than isize::MAX, and alignment to not be power of two.

[RalfJung]

vtables are special memory regions generated by the compiler. It is UB to point a vtable pointer to anything but a compiler-generated vtable. So it is impossible to even have a vtable that violates those properties.

[celinval]

Actually, the reference states that the requirement for raw pointers is subject to debate:

• dyn Trait metadata must be a pointer to a compiler-generated vtable for Trait. (For raw pointers, this requirement remains a subject of some debate.)

But is it?

[RalfJung]

Yes, and this thread is part of the debate. :)

[VorpalBlade]

What are the concrete use cases of having non-compiler-generated vtables? The main case I can think of is if another language (or implementation of Rust) wants to interoperate with Rust.

But without a stable ABI that doesn't work for a lot of other reasons anyway. So given that FFI isn't a realistic reason (at least not for quite a long time), what are the other possible use cases? I suspect the existance or lack of such use cases could help inform what Rust wants to do when it comes to vtables.