Stream: project-safe-transmute

Topic: Using const eval for compiler guarantees only

HeroicKatora (Mar 18 2020 at 13:29, on Zulip):

There is a desire to have some part of transmutation embedded in the compiler, as official endorsement of certain casts. The current different crates all embody different underapproximations possible casts, however there is disparity how and which underapproximations are made available within the type system. This problem is exacerbated by validity vs. safety tradeoffs: As far as I'm aware only typic makes a clear distinction between the two it's more involved because of it..
Let's approach the problem from a different angle: The methods mem::zeroed and mem::uninitialized recently have been changed to do layout checks of the target type, and panic at runtime when they are used to definitely create an _invalid_ instance of a type (no checks for safety of course). What if there was a const fn compatible_layout<T, U>() -> bool in the core language that models an RFC.-defined and SemVer compatible relation of types, i.e. an underapproximation of types for which the language wants to opt-in to such a guarantee? And additionally compatible_transmute<T, U> that panics when compatible layout does not hold, i.e. is only unsafe due to _safety_ invariants but not due to validity invariants. The exact set of methods and guarantees subject to discussion.
This might be useful for all multiple cases. For those looking for compile time guarantees through the type system, a sound user-defined trait system safely encapsulates some underapproximation of those transmutes allowed by the standard library. If such a crate were to use compatible_transmute then it would be easier to audit its trait system. Even if its own logic is faulty, the worst is a panic and not undefined behaviour. Note that this would be regarded as a _bug_ in the crate interface, a correct crate should not allow unguaranteed transmutes.
This would also (hopefully) fight the proliferation of private transmutes. Small crates that use only very simple internal casts might not want to depend on a large crate for the job, i.e. don't want to include a whole proc-macro machinery. What is lost in compile time guarantees can in this case be compensated with tests while the simplicity of core functions makes the whole ordeal sound. It could even be made _safe_ by only additionally requiring the Transparent marker trait, and no other.
And lastly, it could serve as a stepping stone for more experiments. Some use cases require byte-to-type transmutes as pure optimization (e.g. in image decoding fills an arbitrary byte buffer but _can_ copy primitive slices directly if the buffer is correctly aligned. It's the caller's responsibility to ensure this for their byte allocation.). Here, a const fn would be more immediately available than depending on trait specialization for such an optimization.

Hanna Kruppe (Mar 18 2020 at 16:34, on Zulip):

I'm generally a fan of compiler intrinsics that add very little magic to enable a lot of useful library code, but I'm not quite sure if the one intrinsic you describe is all that feasible and useful:

HeroicKatora (Mar 18 2020 at 21:43, on Zulip):

There are of course some open questions to answer but I gave them some consideration. 'T is layout compatible with U' should be interpreted as 'all valid bit-patterns of T are valid bit-patterns of U' in the use above.

  1. Casting of references needs to be considered separately from transmutes in any case, even more so for a trait solution. There is an overlap between the two but neither is contained in the other. Consider that the former does not require Sized of any of its arguments (requireing the unsized locals feature sounds suboptimal), and that references are fundamental (which complicates trait impls for them). On the other side, transmuting of Cell<&'static T> to Cell<&'a T> is sound but doing so behind any kind of references is not. A trait to mark types without any safety invariants contains a subset of the overlap though, I think? (The Transparent trait might be a good candidate, but it's a bit of a misnomer.) Anyways, this seems to hint that solving all three kinds of casts at the same time is more complicated than considering them separately. And maybe that std marker-trait would help, as they permit overlapping impls more easily. That however, is yet another unstable feature and we might be better off with initial implementations that don't create such inter-dependencies.
  2. I consider the intrinsic highly useful if the crates have the concept of Transparent which appears to be a common choice anyways. zerocopy. bytemuck, plain, typic all require it for their safe cast methods—although only the last makes it an explicit trait. As for runtime safey check—by the way, how should we term such code, to not allude to validity—that is a separate issue to consider but I'm quite confident that this can be done as an independent additional follow up with a wrapper. In similar ways as MaybeUninit protects an instance with not-yet-fulfilled validity invariants there could be wrapper to protect one with not-yet-checked safety invariants. Crates that want to build safety checks prior to a standardized wrapper, or some custom safety checking code flow, can still make use of compatible_transmute. It does not eliminate the proof obligations but removes a large part of them. The remaining obligations also do not depend on the Rust language—except for those of types provided in core. As opposed to invalid values, it is not instant UB to make a reference to the not-yet-safety-checked instance, it only might be UB to use any of its methods that could rely on custom invariants.
Lokathor (Mar 18 2020 at 22:28, on Zulip):

MaybeUninit is also the correct wrapper for incomplete safety

HeroicKatora (Mar 19 2020 at 00:31, on Zulip):

I disagree, it's weaker than necessary. In particular, it does not contain any niches of the type. The most concrete wrapper would be a completely opaque ManuallyDrop<T>-like that doesn't provide Deref or any other safe access to the instance. This allows the compiler to assert that value has the layout of the type with all its niches and padding bytes but does not require any safety invariants to hold.

HeroicKatora (Mar 19 2020 at 00:39, on Zulip):

Litmus test: Such a wrapper would allow sound transmutes between itself and T in both directions, meanwhile MaybeUninit would not.

Lokathor (Mar 19 2020 at 09:19, on Zulip):

If the wrapper is specifically "safety not enforced", then how can you soundly remove it so easily?

HeroicKatora (Mar 19 2020 at 10:39, on Zulip):

unsafe fn into_inner, and I said the transmute is sound, not safe.

Hanna Kruppe (Mar 19 2020 at 10:47, on Zulip):

That is an, uh, unusual use of the term "sound". Quoting UGC glossary:

Accordingly, we say that a library (or an individual function) is sound if it is impossible for safe code to cause Undefined Behavior using its public API. Conversely, the library/function is unsound if safe code can cause Undefined Behavior.

HeroicKatora (Mar 19 2020 at 10:52, on Zulip):

It's the use of typic, I merely used it

HeroicKatora (Mar 19 2020 at 10:54, on Zulip):

That said, I agree it's not named in an optimal way (as also the Transparent trait). We should create a thread to agree on such terminology.

HeroicKatora (Mar 19 2020 at 10:55, on Zulip):

As transmute is unsafe, I don't think the quote from the UGC applies.

Hanna Kruppe (Mar 19 2020 at 11:07, on Zulip):

HeroicKatora said:

  1. Casting of references needs to be considered separately from transmutes in any case, even more so for a trait solution. [...]

Yes, they are different operations, but both are strongly desired and there is enough interplay between them that treating them as entirely separate problems might result in a worse set of solutions overall.

As for runtime safey check—by the way, how should we term such code, to not allude to validity—that is a separate issue to consider but I'm quite confident that this can be done as an independent additional follow up with a wrapper. In similar ways as MaybeUninit [...]

Validity does play a role too. For example, if the destination type contains a bool or enums, you to check that the source value corresponds to a valid disciminant. If you're transmuting a reference, you might need to check its alignment (in addition to whatever checks you need w.r.t. the referent). I don't know how to simplify such checks with the kind of wrapper you propose. In general you need field offsets and access raw bytes.

HeroicKatora (Mar 19 2020 at 11:16, on Zulip):

Both of your examples are validity checks and not safety checks. But in a pure safety check you could rely on valid initialization. It would be allowed to load fields of a struct for example, or even to match on an enum's discriminant. You only need to be careful not to use any of the type's methods and impls that may rely on its safety invariants (e.g. by doing recursive safety checks on fields first, or only having Transparent fields).

HeroicKatora (Mar 19 2020 at 11:17, on Zulip):

Your comment does make a point though, that even validity checks might be dynamic.

Lokathor (Mar 20 2020 at 21:07, on Zulip):

Well an example safety check would be the len field of some sort of vec-like thing

Last update: Jul 02 2020 at 12:45UTC