Stream: project-safe-transmute

Topic: typic

Josh Triplett (Dec 30 2019 at 08:38, on Zulip):

Worth reviewing:

Jack Wrenn (Dec 30 2019 at 18:53, on Zulip):

I'm happy to answer questions about it!

Yato (Dec 31 2019 at 18:25, on Zulip):

Based on @Jack Wrenn's amazing work on typic, I created chtrans (checked transmute)

Yato (Dec 31 2019 at 18:25, on Zulip):

A few notable differences, I fully support non-zero types, bool, but I have less support for references right now. (Although I could support references in the same way as typic with minor changes).
There are numerous implementation differences, such as removing references as slots, a custom backwards linked list (which made some other parts easier/cheaper to implement).

Yato (Dec 31 2019 at 18:27, on Zulip):

All this without a huge amount of special casing (such as in typic::transmutation), this is due to a new idea I'm calling Marker. Markers are properties that can be searched for and must be validated. For example, NonZero<N> is the property that the next N bytes when interpreted as an integer is non-zero

Jack Wrenn (Dec 31 2019 at 18:30, on Zulip):

My hope was to spur experimentation in this area, so I'm thrilled to have been a source of inspiration. :D I can't wait to dig into your approach!

Yato (Dec 31 2019 at 18:30, on Zulip):

It was really eye opening to see your implementation, I loved digging through it!

Yato (Dec 31 2019 at 18:31, on Zulip):

It reminded me of when I first began to understand typenum.

Yato (Dec 31 2019 at 18:33, on Zulip):

One limitation that I am now realizing is the handling of privacy, right now if you apply chtrans::repr, you are implicitly making your type transparent. I need to find a way around this. I also don't support fallible transmute just yet. I suspect these two are related.

Jack Wrenn (Dec 31 2019 at 18:37, on Zulip):

Re privacy, did you see this:
In my approach, a type is only transparent if all fields are marked pub.

Yato (Dec 31 2019 at 18:38, on Zulip):

Yes, I was just getting to that. That's what reminded me of this problem.

Yato (Dec 31 2019 at 18:42, on Zulip):

For checking invariants, would it be fine if we just created the type then passed it to Invariants::check, and leaking it if the check fails? That way we don't need to generate at Candidate type (which looks like it is semantically doing the same thing). I'm not sure if the is sound, but it might be simpler to implement.

Jack Wrenn (Dec 31 2019 at 18:47, on Zulip):

Maybe? The Candidate is a doppelganger of the type but without any methods, since methods on the original type might rely on inter-field invariants. My first approach at this did what you describe, but the soundness issue lead me to move away from it.

Yato (Dec 31 2019 at 18:49, on Zulip):

cc @RalfJ

Jack Wrenn (Jan 01 2020 at 20:41, on Zulip):

Following up on my assertion that supporting union was 'just' a matter of adding the layout algorithm for them, the unions branch of Typic now provides a proof-of-concept implementation. Example here:

To encode which unions are layout compatible, I needed my first nightly feature: marker_trait_attr. (Namely, to state that aUA can be transmuted into a UB if every variant of UA can be transmuted into _any_ variant of UB.) So, that's an unfortunate barrier for making this functionality available as a library.

Yato (Jan 01 2020 at 21:04, on Zulip):

I added MaybeUninit, but I need to add full union support. (This shouldn't be too hard, similarly to you it is just extending the macro, and adding the layout algorithm)

On the note of Candidate, I found this in the UCG

The safety invariant is an invariant that safe code may assume all data to uphold. This invariant is used to justify which operations safe code can perform. The safety invariant can be temporarily violated by unsafe code, but must always be upheld when interfacing with unknown safe code. It is not relevant when arguing whether some program has UB, but it is relevant when arguing whether some code safely encapsulates its unsafety -- in other words, it is relevant when arguing whether some library is sound.

So I think it is fine to forge an possibly invalid type as long as it doesn't break any validity invariants, like references must always be aligned. Library invariants don't matter (like Foo.x can never be odd, or Vec.len must not cover uninitialized data. So long as you don't expose this to safe code. Which we don't, only exposed to the Invariants trait, which is unsafe to implement.

Jack Wrenn (Jan 01 2020 at 21:42, on Zulip):

@Yato I just pushed to the enums branch with proof-of-concept support of enum.

I desugar enums to equivalent unions according to RFC#2195 and use my existing union layout machinery. Discriminants are represented using const generics. See:

Jack Wrenn (Jan 01 2020 at 21:46, on Zulip):

And with that, I'm thrilled to have confirmed my suspicion that Rust's type system is already expressive enough to reasonably encode every statically-checkable transmutation I can think of! ^_^

Jack Wrenn (Jan 01 2020 at 21:51, on Zulip):

Caveat for onlookers: this is totally Research Quality™. I haven't yet closely reviewed my low-level layout compatibility checks to make sure I haven't accidentally permitted something I meant to forbid (which is as easy as leaving off a trait bound) or omitted an impl for a transition I meant to allow (in fact, I've definitely omitted impls).

Yato (Jan 01 2020 at 22:55, on Zulip):

Same caveat applies to chtrans

Yato (Jan 01 2020 at 22:57, on Zulip):

One thing I was thinking about wrt unions, are unions allowed to contain uninitialized bytes anywhere in their layout

i.e. is this allowed to be uninitialized

union Foo {
    value: i32
Jack Wrenn (Jan 01 2020 at 22:59, on Zulip):

Unless those bytes are introduced by a variant: no. MaybeUninit<i32> cannot be transmuted to Foo, because Foo's valid bit pattern is identical to i32 (i.e., four arbitrarily initialized bytes).

Yato (Jan 01 2020 at 23:03, on Zulip):

This seems to be related to

Jack Wrenn (Jan 01 2020 at 23:10, on Zulip):

Huh. Regarding #73, I find it very surprising that there could be a valid bit pattern for a union T that wasn't a valid bit pattern for any of T's variants.

Jack Wrenn (Jan 01 2020 at 23:12, on Zulip):

Off the top of my head, it seems like that would pose a significant barrier to providing a mechanism for safe variant access. E.g., it's completely impossible to provide a safe accessor for value in your example, because a valid instance ofFoo could have been constructed from uninitialized bits.

Yato (Jan 01 2020 at 23:15, on Zulip):

Yes, that's why I asked. As I was going through the UCG's guidelines I came across #73 and that surprised me. For now, I will implement unions as a sequence of uninitialized bytes, just to be safe.

Here is the motivating example for unions having byte patterns that fit non of the variants from another thread

Jack Wrenn (Jan 01 2020 at 23:23, on Zulip):

The current rule I have basically this:

Jack Wrenn (Jan 01 2020 at 23:26, on Zulip):

There's some discussion of niche-finding optimizations in that issue. AFAIK, those considerations don't affect us because niche finding can't reach into #[repr(C)] types to find niches.

(At least, that's my understanding from #174.)

Yato (Jan 01 2020 at 23:30, on Zulip):

Ok, I'm going to ask there just to make sure, I don't really want to speculate on this.

Yato (Jan 01 2020 at 23:32, on Zulip):

Layout optimizations does matter for your implementation of enums, wrt Option-like enums, but not wrt unions

Jack Wrenn (Jan 01 2020 at 23:34, on Zulip):

I haven't tried to implement any layout computations for #[repr(Rust)] enums. There aren't any tricky layout optimizations on #[repr(C)] enums.

Yato (Jan 01 2020 at 23:36, on Zulip):

It looks like this doesn't work?

#[typic::repr(C, _)]
enum OptionLike {

So just C-like enums then?

Jack Wrenn (Jan 01 2020 at 23:40, on Zulip):

What do you mean by "work"? OptionLike is repr(C), so we can compute the layout of it without doing any niche finding optimizations. It should be eight bytes. If it weren't repr(C), then it'd only be one byte.

Yato (Jan 01 2020 at 23:45, on Zulip):

Right now that doesn't compile with typic (regardless of what integer type I put in the _)

Jack Wrenn (Jan 01 2020 at 23:47, on Zulip):

Oops. It "works" in "theory". ;) Tbh, I committed after getting discriminants to work and didn't write any test cases for variants with fields. Let me see what I forgot...

Yato (Jan 01 2020 at 23:48, on Zulip):

I get these errors,

error: reaching this expression at runtime will panic or abort
  --> typic\tests\
24 | #[typic::repr(C)]
   | ^^^^^^^^^^^^^^^^^ "pointer-to-integer cast" needs an rfc before being allowed inside constants
   = note: `#[deny(const_err)]` on by default

error[E0080]: evaluation of constant value failed
  --> typic\tests\
24 | #[typic::repr(C)]
   | ^^^^^^^^^^^^^^^^^ referenced constant has errors

error[E0605]: non-primitive cast: `OptionLike` as `isize`
  --> typic\tests\
24 | #[typic::repr(C)]
   | ^^^^^^^^^^^^^^^^^
   = note: an `as` expression can only be used to convert between primitive types. Consider using the `From` trait
Jack Wrenn (Jan 01 2020 at 23:48, on Zulip):

HUH. Ohhhhh. D'oh.

Jack Wrenn (Jan 01 2020 at 23:49, on Zulip):

I forgot you can't use as on non-fieldless variants.

Jack Wrenn (Jan 01 2020 at 23:53, on Zulip):

Yeah, that's going to be slightly inconvenient to work around, but nothing major. (I'd love to see an RFC for safely accessing the discriminants of non-fieldless variants!)

Jack Wrenn (Jan 02 2020 at 00:56, on Zulip):

@Yato: Give things a shot, now. I found a workaround that wasn't as painful as I was expecting. :)

James Munns (Jan 02 2020 at 15:48, on Zulip):

@Jack Wrenn I think @Florian Gilcher had an RFC for this (a [const] function that gets you the ID of the variant)

James Munns (Jan 02 2020 at 15:50, on Zulip):

Ah, wrong on two counts. It was only to get the size of the discriminant in bits, and it was closed without merging:

RalfJ (Jan 02 2020 at 16:48, on Zulip):


I think it is fine to forge an possibly invalid type as long as it doesn't break any validity invariants

That sounds a bit funny? "It can be invalid as long as it's valid"?^^
but it is fine to forge a value violating the safety invariant as long as it does not violate the validity invariant

Yato (Jan 02 2020 at 16:49, on Zulip):

@RalfJ Sorry, got my terms mixed up, but that's what I meant

RalfJ (Jan 02 2020 at 16:49, on Zulip):

@Jack Wrenn

Huh. Regarding #73, I find it very surprising that there could be a valid bit pattern for a union T that wasn't a valid bit pattern for any of T's variants.

That is explicitly desired and by design; the OP of has a motivating use-case for allowing bit patterns in a union that none of the variants allow.

Yato (Jan 02 2020 at 16:50, on Zulip):

As per this comment by @RalfJ, unions do not inherently have any validity constraints. This means that we will have to require the users to opt-in to allow safe union transmute (when transmuting from a union to a non-union type)

RalfJ (Jan 02 2020 at 16:53, on Zulip):

Yeah, that's going to be slightly inconvenient to work around, but nothing major. (I'd love to see an RFC for safely accessing the discriminants of non-fieldless variants!)

EDIT: That's going to be a pain to work around. It's not a theoretical limitation or anything, there's just no easy way to get the discriminant out of a variant without constructing an instance of the variant. I have an idea of how to do it, but it's a bunch of dreadful extra work on the automatic derive that I'm not eager to do tonight. :(

also AFAIK the discriminant of dataful variants is entirely unspecified in both value and representation, so code relying on it or messing with it in any way is probably incorrect. I am not an export on representation details though, so maybe more things are fixed than I know. but please be extremely conservative around enums and only assume things that are explicitly RFC'd! Notice that the unsafe-code-guidelines repo is not RFC'd.

RalfJ (Jan 02 2020 at 16:55, on Zulip):

ah and while we are speaking about unions, one of the things that is not specified for them is the start offset of the fields. for non-repr(C) unions, fields are allowed to be at non-0 offset.

Yato (Jan 02 2020 at 17:00, on Zulip):

@RalfJ We are only dealing with repr(C) types, not even attempting anything on repr(Rust) types. Right now enum's repr(C) representation is guaranteed by RFC#2195

RalfJ (Jan 02 2020 at 17:03, on Zulip):

oh I see. yes that guarantees many things.

Ryan Levick (Jan 03 2020 at 13:13, on Zulip):

@Jack Wrenn @Yato Just catching up now. Very, very cool! I think this type of approach has a lot of promise. I'm guessing one of the biggest issues would be error messages. One extra downside one considering this as a possibility for inclusion in the std lib is that such an approach doesn't really have any precedence in the std lib. Would it be considerable to magical?

Ryan Levick (Jan 03 2020 at 13:34, on Zulip):

@Lokathor @gnzlbg @Yato @Jack Wrenn I know I've asked this before, but I'd like to collect use cases for the motivation for this feature. My main motivating use case is viewing bytes in a structured way for parsing network protocols in an extremely efficient way. As such a feature that allowed [u8; size_of::<T>] to T transmutes (when the array is well aligned) would probably be sufficient for me. I personally don't really need arbitrary casting between byte compatible types.

The other motivation I've seen is from @gnzlbg around packing and unpacking of simd types to and from arrays (of floating points). I don't know enough about simd to fully understand this use case but if you're using simd instructions I can understand why performance is a concern.

What are some other use cases?

gnzlbg (Jan 03 2020 at 13:37, on Zulip):

@Ryan Levick I think my RFC had a collection of examples at the beginning.

gnzlbg (Jan 03 2020 at 13:38, on Zulip):

Feel free to extract those and put them wherever they might be more easily accessible.

gnzlbg (Jan 03 2020 at 13:38, on Zulip):

Compiling a collection of examples of things that we do want to support (and why) would be useful to check whether any design satisfies the constraints.

Ryan Levick (Jan 03 2020 at 13:41, on Zulip):

@gnzlbg The examples feel a bit abstract to me. I'm looking for more higher level descriptions of use cases.

gnzlbg (Jan 03 2020 at 13:43, on Zulip):

Ah ok.

gnzlbg (Jan 03 2020 at 13:43, on Zulip):

The SIMD example is just a sub-case of doing type-punning.

gnzlbg (Jan 03 2020 at 13:44, on Zulip):

That's pretty much what you are doing with network data, file data, or any other example as well.

gnzlbg (Jan 03 2020 at 13:44, on Zulip):

So I suppose you are looking more for examples of useful type punning ?

Ryan Levick (Jan 03 2020 at 13:45, on Zulip):

Yes to prove that we need general type punning capabilities and not specialized ones.

gnzlbg (Jan 03 2020 at 13:46, on Zulip):

The union RFC might also be a good place to search for those. Another example is, e.g., comparing floating-point numbers for equality based on ULPs, using e.g. a pointer address as an usize hash, etc.

gnzlbg (Jan 03 2020 at 13:47, on Zulip):

Some of these are already supported in safe Rust via as, e.g., in the language reference, thin_ptr as usize is guaranteed to not "modify" the bits of the pointer in any way.

gnzlbg (Jan 03 2020 at 13:48, on Zulip):

or 0_isize as usize

gnzlbg (Jan 03 2020 at 13:48, on Zulip):

So there is a certain degree of "safe transmutes" (bit preserving type conversions) in the language already.

gnzlbg (Jan 03 2020 at 13:49, on Zulip):

I see the safe transmute RFC goal as generalizing those for user-defined types.

Ryan Levick (Jan 03 2020 at 13:50, on Zulip):

One reason I'm asking for use cases is whether we can justify things like transmuting for types that require validation (e.g, bool). Is the complexity of supporting such transmutes worth it?

gnzlbg (Jan 03 2020 at 13:50, on Zulip):

e.g. why is 0_isize as usize safe, but (0_isize, 0_isize) as (usize, usize) require an unsafe transmute ?

gnzlbg (Jan 03 2020 at 13:51, on Zulip):

@Ryan Levick good question, my RFC supports that through a different API (a different trait), which does not necessarily need to be part of the first version of the RFC

gnzlbg (Jan 03 2020 at 13:51, on Zulip):

I don't recall what the safe-transmute v2 RFC did

gnzlbg (Jan 03 2020 at 13:52, on Zulip):

but that RFC allowed transmuting types of different sizes, so if you want that, you need some kind of validation

gnzlbg (Jan 03 2020 at 13:53, on Zulip):

e.g. to transmute a fat &[T] into a thin &[T; N]

gnzlbg (Jan 03 2020 at 13:53, on Zulip):

the type of validation you need for that isn't that different of the one required for bool IIRC

gnzlbg (Jan 03 2020 at 13:54, on Zulip):

So I guess my opinion is that I don't know if it is worth it to allow transmutes that require validation, but if we do, then it probably isn't worth it to support some validations but not others.

Ryan Levick (Jan 03 2020 at 13:55, on Zulip):

Yea I agree that any mechanism that supports validation would allow for arbitrary validation

gnzlbg (Jan 03 2020 at 13:56, on Zulip):

The SIMD case could benefit from validation support, e.g., we have SIMD vectors of bools, that people want to transmute from/to byte vectors

gnzlbg (Jan 03 2020 at 13:56, on Zulip):

bool -> u8 is ok, but u8 -> bool requires validation, and so does transmuting a u8xN -> boolxN

gnzlbg (Jan 03 2020 at 13:57, on Zulip):

IIRC packed_simd currently uses TryFrom for those where TryFrom makes sense

Ryan Levick (Jan 03 2020 at 13:58, on Zulip):

FYI: here's a WIP version of v3 of the safe-transmute proposal. It's largely unfinished but the reference guide section is mostly complete:

Ryan Levick (Jan 03 2020 at 13:59, on Zulip):

It would be nice to collect at this into a document of what the current state is. I'd like for @Jack Wrenn's proposal to settle a bit and for us to see how that fits into this.

Jack Wrenn (Jan 03 2020 at 14:03, on Zulip):

Glad to have inspired some thought! I wouldn't call typic a proposal per se, but a demonstration that when Rust's can reason about layouts at inference time, really powerful static guarantees are useful. Implementing this as a library has downsides (like error messages), so there might be a case for having rustc reason 'natively' about layouts.

My motivation is that I view transmute as a soundness footgun. If Rust can improve it's safety guarantees without incurring a runtime cost, it should—and we now know that it can! We can also improve the expressivity of transmute, removing it's unique limitation of not being usable in generic contexts.

My approach (or actually extending the type system to reason about layout natively) also creates an executable semantics of Rust's layout guarantees, which is valuable to have.

Ryan Levick (Jan 03 2020 at 14:08, on Zulip):

@Jack Wrenn thanks for providing some background on your motivation. I think it's fair to say that while we should consider type level reasoning about type layout, but that we shouldn't necessarily expect a _full_ proposal on a safe transmute feature from you (though of course we're happy to have your input and help with that!).

Ryan Levick (Jan 03 2020 at 14:13, on Zulip):

I wonder who the right person is to talk to about the viability of type level constraints as a solution that is likely to be accepted into the std lib. Perhaps @nikomatsakis has some ideas?
Niko, the tl;dr is that @Jack Wrenn has written a proof of concept that allows encoding of a type's layout in the type system (as a trait) such that at the type level we can tell whether two types have compatible layouts. Besides talking with @Josh Triplett about this from the lang team's perspective, is there a person from the compiler team who would be good to talk to about the viability of such an approach to ensure safe transmute?

Yato (Jan 03 2020 at 14:22, on Zulip):

Another issue is performance, as types scale up, the performance of this method dies (but only for really large types). For example, converting between 2 4-kilobyte types took about a minute to compile. If there are any errors you get a full screenful of a type.
I think that Rust can do better, especially wrt error messages and performance. Now a 4-kilobyte type is huge, so this may not be a problem, but doing lots of these checked transmutes could add up in the same way.

Yato (Jan 03 2020 at 14:26, on Zulip):

I personally would not want the type based checked transmutes in std, because it is really complex and subtle. It be far more reliable and maintainable if these computations were baked into the compiler.

gnzlbg (Jan 03 2020 at 14:36, on Zulip):

I wonder who the right person is to talk to about the viability of type level constraints as a solution that is likely to be accepted into the std lib. Perhaps @nikomatsakis has some ideas?

I can imagine how to do this reliably using traits, but haven't looked at typic. I can't imagine a way of doing this reliably and efficiently as a library.

gnzlbg (Jan 03 2020 at 14:37, on Zulip):

@Jack Wrenn do you have any compile-time benchmarks for typic ?

gnzlbg (Jan 03 2020 at 14:37, on Zulip):

e.g. we used to have benchmarks for meta-programming like this in C++, for example, in

gnzlbg (Jan 03 2020 at 14:38, on Zulip):

In a nutshell one just needs to automatically generate a file containing two types, and performing a typic conversions, and measure the compile-time.

gnzlbg (Jan 03 2020 at 14:38, on Zulip):

You then increase the size of these types, e.g., by adding more fields, and plot the compile-time as a function of, e.g., the type fields.

gnzlbg (Jan 03 2020 at 14:39, on Zulip):

You can then also add more types, and plot the compile-times as a functions of the types using typic

Jack Wrenn (Jan 03 2020 at 14:40, on Zulip):

Not yet. If you don't transmute between two types, you shouldn't pay any penalty for computing the layout of those types.

My implementation aims for simplicity in the low level representation, not efficiency. @Yato followed up with a more efficient representation without too much increased complexity.

I like your benchmarking idea!

gnzlbg (Jan 03 2020 at 14:41, on Zulip):

I've thought about writing a tool to "hammer" the rust compiler, by allowing to easily specify production rules for tests like these, but never got the time to do it. There are many language features for which compile-times currently explode once you add more cases (e.g. a match 0_i32 { 0 => ..., 1 => ..., ... i32::MAX => ... } where compile-times grow as a function of the match arms in a non-linear way).

Yato (Jan 03 2020 at 17:16, on Zulip):

I'm not sure if my way is more efficient, just that it has less special casing.

Lokathor (Jan 03 2020 at 18:44, on Zulip):

(What happened to your last list of needs? keep this one in a safe place this time :P )

I've had the following general needs:

Jack Wrenn (Jan 10 2020 at 18:24, on Zulip):

I wonder who the right person is to talk to about the viability of type level constraints as a solution that is likely to be accepted into the std lib. Perhaps @nikomatsakis has some ideas?

I can imagine how to do this reliably using traits, but haven't looked at typic. I can't imagine a way of doing this reliably and efficiently as a library.

Incidentally, I think I've encountered an obstacle in my implementation of enums. In short, there's one case where I need to encode a type-level Filter operation. However, I don't know how to that without overlapping implementations. Overlapping implementations are supported with #[marker], but _not_ on traits with associated types. A Filter-like trait needs an associated type to encode its output.

A brief experiment suggests that chalk is already capable of expressing this.

Jack Wrenn (Jan 10 2020 at 22:03, on Zulip):

Actually, disregard the above! I think I found a workaround.

Jack Wrenn (Jan 11 2020 at 16:54, on Zulip):

For the curious, I've started modeling the type-level compatibility checking rules with Prolog. Prolog is really a fantastic way to experiment with typelevel programming in Rust. Rust's type system is very similar to Prolog, and Prolog predicates can be written in a way that admits straight-forward translation to Rust code. The repo is here:

Lokathor (Jan 11 2020 at 20:09, on Zulip):

I'm nervous about this being computed at compile time every build.

How have the build times been with this approach?

Jack Wrenn (Jan 11 2020 at 20:52, on Zulip):

I haven't benchmarked anything, but in the process of developing and testing typic, compilation has been basically instantaneous. There's a few costs to consider:

The library itself:

  1. typic itself builds nearly instantly, but relies on typic-derive
  2. typic-derive requires syn, which has significant one-off build time.

Clients using the library:

  1. For types marked with #[typic::repr(C)], a trait implementation is generated with an associated type that corresponds to the high-level structure of that type. It's akin to frunk::Generic. This is just a syntactic transformation of the type. There is no type-level computation happening yet.
  2. The low-level representations of types are only computed if the user tries to transmute between them. How expensive this is depends on the complexity of the types. For small types, it's basically instantaneous.
Jack Wrenn (Jan 11 2020 at 20:56, on Zulip):

That last item is really the only expensive operation. The good news is that it's paid on-demand, only for the types you try to transmute between.

I'm unsure to what extent this cost is paid on every build. It might be the case that incremental compilation caches most of the computation. I'll have to look into it.

Yato (Jan 13 2020 at 01:21, on Zulip):


pub struct T6<T> {
    pub a: T,
    pub b: T,
    pub c: T,
    pub d: T,
    pub e: T,
    pub f: T,

pub fn foo(x: T6<u64>) -> T6<u64> {
    use typic::transmute::TransmuteFrom;

took 7.72 s to build on my computer (just this, nothing else). So performance is definitely a concern.

Yato (Jan 13 2020 at 01:24, on Zulip):

Incremental compilation helps, but Rust need to redo all the work for every call to transmute_from

Yato (Jan 13 2020 at 01:43, on Zulip):

That said, I just tested my version, and a the equivalent code (shown below) built in under a second

pub struct T6<T> {
    pub a: T,
    pub b: T,
    pub c: T,
    pub d: T,
    pub e: T,
    pub f: T,}

pub fn tac(x: T6<u64>, (): ()) -> T6<u64> {
    use chtrans::Reinterpret;
Yato (Jan 13 2020 at 01:45, on Zulip):

(although I did find out that I didn't support generic types till now, so there's that)

Yato (Jan 13 2020 at 01:48, on Zulip):

Just tried to push it a bit, and I broke the default recursion limit :)

This took ~9 s


pub struct T4<T> {
    pub a: T,
    pub b: T,
    pub c: T,
    pub d: T,
    pub e: T,
    pub f: T,
    pub g: T,
    pub h: T,

pub struct T8<T> {
    pub a: T,
    pub b: T,
    pub c: T,
    pub d: T,
    pub e: T,
    pub f: T,
    pub g: T,
    pub h: T,

pub fn foo(x: T8<T4<u64>>, (): ()) -> T4<T8<u64>> {
    use chtrans::Reinterpret;
Yato (Jan 13 2020 at 01:56, on Zulip):

the largest I'm willing to try took ~27s

pub struct Big {
    pub _a: [u64; 1],
    pub _b: [u64; 2],
    pub _c: [u64; 3],
    pub _d: [u64; 4],
    pub _e: [u64; 5],
    pub _f: [u64; 6],
    pub _g: [u64; 7],
    pub _h: [u64; 8],
    pub a: [u64; 9],
    pub b: [u64; 10],
    pub c: [u64; 11],
    pub d: [u64; 12],
    pub e: [u64; 13],
    pub f: [u64; 14],
    pub g: [u64; 15],
    pub h: [u64; 16],

pub fn foo(x: Big) -> Big {
    use chtrans::Reinterpret;

Rust seems to be good at caching results sometimes, so having a lot of different types helps kill build times.

Jack Wrenn (Jan 14 2020 at 00:50, on Zulip):

Yikes! Are there any mechanisms for tracing trait resolution? I'd be very interested to know how the time is being spent.

Yato (Jan 14 2020 at 03:44, on Zulip):

I am not aware of any

gnzlbg (Jan 17 2020 at 16:33, on Zulip):

Trait resolution is just very poorly suited for this

gnzlbg (Jan 17 2020 at 16:33, on Zulip):

This problem is common enough that it deserves a good solution

Lokathor (Jan 18 2020 at 01:14, on Zulip):

Is there a specific proposal to go with that?

gnzlbg (Jan 20 2020 at 12:51, on Zulip):

There is the Compatible<T> rfc

gnzlbg (Jan 20 2020 at 12:52, on Zulip):

and the safe-transmute v2/v3 rfcs

gnzlbg (Jan 20 2020 at 12:54, on Zulip):

but the transitive closure in the Compatible<T> RFC only works for transmuting types that have the same size

gnzlbg (Jan 20 2020 at 12:55, on Zulip):

I don't think there is a good solution with transitive closure for transmuting types of different sizes into each other

gnzlbg (Jan 20 2020 at 12:55, on Zulip):

although there are some

gnzlbg (Jan 20 2020 at 12:57, on Zulip):

e.g. one could make the TryCompatible<T> trait in the compatible-RFC have transitive closure, by eliminating the associated error type

gnzlbg (Jan 20 2020 at 12:58, on Zulip):

at the cost of loosing the ability to specify custom error conditions

Jack Wrenn (Feb 05 2020 at 21:25, on Zulip):

I was curious how far a Safer Transmute RFC could get in which every example was explorable with Typic, and the answer is pretty far:

All of the guidance in that document is implemented in Typic. The examples that ought to work actually compile and run. The examples that ought to fail actually produce compile errors.

(Well, except that the automatic Transparent impl currently happens a little more conservatively than I've described in that document: the implementation requires that all fields are pub, not merely that they have visibility greater than or equal to their parent type, as specified. I'll fix that tomorrowish.)

Jack Wrenn (Feb 05 2020 at 21:32, on Zulip):

@Yato The new implementation is also much faster. Your Big testcase now takes under a second on my laptop. With incremental builds thereafter, I don't notice any slowdown at all.

Yato (Feb 07 2020 at 16:40, on Zulip):

@Jack Wrenn Cool! What was the big slowdown?

Yato (Feb 07 2020 at 16:51, on Zulip):

Looks like you went through a big rewrite recently, can I get a brief summary of what changed? It looks like you moved away from frunk into some custom type encodings like I did, but it's a big change and I'm not sure how it all fits together in my brief look at it.

Jack Wrenn (Feb 07 2020 at 17:34, on Zulip):

@Yato The major representational change is that rather than representing type layouts as an HList of one-byte slots, the leaves of the new low-level representation have both a Kind and a Size.

So, eight adjacent padding bytes is no longer an HList of eight Uninitialized slots. It's now one slot with Kind=Uninitialized and Size=U8.

I'm guessing this change is responsible for the speedup, but I never actually traced trait resolution on the initial version, so I can't be absolutely sure.

Jack Wrenn (Feb 07 2020 at 18:17, on Zulip):

As far as the public API goes, I refocused typic on strictly on infallible conversions. The public API of the crate exactly models what I think the public API of an initial, general, safe transmute RFC should be:

A general solution for fallible conversions is much trickier, because error reporting is hard to get right.

Lokathor (Feb 07 2020 at 22:33, on Zulip):

What's so hard to get right?

Jack Wrenn (Feb 07 2020 at 23:00, on Zulip):

Let's say we want to transmute from u16 to Constrained:

struct Constrained(NonZeroU8, bool);

On a little-endian system, this is only a valid transmutation if our u16 is greater than 0 and less than 512. In all other cases, a general-purpose, fallible, sound transmuter should be able to report which bytes of our source type violated the layout of the destination type.

A general-purpose, fallible, safe transmuter should be able to communicate why user-defined invariants are violated, too. (E.g., if Constrained also required that it's first field was an odd number.)

Shnatsel (Feb 08 2020 at 19:01, on Zulip):

Is the granularity of "alignment", "size" and "type invariants" not enough?

Shnatsel (Feb 08 2020 at 19:05, on Zulip):

I don't think you can reasonably report what type invariant exactly was broken in code that's this generic.

Shnatsel (Feb 08 2020 at 19:06, on Zulip):

I've just been looking into this in the context of and that's the only errors that any code I can imagine really cares to distinguish between. If it's an alignment problem, the fix is just copy the thing and make the alignment correct; if it's a size or broken type invariant problem, then we reject the input as invalid. That's it.

Shnatsel (Feb 08 2020 at 19:06, on Zulip):

OTOH I like the idea of keeping the scope contained and getting infallible conversions right first, then experimenting with fallible ones

Yato (Feb 09 2020 at 06:08, on Zulip):

Yes, encoding the size would speed things up, especially because typenum has O(log n) operations as opposed to mine at O(n).

I agree that we should only focus on infallible conversions as a start.

Jack Wrenn (Mar 09 2020 at 17:40, on Zulip):

So, I'm thinking about how bit validity interacts with mutation.

Consider these two type definitions:

struct Foo(pub NonZeroU8);
struct Bar(pub u8);

Given these types,

let mut foo: Foo = ...;
  let mut bar : &mut Bar = transmute(&mut foo);
  bar.0 = 0
assert_eq!(foo.0.get(), 0); // ack!!!

This suggests a rule: bit validity can be widened with a value-to-value transmute, but is totally invariant for a mutable-reference-to-mutable-reference transmute.

Jack Wrenn (Mar 09 2020 at 17:41, on Zulip):

Except, now let's consider:

struct Foo(pub NonZeroU8);
struct Bar(pub AtomicU8);

Given those types,

let foo = Foo(NonZeroU8::new(1).unwrap());
  let bar: &Bar = core::mem::transmute(&foo);, Ordering::Relaxed);
assert_eq!(foo.0.get(), 0); // ack!!!

I'm not really sure what rule is suggested by this.

Jack Wrenn (Mar 09 2020 at 17:54, on Zulip):

A very coarse rule is that validity is invariant whenever references (shared or unique) are involved.
I think the actual rule is something like: &T&U is safe only if U does not contain any UnsafeCells.

Joshua Liebow-Feeser (Mar 09 2020 at 17:57, on Zulip):

As for the first point, the rule I use in zerocopy is that &mut Foo -> &mut Bar requires that Foo -> Bar and Bar -> Foo since the reference allows you to both view a Foo as a Bar and also store a Bar in the location that may be later viewed as a Foo. There are more constraints besides (as you've pointed out with your atomic example), but those requirements exist at a minimum.

Jack Wrenn (Mar 09 2020 at 18:12, on Zulip):

Typic used a similar rule for a while, but ultimately abandoned it, because it forbids safe transmutations between types of different sizes. Given:

type A0 = [u8; 0];
type A9 = [u8; 9];

...we have the following transmutations:

  1. A9A0
  2. A0A9
  3. &A9&A0
  4. &A0&A9

That rule eliminates transmute №3, since A0A9.

Lokathor (Mar 09 2020 at 20:12, on Zulip):

I wouldn't say that A9 -> A0 is valid in the first place

Jack Wrenn (Mar 09 2020 at 22:26, on Zulip):

Why not?

Lokathor (Mar 10 2020 at 02:29, on Zulip):

I would not allow different sized owned casting

Lokathor (Mar 10 2020 at 02:30, on Zulip):

it's relatively poorly defined and you can generally have the user reference manipulate things to get the cast they want if they really do want to truncate things

Jack Wrenn (Mar 10 2020 at 02:31, on Zulip):

It's maybe a bit weird, but not at all unsound. :shrug:

Lokathor (Mar 10 2020 at 02:34, on Zulip):

On the scale of good APIs there's a small margin just above "strictly speaking sound" where the API is still a footgun

Jack Wrenn (Mar 10 2020 at 02:38, on Zulip):

Sure. Right now, Typic is scoped to just telling you when you're doing something unstable, unsafe, or unsound—but not insensible.

Jack Wrenn (Mar 10 2020 at 02:39, on Zulip):

Although I've actually legitimately used shrinking transmutes of arrays!

Lokathor (Mar 10 2020 at 02:52, on Zulip):

Sure, there are correct uses, but usually you should sub-slice it down to a desired size and use the sub-slice

HeroicKatora (Mar 10 2020 at 09:53, on Zulip):

The cast A9 -> A0, in that case, seems as if it should be implemented as a conversion by reference followed by deref+Copy to me.
But it makes a point, that this is not always possible and requires A0: Copy.

Lokathor (Mar 10 2020 at 16:40, on Zulip):

You also probably really shouldn't be transmuting non-Copy data in the first place ;P

Last update: Apr 05 2020 at 00:55UTC