Stream: project-safe-transmute

Topic: mem-markers v0.1


Ryan Levick (May 14 2020 at 17:27, on Zulip):

Hey all. I'm thinking about making a public announcement to mem-markers so we can get people discussing it. Please let me know if you think that releasing will lead to quality discussion. https://github.com/rust-secure-code/mem-markers
Of particular interest is how we handle this: https://github.com/rust-secure-code/mem-markers/issues/6 It seems like this is in a way a meta trait. For instance, for zeroable, we need to decide if the trait means, this type is fine to be viewed at any point as zeros OR one particular state is has is zeros but it has invariants it keeps track off so don't just std::mem::zeroed initialize it.

Lokathor (May 14 2020 at 21:17, on Zulip):

In general, more eyes is probably a good idea. for the zeroed thing i'll keep the discussion of it inside that thread so it doesn't get spread out.

Thom Chiovoloni (May 19 2020 at 21:52, on Zulip):

That issue was confusing to me, as it's not clear if it means "This trait indicates a type has no internal validity requirements" or if it means it's repr(transparent) around a wrapped type and imposes no additional constraints upon the wrapped type. Different comments seem to be discussing different things

Casey Rodarmor (May 25 2020 at 06:28, on Zulip):

I would definitely like to use this!

I am writing a serialization library that casts to and from bytes, and was just about to write versions of NoUninit, FromBytes, and ToBytes myself.

I notice that FromBytes isn't implemented for f64 and f32, but I think perhaps it can be, since f32/64::from_bits is using a transmute from u32/64 under the hood.

One thing that I would like to see would be a trait that asserts that a type has alignment == 1. In my case, my format does not include padding, so conversion from byte strings with any alignment must be safe.

Would you be interested in an Unaligned trait via PR?

Are you planning on publishing the crate to crates.io?

I think you might find the zerocopy crate interesting, which has some similar traits: https://docs.rs/zerocopy/0.3.0/zerocopy/

Lokathor (May 25 2020 at 06:35, on Zulip):

I think that traits for align==X is certainly possible.

Lokathor (May 25 2020 at 06:38, on Zulip):

note that owned casts (such as [u8;4] to u32) can always be done safely even if the target type is a higher alignment because when the value is moved across the casting function call and back it implicitly moves to a location with the necessary alignment.

Ryan Levick (May 25 2020 at 14:24, on Zulip):

Would you be interested in an Unaligned trait via PR?

Absolutely! I'm not so sure if I love the name as it's not necessarily _un_aligned but rather always aligned.

Ryan Levick (May 25 2020 at 14:25, on Zulip):

Are you planning on publishing the crate to crates.io?

Yep, I'll probably announce this today on Twitter and then publish 0.1 soon

Ryan Levick (May 25 2020 at 14:29, on Zulip):

pinging @Lokathor @Joshua Liebow-Feeser @Jack Wrenn since I'm probably going to announce mem-markers today, and there's naturally a bit of overlap with your crates. If people ask what the relationship is, my official response is that mem-markers is much lower level and more general purpose. It's meant to allow programmers to explicitly state memory invariants of their types as traits which can help with safely implementing transmute but is also useful in other situations. Thoughts?

Ryan Levick (May 25 2020 at 14:42, on Zulip):

Also going to ping @RalfJ since this is highly relevant to unsafe. @RalfJ, please let me know what you think of the overall direction of the crate. The idea is to take these traits and eventually move most if not all into the std lib so that people can more easily express what type of memory invariants their types have. https://github.com/rust-secure-code/mem-markers

Ryan Levick (May 25 2020 at 14:42, on Zulip):

And of course, @Josh Triplett since he was helping me with the safe transmute pre-RFC which led to this work

Jack Wrenn (May 25 2020 at 14:56, on Zulip):

@Ryan Levick What do you mean by "lower level"? Zerocopy and typic also define general-purpose marker traits. At a glance, mem-markers seems pretty similar to zerocopy.

Typic also has the goal of being useful outside of transmutes, like communicating abi-breaking changes. In the latest stab at stability marker traits, you express lower and upper bounds for a type's alignment, validity and size. So, this lets you do clever things like expressing that you _might_ add a field to a struct where there once was padding.

Ryan Levick (May 25 2020 at 15:02, on Zulip):

Hmmm then perhaps there is more overlap then I had thought. mem-markers is meant to be _only_ markers (and derives for those markers) with essentially no functionality built in. This is to encourage its usage across many different libraries that might have wide range of uses. This seems like a much easier sell for moving into the std lib as its as flexible as you can get and ultimately pretty forward proof. My thought was that zerocopy and Typic could be built on top of mem-markers, and that the existing marker traits in those crates could be deprecated in favor of the traits in mem-markers

Ryan Levick (May 25 2020 at 15:03, on Zulip):

For example, mem-markers will never expose a safe-transmute function, but one could easily be written using it.

Ryan Levick (May 25 2020 at 15:04, on Zulip):

@Jack Wrenn For some libraries at work, I could easily imagine us using mem-markers (because it's so straight forward), but the use of Typic would be a harder sell since it does more

Jack Wrenn (May 25 2020 at 15:06, on Zulip):

Typic could never be implemented atop the mem-markers traits, unfortunately; it's basis is Frunk's Generic trait and another trait that encodes any repr modifiers.

Lokathor (May 25 2020 at 15:07, on Zulip):

I think the position should be that mem-markers aims to replace most or all of the functionality/purpose of the existing casting crates.

Lokathor (May 25 2020 at 15:09, on Zulip):

we're moving things out of crates.io crates and into core and the compiler.

Ryan Levick (May 25 2020 at 15:10, on Zulip):

It seems like @Jack Wrenn is of the opinion that typic and men-markers overlap completely. Is that true, Jack?

Ryan Levick (May 25 2020 at 15:12, on Zulip):

I’m not sure though if the hope that typic moves into core, but that is directly a hope of men-markers

Ryan Levick (May 25 2020 at 15:13, on Zulip):

(I can write up something for the readme comparing these crates and send it around for people to give their feedback)

Jack Wrenn (May 25 2020 at 15:13, on Zulip):

Typic's approach is strictly more powerful than mem-markers. There aren't any markers you can express with the mem-markers approach that can't be expressed with typic (but the reverse isn't true).

The mem-markers approach of picking a few marker traits and trying to gate their derives on macro-based checks is substantially similar to zerocopy.

Ryan Levick (May 25 2020 at 15:16, on Zulip):

I’d agree with that. The advantage of men-markers I’d argue is simplicity (both implementation wise and conceptually).

Jack Wrenn (May 25 2020 at 15:18, on Zulip):

And yes, my (perhaps unrealistic) hope would be of getting a safe transmute into core, that's as easy to use as the existing transmute intrinsics but with none of the dangers. I think typic's layout stability markers are also much more expressive, but they're not checkable without its safe-transmute machinery. (For a type T, you provide two types A and B. You promise that T is always transmutable _into_ A, and always transmutable _from_ B. A change to T that breaks these promises is a breaking change. This is enforced by typic.)

Jack Wrenn (May 25 2020 at 15:20, on Zulip):

@Ryan Levick If we instead integrated a frunk-style Generic trait into core, plus a marker trait that's implemented for things that are repr(C), both the mem-markers and typic approaches could be built atop it.

Ryan Levick (May 25 2020 at 15:24, on Zulip):

:thinking: I’m not saying that’s a bad idea, I’m just saying I’m skeptical enough that I think having another approach would be worth exploring. I agree that the men-markers approach is related to zero-copy but it’s a slightly simplified version. I think having that as a “competitor” (in the friendliest sense of the word) to typic is worth it 😀. @Jack Wrenn do you think releasing men-markers will make the Rust ecosystem worse?

Jack Wrenn (May 25 2020 at 15:26, on Zulip):

Oh, not at all. I just mean: if you want to put something in core, there's a slightly different foundation that accommodates both of our approaches. I don't want to close the door on _ever_ providing a safe transmute function.

Ryan Levick (May 25 2020 at 15:31, on Zulip):

Yea I'm not saying that I necessarily want mem-markers to go into core _as it is right now_. I have some reservations about providing type level programming primitives in core, but I don't think those reservations are relevant here.
So Typic covers the same purposes as mem-markers and more (e.g., providing a safe transmute function). Typic aims to get there by using type level programming while mem-markers aims for a less powerful solution using simple marker traits and derive macros. While we may each have our favorites in this race, we're both willing to say that the other should exist to make sure we ultimately end up with the right solution.

Ryan Levick (May 25 2020 at 15:34, on Zulip):

I'll add this to the readme. @Lokathor thoughts on how mem-markers relates to bytemuck?

Ryan Levick (May 25 2020 at 15:35, on Zulip):

And @Joshua Liebow-Feeser I assume your opinion is that you hope zerocopy will eventually be built on top of typic (or deprecated in favor of it)?

Ryan Levick (May 25 2020 at 15:42, on Zulip):

@Jack Wrenn how would one express in typic that a type is zeroable?

Jack Wrenn (May 25 2020 at 15:48, on Zulip):

Something like this:

#[repr(u8)] enum Zero { Z = 0u8; }

pub trait Zeroable {}

impl<T> Zeroable for T
where T: TransmuteFrom<Array<Zero, SizeOf<T>>>
{}
Ryan Levick (May 25 2020 at 15:49, on Zulip):

Is the plan to expose that trait in typic or leave that to another crate?

Ryan Levick (May 25 2020 at 15:52, on Zulip):

Because if the plan is _not to_ then really typic could be seen as lower-level than mem-markers, and one could imagine implementing mem-markers in terms of typic. But even still, having a Zeroable trait that everyone can use no matter its implementation is useful

Ryan Levick (May 25 2020 at 15:57, on Zulip):

Comparison of crates in README: https://github.com/rust-secure-code/mem-markers/pull/19

Jack Wrenn (May 25 2020 at 15:59, on Zulip):

Yeah, I agree! Typic's approach is super expressive.

As its developer but not end-user, I've waffled on how much of typic's guts to expose. I'd like Typic to demonstrate that it provides a _really_ powerful foundation for other abstractions like Zeroable, but I also want to minimize the allergic reactions that would surely occur if it even appeared like I was proposing exposing typenum in the core library.

So, TransmuteFrom and TransmuteInto don't expose any gnarly type-level programming stuff, but surfacing SizeOf does.

Having talked to Josh a bit about the future of typic, it makes sense from an end-user perspective to provide some nice shorthands like FromBytes, FromZeros and AsBytes.

Ryan Levick (May 25 2020 at 16:04, on Zulip):

Having talked to Josh a bit about the future of typic, it makes sense from an end-user perspective to provide some nice shorthands like FromBytes, FromZeros and AsBytes.

Its obviously still super early, but one thing that would make Typic more digestable for use in projects at my work would be for there to be a minimal typic which only exposes TransmuteFrom/TransmuteInto and related low-level machinary for expressing invariants. Then have something like mem-makers which exposes a bunch of short hand traits implememented in terms of the low level machinary. And _then_ a high level zerocopy style crate which packages all this up and exposes a nice (and maybe slightly opinionated) set of APIs.

Most would use the high level APIs, those doing something a bit trickier might use the mid level (mem-markers) style crate, and only in really desparate times would you need to reach for the lowest level crate

Ryan Levick (May 25 2020 at 16:06, on Zulip):

The thing is that the mid-level mem-markers crate being implemented in terms of the low level machinary is really just an implementation detail. I could imagine releasing mem-markers as it is today, and in a non-breaking way, moving to implementation being done in terms of low level typic machinary

Lokathor (May 25 2020 at 16:06, on Zulip):

My hope is that mem-markers and other work with core library functions can completely move bytemuck into core

Lokathor (May 25 2020 at 16:08, on Zulip):

I'd probably write a 2.0 version that is re-done on top of core and keeps the few things that probably won't end up in core as soon, like the offset_of! macro

Jack Wrenn (May 25 2020 at 16:10, on Zulip):

The phrases "high level" and "low level" seem a bit wonky to me.

In one respect, typic's TransmuteFromis "low level" because things like FromBytes can be implemented on top of it (but not the other way around).

But, in another respect, using TransmuteFromis as simple as calling .transmute(). There's no worrying about implementing marker traits, or proving correctness yourself. TransmuteFrom feels like a very high-level API.

Typic's notion of "low level" basically includes all the things it doesn't expose (for the most part): size, alignment and validity calculations on individual types.

Ryan Levick (May 25 2020 at 16:16, on Zulip):

I think that's a fair criticism of the terms low-level and high-level. The fact remains that exposing "high-level" marker traits is useful. For example, the zeroable trait we discussed above. But I suppose you would hope to see those directly in typic at some point

Jack Wrenn (May 25 2020 at 16:18, on Zulip):

Yeah, they kinda sorta already exist as a documentation example: https://docs.rs/typic/0.2.3/typic/layout/index.html

Ryan Levick (May 25 2020 at 16:19, on Zulip):

Yea but the utility isn't from being easy to create them yourself. It's from having a ecosystem-wide agreed upon set of traits for expressing these concepts. There's much less utility when each crate has to define their own AsBytes,Zeroable, etc.

Ryan Levick (May 25 2020 at 16:22, on Zulip):

I guess what I'm coming to realize is that mem-markers are useful no matter if they're implemented like they are today or implemented in terms of typic. They exist as a separate crate as to not tie themselves unnecessarily to a particular implementation. Perhaps, they will one day be built on typic primitives, but that should be transparent to the user.

Jack Wrenn (May 25 2020 at 16:25, on Zulip):

Yes, however, one thing that will make that very difficult to ever do is typic takes a blanket impl approach. Turning a non-blanket impl'd trait into a blanket-impl'd trait is a breaking change.

Ryan Levick (May 25 2020 at 16:28, on Zulip):

That's a good point. Though I'm having a hard time picturing what exactly would break.

Jack Wrenn (May 25 2020 at 16:31, on Zulip):

There's a distinction between it being safe to FromBytes a type, and it being stable to do so.

When a trait like FromBytes is opt-in, implementing it implies that it's both safe and stable to rely on that behavior.

Ryan Levick (May 25 2020 at 16:32, on Zulip):

All mem-marker types are currently opt-in. Are you saying that typic won't allow for opt-in traits?

Jack Wrenn (May 25 2020 at 16:33, on Zulip):

Yep. An trait with a blanket implementation cannot also have opt-in implementations (well, until #[marker] is stabilized).

Jack Wrenn (May 25 2020 at 16:33, on Zulip):

How typic handles this is its traits take an extra Options parameter. By 'default', that parameter is (), which means typic should perform all checks. If you want to neglect the stability check (which is the only check that can be safely neglected), you must state so utterly explicitly: e.g., T: TransmuteFrom<U, NeglectStability>

Jack Wrenn (May 25 2020 at 16:35, on Zulip):

So, basically, typic distinguishes between whether something can soundly be done, and whether it can stable-y be done.

Ryan Levick (May 25 2020 at 16:38, on Zulip):

Yea mem-markers uses different traits for that (or at least will at some point).

Ryan Levick (May 25 2020 at 16:51, on Zulip):

Typic could be used to verify validity of the derive macros in mem-markers but yea you're right that the mem-markers themselves can't be implemented in terms of typic primitives.

RalfJ (May 29 2020 at 15:44, on Zulip):

Ryan Levick said:

Also going to ping RalfJ since this is highly relevant to unsafe. RalfJ, please let me know what you think of the overall direction of the crate. The idea is to take these traits and eventually move most if not all into the std lib so that people can more easily express what type of memory invariants their types have. https://github.com/rust-secure-code/mem-markers

(without having read this entire thread here)
I am pretty swamped I'm afraid, so I can't do an in-depth review. the approach of pinning down interesting properties of types and reflecting them in traits makes a lot of sense to me. some of it however will rely on things like "is there padding" which is not easily available a a trait bound.

Regarding the concrete traits that you have:

RalfJ (May 29 2020 at 15:45, on Zulip):

As a general advice, it would help to list for each trait two examples: a type that satisfies the trait, and a type that does not -- picked as well as possible to indicate the distinction that this trait is making.

Lokathor (May 29 2020 at 16:53, on Zulip):

if you can't rely on an i32 being init both unsafe and safe are in an immense amount of trouble.

RalfJ (May 29 2020 at 17:36, on Zulip):

@Lokathor to be clear, the safety invariant definitely says that it is init

RalfJ (May 29 2020 at 17:36, on Zulip):

but whether the validity invariant also says that is up for debate, and I dont think answering this with "no" (i.e., allowing uninit) will necessarily cause trouble

RalfJ (May 29 2020 at 17:36, on Zulip):

public APIs have to handle all safe instances of a type, not all valid ones.

Lokathor (May 29 2020 at 17:47, on Zulip):

I think that the mem-markers should exclusively work in terms of safe, or at least have any "valid but unsafe" situations specifically marked and documented and such.

but that hasn't been specifically decided upon yet.

RalfJ (May 30 2020 at 07:43, on Zulip):

Fair. But I think the crate should be explicit about that.

Last update: Jun 05 2020 at 23:05UTC