Stream: project-safe-transmute

Topic: Marker Traits


Ryan Levick (Mar 12 2020 at 10:14, on Zulip):

I've created a repo for defining base marker traits that can be shared among all implementations of safe transmute. Again, the idea is that this repo would be the launching pad for any traits that would wind up in the std lib. Here is the first issue for discussing FromBytes/AsBytes https://github.com/rylev/mem-markers/issues/1 (note: this is on my personal GitHub until we can transfer to the rust-secure-code organization)

Ryan Levick (Mar 12 2020 at 16:37, on Zulip):

@Jack Wrenn @Joshua Liebow-Feeser one thing I would love to talk more about is how useful these marker traits are or are not to Typic

Ryan Levick (Mar 12 2020 at 17:10, on Zulip):

Thanks to @Shnatsel the repo has been moved: https://github.com/rust-secure-code/mem-markers

Shnatsel (Mar 12 2020 at 17:14, on Zulip):

I've invited some people as admins to this repo but I'm sure I'm forgetting someone. If you want write access and did not receive an invite, ping me

Ryan Levick (Mar 20 2020 at 11:35, on Zulip):

I'd like to start implementing some of the traits. Does anyone have any objections to this: https://github.com/rust-secure-code/mem-markers/issues/1. I assume this crate also will support custom derives for each of them. Any objections to that?

Ryan Levick (Mar 20 2020 at 11:36, on Zulip):

@Jack Wrenn @Joshua Liebow-Feeser I'd also love to hear from you two since it's the least clear how well this plays into a typic style solution

Jack Wrenn (Mar 20 2020 at 17:15, on Zulip):

I've left comments addressing your question on the github issue: https://github.com/rust-secure-code/mem-markers/issues/1#issuecomment-601810244

Lokathor (Mar 20 2020 at 21:09, on Zulip):

I think that ToBytes and FromBytes are perhaps too broad and could be broken up into sub-markers and then you have ToBytes any time you have NoUninit and also FixedLayout

Lokathor (Mar 20 2020 at 21:11, on Zulip):

But they should exist somewhere, so feel free to go ahead.

Ryan Levick (Apr 15 2020 at 15:42, on Zulip):

Finally got to implementing some of the marker traits stuff. Here's a PR that implements FixedLayout (on any type that has a fixed layout): https://github.com/rust-secure-code/mem-markers/pull/7 It's not complete but puts the beginnings in place. I want to work on a NoUninit trait but I'm not sure how I'll do it. It's to tell if fields are NoUninit, but we'd need to have some idea of size of types to calculate if there is padding. Any ideas on this are welcome :-)

Ryan Levick (Apr 15 2020 at 15:49, on Zulip):

It might be easier to do a ByteComplete trait which says that a type is valid no matter what bytes compromise it. I believe FromBytes: FixedLayout + ByteComplete

Ryan Levick (Apr 16 2020 at 14:17, on Zulip):

I unfortunately can't come up with a way to calculate type sizes at compile time and make compile decisions based on them. So I'm not sure how ToBytes can be done.

Ryan Levick (Apr 16 2020 at 14:35, on Zulip):

I guess I will go with how zerocopy does it where they divide by zero in a const expression

Ryan Levick (Apr 16 2020 at 15:34, on Zulip):

@Lokathor NoUninit means 99% of the time no padding. But are there any cases where it's interesting to know a type is NoUninit but is ?HasPadding?

Lokathor (Apr 16 2020 at 15:39, on Zulip):

I don't think a type can even have padding bytes that are not uninit

Ryan Levick (Apr 16 2020 at 15:48, on Zulip):

@Lokathor I mean a type can have uninit bytes that are not padding

Ryan Levick (Apr 16 2020 at 15:49, on Zulip):

So there is a subtle distinction between a type with no padding and a type with no uninit bytes.

Lokathor (Apr 16 2020 at 15:50, on Zulip):

oh, right

Lokathor (Apr 16 2020 at 15:50, on Zulip):

yeah NoUninit is the stronger claim

Ryan Levick (Apr 16 2020 at 15:52, on Zulip):

So the question is, should we make a distinction?

Ryan Levick (Apr 16 2020 at 15:52, on Zulip):

Should there be a NoPadding trait and a NoUninit: NoPadding trait?

Ryan Levick (Apr 16 2020 at 15:53, on Zulip):

MaybeUninit<u8> is NoPadding but it's not NoUninit

Lokathor (Apr 16 2020 at 16:12, on Zulip):

I don't see the value in NoPadding on it own

Lokathor (Apr 16 2020 at 16:13, on Zulip):

it doesn't let you do anything different

Ryan Levick (Apr 16 2020 at 16:13, on Zulip):

The only thing I can see there is that you can be guranteed that it's packed. But I guess if you cared you would do repr(packed)

Ryan Levick (Apr 16 2020 at 16:15, on Zulip):

I'm trying to think of these as more than just traits for safe transmute. There will be other uses for such traits outside of straight transmute

Lokathor (Apr 16 2020 at 19:48, on Zulip):

Sure, even outside of a transmuting context, I can't think of a single way to be able to do anything with a type once you know that it has no padding bytes, compared to what you could do before you knew that.

In other words, if you have T: Foo + Bar and then make it T: Foo + Bar + NoPadding, I can't imagine a single new thing that you'd be able to do just because of adding that extra bound of NoPadding.

Josh Triplett (Apr 16 2020 at 22:07, on Zulip):

The only things I can think of are effectively tantamount to transmuting, yeah.

Josh Triplett (Apr 16 2020 at 22:07, on Zulip):

I can't think of something you can do with a type if you know that it has no padding but has undefined values for some of its fields, or vice versa.

Josh Triplett (Apr 16 2020 at 22:07, on Zulip):

Only things you can do if you know both that it has no padding and has no undefined values.

Josh Triplett (Apr 16 2020 at 22:08, on Zulip):

Or things you can do if you know specifics of the types of its fields.

Ryan Levick (Apr 17 2020 at 08:10, on Zulip):

It'd be great for people to check out the latest on master. We now have the beginnings of FixedLayout, NoUninit, ByteComplete, FromBytes and ToBytes. In particular do we think that these traits capture their invariants properly? Are there other invariants that are subtly being captured that would better be moved into another trait. Obviously invariants about size and alignment are not captured, but I believe everything else for a safe transmute is there (though of course a transmute that relies on these traits plus size and alignment invariants is naturally conservative).

Ryan Levick (Apr 17 2020 at 08:15, on Zulip):

I'm not sure we're ready to answer this: but I'm constantly wondering if these traits would be useful in std. @Jack Wrenn and @Joshua Liebow-Feeser are going down the route of capturing layout in the type system so we can make adhoc judgement of whether two types are safely transmutable. This a great and super powerful/flexible way to address transmute. The mem markers in the mem-markers repo are a bit less fine grained and thus a bit less powerful in what they can do then Typic, but perhaps they provide guarantees around invariants that are also interesting outside of transmute and would thus be useful. Anyway, I'd like to start capturing usecases outside of transmute that these traits would be useful for.

Ryan Levick (Apr 17 2020 at 08:18, on Zulip):

For example, while it's not complete knowing a type is FixedLayout is useful for ensuring safe FFI. A type must be fixed layout for it to be safe to marshal across FFI boundaries.

Ryan Levick (Apr 17 2020 at 08:57, on Zulip):

@Lokathor Can I get your (or anyone else's) thoughts on the question of the usefulness of ByteComplete as distinguished from FromBytes: https://github.com/rust-secure-code/mem-markers/issues/14

Ryan Levick (Apr 17 2020 at 09:26, on Zulip):

@Josh Triplett @Lokathor trying my hardest to think of cases where it's useful to know that a type is NoPadding . I have one though it's a bit of a stretch. Say you need to allocate a buffer of memory to fit a type and you happen to know the sum of the sizes of the fields of that type, you can know that allocating size_of_fields bytes is enough space for the parent type as a whole because it has no padding. So there's no reason for an extra call to size_of. But seeing as how size of is a const function, I guess this doesn't matter.

Last update: Jul 02 2020 at 13:45UTC