I've created a repo for defining base marker traits that can be shared among all implementations of safe transmute. Again, the idea is that this repo would be the launching pad for any traits that would wind up in the std lib. Here is the first issue for discussing FromBytes/AsBytes https://github.com/rylev/mem-markers/issues/1 (note: this is on my personal GitHub until we can transfer to the rust-secure-code organization)
@Jack Wrenn @Joshua Liebow-Feeser one thing I would love to talk more about is how useful these marker traits are or are not to Typic
Thanks to @Shnatsel the repo has been moved: https://github.com/rust-secure-code/mem-markers
I've invited some people as admins to this repo but I'm sure I'm forgetting someone. If you want write access and did not receive an invite, ping me
I'd like to start implementing some of the traits. Does anyone have any objections to this: https://github.com/rust-secure-code/mem-markers/issues/1. I assume this crate also will support custom derives for each of them. Any objections to that?
@Jack Wrenn @Joshua Liebow-Feeser I'd also love to hear from you two since it's the least clear how well this plays into a typic style solution
I've left comments addressing your question on the github issue: https://github.com/rust-secure-code/mem-markers/issues/1#issuecomment-601810244
I think that ToBytes and FromBytes are perhaps too broad and could be broken up into sub-markers and then you have ToBytes any time you have NoUninit and also FixedLayout
But they should exist somewhere, so feel free to go ahead.
Finally got to implementing some of the marker traits stuff. Here's a PR that implements FixedLayout (on any type that has a fixed layout): https://github.com/rust-secure-code/mem-markers/pull/7 It's not complete but puts the beginnings in place. I want to work on a NoUninit trait but I'm not sure how I'll do it. It's to tell if fields are NoUninit, but we'd need to have some idea of size of types to calculate if there is padding. Any ideas on this are welcome :-)
It might be easier to do a
ByteComplete trait which says that a type is valid no matter what bytes compromise it. I believe
FromBytes: FixedLayout + ByteComplete
I unfortunately can't come up with a way to calculate type sizes at compile time and make compile decisions based on them. So I'm not sure how ToBytes can be done.
I guess I will go with how zerocopy does it where they divide by zero in a const expression
@Lokathor NoUninit means 99% of the time no padding. But are there any cases where it's interesting to know a type is NoUninit but is ?HasPadding?
I don't think a type can even have padding bytes that are not uninit
@Lokathor I mean a type can have uninit bytes that are not padding
So there is a subtle distinction between a type with no padding and a type with no uninit bytes.
yeah NoUninit is the stronger claim
So the question is, should we make a distinction?
Should there be a
NoPadding trait and a
NoUninit: NoPadding trait?
MaybeUninit<u8> is NoPadding but it's not NoUninit
I don't see the value in NoPadding on it own
it doesn't let you do anything different
The only thing I can see there is that you can be guranteed that it's packed. But I guess if you cared you would do
I'm trying to think of these as more than just traits for safe transmute. There will be other uses for such traits outside of straight transmute
Sure, even outside of a transmuting context, I can't think of a single way to be able to do anything with a type once you know that it has no padding bytes, compared to what you could do before you knew that.
In other words, if you have
T: Foo + Bar and then make it
T: Foo + Bar + NoPadding, I can't imagine a single new thing that you'd be able to do just because of adding that extra bound of
The only things I can think of are effectively tantamount to transmuting, yeah.
I can't think of something you can do with a type if you know that it has no padding but has undefined values for some of its fields, or vice versa.
Only things you can do if you know both that it has no padding and has no undefined values.
Or things you can do if you know specifics of the types of its fields.
It'd be great for people to check out the latest on master. We now have the beginnings of FixedLayout, NoUninit, ByteComplete, FromBytes and ToBytes. In particular do we think that these traits capture their invariants properly? Are there other invariants that are subtly being captured that would better be moved into another trait. Obviously invariants about size and alignment are not captured, but I believe everything else for a safe transmute is there (though of course a transmute that relies on these traits plus size and alignment invariants is naturally conservative).
I'm not sure we're ready to answer this: but I'm constantly wondering if these traits would be useful in std. @Jack Wrenn and @Joshua Liebow-Feeser are going down the route of capturing layout in the type system so we can make adhoc judgement of whether two types are safely transmutable. This a great and super powerful/flexible way to address transmute. The mem markers in the mem-markers repo are a bit less fine grained and thus a bit less powerful in what they can do then Typic, but perhaps they provide guarantees around invariants that are also interesting outside of transmute and would thus be useful. Anyway, I'd like to start capturing usecases outside of transmute that these traits would be useful for.
For example, while it's not complete knowing a type is FixedLayout is useful for ensuring safe FFI. A type must be fixed layout for it to be safe to marshal across FFI boundaries.
@Lokathor Can I get your (or anyone else's) thoughts on the question of the usefulness of
ByteComplete as distinguished from
@Josh Triplett @Lokathor trying my hardest to think of cases where it's useful to know that a type is NoPadding . I have one though it's a bit of a stretch. Say you need to allocate a buffer of memory to fit a type and you happen to know the sum of the sizes of the fields of that type, you can know that allocating size_of_fields bytes is enough space for the parent type as a whole because it has no padding. So there's no reason for an extra call to size_of. But seeing as how size of is a const function, I guess this doesn't matter.