Stream: t-lang/wg-unsafe-code-guidelines

Topic: Safety of inspecting bytes of a repr(rust) type.


Lokathor (Sep 20 2019 at 02:23, on Zulip):

Suppose we have an unsafe marker trait for type with no padding bytes:

pub unsafe trait NoPaddingBytes {}

And then we implement it on some of our favorite types. Particularly, say i32 and (u8,u8)

Is it safe/sound to then have a function to convert a reference to a NoPaddingBytes value into a properly sized byte slice.

pub fn bytes_of<T: NoPaddingBytes>(t: &T) -> &[u8] {
  unsafe { core::slice::from_raw_parts(t as *const T as *const u8, size_of::<T>()) }
}
rkruppe (Sep 20 2019 at 06:31, on Zulip):

I don't see any immediate problem if (can you add scottmcm's rationale here?) if the NoPaddingBytes impl is justified. But for virtually all repr(rust) types, layout is sufficently unspecified that claiming it has no padding bytes wouldn't be justified.

gnzlbg (Sep 20 2019 at 09:37, on Zulip):

and (u8, u8) is repr(rust)

Lokathor (Sep 20 2019 at 14:24, on Zulip):

(u8,u8) will never have padding bytes, yet it is repr(rust) and inspecting the individual bytes is undefined. Or at least, uh, I think "under defined" is the term that's been used a bit lately, perhaps that. (u8,u8) is always 2 bytes big and neither of the bytes is a padding byte, but you still can't turn &(u8,u8) into &[u8] and look around inside because it's undefined if tuple.0 is the same as punned_slice[0].

rkruppe (Sep 20 2019 at 17:19, on Zulip):

I don't believe it is actually defined that (u8, u8) has no padding. Not even in the UCG which is all provisional and contains errors as well as proposals not yet blessed by the lang team.

Lokathor (Sep 20 2019 at 20:54, on Zulip):

if there's ever a secret padding byte somehow I think you'll get torches and pitchforks

Lokathor (Sep 20 2019 at 20:54, on Zulip):

but sure, perhaps it isn't really defined

gnzlbg (Sep 21 2019 at 07:42, on Zulip):

if you don't want padding, make it #[repr(C)]

gnzlbg (Sep 21 2019 at 07:43, on Zulip):

_I think_ there is an issue open for homogeneous aggregates

gnzlbg (Sep 21 2019 at 07:45, on Zulip):

ucg#36

gnzlbg (Sep 21 2019 at 07:48, on Zulip):

(u8, u8, u8, u8) has size 4 but alignment 1

gnzlbg (Sep 21 2019 at 07:48, on Zulip):

and right now repr(Rust) could change that to size 4 alignment 4

gnzlbg (Sep 21 2019 at 07:49, on Zulip):

it also could change that to size 8 alignment 8 AFAICT

gnzlbg (Sep 21 2019 at 07:49, on Zulip):

which gives you 4 bytes of trailing padding

Lokathor (Sep 21 2019 at 08:15, on Zulip):

I can't make a tuple be repr(C) XD

gnzlbg (Sep 21 2019 at 09:52, on Zulip):

#[repr(C)] pub Foo(u8, u8, u8, u8);

rkruppe (Sep 21 2019 at 10:06, on Zulip):

That solves... nothing. It gives no interop with tuple-based APIs, it is a different type (different kind even, so no a priori reason to expect them to have similar rules), and it loses all the convenience that motivate having tuples as built-ins in the first place.

gnzlbg (Sep 21 2019 at 11:01, on Zulip):

Well of course you can't interface with tuple based APIs with it

gnzlbg (Sep 21 2019 at 11:01, on Zulip):

a tuple-based API expects a repr(Rust) tuple, which might have padding anywhere

gnzlbg (Sep 21 2019 at 11:05, on Zulip):

They want a different type with a different layout that has no padding

gnzlbg (Sep 21 2019 at 11:05, on Zulip):

That's incompatible with repr(Rust)

gnzlbg (Sep 21 2019 at 11:06, on Zulip):

Even if we were to support the syntax #[repr(C)] (u8, u8) that would be a different type than (u8, u8)

gnzlbg (Sep 21 2019 at 11:06, on Zulip):

so doing that fixes nothing

gnzlbg (Sep 21 2019 at 11:06, on Zulip):

A tuple-struct solves the same problem in the same way but already works on stable Rust

centril (Sep 21 2019 at 14:13, on Zulip):

Even if we were to support the syntax #[repr(C)] (u8, u8) that would be a different type than (u8, u8)

Would be open to #[repr(C) (T, U, ...) if there's desire for that

rkruppe (Sep 21 2019 at 14:16, on Zulip):

If we're talking about working on stable Rust, tuple struct don't have any more layout guarantees than tuples atm (none de jure, very little in UCG other than the special cases for zero and one fields)

rkruppe (Sep 21 2019 at 14:18, on Zulip):

And #[repr(C)] (T, U, ...) is more confusing than helpful. More distinctions in the type system generally hurt interop, and perpetuating the idea repr-ness is a property of every type rather than something that means specific things for specific user-defined types. Either we provide deterministic, predictable layout of tuples or we don't.

Lokathor (Sep 21 2019 at 16:12, on Zulip):

In this case, the creation of any new types is not desired.

I want to have a function like

fn bytes_of<T:SomeBound>(t: &T) -> &[u8] {}

and then implement SomeBound on all appropriate core/alloc/std types. Currently the bound is Pod since most other casting functions just use Pod.

For the casting case of &T -> &[u8] specifically I wanted to investigate if some sort of lesser bound would usefully be able to add a lot of types that are readable as just bytes but aren't Pod (such as NonNull, NomZeroUsize, etc).

rkruppe (Sep 21 2019 at 16:20, on Zulip):

Isn't that just zerocopy::AsBytes?

gnzlbg (Sep 21 2019 at 16:24, on Zulip):

I'm unconvinced about guaranteeing something for tuples only

gnzlbg (Sep 21 2019 at 16:24, on Zulip):

this is essentially the discussion about homogeneous agregates

gnzlbg (Sep 21 2019 at 16:25, on Zulip):

and all of it applies to homogeneous structs, tuple structs, tuples, and arrays

gnzlbg (Sep 21 2019 at 16:26, on Zulip):

rkruppe 4:16 PM
If we're talking about working on stable Rust, tuple struct don't have any more layout guarantees than tuples atm

#[repr(C)] tuple structs do

gnzlbg (Sep 21 2019 at 16:28, on Zulip):

guaranteeing field order would already be enough for union type punning and transmute (if the transmute succeeds)

gnzlbg (Sep 21 2019 at 16:29, on Zulip):

so if and only if the transmute form (T, T) to [T; N] succeeds, then (T, T).i can be accessed via [T; N][i]`

gnzlbg (Sep 21 2019 at 16:30, on Zulip):

we could make that transmute always succeed by guaranteeing that their sizes match, but the size of arrays is constrained by what C specifies

gnzlbg (Sep 21 2019 at 16:31, on Zulip):

that would still allow us raising the alignment of aggregates up to the aggregate size if we wanted, but we can't raise it further than that

gnzlbg (Sep 21 2019 at 16:31, on Zulip):

so we could raise the alignment of (u8, u8) to 2

gnzlbg (Sep 21 2019 at 16:33, on Zulip):

if we don't want to limit that, we could maybe guarantee that the first field is at offset 0

gnzlbg (Sep 21 2019 at 16:34, on Zulip):

that would allow raising the alignment of (u8, u8) to 4, which gives it size 4 , and still allow going from a tuple to an array by using an #[repr(C)] union U { t: (u8, u8), a: [u8; 2] }

gnzlbg (Sep 21 2019 at 16:37, on Zulip):

So for getting that union to work for type-punning, we just need:

If we wanted to guarantee that transmute works, we could extend those guarantees with:

Lokathor (Sep 21 2019 at 17:04, on Zulip):

@rkruppe yes, it's one of the things zerocopy does

Josh Triplett (Sep 22 2019 at 19:02, on Zulip):

@Lokathor I'm fairly sure there's an established crate with that exact SomeBound and associated functions you'd expect.

Lokathor (Sep 22 2019 at 19:10, on Zulip):

Oh, sure, and I own one of the best ones in the field, which I recently extracted into its own crate at the request of a user (announcing: bytemuck-0.1 is now on crates.io)

The question isn't if we can do it at all, the question is how much we can loosen that bound while remaining sound.

gnzlbg (Sep 22 2019 at 19:26, on Zulip):

Depends on how much you want to rely on the rust compiler not having serious performance regressions.

gnzlbg (Sep 22 2019 at 19:26, on Zulip):

There are arguments for changing the layout of (u8, u8) slightly, e.g., increase its alignment.

gnzlbg (Sep 22 2019 at 19:27, on Zulip):

But that's not free, since it might end up raising the size of some other types

gnzlbg (Sep 22 2019 at 19:29, on Zulip):

Just because the language does not guarantee something, does not mean that the behavior will change. It might or might not change. If you need to make an assumption to get something done, and you are aware of the issues, you often can write code that tests your assumptions, so that if they were to change, you'd get notified.

Lokathor (Sep 22 2019 at 20:05, on Zulip):

however, when people use a crate as a dependency the crate's tests aren't run on their machine before you're allowed to build the crate

Lokathor (Sep 22 2019 at 20:06, on Zulip):

so while i might accept such a risk for myself, it would clearly be irresponsible to ship that as a lib

Josh Triplett (Sep 22 2019 at 21:50, on Zulip):

What about a compile-time assert of some kind?

Josh Triplett (Sep 22 2019 at 21:51, on Zulip):

If there isn't a good mechanism for that, I'd support adding one.

Lokathor (Sep 22 2019 at 23:04, on Zulip):

There are compile time asserts

in this case,

However, asserting the linearity of the tuple type would be less of a sure thing. Assuming that all (u8,u8) have the same layout then you could do it by making some demo values and asserting that the fields land in the right spots in memory. However, you cannot currently const transmute

gnzlbg (Sep 23 2019 at 06:06, on Zulip):

however, when people use a crate as a dependency the crate's tests aren't run on their machine before you're allowed to build the crate

That's what assert! is for

gnzlbg (Sep 23 2019 at 06:07, on Zulip):

assert!(size_of::<(u8,u8)>() == 2);

gnzlbg (Sep 23 2019 at 06:07, on Zulip):

With optimizations on, the compiler should be able to remove the assert

gnzlbg (Sep 23 2019 at 06:08, on Zulip):

However, you cannot currently const transmute

Of course you can

Lokathor (Sep 25 2019 at 03:15, on Zulip):

stable? or just with some feature?

gnzlbg (Sep 25 2019 at 06:32, on Zulip):

you can use unions in const fns

Lokathor (Sep 25 2019 at 16:56, on Zulip):

That only works with specific types, as I recall, you can't do it generically.

Thom Chiovoloni (Sep 25 2019 at 17:10, on Zulip):

You sorta can const transmute by using a union in a macro

Thom Chiovoloni (Sep 25 2019 at 17:11, on Zulip):

Example: https://github.com/thomcc/bad3d/blob/master/t3m/src/macros.rs#L141-L159 (it's gross, admittedly)

gnzlbg (Sep 25 2019 at 17:50, on Zulip):

FWIW you don't need the ARR const there

gnzlbg (Sep 25 2019 at 17:51, on Zulip):

ah, or are you using it for type checking ?

gnzlbg (Sep 25 2019 at 17:51, on Zulip):

it might give better errors than omitting it

Thom Chiovoloni (Sep 25 2019 at 18:19, on Zulip):

Yeah, it's for type checking and so that i don't run code passed into the macro inside an unsafe block

Thom Chiovoloni (Sep 25 2019 at 20:03, on Zulip):

admittedly it is a const unsafe block so it probably doesn't really matter... well, maybe. but better safe than sorry

Last update: Nov 19 2019 at 17:45UTC