Stream: t-lang/wg-unsafe-code-guidelines

Topic: repr of slices/trait objs


gnzlbg (Nov 15 2018 at 18:00, on Zulip):

@Nicole Mazzuca one can communicate with C without specifying the representation of slices and trait objects.

gnzlbg (Nov 15 2018 at 18:01, on Zulip):

I can query the slice pointer and length and pass those to C directly, or put them in a repr(C) struct.

rkruppe (Nov 15 2018 at 18:01, on Zulip):

that's of no help of a &[T] is embedded in a larger data structure you'd otherwise want to share with C

gnzlbg (Nov 15 2018 at 18:02, on Zulip):

you'd have to create a struct on the C side of things anyways, given that C does not have slices or trait objects :/

gnzlbg (Nov 15 2018 at 18:03, on Zulip):

ah, no, i see what you mean

gnzlbg (Nov 15 2018 at 18:04, on Zulip):

still, you could memcpy it into a struct that's repr(C), except for the &[T] pieces, which you translate to pointer+length

gnzlbg (Nov 15 2018 at 18:05, on Zulip):

i mean, you would only be able to use &[T] in C FFI as long as the C code defines it in the exact same way than we do

Nicole Mazzuca (Nov 15 2018 at 18:09, on Zulip):

this is correct

Nicole Mazzuca (Nov 15 2018 at 18:10, on Zulip):

but I'd also argue it's useful to have the repr(C) guarantee for ease of FFI

Nicole Mazzuca (Nov 15 2018 at 18:10, on Zulip):

not necessary, but helpful

Nicole Mazzuca (Nov 15 2018 at 18:10, on Zulip):

(as opposed to saying that the first word is the pointer, which isn't)

Gankro (Nov 15 2018 at 19:23, on Zulip):

You should always be using cbindgen/rust-bindgen for ffi so keeping the repr right shouldn’t be a burden

Nicole Mazzuca (Nov 15 2018 at 19:57, on Zulip):

?

Nicole Mazzuca (Nov 15 2018 at 19:57, on Zulip):

I mean for C APIs written in Rust.

Nicole Mazzuca (Nov 15 2018 at 19:59, on Zulip):

I want to be able to declare

template <typename T>
struct slice {
  T* ptr;
  size_t length;
};

extern "C" {
  void print_int_slice(slice<int const> x);
}

and then in rust

#[no_mangle]
pub extern "C" fn print_int_slice(x: &[i32]) {
  ...
}

or w/e

RalfJ (Nov 15 2018 at 20:02, on Zulip):

I think for slices and trait objects we could commit to something. (Though we shouldnt fix vtable layout.) The original statement was about fat pointers in general, there we should be careful.

Nicole Mazzuca (Nov 15 2018 at 20:03, on Zulip):

I don't know why I'd want to be careful here.

RalfJ (Nov 15 2018 at 20:04, on Zulip):

didn't you propose custom DST? then we could have fat ptrs with nobody-knows-what as metadata

Nicole Mazzuca (Nov 15 2018 at 20:04, on Zulip):

sure

Nicole Mazzuca (Nov 15 2018 at 20:04, on Zulip):

why's that make a difference?

RalfJ (Nov 15 2018 at 20:05, on Zulip):

uh, if we say all fat ptrs are two pointer sizes large, then that restricts your custom DST proposal to ptr-sized metadata?

Nicole Mazzuca (Nov 15 2018 at 20:05, on Zulip):

I didn't say that?

RalfJ (Nov 15 2018 at 20:05, on Zulip):

I didnt say you did

Nicole Mazzuca (Nov 15 2018 at 20:05, on Zulip):

right, but why should we be careful about committing to the thing I said

Nicole Mazzuca (Nov 15 2018 at 20:06, on Zulip):

obviously we can't commit to two pointers, that's a non-starter

RalfJ (Nov 15 2018 at 20:06, on Zulip):

okay I guess I have some context here that I thought we share but don't... one sec

RalfJ (Nov 15 2018 at 20:06, on Zulip):

https://github.com/rust-rfcs/unsafe-code-guidelines/issues/16#issuecomment-439106988

RalfJ (Nov 15 2018 at 20:07, on Zulip):

What you proposed sounds absolutely reasonable, that's why I said that for slices (and trait objects) we can commit to something

RalfJ (Nov 15 2018 at 20:07, on Zulip):

at least for layout. I will leave ABI questions to the experts.^^

Nicole Mazzuca (Nov 15 2018 at 20:07, on Zulip):

right, but we don't need to worry about "just for slices and trait objects"

Nicole Mazzuca (Nov 15 2018 at 20:08, on Zulip):

I would argue we should define it forall Metadata

RalfJ (Nov 15 2018 at 20:08, on Zulip):

I'd rather not define anything for custom DST

RalfJ (Nov 15 2018 at 20:08, on Zulip):

just to keep all options open

Nicole Mazzuca (Nov 15 2018 at 20:08, on Zulip):

hmm

Nicole Mazzuca (Nov 15 2018 at 20:08, on Zulip):

I personally disagree, but not super strongly

RalfJ (Nov 15 2018 at 20:09, on Zulip):

I dont care strongly either^^

rkruppe (Nov 15 2018 at 20:09, on Zulip):

As I already wrote on the issue I think we can't define layout in terms DynamicallySized because it doesn't even exist yet

RalfJ (Nov 15 2018 at 20:09, on Zulip):

(I dont care strongly about basically any layout stuff, except where it interacts with validity invariants^^)

rkruppe (Nov 15 2018 at 20:09, on Zulip):

Unless of course you propose waiting to define anything until we have it

Nicole Mazzuca (Nov 15 2018 at 20:10, on Zulip):

oh, if you want to say that, sure

Nicole Mazzuca (Nov 15 2018 at 20:10, on Zulip):

I mean right when custom DSTs are implemented, this should be defined.

Nicole Mazzuca (Nov 15 2018 at 20:10, on Zulip):

I want to be able to use custom DSTs in C APIs

rkruppe (Nov 15 2018 at 20:10, on Zulip):

yeah i agree

Gankro (Nov 16 2018 at 12:39, on Zulip):

eddyb is reluctant to define {ptr, len} because we actually always pass it as two values in the rust abi, even on x86 which passes structs on the stack

rkruppe (Nov 16 2018 at 12:40, on Zulip):

we could just... not do that... in the C calling convention

Gankro (Nov 16 2018 at 12:45, on Zulip):

alternatively we could "just" teach cbindgen that extern fn hello(data: &[T]) is void hello(T* data_ptr, size_t data_len);

Gankro (Nov 16 2018 at 12:46, on Zulip):

Which honestly seems like more idiomatic C code

Gankro (Nov 16 2018 at 12:46, on Zulip):

but is also easier to mess up your usage of

Gankro (Nov 16 2018 at 12:48, on Zulip):

There is also the matter of to what extent we are guaranteeing, accidentally or not, whether {*T, usize} is type-punnable with &[T]

Gankro (Nov 16 2018 at 12:48, on Zulip):

I don't think it's a huge concern?

Gankro (Nov 16 2018 at 12:50, on Zulip):

Basically I think this is the first time we're saying "hey these are the same everywhere" instead of "hey this is specifically the valid FFI bridging, but in actual Rust code don't rely on this"

rkruppe (Nov 16 2018 at 12:51, on Zulip):

i think that type-punability is something we should be intentionally guaranteeing, as it's more useful than just passing immediate slices across the FFI boundary -- you can already do that today with a small wrapper that disassembles the slice, but today you can't share a big repr(C) data structure with C if it contains slice pointers

rkruppe (Nov 16 2018 at 12:52, on Zulip):

to be clear this is not really about type punning within Rust, but that type punning becomes legal as a side effect of allowing C to manipulate slice pointers natively

Gankro (Nov 16 2018 at 12:52, on Zulip):

Correct me if I'm wrong but we try to keep the Rust ABI close to the C one so native tooling will be happier, right?

rkruppe (Nov 16 2018 at 12:52, on Zulip):

Uh

rkruppe (Nov 16 2018 at 12:52, on Zulip):

I at least haven't ever heard that articulated as a goal

Gankro (Nov 16 2018 at 12:53, on Zulip):

I might have hallucinated it, I'll check on #compiler discord

gnzlbg (Nov 16 2018 at 13:01, on Zulip):

@Nicole Mazzuca do you think DSTs that do not contain a pointer are useful ?

gnzlbg (Nov 16 2018 at 13:02, on Zulip):

I don't understand why so many people want to specify a layout for custom DSTs right now, when nobody has agreed yet on what custom DSTs actually are

gnzlbg (Nov 16 2018 at 13:02, on Zulip):

say I have a global array, and write a custom DST that has a pointer to this array as an associated const, and only stores a u8 index to index into it

gnzlbg (Nov 16 2018 at 13:04, on Zulip):

that would be a &T that has the same size as an u8 - if we specify that &T is at least pointer sized, and that for custom DSTs that they are all (ptr, Metadata), then these types cannot be written.

rkruppe (Nov 16 2018 at 13:04, on Zulip):

if there's no data pointer component, why do you want to bludgeon it into the shame of a reference to an dynamically sized value? in your example, why not just have a newtype around an u8?

gnzlbg (Nov 16 2018 at 13:05, on Zulip):

there is a data pointer component, it is just not stored in the DST

rkruppe (Nov 16 2018 at 13:06, on Zulip):

out of all the features of pointers, pretty much only the ability to return such a thing from Index::index (and variants) seems relevant, everything else (e.g. different kinds of pointers like raw vs reference vs Rc vs ..., the lifetime parameter some of these have, etc.) seems irrelevant

gnzlbg (Nov 16 2018 at 13:06, on Zulip):

i don't really know if supporting this use case is worth it, or even useful, hence why I asked

rkruppe (Nov 16 2018 at 13:06, on Zulip):

and even the Index::index thing seems dubious because you'd artificially constrain yourself to returning &'a YourCustomThingy, when you could return &'static YourCustomThingy

rkruppe (Nov 16 2018 at 13:07, on Zulip):

It really sounds like you want a custom pointer type, not a custom DST

gnzlbg (Nov 16 2018 at 13:08, on Zulip):

You could write a custom pointer type than then has size == 1, but if you then &* you would get a pointer-sized reference

rkruppe (Nov 16 2018 at 13:09, on Zulip):

why is that a problem?

gnzlbg (Nov 16 2018 at 13:10, on Zulip):

you want to store it somewhere, and would prefer to store a reference type with size == 1.

rkruppe (Nov 16 2018 at 13:12, on Zulip):

if that place where you store it is under your control, there are ways to be reasonably generic over different pointer types (e.g. for starters P: Copy+Deref). if that place is not under your control and e.g. hardcodes &T, it could already be doing a ton of things to this pointer that cannot be generalized to things which claim to be pointers but aren't actually an address

Gankro (Nov 16 2018 at 13:12, on Zulip):

report back from discord: I had misunderstood the ABI concerns. eddyb's concerns seem to all revolve around wanting to better understand the custom DST plan to avoid any special-casing of the implementation

gnzlbg (Nov 16 2018 at 13:12, on Zulip):

I really don't understand why custom DSTs have to be constrained to have a pointer and meta-data members internally, in any particular order

gnzlbg (Nov 16 2018 at 13:13, on Zulip):

why do they even need a metadata: () member at all, if its just (), can't it be omitted in the type ?

gnzlbg (Nov 16 2018 at 13:14, on Zulip):

and same for the pointer, given that we don't even have a merged RFC for custom DSTs, why do we have to specify right now that custom DSTs need to always have a pointer, and that it has to be the first member

gnzlbg (Nov 16 2018 at 13:15, on Zulip):

even if we were to never support DSTs without a memory address, why a pointer and not an usize

gnzlbg (Nov 16 2018 at 13:15, on Zulip):

or a NonNull<*()>

rkruppe (Nov 16 2018 at 13:16, on Zulip):

ok some terminology for starters: if it doesn't have a thin data pointer component and some other components that help interpret that data pointer, it's not a generalization of the concept of DSTs, it's a generalization of the concept of pointers period

rkruppe (Nov 16 2018 at 13:16, on Zulip):

now your question might be, why generalize DSTs specifically and not pointers in general

rkruppe (Nov 16 2018 at 13:17, on Zulip):

and one reason is, there's a ton of things that can be done generically over all DSTs but not necessarily over arbitrary things that are in some shape or form "like pointers". for example, ThinBox<T: ?Sized>

rkruppe (Nov 16 2018 at 13:17, on Zulip):

why do they even need a metadata: () member at all, if its just (), can't it be omitted in the type ?

this is just so we can talk about all pointers without special casing for pointers to sized types

rkruppe (Nov 16 2018 at 13:18, on Zulip):

even if we were to never support DSTs without a memory address, why a pointer and not an usize

irrelevant but a raw pointer is slightly more natural since most of the things you'd do with it are pointer things not address things

rkruppe (Nov 16 2018 at 13:20, on Zulip):

("irrelevant" because raw pointers and usizes are equivalent for layout and ABI so it's just a matter of where you cast, if at all)

gnzlbg (Nov 16 2018 at 13:20, on Zulip):

if I can write my own DSTs, I can provide APIs that expose them and completely hide where they come from

rkruppe (Nov 16 2018 at 13:21, on Zulip):

what wouuld ThinBox<YourU8IndexIntoGlobalArray> or Rc<YourU8IndexIntoGlobalArray> even mean?

gnzlbg (Nov 16 2018 at 13:22, on Zulip):

that's the question

rkruppe (Nov 16 2018 at 13:23, on Zulip):

for custom DSTs as defined in the various DSTs it's completely clear what Rc, ThinBox, etc. mean. for your example i don't even think it can be made to make sense

gnzlbg (Nov 16 2018 at 13:23, on Zulip):

@rkruppe those imply some sort of ownership, but I don't have to expose a way to create T, just expose &T

rkruppe (Nov 16 2018 at 13:24, on Zulip):

that's why I say you want a new kind of pointer, instead of a new kind of type that can be referenced by any pointer

gnzlbg (Nov 16 2018 at 13:24, on Zulip):

ah I see

gnzlbg (Nov 16 2018 at 13:25, on Zulip):

I thought custom DSTs would also allow implementing &T

gnzlbg (Nov 16 2018 at 13:25, on Zulip):

or that it would be an aim of that at some point

Gankro (Nov 16 2018 at 13:25, on Zulip):

first I heard of that

rkruppe (Nov 16 2018 at 13:25, on Zulip):

no, just as the existing DSTs are str and [T] and dyn Bounds and so on, not &str, &[T], &dyn Bounds

gnzlbg (Nov 16 2018 at 13:27, on Zulip):

so yeah, i'd like a way to create custom pointers then

gnzlbg (Nov 16 2018 at 13:29, on Zulip):

thinking about DSTs, I guess it also wouldn't be possible to have DSTs where the meta-data is encoded in the pointer then ?

gnzlbg (Nov 16 2018 at 13:30, on Zulip):

as in, the metadata has to be a distinct field of the DST

Gankro (Nov 16 2018 at 13:31, on Zulip):

this seems very off-topic

rkruppe (Nov 16 2018 at 13:31, on Zulip):

yeah best take this question to a specific proposal about custom DSTs

gnzlbg (Nov 16 2018 at 13:31, on Zulip):

so specifying custom DSTs as (*T, Metadata) does not prevent this?

Last update: Nov 19 2019 at 19:05UTC