Stream: t-lang

Topic: Taking mutable references via programmatic operation


Josh Triplett (Dec 04 2019 at 20:35, on Zulip):

I had an idea yesterday for an operation that I think makes sense and would be sound, but isn't currently possible in Rust. I'm trying to figure out the exact semantics and whether I'm missing anything that would make it unsound.

Josh Triplett (Dec 04 2019 at 20:35, on Zulip):

How mutable references work today:

fn main() {
    let mut x: u32 = 42;
    {
        let y: &mut u32 = &mut x;
        // cannot touch x here
        *y = 10;
    }
    x *= 2;
    println!("{}", x); // prints 20
}
Josh Triplett (Dec 04 2019 at 20:35, on Zulip):

But you can't, today, write a function with the same semantics as the operation &mut x.

Josh Triplett (Dec 04 2019 at 20:36, on Zulip):

Hypothetical:

fn main() {
    let mut x: u32 = 42;
    {
        let y: &mut u32 = x.get_mut_ref();
        // cannot touch x here.
        *y = 10;
    }
    x *= 2;
    println!("{}", x); // prints 20
}
Josh Triplett (Dec 04 2019 at 20:37, on Zulip):

The semantics I'm looking for would also allow writing a function that could return a mutable reference of a different type that was still tied to the owned value it mutably-referenced.

simulacrum (Dec 04 2019 at 20:38, on Zulip):

I'm a bit confused -- isn't this already true of all methods that take &mut self and return &mut U?

Josh Triplett (Dec 04 2019 at 20:38, on Zulip):

For instance:

fn main() {
    let mut x: u32 = 42;
    {
        let y: &mut [u8; 4] = x.get_mut_ref_to_bytes();
        // cannot touch x here.
        y[0] = 10; // assuming little-endian for the sake of example
    }
    x *= 2;
    println!("{}", x); // prints 20
}
simulacrum (Dec 04 2019 at 20:41, on Zulip):

(e.g., for your last example, https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=0aebe3e2d489b2a645b836f2d9b49a1e)

Josh Triplett (Dec 04 2019 at 20:41, on Zulip):

@simulacrum Wait, that's legal?

Josh Triplett (Dec 04 2019 at 20:41, on Zulip):

You can consume &mut self and return &mut OtherType?

simulacrum (Dec 04 2019 at 20:42, on Zulip):

My understanding is yes, for sure. (Indeed, this is a frequent pain point)

Josh Triplett (Dec 04 2019 at 20:42, on Zulip):

Do the lifetime semantics actually work just like you have a &mut referencing that owned type?

simulacrum (Dec 04 2019 at 20:42, on Zulip):

the lifetime of both &mut is the same

Josh Triplett (Dec 04 2019 at 20:42, on Zulip):

And you can't mutably reference the original type anymore, because that would alias? Because it seems like the transmute you show produces aliased &mut values which would be UB...

simulacrum (Dec 04 2019 at 20:43, on Zulip):

it does not, because it consumes the previous &mut (or so I understand it)

Josh Triplett (Dec 04 2019 at 20:43, on Zulip):

I didn't know you could "consume" a &mut.

Josh Triplett (Dec 04 2019 at 20:43, on Zulip):

That's useful, and if that's the semantic, that's incredibly helpful. Is there somewhere I could read more about that semantic and verify?

simulacrum (Dec 04 2019 at 20:43, on Zulip):

I'm a bit confused, I guess

XAMPPRocky (Dec 04 2019 at 20:43, on Zulip):

Damn, that's a pretty cool way to have bit/byte fiddling.

simulacrum (Dec 04 2019 at 20:43, on Zulip):

Like, which semantic is this?

rkruppe (Dec 04 2019 at 20:43, on Zulip):

On a basic level, &mut T does not implement Copy, so it is moved. Reborrows make that slightly complicated though.

Josh Triplett (Dec 04 2019 at 20:43, on Zulip):

This is in the context of the "safe transmute" proposal.

Josh Triplett (Dec 04 2019 at 20:44, on Zulip):

@simulacrum The standard mutable reference semantic. "You can only take a mutable reference to a value you own, once you've taken a mutable reference you can't use the original until you drop that mutable reference, once you drop that mutable reference you can use the original owned value again".

simulacrum (Dec 04 2019 at 20:45, on Zulip):

so I think that's not really how I'd say that (it's sort of a "beginners view" imo)

simulacrum (Dec 04 2019 at 20:45, on Zulip):

there's no difference between Foo<'a, T> and &mut T

simulacrum (Dec 04 2019 at 20:46, on Zulip):

(modulo GATs not being a thing)

Josh Triplett (Dec 04 2019 at 20:46, on Zulip):

I thought the difference was that you can have more than one &T or Foo<'a, T> at a time, but not more than one &mut T?

simulacrum (Dec 04 2019 at 20:47, on Zulip):

well, &mut T is basically just an owned T

simulacrum (Dec 04 2019 at 20:47, on Zulip):

borrow check only cares about the lifetime, not the type

Josh Triplett (Dec 04 2019 at 20:48, on Zulip):

So what prevents having a &mut T and a &T to the same T at the same time?

rkruppe (Dec 04 2019 at 20:48, on Zulip):

Creating a (shared or mutable) reference is a special operation that the borrow checker understands to create what's internally called a "loan" of the place (which can itself be shared or mutable/exclusive), but this loan is linked to a lifetime 'a and thus to all values whose type contains 'a, regardless of what that type is.

simulacrum (Dec 04 2019 at 20:50, on Zulip):

this is actually sort of why a signature like fn(&mut T) -> &U is really painful, too

rkruppe (Dec 04 2019 at 20:50, on Zulip):

A nice resource for this might be https://nikomatsakis.github.io/rust-belt-rust-2019/

XAMPPRocky (Dec 04 2019 at 20:50, on Zulip):

🤯

simulacrum (Dec 04 2019 at 20:50, on Zulip):

as borrow check doesn't know that you've "lost" the mutable ref

Josh Triplett (Dec 04 2019 at 20:50, on Zulip):

OK. So once you have a mutable/exclusive loan of the location, you can safely and without UB turn it into a different mutable/exclusive loan of the location?

Josh Triplett (Dec 04 2019 at 20:51, on Zulip):

@simulacrum Ah. So fn foo(&mut T) -> &T kinda gives you a non-mut reference "backed" by an exclusive loan?

simulacrum (Dec 04 2019 at 20:51, on Zulip):

correct, so long as that "turn it into" is a move-like operation (i.e., happens once)

simulacrum (Dec 04 2019 at 20:51, on Zulip):

Yes

Josh Triplett (Dec 04 2019 at 20:52, on Zulip):

Does the compiler actually enforce that it's a move-like operation? Or do you just have to never touch self again once you've transmuted?

simulacrum (Dec 04 2019 at 20:52, on Zulip):

https://github.com/rust-embedded/wg/blob/master/rfcs/0377-mutex-trait.md#the-rust-standard-library-mutex-uses-self-for-the-lock-method-why-choose-mut-self may also be interesting for you to read (sort of tangentially related)

simulacrum (Dec 04 2019 at 20:52, on Zulip):

Does the compiler actually enforce that it's a move-like operation? Or do you just have to never touch self again once you've transmuted?

In the case, I showed, yes -- mem::transmute cannot be called more than once, you'll get a "use after move"

Josh Triplett (Dec 04 2019 at 20:53, on Zulip):

Ah, because you're "moving" &mut self?

simulacrum (Dec 04 2019 at 20:53, on Zulip):

exactly

Josh Triplett (Dec 04 2019 at 20:53, on Zulip):

Same semantics as "consuming" any other owned value?

rkruppe (Dec 04 2019 at 20:53, on Zulip):

transmute moves its argument and &mut T isn't Copy so I think it should be fine? But I am never sure where reborrows can happen so :shrug:

Josh Triplett (Dec 04 2019 at 20:53, on Zulip):

So, wait a moment...

simulacrum (Dec 04 2019 at 20:53, on Zulip):

to my knowledge we never reborrow in a function call that takes a non-reference type, but I may be wrong about that (e.g., around impl AsRef)

Josh Triplett (Dec 04 2019 at 20:54, on Zulip):

What happens if I write:

let x: &mut T = &mut some_t_I_own;
foo(x); // takes &mut T
bar(x); // takes &mut T
Josh Triplett (Dec 04 2019 at 20:54, on Zulip):

Does bar(x) fail to compile because foo(x) consumed x?

XAMPPRocky (Dec 04 2019 at 20:54, on Zulip):

@simulacrum So we could have u32::as_mut_le_bytes and it would be entirely safe?

rkruppe (Dec 04 2019 at 20:55, on Zulip):

@Josh Triplett If that code works, then it's because it was reborrowed (i.e. &mut *x was inserted) at the call sites

Josh Triplett (Dec 04 2019 at 20:55, on Zulip):

@XAMPPRocky I don't think as_mut_le_bytes is implementable unless you're on an le platform. ;)

Josh Triplett (Dec 04 2019 at 20:55, on Zulip):

@rkruppe Ah.

Josh Triplett (Dec 04 2019 at 20:55, on Zulip):

OK then.

Josh Triplett (Dec 04 2019 at 20:55, on Zulip):

Well, I have a safe transmute proposal to rewrite then!

simulacrum (Dec 04 2019 at 20:55, on Zulip):

simulacrum So we could have u32::as_mut_le_bytes and it would be entirely safe?

Well, I mean, yes, but it's unlikely to be what you want

Josh Triplett (Dec 04 2019 at 20:56, on Zulip):

Thank you very much for the help!

XAMPPRocky (Dec 04 2019 at 20:56, on Zulip):

@Josh Triplett Ah yes true, we could have u32::as_mut_ne_bytes though.

simulacrum (Dec 04 2019 at 20:56, on Zulip):

there's actually an issue to that effect, let me try to dig that up

Josh Triplett (Dec 04 2019 at 20:56, on Zulip):

@XAMPPRocky I'm planning to spell that just as_mut_bytes. ;)

Josh Triplett (Dec 04 2019 at 20:56, on Zulip):

(or possibly as_bytes_mut)

XAMPPRocky (Dec 04 2019 at 20:58, on Zulip):

I guess if there's only one method omitting ne would be fine. Though I would wonder if it's also possible to have it always work for a specific endianness?

simulacrum (Dec 04 2019 at 20:58, on Zulip):

https://github.com/rust-lang/rust/issues/64464

XAMPPRocky (Dec 04 2019 at 21:02, on Zulip):

Is it possible to create a slice that goes in reverse of another slice?

XAMPPRocky (Dec 04 2019 at 21:03, on Zulip):
| 0 | 1 | 2| 3 | 4 |
      <--------
gnzlbg (Dec 04 2019 at 21:03, on Zulip):

Hypothetical:

What am I missing? This just works.

simulacrum (Dec 04 2019 at 21:03, on Zulip):

No -- that issue mentions as such, slices always go "forwards"

simulacrum (Dec 04 2019 at 21:04, on Zulip):

@gnzlbg @Josh Triplett was not aware/misunderstanding something, I think

Josh Triplett (Dec 04 2019 at 21:04, on Zulip):

@XAMPPRocky No, slices have to go forwards.

gnzlbg (Dec 04 2019 at 21:05, on Zulip):

Reading the whole thread. In C++ T& aren't values, but in Rust, a &mut T is just a normal value, and you can just move it around. Maybe that was causing the confusion?

rkruppe (Dec 04 2019 at 21:06, on Zulip):

I think the cause of confusion is thinking of references as more special than they are, not realizing the generality of lifetime variables. A common mistake sadly encouraged by almost all of our documentation.

Josh Triplett (Dec 04 2019 at 21:06, on Zulip):

I definitely didn't have any confusion with C++ references, since I'm not a C++ developer. ;)

Josh Triplett (Dec 04 2019 at 21:06, on Zulip):

@gnzlbg I didn't quite expect it &mut T itself to have "move" semantics, and thus didn't realize this worked without mutable aliasing. I also didn't realize that a function from &mut self to &mut U would preserve the exclusivity of the borrow (or really anything about the borrow, if it used transmute).

gnzlbg (Dec 04 2019 at 21:07, on Zulip):

I find useful to think of &mut T as just type MutRef = &mut T;

Josh Triplett (Dec 04 2019 at 21:08, on Zulip):

That's definitely the semantic I wanted, but it somehow felt too good to be true that it actually worked that way.

gnzlbg (Dec 04 2019 at 21:08, on Zulip):

If you mentally replace it everywhere with MutRef, it looks like just a normal type name, and same rules applies

gnzlbg (Dec 04 2019 at 21:08, on Zulip):

(to be fully correct you need to add the life-time, but...)

gnzlbg (Dec 04 2019 at 21:09, on Zulip):

FWIW I had this aha moment at some point as well

gnzlbg (Dec 04 2019 at 21:09, on Zulip):

I don't know why but I also though that references were more special than they are (in C++ they are, and I came with a C++ background, so probably just extrapolated that to Rust).

rkruppe (Dec 04 2019 at 21:10, on Zulip):

Building on @gnzlbg's suggestion, I would describe &mut as struct MutRef<'a, T>(*mut T, PhantomInvariant<'a>); with some syntactic sugar. Except we don't have PhantomInvariant. (Can it even be defined without involving &mut?)

simulacrum (Dec 04 2019 at 21:11, on Zulip):

I think so -- via trait objects

simulacrum (Dec 04 2019 at 21:11, on Zulip):

in particular IIRC dyn Trait<'a> is invariant over the 'a because it could be &'a mut () internally or so

rkruppe (Dec 04 2019 at 21:12, on Zulip):

oh right

Josh Triplett (Dec 04 2019 at 21:13, on Zulip):

The confusion I had was that I assumed &mut T being passed to a &mut T parameter had something to do with T (or more to the point with the implicit lifetime 'a in the &'a mut T, and wasn't just completely opaque, because the compiler would need to track something something lifetimes.

Josh Triplett (Dec 04 2019 at 21:13, on Zulip):

This is a major improvement in my understanding. :)

lqd (Dec 04 2019 at 21:14, on Zulip):

PhantomData<Cell<&'a ()>> is also invariant over the 'a IIRC

simulacrum (Dec 04 2019 at 21:15, on Zulip):

(I think it certainly can -- with re-borrowing -- but that's sort of handwavy since there's no good reference to my knowledge when reborrowing can or can't occur

gnzlbg (Dec 04 2019 at 21:18, on Zulip):

PhantomData<Cell<&'a ()>> is also invariant over the 'a IIRC

Yes that's correct.

Lokathor (Dec 05 2019 at 06:23, on Zulip):

@Josh Triplett for reference, bytemuck already lets you cast &mut T and &mut [T] between the Pod types.

Josh Triplett (Dec 05 2019 at 08:08, on Zulip):

One other question on this, now that I understand mutable references a little better:
Is it legal, and semantically correct, to consume a mutable reference, produce a non-mutable reference with the same loan, then transmute that to a mutable reference with the same loan, and return that?

Josh Triplett (Dec 05 2019 at 08:08, on Zulip):

For instance, could I go from &mut T to &T to &U to &mut U, within a method that consumes a &mut T and returns a &mut U?

rkruppe (Dec 05 2019 at 08:34, on Zulip):

Not legal under stacked borrows I believe, or if there's a way to do it, it greatly depends on how exactly go from one reference type to another. But this has nothing to do with lifetimes, just with the principle that making a mutable reference from a shared one plays havoc with aliasing rules we want to have.

Josh Triplett (Dec 05 2019 at 08:37, on Zulip):

@rkruppe Oh, interesting. In that case, I have a follow-on question...

Josh Triplett (Dec 05 2019 at 08:38, on Zulip):

This is related to https://internals.rust-lang.org/t/pre-rfc-v2-safe-transmute/11431/1

Josh Triplett (Dec 05 2019 at 08:38, on Zulip):

Suppose I have a trait that provides a function from &T to &U, which is a safe wrapper around std::mem::transmute with requirements that ensure it's safe.

Josh Triplett (Dec 05 2019 at 08:38, on Zulip):

Suppose I also have traits that provide safe functions from &T to and from byte slices.

Josh Triplett (Dec 05 2019 at 08:39, on Zulip):

I'd like to have this function (inside the ToBytes trait):

    fn cast_mut<T: FromBytes>(&mut self) -> Result<&mut T, FromBytesError>
      where Self: FromAnyBytes { /*...*/ }
Josh Triplett (Dec 05 2019 at 08:40, on Zulip):

The FromBytes trait provides a method from &[u8] to &T.

Josh Triplett (Dec 05 2019 at 08:40, on Zulip):

It doesn't provide a method to get a &mut T because that's not safe without additional guarantees (namely those enforced by cast_mut above).

Josh Triplett (Dec 05 2019 at 08:42, on Zulip):

Does that make that function un-implementable, even though it otherwise has all the requirements it needs to do so safely?

Josh Triplett (Dec 05 2019 at 08:44, on Zulip):

It would be possible to have cast_mut transmute directly from &mut Self to &mut T without going via T::from_bytes, but that would require T: FromAnyBytes rather than T: FromBytes, which would exclude types like bool that need to validate the bytes first.

rkruppe (Dec 05 2019 at 08:54, on Zulip):

Couldn't you downgrade the &mut Self to &Self, try the FromBytes cast, and if it succeeds do the transmute from self to &mut T directly, ignoring the &T you got from FromBytes?

rkruppe (Dec 05 2019 at 08:55, on Zulip):

Alternatively, have FromBytes provide the validation separately from the cast (presumably the cast could then be a provided method)

Josh Triplett (Dec 05 2019 at 08:55, on Zulip):

...we could, yes. That seems wasteful, but that would work.

Josh Triplett (Dec 05 2019 at 08:55, on Zulip):

I guess if it gets inlined the waste won't actually matter.

Josh Triplett (Dec 05 2019 at 09:00, on Zulip):

And factoring out the validation seems nice, too.

Josh Triplett (Dec 05 2019 at 09:00, on Zulip):

Is there a way to have a trait method that you can't override, and can only take the default implementation of? Or would that require having a sealed trait that's implemented for the unsealed trait?

Lokathor (Dec 05 2019 at 09:01, on Zulip):

Josh I think your whole plan is fundamentally a bad path

Lokathor (Dec 05 2019 at 09:02, on Zulip):

casting should always be cheap

Lokathor (Dec 05 2019 at 09:02, on Zulip):

if you let user logic in that's a bad time

Lokathor (Dec 05 2019 at 09:05, on Zulip):

core::mem::transmute is unsafe because it's too permissive, but it's fundamentally the correct design from an API usage perspective

Lokathor (Dec 05 2019 at 09:07, on Zulip):

unfortunately it's 2am my zone, so if you want more you'll have to catch me again some time

Josh Triplett (Dec 05 2019 at 09:13, on Zulip):

@Lokathor Casting should always be cheap, yes. But, for instance, if the options are "you can't safely cast a byte to bool" or "you can cast a byte to bool with a single check to make sure the byte is 0 or 1", I'll take the latter.

Josh Triplett (Dec 05 2019 at 09:14, on Zulip):

Casting from bytes to u32 should optimize down to no code, because any possible bytes are a valid u32.

Josh Triplett (Dec 05 2019 at 09:14, on Zulip):

Casting from bytes to bool has to either validate or be unsafe.

Josh Triplett (Dec 05 2019 at 09:18, on Zulip):

@Lokathor It's 1am my time as well, and I'm going to sleep too. We can talk more another time; I'd very much like to get your feedback.

gnzlbg (Dec 05 2019 at 09:43, on Zulip):

For instance, could I go from &mut T to &T to &U to &mut U, within a method that consumes a &mut T and returns a &mut U?

You can't go from & to &mut, but you can go from &mut -> usize -> &mut.

gnzlbg (Dec 05 2019 at 09:44, on Zulip):

Casting from bytes to u32 should optimize down to no code, because any possible bytes are a valid u32.

Is that so? You can't safely cast a MaybeUninit<u32> to an u32 (or at least, not today).

Josh Triplett (Dec 05 2019 at 09:46, on Zulip):

@gnzlbg Yes, that's so. These traits specifically support casting from &[u8], which is necessarily already initialized.

Josh Triplett (Dec 05 2019 at 09:47, on Zulip):

Also, yes, all bit patterns that the storage of a u32 can take are valid for u32. The problem with MaybeUninit is the UB of reading uninitialized memory. That doesn't apply to [u8].

gnzlbg (Dec 05 2019 at 09:48, on Zulip):

Also, yes, all bit patterns that the storage of a u32 can take are valid for u32.

This is wrong, or at least, this is not the case today. For example, the 0xUU bitpattern isn't valid for u32.

Josh Triplett (Dec 05 2019 at 09:48, on Zulip):

I'm talking about real machines with real bits. :P

gnzlbg (Dec 05 2019 at 09:48, on Zulip):

Ah, I was talking about Rust.

Josh Triplett (Dec 05 2019 at 09:49, on Zulip):

Not the LLVM abstract model and the concept of "undef", which, again, doesn't apply to [u8].

gnzlbg (Dec 05 2019 at 09:49, on Zulip):

I didn't say anything about LLVM.

Josh Triplett (Dec 05 2019 at 09:50, on Zulip):

If you have a [u8] and it points to unitialized memory, you've already done something wrong. The operation from &[u8] to &u32 doesn't do anything wrong.

Josh Triplett (Dec 05 2019 at 09:50, on Zulip):

0xUU isn't a bit pattern.

gnzlbg (Dec 05 2019 at 09:50, on Zulip):

If you have a [u8] and it points to unitialized memory, you've already done something wrong. The operation from &[u8] to &u32 doesn't do anything wrong.

That's correct.

gnzlbg (Dec 05 2019 at 09:51, on Zulip):

0xUU isn't a bit pattern.

What's the bitpattern of MaybeUninit::<u8>::uninit() ?

Josh Triplett (Dec 05 2019 at 09:52, on Zulip):

Something in the range 0x00 to 0xFF and you don't know what.

Josh Triplett (Dec 05 2019 at 09:52, on Zulip):

I'm not (today) attempting to have an argument about the UB of reading uninitialized memory.

gnzlbg (Dec 05 2019 at 09:53, on Zulip):

Something in the range 0x00 to 0xFF and you don't know what.

This is incorrect.

Josh Triplett (Dec 05 2019 at 09:54, on Zulip):

Please don't attempt to define the words I'm using. Feel free to define words you're using.

Josh Triplett (Dec 05 2019 at 09:54, on Zulip):

Let's try this again. It's 1:53am. I have zero interest in hearing yet another treatise on UB.

gnzlbg (Dec 05 2019 at 09:55, on Zulip):

I use the rust abstract machine definitions, which defines all rust bits as being 0, 1, or U.

Josh Triplett (Dec 05 2019 at 09:55, on Zulip):

I'm talking about hardware, and bit patterns that can actually appear in memory, and a cast operation that goes from &[u8] to &T. So I don't have to care about uninitialized memory, thankfully.

Josh Triplett (Dec 05 2019 at 09:56, on Zulip):

Please take it to a thread that's about uninitialized memory. Not everyone talking about bit patterns in actual memory is talking about an abstract machine.

gnzlbg (Dec 05 2019 at 09:56, on Zulip):

Your assumption that MaybeUninit::<u8>::uninit() is a bit pattern in range 0x00 and 0xFF is wrong, and would mean that:

let x = unsafe { MaybeUninit::<u8>::uninit().assume_init() };

is a safe operation, and that

let x = MaybeUninit::<u8>::uninit();
unsafe { assert_eq!(x.assume_init(), x.assume_init()) };

never panics. Yet the .assume_init() is UB, and anything can happen.

Josh Triplett (Dec 05 2019 at 10:00, on Zulip):

Please stop trying to redefine the terms I'm using, and please stop assuming I don't understand Rust just because I'm not using the terms you prefer.

Josh Triplett (Dec 05 2019 at 10:01, on Zulip):

Bytes in memory always have a value. The Rust abstract machine defines the behavior of the compiler, not the hardware. And for the purposes of this thread, I don't care about uninitialized memory because I'm dealing with [u8].

gnzlbg (Dec 05 2019 at 10:22, on Zulip):

I agreed with you on the first line of this conversation that for [u8] there is no problem. Yet you keep repeating that all bitpatterns are valid for u32, which I think is incorrect for the purpose of designing a safe transmute trait which appears to be your goal.

gnzlbg (Dec 05 2019 at 10:24, on Zulip):

I'm not using the terms you prefer.

I'm using the terms most people talking about this have agreed on and that we have in writing. If you want to use different terms, open an issue.

pnkfelix (Dec 05 2019 at 10:56, on Zulip):

@simulacrum wrote:

there's no difference between Foo<'a, T> and &mut T

by the way: This is not quite true. Namely, Foo may be covariant or invariant with respect to T, while &'a mut T is always invariant with respect to T. (The variance of Foo is based inferred on the ADT-definition of Foo; there's currently no way to declare variance explicitly.)

centril (Dec 05 2019 at 14:23, on Zulip):

(See https://doc.rust-lang.org/nightly/nomicon/phantom-data.html for variances)

centril (Dec 05 2019 at 14:24, on Zulip):

@Josh Triplett in case you haven read it, @RalfJ's https://www.ralfj.de/blog/2019/07/14/uninit.html is a good read

centril (Dec 05 2019 at 14:31, on Zulip):

Also relevant:
- https://arxiv.org/abs/1903.00982
- https://aaronweiss.us/pubs/popl19-src-oxide-slides.pdf

Lokathor (Dec 05 2019 at 21:25, on Zulip):

I don't think that safe transmuting uninit to init was ever on the table, or being proposed.
In the absence of a freeze intrinsic, that's clearly abstract machine UB.
We are pre-supposing that you have initialized memory in one type and want it in a new type.
The problem of uninitialized memory is an entirely different beast to attack, in some other topic elsewhere.

Lokathor (Dec 05 2019 at 22:07, on Zulip):

That said, accidentally reading uninit memory is a valid concern, which absolutely limits which types can be part of any particular safe-casting framework, for example #[repr(C)]struct Ex {a: u16, b:u8} can't be safe cast to [u8;4] even though the size/align would otherwise allow it, because you'd be exposing an uninit byte in the index 3 position of the output array.

Josh Triplett (Dec 05 2019 at 22:17, on Zulip):

@Lokathor That's why the ToBytes type we proposed doesn't allow implementations for that type, because it contains padding (at the end).

Josh Triplett (Dec 05 2019 at 22:18, on Zulip):

We're carefully defining the semantics to avoid reading uninitialized memory, rather than attempting to define semantics for reading uninitialized memory.

Lokathor (Dec 05 2019 at 23:27, on Zulip):

Right, that was an, "in gnzlbg's defense they've partly got a point" sort of note ;3

RalfJ (Dec 07 2019 at 09:10, on Zulip):

Not the LLVM abstract model and the concept of "undef", which, again, doesn't apply to [u8].

@Josh Triplett of course it does? I do not understand why you bring up u8 as if it was a special case. It is not.

I'm talking about real machines with real bits. :P

So you are writing assembly code then? Because if you are writing Rust, real machines don't matter. (Well yeah I know I am exaggerating but it's not entirely wrong either.)

RalfJ (Dec 07 2019 at 09:11, on Zulip):

Bytes in memory always have a value. The Rust abstract machine defines the behavior of the compiler, not the hardware.

True. But as long as you are writing Rust code, it is fundamentally impossible to use hardware-level concepts to reason about your code.

RalfJ (Dec 07 2019 at 09:11, on Zulip):

So if you want to make an argument about how your Rust code behaves, the only option you have is the Rust Abstract machine. A safe function that transmutes MaybeUninit<u8> to u8 is unsound, can cause UB on the Rust Abstract Machine, and can lead to real bugs in the compiled program. It is of upmost importance that a safe transmute proposal avoids such transmutes.

RalfJ (Dec 07 2019 at 09:11, on Zulip):

https://github.com/rust-embedded/wg/blob/master/rfcs/0377-mutex-trait.md#the-rust-standard-library-mutex-uses-self-for-the-lock-method-why-choose-mut-self may also be interesting for you to read (sort of tangentially related)

@simulacrum that is really weird... why would one ever use Mutex<RefCell<T>>? That's two layers of interior mutability...
Not sure I buy the deadlock-safety argument; we can still create deadlocks through inconsistent lock ordering.
And then I can also still reentrantly acquire the same mutex with their &mut &Mutex scheme so how does it even help...?

@rkruppe

Except we don't have PhantomInvariant. (Can it even be defined without involving &mut?)

Alternative def.n: type PhantomInvariant<'a> = fn(&'a ()) -> &'a ()

Josh Triplett (Dec 07 2019 at 09:14, on Zulip):

First of all, my point was that once you have a &[u8] (or some other type that isn't MaybeUninit) you have real memory that can't be uninitialized (or you've done something undefined).

Second, I'm extremely done with this thread, with the "Rust Abstract Machine", and with formal methods in general.

RalfJ (Dec 07 2019 at 09:15, on Zulip):

Second, I'm extremely done with this thread, with the "Rust Abstract Machine", and with formal methods in general.

Well that's not very helpful. LLVM (and every C compiler) is full of soundness bugs because of such an attitude. I hope we can avoid Rust repeating all the same mistakes.

RalfJ (Dec 07 2019 at 09:17, on Zulip):

First of all, my point was that once you have a &[u8] (or some other type that isn't MaybeUninit) you have real memory that can't be uninitialized (or you've done something undefined).

Such an argument can be made on the R-AM just fine, and I agree for the case of u8. Not sure what you mean by "real", but the safety invariant of u8 rules out uninitialized bit patterns. However that's not true for every type; e.g. (u8, u16) does contain an uninitialized byte and thus a transmute from (u8, u16) to [u8; 4] is UB.

RalfJ (Dec 07 2019 at 09:18, on Zulip):

(This is assuming our current conservative rules, see https://github.com/rust-lang/unsafe-code-guidelines/issues/71. But even if that transmute is not UB any more one day it is still unsafe.)

Josh Triplett (Dec 07 2019 at 09:23, on Zulip):

I feel like we're retreading previous territory. I'm aware that padding is an issue, the proposal in the thread I previously linked to goes to great lengths to address that.

RalfJ (Dec 07 2019 at 09:29, on Zulip):

I feel like we're retreading previous territory. I'm aware that padding is an issue, the proposal in the thread I previously linked to goes to great lengths to address that.

Great! Sorry for not having read up on the entire context. I was mostly reacting to individual statements here that (at least without context) are thoroughly incompatible with the UCG's view of this -- in particular the part about "real machines" and "0xUU isnt a bit pattern". Some of this is likely "just" temrinology, but then figuring out terminology is a big part of formally defining anything.
I misunderstood what you mean by "undef doesnt apply to [u8]"; sorry for that.

Josh Triplett (Dec 07 2019 at 09:39, on Zulip):

It's alright. I'm a bit frustrated at the moment and it isn't your fault. I'm not dismissing formal methods, I just don't always want to have every discussion about code behavior forced into formal methods.

Last update: May 27 2020 at 22:15UTC