Stream: general

Topic: zeroed vs MaybeUninit


simulacrum (Nov 15 2019 at 20:41, on Zulip):

@oli done -- not very intuitive

simulacrum (Nov 15 2019 at 20:41, on Zulip):

you want to do "view source / edit topic" and say "all later"

oli (Nov 15 2019 at 20:42, on Zulip):

oh, I found the editing, but not something that would do the whole shebang

oli (Nov 15 2019 at 20:45, on Zulip):

Anyway, wrt safe uses of zeroed, instead of asking whether dangerous API or obviously dangerous API is worse, shouldn't we ask "can we figure out a way to make zero init safe for the types where it is?"?

simulacrum (Nov 15 2019 at 20:47, on Zulip):

I think there's definitely some discussion around that and an auto trait for it

simulacrum (Nov 15 2019 at 20:47, on Zulip):

in theory we could get pretty much 100% fidelity since the compiler knows all cases of "nonzero"

oli (Nov 15 2019 at 20:48, on Zulip):

omg right, we could even detect enums that can be zeroed

oli (Nov 15 2019 at 20:48, on Zulip):

although for repr(Rust) types that's bad

simulacrum (Nov 15 2019 at 20:48, on Zulip):

probably needs someone to write up an RFC, there's probably issues I'm not thinking of :)

oli (Nov 15 2019 at 20:49, on Zulip):

because compiler updates can then break trait resolution

simulacrum (Nov 15 2019 at 20:49, on Zulip):

hm, sure, though I imagine you would perhaps say "don't use mem zeroed generally"

oli (Nov 15 2019 at 20:49, on Zulip):

or even just reordering of enum variants, which isn't a breaking change in any definition I think

simulacrum (Nov 15 2019 at 20:50, on Zulip):

trait Zero and auto trait Zeroable {} and mem::zeroed::<T: Zero>() with the first you need to explicitly implement

oli (Nov 15 2019 at 20:50, on Zulip):

yea, but if a compiler update changes Option<usize>::None into Option<usize>::Some(0), that's bad

simulacrum (Nov 15 2019 at 20:50, on Zulip):

sure, we'd error on the impl Zero for Foo

simulacrum (Nov 15 2019 at 20:50, on Zulip):

and have some guidelines -- maybe e.g. enums are never zero-safe

oli (Nov 15 2019 at 20:50, on Zulip):

oh it's all opt-in?

simulacrum (Nov 15 2019 at 20:50, on Zulip):

right, yeah, the safety aspect

simulacrum (Nov 15 2019 at 20:50, on Zulip):

but it's always safe to opt-in

oli (Nov 15 2019 at 20:51, on Zulip):

well repr(C) enums can be zero safe

simulacrum (Nov 15 2019 at 20:52, on Zulip):

sure - basically I'm saying that you'd say that for some types maybe we auto do it

simulacrum (Nov 15 2019 at 20:52, on Zulip):

e.g. a repr C struct with u32 or so fields

oli (Nov 15 2019 at 20:53, on Zulip):

kk

simulacrum (Nov 15 2019 at 20:53, on Zulip):

but if you get into repr(Rust) types, maybe we error out unless explicitly enabled

simulacrum (Nov 15 2019 at 20:53, on Zulip):

(and that explicit enabler is not unsafe ever, just a stability promise in some sense)

oli (Nov 15 2019 at 20:54, on Zulip):

the compiler can't make that stability promise

simulacrum (Nov 15 2019 at 20:55, on Zulip):

So what I'm getting at is that I'm saying "I will not make this non-zeroable"

simulacrum (Nov 15 2019 at 20:55, on Zulip):

e.g. maybe it's repr(Rust) struct

simulacrum (Nov 15 2019 at 20:55, on Zulip):

but I know that it'll always be valid at zero

oli (Nov 15 2019 at 20:55, on Zulip):

ah

oli (Nov 15 2019 at 20:55, on Zulip):

right, but what a zeroed value means may change arbitrarily

simulacrum (Nov 15 2019 at 20:55, on Zulip):

well, for some things, yes

simulacrum (Nov 15 2019 at 20:56, on Zulip):

but e.g. a u32 I could imagine we promise is always safe to zero

oli (Nov 15 2019 at 20:56, on Zulip):

yea, also any struct array or tuple I think

simulacrum (Nov 15 2019 at 20:56, on Zulip):

so if I have struct Foo(u32); I could promise that's safe to zero

oli (Nov 15 2019 at 20:56, on Zulip):

I only see problems with enums

simulacrum (Nov 15 2019 at 20:56, on Zulip):

well, things containing enums, right?

oli (Nov 15 2019 at 20:56, on Zulip):

right

oli (Nov 15 2019 at 20:56, on Zulip):

recursively

simulacrum (Nov 15 2019 at 20:57, on Zulip):

Do we optimize structs ever today by reusing padding fields?

simulacrum (Nov 15 2019 at 20:57, on Zulip):

I guess probably no

simulacrum (Nov 15 2019 at 20:57, on Zulip):

since you need to be able to take references

simulacrum (Nov 15 2019 at 20:58, on Zulip):

and in any case we'd never make zero magically invalid I suspect

simulacrum (Nov 15 2019 at 20:59, on Zulip):

yeah, I agree that only enums are presently an issue

simulacrum (Nov 15 2019 at 20:59, on Zulip):

though maybe e.g. references are also interesting

simulacrum (Nov 15 2019 at 20:59, on Zulip):

since we have "more than one" bit in them which we currently might assume we're free to use for enum optimization (eventually)

simulacrum (Nov 15 2019 at 21:00, on Zulip):

due to them being well-aligned

simulacrum (Nov 15 2019 at 21:00, on Zulip):

so &Align16 has not just zero as an invalid bit pattern

Lokathor (Nov 15 2019 at 21:00, on Zulip):

wait slow down, https://github.com/rust-lang/rust/pull/66059 Ralf is already working on type based panic if it would be invalid

simulacrum (Nov 15 2019 at 21:01, on Zulip):

right, but that's just at runtime

simulacrum (Nov 15 2019 at 21:01, on Zulip):

we're discussing making zeroed entirely safe

Lokathor (Nov 15 2019 at 21:01, on Zulip):

I'm not sure if that affects the rest of this since you said quite a bit in like 5min XD

Lokathor (Nov 15 2019 at 21:01, on Zulip):

oh, fully safe at compile time

Lokathor (Nov 15 2019 at 21:01, on Zulip):

"yes please"

Lokathor (Nov 15 2019 at 21:01, on Zulip):

with a trait!

simulacrum (Nov 15 2019 at 21:03, on Zulip):

I wonder if we could even get e.g. #[repr(ptr_packed)] struct Foo(&Align16, bool, bool, bool, bool); such that the bools go into the ptr 'slots' transparently

Lokathor (Nov 15 2019 at 21:04, on Zulip):

wait how does &Align16 have an invalid bit pattern other than zero?

simulacrum (Nov 15 2019 at 21:04, on Zulip):

you know that the bottom bits of the pointer can never be anything than zero, right?

simulacrum (Nov 15 2019 at 21:04, on Zulip):

e.g. 0x1 is not a valid value for that reference

simulacrum (Nov 15 2019 at 21:06, on Zulip):

(nor 0x.....1, etc.)

Lokathor (Nov 15 2019 at 21:16, on Zulip):

ahhh

Lokathor (Nov 15 2019 at 21:16, on Zulip):

right, so it has a number of bits of niche

Lokathor (Nov 15 2019 at 21:16, on Zulip):

does Rust actually use that currently?

simulacrum (Nov 15 2019 at 21:16, on Zulip):

I believe this is not generally true (currently)

simulacrum (Nov 15 2019 at 21:17, on Zulip):

i.e., we don't use it

simulacrum (Nov 15 2019 at 21:17, on Zulip):

because it means you can't take a reference to that field

simulacrum (Nov 15 2019 at 21:17, on Zulip):

(since then unrelated code would need to not "just" dereference it, which has a cost)

centril (Nov 15 2019 at 22:40, on Zulip):

I think there's definitely some discussion around that and an auto trait for it

An auto trait would easily brick a crate like https://docs.rs/refl/0.2.0/refl/struct.Id.html (if it used *mut T instead of function pointers, which a previous iteration did) rendering it unsound

centril (Nov 15 2019 at 22:41, on Zulip):

it gives you the ability to conjure up values for types that the crate author didn't want to enable

centril (Nov 15 2019 at 22:41, on Zulip):

so it cannot be a safe API

simulacrum (Nov 15 2019 at 22:41, on Zulip):

hm, I think that's not entirely true

simulacrum (Nov 15 2019 at 22:42, on Zulip):

i.e., we could presumably say that such types would need to be disabled somehow for it

simulacrum (Nov 15 2019 at 22:42, on Zulip):

in any case, I think saying that the things you need to guarantee for safety are just "library conditions on the existence of this type" are much easier to satisfy, in general, than "make sure there are no references etc in here anywhere"

centril (Nov 15 2019 at 22:42, on Zulip):

auto traits work structurally, so you'll need some structural condition to make the implementations unavailable

simulacrum (Nov 15 2019 at 22:43, on Zulip):

I guess that's true, since it would be backwards incompatible

simulacrum (Nov 15 2019 at 22:43, on Zulip):

but even if it stays unsafe but we reduce the things you need to prove that's still pretty amazing

simulacrum (Nov 15 2019 at 22:43, on Zulip):

And Ralf's work here is already moving us closer

centril (Nov 15 2019 at 22:44, on Zulip):

Ralf's PR is cursed ^^

simulacrum (Nov 15 2019 at 22:44, on Zulip):

well it just broke cargo

simulacrum (Nov 15 2019 at 22:44, on Zulip):

i.e., revealed unsoundness in some dep

centril (Nov 15 2019 at 22:45, on Zulip):

fun stuff

RalfJ (Nov 16 2019 at 09:27, on Zulip):

@simulacrum

hm, but so does uninitialized, right? My impression was that the two are essentially equivalent...

no. mem::uninitialized is pretty much always wrong, including in FFI. there are very few types for this this is not insta-UB. (basically, let m: MaybeUninit<T> = mem::uninitialized(); is okay, but that's about it)

RalfJ (Nov 16 2019 at 09:28, on Zulip):

sure, I understand that, but I'd expect us to push users towards MaybeUninit::zeroed().assume_init(). I guess that's not really better in any way.

actually mem::zeroed will do better run-time checks once https://github.com/rust-lang/rust/pull/66059 lands, so between the two of them, mem::zeroed is better ;)
MaybeUninit provides type-based tracking of initializedness; if you don't need that for correctness (which can indeed be the case if 0-initialization is allowed for your types), there is no reason to use it.

though arguably you should pass the ptr into FFI if you can rather than the zeroed struct (and then afterwards assume_init())

yes.

RalfJ (Nov 16 2019 at 09:32, on Zulip):

so I am actually fine with making mem::zeroed a const fn (maybe we could hack something to do a full validity check for such values, which generally for perf reasons we do not do in CTFE). however I'd prefer to not implement the init intrinsic in CTFE, it's quite a mess; instead we should implement mem::zeroed as MaybeUninit::zeroed().assume_init() and make the latter a const fn.

Elichai Turkel (Nov 21 2019 at 09:21, on Zulip):

@oli I like your idea :) I think using repr(C) is better than a Zeroable trait.
(although it might push some users to start putting repr(C) just to be able to zeroize)
that way we can just lint if mem::zeroed() is used on any non repr(C) type

Lokathor (Nov 23 2019 at 14:15, on Zulip):

repr(C) is not an appropriate way to track if a type can be zeroed

Last update: Dec 12 2019 at 00:50UTC