Stream: t-lang/wg-unsafe-code-guidelines

Topic: mmap concerns


Jake Goulding (Mar 27 2019 at 12:36, on Zulip):

We are having a good discussion over in https://rust-lang.zulipchat.com/#narrow/stream/187831-t-compiler.2Fwg-self-profile/topic/mmap.20concerns, but I figured this WG might have something to add.

TL;DR, I think that any usage of file-backed-mmap in Rust leads to UB. This is because we get a *mut T from mmap. If we convert it to a &T or &mut T, the rules of references could be broken by any arbitrary process writing to the backing file. This would be reflected in the Rust process, effectively causing an immutable reference to mutate or a mutable reference to mutate by "not us".

I'd love to know that my concerns are unwarranted!

Jake Goulding (Mar 27 2019 at 13:04, on Zulip):

@rkruppe did point out that we might be able to get a &[Cell<...>] (or equivalent) from mmap, which might sidestep all concerns.

rkruppe (Mar 27 2019 at 13:09, on Zulip):

Well, all the mutable-xor-shared concerns. There is also the question of how concurrent modification by other processes interacts with the memory consistency model, in particular whether volatile or atomic accesses are needed.

nagisa (Mar 27 2019 at 13:39, on Zulip):

These concerns are possibly better considered from the standpoint of shared memory and we at Standard thought a lot about it. The most trivial example would be two distinct virtual memory pages backed by the same physical memory page.

nagisa (Mar 27 2019 at 13:39, on Zulip):

basically the conclusion we arrived at is that the mmaped shared memory shall only be accessed via raw pointers.

nagisa (Mar 27 2019 at 13:40, on Zulip):

For now, anyway.

nagisa (Mar 27 2019 at 13:41, on Zulip):

We also considered using &[{Unsafe,}Cell<_>] and friends, but we found that to be too inconvenient for other reasons (we want to be able to do atomic operations on the memory...)

gnzlbg (Mar 27 2019 at 15:23, on Zulip):

The problem with mmap is that two consecutive reads can return different values in a sequential program. Because of this, if the reads are not volatile, AFAICT the behavior is undefined. Whether you are reading from mmap'ed memory that's modified by different processes, whether the same physical memory page is mapped to two different virtual memory pages within the same process, or whether you are reading from a register that's modified by a sensor, does not really matter.

gnzlbg (Mar 27 2019 at 15:26, on Zulip):

Somebody made the argument that multiple processes can be considered multiple threads of execution, and therefore atomic read / stores are enough, but that's IMO a long shot. Multiple processes != multiple threads, and unless our memory model ends up being multi-process (which is something that AFAICT no practical PL memory model does), atomic reads / writes guarantees don't apply here. Synchronization across threads does not matter if the program is sequential, and a "sufficiently smart" compiler could prove that the program is single threaded and turn all atomic read / writes into normal read / writes.

Florian Gilcher (Mar 27 2019 at 15:30, on Zulip):

Isn't that basically a similar problem to what the embedded working group approaches with libs like volatile cell? https://github.com/japaric/vcell ?

gnzlbg (Mar 27 2019 at 16:16, on Zulip):

Yep, that's pretty much it.

gnzlbg (Mar 27 2019 at 16:18, on Zulip):

There is also: https://docs.rs/voladdress/0.2.3/voladdress/ which lets you handle slices as well

gnzlbg (Mar 27 2019 at 16:20, on Zulip):

note how weird the API of VolBlock there has to be

gnzlbg (Mar 27 2019 at 16:21, on Zulip):

basically when indexing into an element of a volatile array to get a reference, you need a volatile reference that performs volatile load / stores on access

Jake Goulding (Mar 27 2019 at 17:59, on Zulip):

Isn't that basically a similar problem to what the embedded working group approaches with libs like volatile cell? https://github.com/japaric/vcell ?

I'm glad I'm not the only one who had this thought (and that it appears to be valid)

Jake Goulding (Mar 27 2019 at 18:01, on Zulip):

two consecutive reads can return different values

One argument made in the other thread was that in this case, data is never read via mmap. It's write-once-read-never. Does that change the situation?

gnzlbg (Mar 28 2019 at 07:55, on Zulip):

Well the problem is that your program behaves differently depending on how its optimized

gnzlbg (Mar 28 2019 at 07:56, on Zulip):

If one does not care about that, then one does not really care about UB I guess

RalfJ (Mar 28 2019 at 13:30, on Zulip):

@Jake Goulding there's been a long discussion about mmap on a forum... last year or so. let me find that.

RalfJ (Mar 28 2019 at 13:32, on Zulip):

here you go: https://users.rust-lang.org/t/how-unsafe-is-mmap/19635

RalfJ (Mar 28 2019 at 13:33, on Zulip):

given that no H/W IO stuff is involved, I think that atomics and not volatile are the right tool here

RalfJ (Mar 28 2019 at 13:33, on Zulip):

notice that "two consecutive reads can return different values" is an argument for atomic accesses just as much as it is one for volatile accesses

RalfJ (Mar 28 2019 at 13:34, on Zulip):

mmap can share memory with threads that run in a different address space (aka process), but I dont see why that would make a fundamental difference compared to threads running in the same address space -- the relevant part is that the physical address spaces backing the virtual pages overlap.

RalfJ (Mar 28 2019 at 13:35, on Zulip):

also @Jake Goulding one terminology nit:

any usage of file-backed-mmap in Rust leads to UB

it is UB only if the file gets mutated. you make it sound like just the act of mmap'ing and taking a reference leads to UB, that is not true. (sorry if I misunderstood you.)

RalfJ (Mar 28 2019 at 13:36, on Zulip):

It's write-once-read-never. Does that change the situation?

no, it just makes it harder to come up with examples.^^ "other parties can observe when the write happens" also means we have to use atomics/volatile.

gnzlbg (Mar 28 2019 at 15:51, on Zulip):

@RalfJ I would like to hear your argument about why atomics are not enough for when HW I/O is involved (or where you suggesting that they are?), you can think of HW I/O as some process with its own threads, on a separate CPU, writing to some memory location that your process is able to read.

gnzlbg (Mar 28 2019 at 15:53, on Zulip):

(deleted)

gnzlbg (Mar 28 2019 at 17:38, on Zulip):

AFAIK the difference between volatile and atomics is that volatile guarantees that the accesses will happen exactly in the order specified in the program, while atomics guarantees that your program will behave "as if" these accessed happened and "as if" they happened in that order - the difference between "exactly" and "as if" meaning that the compiler can eliminate consecutive read/writes for atomics (or dead ones) but not for volatile

gnzlbg (Mar 28 2019 at 17:40, on Zulip):

for memory mapped I/O whether the difference between "exactly" and "as if" matters is subtle - consider:

let x: *mut T;
let y: *mut T;
assert!(x != y);
let y_val = y.read();
assert!(y_val != 42);
x.write(42);
if y.read() == 42 {  /* dead code ? */ }
gnzlbg (Mar 28 2019 at 17:41, on Zulip):

with atomics, the two reads to y could be coalesced, and the if optimized away

gnzlbg (Mar 28 2019 at 17:42, on Zulip):

but while x and y point to two different virtual addresses, they can refer to the same physical memory even within the same process - if the reads are volatile, the if won't be removed, and its block will be executed when this is the case

gnzlbg (Mar 28 2019 at 17:53, on Zulip):

for this code to be correct using atomics, the y.read()s cannot be coalesced into one, that at least rules out using relaxed ordering

gnzlbg (Mar 28 2019 at 18:11, on Zulip):

but yeah, I think I see now that atomics (with the appropriate ordering) should be sufficient

Jake Goulding (Mar 28 2019 at 19:14, on Zulip):

it is UB only if the file gets mutated. you make it sound like just the act of mmap'ing and taking a reference leads to UB, that is not true. (sorry if I misunderstood you.)

@RalfJ you understood me correctly. I was under the impression that Rust was taking the position of "if any code flow path could cause UB, it is always UB".

RalfJ (Mar 29 2019 at 09:21, on Zulip):

@gnzlbg

I would like to hear your argument about why atomics are not enough for when HW I/O is involved (or where you suggesting that they are?), you can think of HW I/O as some process with its own threads, on a separate CPU, writing to some memory location that your process is able to read.

Atomics do not guarantee that every read/write hits the memory bus verbatim. For example, the compiler can optimize x.store(5, some_ordering); let val = x.load(some_other_ordering) assuming that val == 5. Atomics still assume that the observer is another thread running in the same memory model. In fact if the compiler can prove that x is never shared with another thread, it can even downgrade the atomic accesses to non-atomic accesses.
volatile OTOH is for the case where the observer basically literally sits on your memory bus and can observe every single read and write. It seems intuitively like that is a stronger observer and hence it should be legal to replace (relaxed!) atomic accesses with volatile, but that is not how memory models are defined. It might be possible to define them like that, but I know of no research looking into this. (and for non-relaxed atomic accesses this cannot work because of the synchronization they induce.)

RalfJ (Mar 29 2019 at 09:23, on Zulip):

(This is basically what you said about "exactly" vs "as if", but IMO this answers the question about why atomics are not good enough for memory-mapped IO)

gnzlbg (Mar 29 2019 at 09:24, on Zulip):

The question is then for mmapped files for interprocess communication (or virtual memory shenanigans), which atomic orderings are necessary ?

RalfJ (Mar 29 2019 at 09:25, on Zulip):

for this code to be correct using atomics, the y.read()s cannot be coalesced into one, that at least rules out using relaxed ordering

I am afraid the area of virtual memory effects is entirely unspecified, so I dont think any standard literally says anything about your program. Anything we can say here is wild speculation.
Any formal model I am aware of would allow the compiler to assume that objects at distinct virtual addresses are independent of each other.

gnzlbg (Mar 29 2019 at 09:26, on Zulip):

in the same way that the compiler can assume that another process won't mutate memory that you are using behind your back ?

RalfJ (Mar 29 2019 at 09:27, on Zulip):

I suppose that is comparable, yes

RalfJ (Mar 29 2019 at 09:27, on Zulip):

if you start messing with /dev/mem, you asked for it. basically. ;)

gnzlbg (Mar 29 2019 at 09:27, on Zulip):

unless you use volatile

RalfJ (Mar 29 2019 at 09:27, on Zulip):

The question is then for mmapped files for interprocess communication (or virtual memory shenanigans), which atomic orderings are necessary ?

Ignore the fact that this is interprocess. The orderings are the same as within one process. I mean, it's two threads sharing some of their address space that are synchronizing here.

RalfJ (Mar 29 2019 at 09:27, on Zulip):

unless you use volatile

yes. not that any formal model would cover this, but volatile should be sufficiently restricted to achieve this.

gnzlbg (Mar 29 2019 at 09:28, on Zulip):

i was hoping that there was a way to do this with a strong memory ordering

gnzlbg (Mar 29 2019 at 09:30, on Zulip):

@RalfJ so @Jake Goulding can't do better than using volatile loads / stores, unless, they are ensuring synchronization in some way ?

RalfJ (Mar 29 2019 at 09:32, on Zulip):

well I think I'd use atomic accesses, I think

RalfJ (Mar 29 2019 at 09:34, on Zulip):

or do you mean if the same file gets mapped twice?

RalfJ (Mar 29 2019 at 09:35, on Zulip):

I think at that point I am just out, all I can do is speculate. We are way off in uncharted territory, formally speaking, and I do not know enough about LLVM to tell if it would exploit pointer comparisons assuming the virtual memory mapping is injective.

gnzlbg (Mar 29 2019 at 09:36, on Zulip):

i mean mapping a file once, but another process (or thread can also map it), and then two processes (or threads) modify it concurrently

RalfJ (Mar 29 2019 at 09:36, on Zulip):

also I'm afraid I'll have to leave now, got a lot of stuff to do. I'll hopefully be back online in the evening.

Last update: Nov 19 2019 at 18:00UTC