Stream: wg-ffi-unwind

Topic: weekly meeting


nikomatsakis (Oct 31 2019 at 17:36, on Zulip):

Hey @WG-ffi-unwind -- are we going to have a sync this week?

nikomatsakis (Oct 31 2019 at 17:38, on Zulip):

er, I see @Kyle Strand you wrote a few comments in the "lang team sync" topic...

nikomatsakis (Oct 31 2019 at 17:38, on Zulip):

let's use this one for weekly meetings instead? seems like a better name :)

nikomatsakis (Oct 31 2019 at 17:38, on Zulip):

so where are we at

nikomatsakis (Oct 31 2019 at 17:39, on Zulip):

I guess @Kyle Strand I should just merge https://github.com/rust-lang/project-ffi-unwind/pull/16 ?

Kyle Strand (Oct 31 2019 at 17:40, on Zulip):

Sounds good!

nikomatsakis (Oct 31 2019 at 17:42, on Zulip):

I think this is basically the same content as https://github.com/rust-lang/rfcs/pull/2797, right?

Kyle Strand (Oct 31 2019 at 17:42, on Zulip):

I feel like the PR is close to having agreement, which is nice (I am guessing Centril is holding off on approving until the scope questions are addressed)

nikomatsakis (Oct 31 2019 at 17:42, on Zulip):

I haven't revisited since my last round of comments

nikomatsakis (Oct 31 2019 at 17:42, on Zulip):

did you modify the text at all in response to those?

nikomatsakis (Oct 31 2019 at 17:42, on Zulip):

I should check

Kyle Strand (Oct 31 2019 at 17:43, on Zulip):

I guess Kyle Strand I should just merge https://github.com/rust-lang/project-ffi-unwind/pull/16 ?

You should always merge my PRs :wink:

Kyle Strand (Oct 31 2019 at 17:43, on Zulip):

No, I haven't made any modifications since then

nikomatsakis (Oct 31 2019 at 17:43, on Zulip):

haven't written anything up yet, but my "summarizing discussions" file and the new PR itself might be sufficient anyway

@Kyle Strand which "summarizing discussions" file did you mean? oh, the one that summarized all of zulip?

Kyle Strand (Oct 31 2019 at 17:43, on Zulip):

But those were the only two outstanding concerns

Kyle Strand (Oct 31 2019 at 17:43, on Zulip):

yep

nikomatsakis (Oct 31 2019 at 17:43, on Zulip):

ok, I feel like there was some discussion I thought would be good to summarize specifically..not sure which it was now :)

nikomatsakis (Oct 31 2019 at 17:44, on Zulip):

"catching foreign exceptions", that topic

nikomatsakis (Oct 31 2019 at 17:44, on Zulip):

I guess the other lang team update is that we've extended the set of arguments somewhat around "C unwind" vs "C noexcept" etc

Kyle Strand (Oct 31 2019 at 17:45, on Zulip):

Yes

nikomatsakis (Oct 31 2019 at 17:46, on Zulip):

oh I see there have been some new additions as well

nikomatsakis (Oct 31 2019 at 17:46, on Zulip):

I have to read those and see if I find them persuasive

nikomatsakis (Oct 31 2019 at 17:46, on Zulip):

one thing I would like to have is a list of which functions use "C" but actually can unwind

nikomatsakis (Oct 31 2019 at 17:47, on Zulip):

I guess longjmp is such an example :)

nikomatsakis (Oct 31 2019 at 17:47, on Zulip):

(at least on MSVC)

Kyle Strand (Oct 31 2019 at 17:48, on Zulip):

I have to read those and see if I find them persuasive

FWIW I believe the HackMD updates capture the important bits of the conversation; reading the whole thread is likely not an optimal use of time

nikomatsakis (Oct 31 2019 at 17:48, on Zulip):

I do not mean to read any threads :)

Kyle Strand (Oct 31 2019 at 17:49, on Zulip):

Perfect!

nikomatsakis (Oct 31 2019 at 17:49, on Zulip):

which thread are you referring to, "noexcept-like feature"? :)

nikomatsakis (Oct 31 2019 at 17:49, on Zulip):

Adding the guarantee to the language that extern "C" do not unwind has the same expressive power as C++ noexcept keyword, but much worse ergonomics.

Kyle Strand (Oct 31 2019 at 17:49, on Zulip):

Yes

nikomatsakis (Oct 31 2019 at 17:49, on Zulip):

I am struggling with this statement :)

nikomatsakis (Oct 31 2019 at 17:49, on Zulip):

in particular, I'd lke to see examples of the ergonomics?

acfoltzer (Oct 31 2019 at 17:49, on Zulip):

hey, sorry I'm late. catching up now

nikomatsakis (Oct 31 2019 at 17:49, on Zulip):

maybe they're below, but I got lost in the proof..

Kyle Strand (Oct 31 2019 at 17:49, on Zulip):

Oh, for that it might actually be useful to look at the examples gnzlbg gave in the thread

Kyle Strand (Oct 31 2019 at 17:50, on Zulip):

https://rust-lang.zulipchat.com/#narrow/stream/210922-wg-ffi-unwind/topic/noexcept-like.20feature/near/179520449

Kyle Strand (Oct 31 2019 at 17:50, on Zulip):

and

https://rust-lang.zulipchat.com/#narrow/stream/210922-wg-ffi-unwind/topic/noexcept-like.20feature/near/179444817

nikomatsakis (Oct 31 2019 at 17:50, on Zulip):

okI think the argument is:

nikomatsakis (Oct 31 2019 at 17:51, on Zulip):

you might like to guarantee "no unwinding" in cases where no C code is involved

Kyle Strand (Oct 31 2019 at 17:51, on Zulip):

Yes

nikomatsakis (Oct 31 2019 at 17:51, on Zulip):

that's an interesting argument

Kyle Strand (Oct 31 2019 at 17:51, on Zulip):

and the idea is that introducing this feature "accidentally" by way of ABI strings would be detrimental

Kyle Strand (Oct 31 2019 at 17:51, on Zulip):

"detrimental" may be too strong

Kyle Strand (Oct 31 2019 at 17:53, on Zulip):

Generally, I do think that if/when we introduce a feature to cause functions to automatically abort-on-panic, it should be applicable to functions regardless of the ABI

acfoltzer (Oct 31 2019 at 17:54, on Zulip):

okay, caught up on this thread at least. I have had even less time than I was expecting given our current demo crunch. I'm excited by #16!

nikomatsakis (Oct 31 2019 at 17:54, on Zulip):

That seems true but it doesn't mean we can't also the ABI

nikomatsakis (Oct 31 2019 at 17:55, on Zulip):

but I see the point

nikomatsakis (Oct 31 2019 at 17:55, on Zulip):

I've summarized here:

You might like to avoid unwinding of Rust functions, too

We discussed how unsafe code in particular must be careful to be "unwind safe", but this applies not just to "C" functions but really to any Rust function. Therefore, we might like to have a feature that guarantees that some callee cannot unwind. But do we really want to use the ABI string to contrl this in the more general case?

Kyle Strand (Oct 31 2019 at 17:55, on Zulip):

I think the quote from Nick L that I added to the HackMD doc captures the concern: "FFI is not a sandbox"

Kyle Strand (Oct 31 2019 at 17:56, on Zulip):

which is to say, the lang team has been planning to make it a sandbox for a long time, but at least one LLVM dev thought that sounded like the wrong approach. Not sure what weight to assign that fact.

acfoltzer (Oct 31 2019 at 17:57, on Zulip):

sandbox in what sense, here?

Kyle Strand (Oct 31 2019 at 17:57, on Zulip):

Sandboxing unwinding ops to prevent them from escaping

nikomatsakis (Oct 31 2019 at 17:57, on Zulip):

I mean it doesn't actually "prevent", of course, since it would be "UB"

Kyle Strand (Oct 31 2019 at 17:57, on Zulip):

by changing the semantics of a function (i.e. inserting a shim that does something)

nikomatsakis (Oct 31 2019 at 17:57, on Zulip):

I think it's more accurate to say that the ABI is promising not to unwind

nikomatsakis (Oct 31 2019 at 17:57, on Zulip):

it is most definitely not sandboxing it

nikomatsakis (Oct 31 2019 at 17:58, on Zulip):

it doesn't seem that different frm any other type

Kyle Strand (Oct 31 2019 at 17:58, on Zulip):

Oh, there's a misunderstanding here. The "sandbox" concern is about the planned behavior of extern "C", which is to insert an abort-on-unwind shim (which is well defined)

acfoltzer (Oct 31 2019 at 17:58, on Zulip):

got it. yeah, seems like the best we can do is insert shims for the unwinding mechanisms we're aware of, but that's certainly not a guarantee

nikomatsakis (Oct 31 2019 at 17:59, on Zulip):

Oh, there's a misunderstanding here. The "sandbox" concern is about the planned behavior of extern "C", which is to insert an abort-on-unwind shim (which is well defined)

that is only true for

extern "C" fn foo() { }

but not calls to some random extern "C" fn

nikomatsakis (Oct 31 2019 at 17:59, on Zulip):

it seems like a key point

nikomatsakis (Oct 31 2019 at 17:59, on Zulip):

because the strongest argument against extern "C" unwinding has precisely to do with this

Kyle Strand (Oct 31 2019 at 17:59, on Zulip):

True.

nikomatsakis (Oct 31 2019 at 17:59, on Zulip):

at least what I see as the strongest argument

Kyle Strand (Oct 31 2019 at 18:00, on Zulip):

Hm....I would see that as an argument for extern "C" unwinding: since the abort-shim can only be inserted in Rust functions, that would make extern "C" inconsistent.

nikomatsakis (Oct 31 2019 at 18:00, on Zulip):

which is that -Cpanic=abort must either introduce shims on calls to extern "C" functions to abort or else it means that -Cpanic=abort can cause UB

nikomatsakis (Oct 31 2019 at 18:00, on Zulip):

for code that otherwise worked

nikomatsakis (Oct 31 2019 at 18:00, on Zulip):

and then there is the question of the cost of said shims in terms of code size (I am not sure what the answer is there)

Kyle Strand (Oct 31 2019 at 18:00, on Zulip):

I'm talking about the non-panic=abort case, i.e., the behavior originally introduced in 1.24.0

acfoltzer (Oct 31 2019 at 18:01, on Zulip):

those shims might also depend on platform/library calls that are exactly the kinds of things that, say, embedded would want to avoid

nikomatsakis (Oct 31 2019 at 18:01, on Zulip):

Hm....I would see that as an argument for extern "C" unwinding: since the abort-shim can only be inserted in Rust functions, that would make extern "C" inconsistent.

fair, although you could view that as just an argument against adding the shims (to avoid inconsistency)

nikomatsakis (Oct 31 2019 at 18:02, on Zulip):

those shims might also depend on platform/library calls that are exactly the kinds of things that, say, embedded would want to avoid

I suppose that if you are on a platform that has no unwinding mechanism, then that's a different scenario

nikomatsakis (Oct 31 2019 at 18:02, on Zulip):

(not sure if that's what you were implying)

Kyle Strand (Oct 31 2019 at 18:02, on Zulip):

Right -- without the shims, extern "C" would unwind

Kyle Strand (Oct 31 2019 at 18:02, on Zulip):

instead of aborting

nikomatsakis (Oct 31 2019 at 18:02, on Zulip):

the point is that throws of foreign exceptions are out of our control, ultimately, so -Cpanic=abort can't change them to aborts

nikomatsakis (Oct 31 2019 at 18:02, on Zulip):

Right -- without the shims, extern "C" would unwind

it would just be UB al the time

nikomatsakis (Oct 31 2019 at 18:02, on Zulip):

(presuming we added "C unwind", I mean)

Kyle Strand (Oct 31 2019 at 18:03, on Zulip):

Re: panic=abort and calling foreign functions that unwind, I'd like to know whether noexcept in LLVM actually causes undefined behavior there. E.g. we already know it doesn't interfere with longjmp on Windows

nikomatsakis (Oct 31 2019 at 18:04, on Zulip):

@gnzlbg are you around?

acfoltzer (Oct 31 2019 at 18:04, on Zulip):

I don't like appealing to the authority of the C/C++ world, but if my -fno-exceptions code called a function in a library that threw an exception, I wouldn't expect anything other than UB

gnzlbg (Oct 31 2019 at 18:04, on Zulip):

@nikomatsakis more or less

nikomatsakis (Oct 31 2019 at 18:04, on Zulip):

@acfoltzer yeah but that's the scenario I think we're trying to avoid

Kyle Strand (Oct 31 2019 at 18:04, on Zulip):

That's certainly my default assumption in that world :laughing:

nikomatsakis (Oct 31 2019 at 18:04, on Zulip):

that is an alternative though

gnzlbg (Oct 31 2019 at 18:05, on Zulip):

I haven't read this thread

nikomatsakis (Oct 31 2019 at 18:05, on Zulip):

@gnzlbg what I specifically wanted to ask you is

nikomatsakis (Oct 31 2019 at 18:05, on Zulip):

do you have a list somewhere of libc functions that might unwind?

gnzlbg (Oct 31 2019 at 18:05, on Zulip):

yes, I posted it on the other thread, wait a sec

nikomatsakis (Oct 31 2019 at 18:05, on Zulip):

is it like "any pthread function", in the case of cancelation occuring?

nikomatsakis (Oct 31 2019 at 18:05, on Zulip):

aah, sorry

gnzlbg (Oct 31 2019 at 18:06, on Zulip):

http://man7.org/linux/man-pages/man7/pthreads.7.html

nikomatsakis (Oct 31 2019 at 18:06, on Zulip):

I don't like appealing to the authority of the C/C++ world, but if my -fno-exceptions code called a function in a library that threw an exception, I wouldn't expect anything other than UB

to expand on this -- the question is basically whether -Cpanic=abort should be the kind of thing that users have to be very careful and knowledge about (like certain details of target-feature), or is it something you can enable casually?

gnzlbg (Oct 31 2019 at 18:06, on Zulip):

s it like "any pthread function", in the case of cancelation occuring?

No, its more like any C or POSIX API that can do a system call

nikomatsakis (Oct 31 2019 at 18:06, on Zulip):

@gnzlbg presumably longjmp also falls into this category?

nikomatsakis (Oct 31 2019 at 18:06, on Zulip):

No, its more like any C or POSIX API that can do a system call

woah wait hold up :)

gnzlbg (Oct 31 2019 at 18:06, on Zulip):

in that link go for cancellation points, and there is a huge list: poll, open, sleep,

nikomatsakis (Oct 31 2019 at 18:06, on Zulip):

that's a fairly broad category

nikomatsakis (Oct 31 2019 at 18:06, on Zulip):

ok, that's interesting

gnzlbg (Oct 31 2019 at 18:07, on Zulip):

yes, there are a lot of APIs that are required to be cancellation points, and many that are allowed to be

gnzlbg (Oct 31 2019 at 18:07, on Zulip):

e.g. printf can be a cancellation point

gnzlbg (Oct 31 2019 at 18:07, on Zulip):

basically it works like a garbage collector would in say Java

gnzlbg (Oct 31 2019 at 18:08, on Zulip):

when you cancel a thread, the threading run-time will do so, once that thread calls a function from the C run-time

gnzlbg (Oct 31 2019 at 18:08, on Zulip):

if a thread doesn't do that, the run-time has little options to cancel the thread because the thread is doing other stuff

gnzlbg (Oct 31 2019 at 18:08, on Zulip):

so by calling open, you are yielding to the run-time, and might do a context switch, and at that point the runtime might say "you are cancelled"

acfoltzer (Oct 31 2019 at 18:09, on Zulip):

that makes a lot of sense but the implications are kind of horrifying

gnzlbg (Oct 31 2019 at 18:10, on Zulip):

I mean, the "anything that can do a context switch is a cancellation point" is my rule of thumb

gnzlbg (Oct 31 2019 at 18:10, on Zulip):

There are like ~50 or so POSIX APIs that can be cancellation points

gnzlbg (Oct 31 2019 at 18:10, on Zulip):

there are many C APIs that are not. Those 50 APIs are used a lot though.

gnzlbg (Oct 31 2019 at 18:11, on Zulip):

Then there is the issue of what happens if the thread doesn't call a cancellation point for too long

centril (Oct 31 2019 at 18:11, on Zulip):

I feel like the PR is close to having agreement, which is nice (I am guessing Centril is holding off on approving until the scope questions are addressed)

@Kyle Strand I checked my box 2 days ago but it seems rfcbot didn't record it

nikomatsakis (Oct 31 2019 at 18:11, on Zulip):

@gnzlbg thanks I will add those details to the hackmd

nikomatsakis (Oct 31 2019 at 18:11, on Zulip):

so I think i've revised my opinion

nikomatsakis (Oct 31 2019 at 18:11, on Zulip):

I used to think this was open-and-shut

centril (Oct 31 2019 at 18:11, on Zulip):

There; the "RFC" is in FCP now

nikomatsakis (Oct 31 2019 at 18:12, on Zulip):

but now I think that it's unclear

gnzlbg (Oct 31 2019 at 18:12, on Zulip):

there is the question whether we want to support those cancellation points

nikomatsakis (Oct 31 2019 at 18:13, on Zulip):

yes

gnzlbg (Oct 31 2019 at 18:13, on Zulip):

for example, we could say that using pthread_cancel with libstd threads is UB

gnzlbg (Oct 31 2019 at 18:13, on Zulip):

that's kind of the status quo right now

nikomatsakis (Oct 31 2019 at 18:13, on Zulip):

I also used to think that "C noexcept" was a "no brainer" to include, but now I think that is less clear -- the idea of just having "C" (which suppors unwinding in the platform default way) is somewhat appealing, and maybe having some form of no-except that is orthogonal to ABI

nikomatsakis (Oct 31 2019 at 18:13, on Zulip):

right, I think that is one of the options

nikomatsakis (Oct 31 2019 at 18:14, on Zulip):

I mean I think pthread cancelation is a terrible ida anyway :)

nikomatsakis (Oct 31 2019 at 18:14, on Zulip):

but .. maybe that's not for me to decide

nikomatsakis (Oct 31 2019 at 18:14, on Zulip):

I at least didn't appreciate that unwinding was used in this way

Kyle Strand (Oct 31 2019 at 18:14, on Zulip):

I also used to think that "C noexcept" was a "no brainer" to include, but now I think that is less clear -- the idea of just having "C" (which suppors unwinding in the platform default way) is somewhat appealing, and maybe having some form of no-except that is orthogonal to ABI

Yes, I don't believe it's a "no brainer" either :smile:

gnzlbg (Oct 31 2019 at 18:14, on Zulip):

might be a libs-team issue

nikomatsakis (Oct 31 2019 at 18:14, on Zulip):

I feel like it merits a more focused discussion

centril (Oct 31 2019 at 18:14, on Zulip):

but .. maybe that's not for me to decide

@nikomatsakis general point: I think it's fine to have opinionated language design

gnzlbg (Oct 31 2019 at 18:14, on Zulip):

note that if a thread doesn't call a C run-time function for a while, the run-time might send a signal and just kill it

nikomatsakis (Oct 31 2019 at 18:14, on Zulip):

one thing that I think would be highly relevant would be data

centril (Oct 31 2019 at 18:14, on Zulip):

(As an also-Haskeller it is predictable for me to say that however)

nikomatsakis (Oct 31 2019 at 18:15, on Zulip):

I would be interestd -- since @Taylor Cramer cares a lot about this -- to gather data on fuschia builds without -Cpanic=abort, for example

nikomatsakis (Oct 31 2019 at 18:15, on Zulip):

which is kind of a "worse case' :)

nikomatsakis (Oct 31 2019 at 18:16, on Zulip):

might be a libs-team issue

actually it occurs to me that

nikomatsakis (Oct 31 2019 at 18:16, on Zulip):

this is also an argument for #[unwind(allow)]

nikomatsakis (Oct 31 2019 at 18:16, on Zulip):

i.e., if we say that "C" ABI permits unwinding, but the function must be declared to unwind, and by-pointer calls assume it may unwind

centril (Oct 31 2019 at 18:16, on Zulip):

@nikomatsakis huh?

nikomatsakis (Oct 31 2019 at 18:17, on Zulip):

then the libc functions can become #[unwind(allow)]

nikomatsakis (Oct 31 2019 at 18:17, on Zulip):

in a backwards compatible way

gnzlbg (Oct 31 2019 at 18:17, on Zulip):

yes, that was also another possible design

centril (Oct 31 2019 at 18:17, on Zulip):

libc's backwards compatibility policies are not something I care about

Kyle Strand (Oct 31 2019 at 18:18, on Zulip):

Niko, do you mean that without the unwind(allow) annotation, extern "C" fn ... { /* Rust code */ } would insert the abort-shim?

Kyle Strand (Oct 31 2019 at 18:18, on Zulip):

libc's backwards compatibility policies are not something I care about

Noted, but that's not universal!

nikomatsakis (Oct 31 2019 at 18:18, on Zulip):

yes but also

nikomatsakis (Oct 31 2019 at 18:18, on Zulip):

it's not just about libc back-compat

Kyle Strand (Oct 31 2019 at 18:19, on Zulip):

Centril, I believe you previously objected to extern "C" <Rust fn> inserting the shim, but extern "C" being assumed to "possibly unwind", being inconsistent?

nikomatsakis (Oct 31 2019 at 18:19, on Zulip):

I guess it's a variant of the "it's simpler to have one ABI that supports the things the ABI supports"

centril (Oct 31 2019 at 18:19, on Zulip):

@nikomatsakis btw, 10 min to pre-triage

nikomatsakis (Oct 31 2019 at 18:19, on Zulip):

i.e., all extern "C" functions can unwind (the ABI permits it) -- but if we know the callee, we may know they won't

nikomatsakis (Oct 31 2019 at 18:19, on Zulip):

you could also argue for the opposite default

nikomatsakis (Oct 31 2019 at 18:20, on Zulip):

which is where @Amanieu started

nikomatsakis (Oct 31 2019 at 18:20, on Zulip):

ok, I'm seeing the case for that a bit stronger

nikomatsakis (Oct 31 2019 at 18:21, on Zulip):

I want to spend some time thinking on how to re-organize the hackmd now to better bring these ideas out

centril (Oct 31 2019 at 18:22, on Zulip):

@Kyle Strand "the shim" ?

centril (Oct 31 2019 at 18:22, on Zulip):

the abort on unwind shim?

centril (Oct 31 2019 at 18:22, on Zulip):

or some other?

centril (Oct 31 2019 at 18:22, on Zulip):

That sentence wasn't clear enough for me to understand & answer

Kyle Strand (Oct 31 2019 at 18:23, on Zulip):

Sorry, yes, the abort-on-unwind shim.

centril (Oct 31 2019 at 18:23, on Zulip):

@nikomatsakis From a purely compiler & spec POV I think a distinct ABI string is simplest at least; it is using an existing infra rather than inventing a new one

centril (Oct 31 2019 at 18:24, on Zulip):

@Kyle Strand the abort-on-unwind shim is mandatory if unwinding is not allowed for soundness

Kyle Strand (Oct 31 2019 at 18:24, on Zulip):

nikomatsakis From a purely compiler & spec POV I think a distinct ABI string is simplest at least; it is using an existing infra rather than inventing a new one

Removing nounwind from LLVM code generation for extern "C", and stabilizing the unwind(allowed/abort) attributes, is even simpler, isn't it?

centril (Oct 31 2019 at 18:25, on Zulip):

I don't believe so

nikomatsakis (Oct 31 2019 at 18:26, on Zulip):

@centril when you invoke a function, we already have a unique type that represents known callees

centril (Oct 31 2019 at 18:26, on Zulip):

The "Rust" ABI already does switch nounwind off, so that logic already exists wrt. pattern matching on ABIs

nikomatsakis (Oct 31 2019 at 18:26, on Zulip):

so it's not hard to distinguish "call by pointer" from "call to known function"

Kyle Strand (Oct 31 2019 at 18:26, on Zulip):

Kyle Strand the abort-on-unwind shim is mandatory if unwinding is not allowed for soundness

I'm talking about the case where unwinding is allowed. I remember you saying that this combination would be inconsistent:

Kyle Strand (Oct 31 2019 at 18:27, on Zulip):

...and I believe that's the combination Niko is describing above.

Kyle Strand (Oct 31 2019 at 18:27, on Zulip):

i.e., if we say that "C" ABI permits unwinding, but the function must be declared to unwind, and by-pointer calls assume it may unwind

in this message

centril (Oct 31 2019 at 18:27, on Zulip):

that seems inconsistent yes

Kyle Strand (Oct 31 2019 at 18:28, on Zulip):

Okay, glad I remembered correctly :smile:

nikomatsakis (Oct 31 2019 at 18:28, on Zulip):

I thought so, but now i'm not so sure.

nikomatsakis (Oct 31 2019 at 18:28, on Zulip):

It also depends a lot on the defaults.

centril (Oct 31 2019 at 18:28, on Zulip):

@nikomatsakis You are talking about the distinction between FnDef and FnPtr?

nikomatsakis (Oct 31 2019 at 18:29, on Zulip):

But it seems like a big part of this question is also a matter of costs -- like, if the worst part of "C unwinds" is the need for an abort shim on call sites, how much does that really cost in actual programs? can we readily measure that?

nikomatsakis (Oct 31 2019 at 18:29, on Zulip):

it seems eminently measurable

centril (Oct 31 2019 at 18:30, on Zulip):

@nikomatsakis at any rate, we have pre-triage now :slight_smile:

nikomatsakis (Oct 31 2019 at 18:30, on Zulip):

yes, I know.

gnzlbg (Oct 31 2019 at 18:33, on Zulip):

one open question with the unwind by default, nounwind opt-in, is how to apply #[unwind(aborts)] to function pointers

gnzlbg (Oct 31 2019 at 18:33, on Zulip):

but that's more a syntactic issue

Amanieu (Oct 31 2019 at 19:07, on Zulip):

For #[unwind(aborts)], it is the responsibility of the callee to ensure that it aborts instead of unwinding. From the caller's point of view it's just a nounwind function. So there isn't much to do with function pointers: by default they can unwind and specifying nounwind function pointers probably isn't worth the trouble.

gnzlbg (Oct 31 2019 at 19:17, on Zulip):

maybe, that was an unresolved question last time

gnzlbg (Oct 31 2019 at 19:18, on Zulip):

in case we wanted to do that in the future #[unwind(aborts)] can just be allowed when specifying function pointers, that would be a backward compatible change

acfoltzer (Nov 07 2019 at 18:27, on Zulip):

hey folks, I'm going to be a bit late to the sync this week as we have a contractor at the house still

nikomatsakis (Nov 07 2019 at 18:33, on Zulip):

:wave:

nikomatsakis (Nov 07 2019 at 18:33, on Zulip):

Hey @WG-ffi-unwind -- quick sync? I'm not sure if anything happened this week, though

nikomatsakis (Nov 07 2019 at 18:34, on Zulip):

I would be happy to spend a bit of time now on looking over the hackmd

acfoltzer (Nov 07 2019 at 18:49, on Zulip):

okay, contractors just finished up. I could take a look at a hackmd—which one?

nikomatsakis (Nov 07 2019 at 18:51, on Zulip):

https://hackmd.io/ymsEL6OpR6OSMoFr1As1rw

nikomatsakis (Nov 07 2019 at 18:52, on Zulip):

I've not done anything, I got distracted by a million other pings :)

nikomatsakis (Nov 07 2019 at 18:52, on Zulip):

But I was saying last week that I felt like the organization wasn't right anymore

nikomatsakis (Nov 07 2019 at 18:52, on Zulip):

Let me take 10 minutes to see...

acfoltzer (Nov 07 2019 at 18:52, on Zulip):

gotcha, taking a look now. I likewise have been on other things for the last week, but we're getting close to the end of this demo sprint

acfoltzer (Nov 07 2019 at 18:53, on Zulip):

also the Cranelift PR for eh_frame data is active once again, so this is really going to be occupying my attention again soon

nikomatsakis (Nov 07 2019 at 18:54, on Zulip):

what I didn't like about this framing

nikomatsakis (Nov 07 2019 at 18:54, on Zulip):

for one thing I didnj't like the lsit of two options

nikomatsakis (Nov 07 2019 at 18:55, on Zulip):

I thnk i'd like to sort of collapse the list of arguments into broader "considerations"

nikomatsakis (Nov 07 2019 at 18:55, on Zulip):

I'm creating a second hackmd to play around

nikomatsakis (Nov 07 2019 at 18:58, on Zulip):

@Taylor Cramer -- you mentioned that Fuschia had some numbers regarding the cost of -Cpanic=abort vs -Cpanic=unwind somewhere?

nikomatsakis (Nov 07 2019 at 18:58, on Zulip):

I know that @gnzlbg had some smaller measurements

gnzlbg (Nov 07 2019 at 19:02, on Zulip):

I only had some synthetic examples, where the impact was massive, but it is hard to extrapolate from there to a real application. An App where ~0% of the code does C FFI calls won't probably see any differences, but an FFI wrapper might or might not see some, depending on other factors like whether it uses variables with destructors in the frames doing FFI calls, etc.

gnzlbg (Nov 07 2019 at 19:03, on Zulip):

Maybe it would be interesting to benchmark, e.g., mozjpeg or similar with -C panic=abort and -C panic=unwind.

nikomatsakis (Nov 07 2019 at 19:03, on Zulip):

I agree

Taylor Cramer (Nov 07 2019 at 19:04, on Zulip):

cc @tmandry who is the better person to talk with about fuchsia stuff going forwards

gnzlbg (Nov 07 2019 at 19:04, on Zulip):

It's a wrapper over mozjpeg-sys, so the impact there can be big, but if it turns out it is small, it is probably safe to conclude that this won't impact large apps except for maybe pathological cases

Taylor Cramer (Nov 07 2019 at 19:04, on Zulip):

and is the one who collected the most recent data on this

Taylor Cramer (Nov 07 2019 at 19:05, on Zulip):

We (fuchsia) also have a decently large amount of FFI, and so would probably be an interesting benchmark for this specific case as well

nikomatsakis (Nov 07 2019 at 19:06, on Zulip):

yes, it seems like a perfect case to measure

nikomatsakis (Nov 07 2019 at 19:06, on Zulip):

of course the cost of -Cpanic=unwind is greater than the cost would be to just have abort guards on FFI calls

nikomatsakis (Nov 07 2019 at 19:06, on Zulip):

but it's a good and easy to measure upper bound

nikomatsakis (Nov 07 2019 at 19:06, on Zulip):

(I think it's an upper bound?)

acfoltzer (Nov 07 2019 at 19:09, on Zulip):

the addendum in the first hackmd about nounwind fn foo() is really interesting; it hadn't occurred to me but of course makes a lot of sense that people would use a nounwind C ABI to implement targeted no-panic guarantees

Kyle Strand (Nov 07 2019 at 19:17, on Zulip):

Hey all, sorry I missed this sync session, and generally haven't been very helpful this week. @nikomatsakis let me know if you think it would be valuable for me to attend the lang meeting; I believe we don't have a real update since last meeting, though.

nikomatsakis (Nov 07 2019 at 19:19, on Zulip):

I think it's ok to say "no update, we're still working on preparing this core question"

nikomatsakis (Nov 07 2019 at 19:20, on Zulip):

I mean I also think this is a pretty important question to get right!

nikomatsakis (Nov 07 2019 at 19:20, on Zulip):

"the rest is details"

nikomatsakis (Nov 07 2019 at 19:20, on Zulip):

I think we've kind of covered the space

nikomatsakis (Nov 07 2019 at 19:20, on Zulip):

in terms of considerations

nikomatsakis (Nov 07 2019 at 19:29, on Zulip):

ok so I've been working on this alternate form

nikomatsakis (Nov 07 2019 at 19:30, on Zulip):

what I am trying to do is to factor out some of the "considerations" that are common, without getting into too much detail about specific approaches

nikomatsakis (Nov 07 2019 at 19:30, on Zulip):

what I wanted to do next is to create a section where I talk about the various options and try to explain their pros/cons in terms of those considerations

nikomatsakis (Nov 07 2019 at 19:30, on Zulip):

I'm running out of time to work on that :)

Kyle Strand (Nov 07 2019 at 19:46, on Zulip):

meaning of C == meaning of the "C" ABI specifier ?

Kyle Strand (Nov 07 2019 at 19:47, on Zulip):

I will try to spend some time looking at that in the next few days

tmandry (Nov 08 2019 at 07:58, on Zulip):

the rough numbers I have are, panic=abort reduces the size of a binary by 10%-20%

gnzlbg (Nov 08 2019 at 10:59, on Zulip):

@tmandry it might be interesting to try building fuchsia with -C panic=unwind and a nightly toolchain from July, and then compare that with the current nightly toolchain.

gnzlbg (Nov 08 2019 at 10:59, on Zulip):

IIRC the current toolchain does not emit nounwind for extern C, while the one in July should do that.

gnzlbg (Nov 08 2019 at 11:00, on Zulip):

That would show the impact of this particular change there.

tmandry (Nov 08 2019 at 11:01, on Zulip):

the last numbers I gathered were from July/August timeframe

tmandry (Nov 08 2019 at 11:01, on Zulip):

@gnzlbg if you can point me to a particular change to apply/revert I could also do that

tmandry (Nov 08 2019 at 11:01, on Zulip):

the numbers might be a little noisy with other changes mixed in there

gnzlbg (Nov 08 2019 at 11:03, on Zulip):

@tmandry let me check

gnzlbg (Nov 08 2019 at 11:04, on Zulip):

@RalfJ I see https://github.com/rust-lang/rust/pull/65346 and https://github.com/rust-lang/rust/pull/65020 landed but it appears that none of these removed nounwind from all extern "C" definitions and declarations, did that happen somewhere already?

gnzlbg (Nov 08 2019 at 11:04, on Zulip):

Ah https://github.com/rust-lang/rust/pull/63909 is not merged.

gnzlbg (Nov 08 2019 at 11:04, on Zulip):

@tmandry you might want to apply https://github.com/rust-lang/rust/pull/63909 to the current nightly.

gnzlbg (Nov 08 2019 at 11:05, on Zulip):

then you can just compare the current nightly with that PR, in both cases with -C panic=unwind

gnzlbg (Nov 08 2019 at 11:06, on Zulip):

that should show the effect of making extern "C" definitions and declarations allow unwinding by default

nikomatsakis (Nov 14 2019 at 18:28, on Zulip):

I might be late and/or miss a meeting today, @WG-ffi-unwind

Kyle Strand (Nov 14 2019 at 18:44, on Zulip):

Shoot, I forgot this was right after the release team meeting.

Kyle Strand (Nov 14 2019 at 18:45, on Zulip):

Want to sync up now?

nikomatsakis (Nov 14 2019 at 18:47, on Zulip):

OK, I'm around now -- not sure that there is much to say

nikomatsakis (Nov 14 2019 at 18:47, on Zulip):

At this point I still really want to see some data

nikomatsakis (Nov 14 2019 at 18:47, on Zulip):

It looks like I've gotten "close" to implementing the changes needed, I have to incorporate what @Amanieu mentioned

nikomatsakis (Nov 14 2019 at 18:47, on Zulip):

And then probably bug @tmandry to find out how to replicate their numbers but with a local build?

nikomatsakis (Nov 14 2019 at 18:47, on Zulip):

Not sure how hard that will be

nikomatsakis (Nov 14 2019 at 18:47, on Zulip):

Or I guess it doesn't have to be me who does that

nikomatsakis (Nov 14 2019 at 18:48, on Zulip):

In fact it seems like it would be better if it were someone else :)

nikomatsakis (Nov 14 2019 at 18:48, on Zulip):

Ah one thing I did want to talk to you about @Kyle Strand was that I think we are ripe for an internals blog post

nikomatsakis (Nov 14 2019 at 18:48, on Zulip):

er, "Inside Rust"

nikomatsakis (Nov 14 2019 at 18:48, on Zulip):

Maybe introducing the group with a bit of background, talking about how the first big discussion has to do with "C", and encouraging people to check out the hackmd, come to the zulip channel, something like that?

Kyle Strand (Nov 14 2019 at 18:51, on Zulip):

"All areas of life progressively come to resemble TODO lists" - me, inside my head, often

Kyle Strand (Nov 14 2019 at 18:52, on Zulip):

That sounds good. I'm sorry I haven't been active here this week. I need to take some days off work and catch up on the rest of my life.

nikomatsakis (Nov 14 2019 at 18:52, on Zulip):

:/

nikomatsakis (Nov 14 2019 at 18:52, on Zulip):

Oh please never apologize for that :)

nikomatsakis (Nov 14 2019 at 18:52, on Zulip):

To be clear, I think we're making good progress

nikomatsakis (Nov 14 2019 at 18:52, on Zulip):

I feel like at this point we've really bottomed out the pros/cons around the extern "C" thing fairly deeply

nikomatsakis (Nov 14 2019 at 18:52, on Zulip):

We're missing some data and we need to kind of have a chat and try to "make a call"

Kyle Strand (Nov 14 2019 at 18:53, on Zulip):

My third sentence was intended to be merely informational, not a continuation of the apology :laughing:

nikomatsakis (Nov 14 2019 at 18:54, on Zulip):

Are you interested in trying to write said blog post?

Kyle Strand (Nov 14 2019 at 18:55, on Zulip):

Maybe - I think that a higher-priority use of my time, when I have some, would be to get our "summaries & current status" documents in a good state.

Kyle Strand (Nov 14 2019 at 18:55, on Zulip):

Especially if we hope to get new contributors to the discussion via this blog post

Kyle Strand (Nov 14 2019 at 18:56, on Zulip):

I agree about needing to have a chat to "make the call." I want to talk to Centril more about the "FFI is not a sandbox" point - he left a response in the HackMD saying the verbiage there was "not justified", but I'm still pretty convinced that it's a valid concern.

nikomatsakis (Nov 14 2019 at 18:56, on Zulip):

one thing -- we do now have the two documents

nikomatsakis (Nov 14 2019 at 18:56, on Zulip):

I would prefer to deprecate the first one :)

Kyle Strand (Nov 14 2019 at 18:57, on Zulip):

the two HackMDs?

nikomatsakis (Nov 14 2019 at 18:57, on Zulip):

at least two, but yeah

nikomatsakis (Nov 14 2019 at 18:57, on Zulip):

the one centril commented on is the older one

nikomatsakis (Nov 14 2019 at 18:57, on Zulip):

I'm curious if other people feel the second one is better :)

nikomatsakis (Nov 14 2019 at 18:57, on Zulip):

I don't have the "FFI is not a sandbox" thing in there

Kyle Strand (Nov 14 2019 at 18:57, on Zulip):

two documents about extern "C" considerations? Yes, at most one should be introduced to the master branch of the Repo.

nikomatsakis (Nov 14 2019 at 18:57, on Zulip):

at least not "explicitly", but I do have the

nikomatsakis (Nov 14 2019 at 18:58, on Zulip):

well I do have a section about exposing the "full" ABI

nikomatsakis (Nov 14 2019 at 18:58, on Zulip):

I think that quote could be added, potentially, though I'm not sure how imp't it is

nikomatsakis (Nov 14 2019 at 18:58, on Zulip):

Newer document: "meaning of the C abi"

Kyle Strand (Nov 14 2019 at 18:58, on Zulip):

I think the "sandbox" quote captures something important, but doesn't do so in a clear/unambiguous way (which makes sense because it's just a snippet from a personal email).

nikomatsakis (Nov 14 2019 at 18:59, on Zulip):

I think what could be improved in this doc a bit

nikomatsakis (Nov 14 2019 at 18:59, on Zulip):

hmm have to think about it

nikomatsakis (Nov 14 2019 at 19:00, on Zulip):

well what I find compelling in this structure is reading the options at the end

nikomatsakis (Nov 14 2019 at 19:00, on Zulip):

maybe I should move them to the front :)

nikomatsakis (Nov 14 2019 at 19:00, on Zulip):

and trying to tell different "stories"--

nikomatsakis (Nov 14 2019 at 19:00, on Zulip):

anyway

nikomatsakis (Nov 14 2019 at 19:00, on Zulip):

I haven't made up my mind yet, in any case

nikomatsakis (Nov 14 2019 at 19:01, on Zulip):

Though I am much more sympathetic to the "noexcept is an orthogonal consideration from ABI" POV that I thought I was

nikomatsakis (Nov 14 2019 at 19:01, on Zulip):

but I think it has to come paired with the more permissive defaults

nikomatsakis (Nov 14 2019 at 19:01, on Zulip):

i.e., if extern "C" fns make it UB to unwind by default

nikomatsakis (Nov 14 2019 at 19:01, on Zulip):

I think that combination is strictly worse than "C unwind" ABI

Kyle Strand (Nov 14 2019 at 19:04, on Zulip):

Though I am much more sympathetic to the "noexcept is an orthogonal consideration from ABI" POV that I thought I was

I think that at least partially captures the "FFI is not a sandbox" concern.

Kyle Strand (Nov 14 2019 at 19:04, on Zulip):

They're at least tightly coupled.

Kyle Strand (Nov 14 2019 at 19:04, on Zulip):

I absolutely agree that UB-by-default is strictly worse than not-UB-by-default.

Kyle Strand (Nov 14 2019 at 19:05, on Zulip):

And I think in that sense, making ABI boundaries have some "sandbox" qualities _is_ in line with Rust's goals and language-values.

Kyle Strand (Nov 14 2019 at 19:08, on Zulip):

Or, to remove the word "sandbox" entirely:

Kyle Strand (Nov 14 2019 at 19:09, on Zulip):

In any case, I think this is a reasonable TODO list for me, with a soft deadline of...let's say this time next week:

nikomatsakis (Nov 14 2019 at 19:11, on Zulip):

I think that at least partially captures the "FFI is not a sandbox" concern.

I agree

Kyle Strand (Nov 14 2019 at 19:11, on Zulip):

That's roughly in order of what I think my priorites ought to be.

nikomatsakis (Nov 14 2019 at 19:11, on Zulip):

I don't think I would waste a lot of time scraping zulip

nikomatsakis (Nov 14 2019 at 19:12, on Zulip):

but I do think that updating the repo is maybe good

nikomatsakis (Nov 14 2019 at 19:12, on Zulip):

in particular, our roadmap seems wrong

nikomatsakis (Nov 14 2019 at 19:12, on Zulip):

i.e., we have a "step 0" I thought was settled

nikomatsakis (Nov 14 2019 at 19:12, on Zulip):

but it seems is not

nikomatsakis (Nov 14 2019 at 19:12, on Zulip):

and we've not updated that

nikomatsakis (Nov 14 2019 at 19:12, on Zulip):

("figuring out the overall direction")

Kyle Strand (Nov 14 2019 at 19:18, on Zulip):

What I want to try to do vis-a-vis Zulip is maintain a document that we can point to such that anyone who reads it (whether they're newcomers or haven't visited the stream in a while) should feel equipped to participate in the conversation without needing to read too far back in Zulip history.

Kyle Strand (Nov 14 2019 at 19:18, on Zulip):

I think my original approach of "summarize each week's Zulip activity" may not be the most efficient way to do that, but I'd still like to find some way to accomplish the goal.

nikomatsakis (Nov 14 2019 at 19:21, on Zulip):

Yes.

nikomatsakis (Nov 14 2019 at 19:21, on Zulip):

I do think that's a super good goal

nikomatsakis (Nov 14 2019 at 19:21, on Zulip):

I guess what I'm contending .. hmm, what am I contending

nikomatsakis (Nov 14 2019 at 19:21, on Zulip):

One observation is that while I like the hackmd approach, I do often wish for a "go deeper" option

nikomatsakis (Nov 14 2019 at 19:21, on Zulip):

A wiki is a plausible structure here

nikomatsakis (Nov 14 2019 at 19:21, on Zulip):

(We could even use GH's wikis)

nikomatsakis (Nov 14 2019 at 19:22, on Zulip):

i.e., I'd like to start from "here are the proposals and some pros and cons"

nikomatsakis (Nov 14 2019 at 19:22, on Zulip):

and be able to click from a given pro to learn more about the thing in question

nikomatsakis (Nov 14 2019 at 19:22, on Zulip):

but I also feel a "woah diminishing returns" flag going on

Kyle Strand (Nov 14 2019 at 19:22, on Zulip):

Yes

Kyle Strand (Nov 14 2019 at 19:22, on Zulip):

Hmmm

nikomatsakis (Nov 14 2019 at 19:22, on Zulip):

lots of work to organize this information and not obviously worth the effort :)

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

though if we found a good structure, it might be less work overall

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

and I am deeply concerned about the way our current practices

Kyle Strand (Nov 14 2019 at 19:23, on Zulip):

Well, to be fair, the topics in the stream are decently well named, so anyone interested in the history of a given question can probably find what they want.

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

leave us with a very confusing "design trail"

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

I want to be able to come back to this decision in a year, two years, etc, and clearly lay out what we took into account

Kyle Strand (Nov 14 2019 at 19:23, on Zulip):

Yes!

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

Well, to be fair, the topics in the stream are decently well named, so anyone interested in the history of a given question can probably find what they want.

well I sort of imagine that

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

the wiki page would link to zulip topics

nikomatsakis (Nov 14 2019 at 19:23, on Zulip):

i.e., that's the "deepest"

nikomatsakis (Nov 14 2019 at 19:24, on Zulip):

so the wiki page might just be roughly what's in the current hackmd, but also have links to zulip topics where you can read more

Kyle Strand (Nov 14 2019 at 19:24, on Zulip):

Agreed. That's what I did in my first "summary" doc, but unfortunately that was probably the most time-consuming aspect

nikomatsakis (Nov 14 2019 at 19:24, on Zulip):

or rather, the "sections" of the current hackmd might each be a wiki page

Kyle Strand (Nov 14 2019 at 19:24, on Zulip):

Hm....it would be nice to be able to "tag" old Zulip messages.

nikomatsakis (Nov 14 2019 at 19:24, on Zulip):

well, you could just link to the topics in general

nikomatsakis (Nov 14 2019 at 19:24, on Zulip):

and not try to link to specific messages within

nikomatsakis (Nov 14 2019 at 19:24, on Zulip):

like "this was discussed here"

nikomatsakis (Nov 14 2019 at 19:24, on Zulip):

I'm imagining like

nikomatsakis (Nov 14 2019 at 19:25, on Zulip):

Interaction with -Cpanic=abort and UB

Today, the -Cpanic=abort feature works by changing panic!() to cause an immediate abort at the site of panic. This in turn means we can remove all the "landing pads" and thus remove the impact of unwinding completely.

However, if the source of the unwinding is foreign code that is not possible. Instead, if we wish to avoid UB, we have to add landing pads to the call sites of any foreign function that may unwind, such that we can abort when the unwinding enters into a Rust frame. Note that all designs include ways to indicate foreign functions that cannot unwind, though they vary in defaults and approach. Inserting these landing pads has a certain cost, though it is less than the full cost of -Cpanic=unwind (since in that case landing pads are required on all call sites, not just foreign call sites).

You might ask "why not just have it be UB to invoke a foreign function that unwinds with -Cpanic=abort?" The problem here is that then you can have libraries which compile under both modes but which produce UB when executed with -Cpanic=abort. If we install "abort guards", those same libraries would merely abort at runtime, which is much more noticeable.

If you come from C++, this might not seem like a big deal. After all, mixing -fexceptions code with code with -fno-exceptions code is going to mess everything up. But it's the kind of low-level footgun we'd prefer to avoid in Rust if we can.

Discussed on:

nikomatsakis (Nov 14 2019 at 19:25, on Zulip):

maybe that's a ton of work to keep up to date too :)

nikomatsakis (Nov 14 2019 at 19:26, on Zulip):

I want to be able to come back to this decision in a year, two years, etc, and clearly lay out what we took into account

also, to be clear, I think we're doing way better than ever before here

Kyle Strand (Nov 14 2019 at 19:28, on Zulip):

Agreed - when discussing my original RFCs, I found several old relevant GitHub threads quite late in the process

Kyle Strand (Nov 14 2019 at 19:44, on Zulip):

I think as a first pass, making that map from design questions to Zulip-topics is more important than writing new "summary" verbiage for each question.

Kyle Strand (Nov 14 2019 at 19:44, on Zulip):

The summarizing effort can then be done by anyone at any time

Kyle Strand (Nov 14 2019 at 19:44, on Zulip):

If indeed it's even necessary.

Kyle Strand (Nov 14 2019 at 19:48, on Zulip):

Anyway, for now I'll plan on following my "TODO" list above, with the caveat that I'll try to keep my time investment into "Zulip summary/links" docs minimal.

centril (Nov 14 2019 at 22:36, on Zulip):

@Kyle Strand "not justified" in the sense that a claim is being made like "this is the way it is" without the "why"

Last update: Nov 15 2019 at 09:40UTC