Stream: wg-async-foundations

Topic: meeting 2020.04.28


tmandry (Apr 28 2020 at 17:00, on Zulip):

@WG-async-foundations meeting!

tmandry (Apr 28 2020 at 17:02, on Zulip):

Project board

tmandry (Apr 28 2020 at 17:03, on Zulip):

A few issues were closed this week:

tmandry (Apr 28 2020 at 17:05, on Zulip):

Interestingly, nothing is marked in progress at the moment

tmandry (Apr 28 2020 at 17:05, on Zulip):

I don't think there are any updates to blockers

tmandry (Apr 28 2020 at 17:06, on Zulip):

Let's jump to triage

tmandry (Apr 28 2020 at 17:06, on Zulip):

we have one uncategorized issue

tmandry (Apr 28 2020 at 17:07, on Zulip):

..filed by me :)

tmandry (Apr 28 2020 at 17:07, on Zulip):

pretty quiet week I guess

tmandry (Apr 28 2020 at 17:07, on Zulip):

Tracking issue for generator code quality #71407

tmandry (Apr 28 2020 at 17:07, on Zulip):

not much to say here

tmandry (Apr 28 2020 at 17:07, on Zulip):

there was a question about when tracking issues like this would be closed

tmandry (Apr 28 2020 at 17:08, on Zulip):

..which is valid, I don't know if there's a clear criteria for when to close it

nikomatsakis (Apr 28 2020 at 17:08, on Zulip):

heh, a good point

nikomatsakis (Apr 28 2020 at 17:08, on Zulip):

but I think it'd be ok to close when there are no outstanding ideas left that we know of

tmandry (Apr 28 2020 at 17:08, on Zulip):

yeah, that's roughly what I said

tmandry (Apr 28 2020 at 17:09, on Zulip):

ok, marking "triaged"

tmandry (Apr 28 2020 at 17:09, on Zulip):

so, I think it might be just you and me, @nikomatsakis

tmandry (Apr 28 2020 at 17:10, on Zulip):

otherwise I'd ask if people wanted things to work on

tmandry (Apr 28 2020 at 17:10, on Zulip):

Maybe we can use this time to talk about bigger goals for the rest of the year?

nikomatsakis (Apr 28 2020 at 17:10, on Zulip):

Let's do that

nikomatsakis (Apr 28 2020 at 17:10, on Zulip):

I've been wanting to ahve that conversation for a while

nikomatsakis (Apr 28 2020 at 17:10, on Zulip):

now is as good a time as any

nikomatsakis (Apr 28 2020 at 17:11, on Zulip):

Or, perhaps slightly differently but relatedly, about the overall org + scope of this group

nikomatsakis (Apr 28 2020 at 17:11, on Zulip):

I think those are kind of the same thing

nikomatsakis (Apr 28 2020 at 17:11, on Zulip):

I guess you saw me writing to @Taylor Cramer in #wg-async-foundations > "census" and organization

tmandry (Apr 28 2020 at 17:11, on Zulip):

yep

tmandry (Apr 28 2020 at 17:12, on Zulip):

I think our scope has been pretty narrowly focused so far this year

nikomatsakis (Apr 28 2020 at 17:12, on Zulip):

I guess it seems pretty obvious to me that we're struggling a bit with people / motivation here

nikomatsakis (Apr 28 2020 at 17:12, on Zulip):

I attribute some of that to not having clear goals and advertising around them

tmandry (Apr 28 2020 at 17:12, on Zulip):

agreed

nikomatsakis (Apr 28 2020 at 17:12, on Zulip):

I also think there's clearly some real-estate that we're not "monitoring"

nikomatsakis (Apr 28 2020 at 17:13, on Zulip):

I'm thinking of things like the futures crate, pin-utils crate, but also proposals like the ones from @Yoshua Wuyts

nikomatsakis (Apr 28 2020 at 17:13, on Zulip):

I think a core question is

nikomatsakis (Apr 28 2020 at 17:13, on Zulip):

there are like these different "groups" of activities

nikomatsakis (Apr 28 2020 at 17:13, on Zulip):
nikomatsakis (Apr 28 2020 at 17:13, on Zulip):
nikomatsakis (Apr 28 2020 at 17:13, on Zulip):
nikomatsakis (Apr 28 2020 at 17:13, on Zulip):

I'm not sure if combining those things is fruitful or not

nikomatsakis (Apr 28 2020 at 17:14, on Zulip):

on the one hand, there's obvious overlap, but I've also found that it's often harder to combine those kinds of activites than you might think

nikomatsakis (Apr 28 2020 at 17:14, on Zulip):

e.g., I think it's pretty useful to have a triage meeting focused on the impl side

tmandry (Apr 28 2020 at 17:14, on Zulip):

yeah, they're sort of in a different headspace if you will

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

nikomatsakis said:

I'm thinking of things like the futures crate, pin-utils crate, but also proposals like the ones from Yoshua Wuyts

https://github.com/rust-lang/rust/pull/70834, https://github.com/rust-lang/rust/pull/70817

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

right

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

and yet they sort of inform each other a bit, too

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

so it feels like there should be some amount of "coordination" or visibility or something

tmandry (Apr 28 2020 at 17:15, on Zulip):

right

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

I would add that

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

is probably a 4th thing

nikomatsakis (Apr 28 2020 at 17:15, on Zulip):

documenting is its own headspace

tmandry (Apr 28 2020 at 17:16, on Zulip):

indeed

nikomatsakis (Apr 28 2020 at 17:16, on Zulip):

I'm curious about your own take

nikomatsakis (Apr 28 2020 at 17:16, on Zulip):

like, what parts of the above interest you (if any) :)

tmandry (Apr 28 2020 at 17:17, on Zulip):

I guess one question that naturally arises re: documentation is, now that async is officially a part of the language, should we be seeking to add to the rust lang book

nikomatsakis (Apr 28 2020 at 17:17, on Zulip):

I guess I think that given what we've been saying, it makes sense to try and create a group focused on the polish, and then think about how we will manage the other things (even if that's just acknowledging that they're "on hold" for now)

nikomatsakis (Apr 28 2020 at 17:18, on Zulip):

tmandry said:

I guess one question that naturally arises re: documentation is, now that async is officially a part of the language, should we be seeking to add to the rust lang book

I suspect the answer is medium term, yes, short term, no

tmandry (Apr 28 2020 at 17:18, on Zulip):

yeah, I can see having multiple sub-groups making sense

nikomatsakis (Apr 28 2020 at 17:18, on Zulip):

the problem with multiple sub-groups is that we don't even really have the energy for one effective group right now

tmandry (Apr 28 2020 at 17:18, on Zulip):

right

tmandry (Apr 28 2020 at 17:18, on Zulip):

that's roughly what I was going to say..

tmandry (Apr 28 2020 at 17:18, on Zulip):

but

tmandry (Apr 28 2020 at 17:19, on Zulip):

maybe having regular design meetings about new things that people want

tmandry (Apr 28 2020 at 17:19, on Zulip):

(less frequently than weekly, probably)

tmandry (Apr 28 2020 at 17:19, on Zulip):

would be a good way to pick up some steam?

nikomatsakis (Apr 28 2020 at 17:19, on Zulip):

interesting

nikomatsakis (Apr 28 2020 at 17:20, on Zulip):

I do think that starting out with some kind of effort like this

nikomatsakis (Apr 28 2020 at 17:20, on Zulip):

/me thinks

tmandry (Apr 28 2020 at 17:20, on Zulip):

I mean, polish is important but I think focusing on it completely might have been a mistake

nikomatsakis (Apr 28 2020 at 17:20, on Zulip):

so I think that doing something simple and relatively low effort is a good idea

tmandry (Apr 28 2020 at 17:21, on Zulip):

tmandry said:

I mean, polish is important but I think focusing on it completely might have been a mistake

from a motivation standpoint

tmandry (Apr 28 2020 at 17:21, on Zulip):

well, and maybe others

nikomatsakis (Apr 28 2020 at 17:21, on Zulip):

yeah, perhaps

tmandry (Apr 28 2020 at 17:22, on Zulip):

I think we are clearly lacking some critical features / traits, focusing on those would be a good place to start

nikomatsakis (Apr 28 2020 at 17:22, on Zulip):

I have to go dust off my list of what I think async should do next, but I think the most obvious next steps to me are probably

nikomatsakis (Apr 28 2020 at 17:22, on Zulip):

those seem all like relatively achievable and solid goals

nikomatsakis (Apr 28 2020 at 17:23, on Zulip):

I think that in the medium term -- like, looking more at 2021, I'd be interested in dedicated generator syntax

nikomatsakis (Apr 28 2020 at 17:23, on Zulip):

but I think it's not a good idea to talk about language additions when we can get clear wins from the lib space

tmandry (Apr 28 2020 at 17:23, on Zulip):

yes, agree with all of the above

nikomatsakis (Apr 28 2020 at 17:23, on Zulip):

or at least, it's maybe a good idea to talk about lang additions, and brainstorm

nikomatsakis (Apr 28 2020 at 17:23, on Zulip):

but it's not where I'd be putting most of my energy

nikomatsakis (Apr 28 2020 at 17:23, on Zulip):

some other additions I would consider would be

nikomatsakis (Apr 28 2020 at 17:24, on Zulip):

adding in utilities like mutexes, channels

nikomatsakis (Apr 28 2020 at 17:24, on Zulip):

which are "executor-agnostic", basically

tmandry (Apr 28 2020 at 17:24, on Zulip):

ah, good point

nikomatsakis (Apr 28 2020 at 17:24, on Zulip):

these raise some questions

nikomatsakis (Apr 28 2020 at 17:24, on Zulip):

the most obvious being, where do they live in libstd hierarchy?

nikomatsakis (Apr 28 2020 at 17:24, on Zulip):

the interesting observation here is that this is all @T-libs stuff I'm talking about, so it'd be good to get somebody (maybe @Steven Fackler?) involved in this conversation, who sort of wants to champion that stuff

nikomatsakis (Apr 28 2020 at 17:24, on Zulip):

Ah, there is one language change I would consider:

nikomatsakis (Apr 28 2020 at 17:25, on Zulip):

extending Drop trait with more "async lifecycle" callbacks (cc @David Barsky)

nikomatsakis (Apr 28 2020 at 17:25, on Zulip):

this is roughly the contents of my blog post, typing into zulip now :)

nikomatsakis (Apr 28 2020 at 17:25, on Zulip):

I think part of what has held me up was some uncertainty about "how" to see those things happening

tmandry (Apr 28 2020 at 17:25, on Zulip):

yeah that's one I had in mind also

nikomatsakis (Apr 28 2020 at 17:26, on Zulip):

yeah, I think we should be looking for like "low glitz, max utility"

tmandry (Apr 28 2020 at 17:26, on Zulip):

probably less important than some of the early lib wins, but more important than generator syntax

nikomatsakis (Apr 28 2020 at 17:26, on Zulip):

so

nikomatsakis (Apr 28 2020 at 17:26, on Zulip):

some questions in my mind

tmandry (Apr 28 2020 at 17:26, on Zulip):

re: Mutex, a related but hard question is, could we ever have some kind of AsyncSafe trait

nikomatsakis (Apr 28 2020 at 17:26, on Zulip):

one of them is where things like the book, pin-utils, futures fit in

nikomatsakis (Apr 28 2020 at 17:26, on Zulip):

to be honest, I view the futures crate as kind of "done"

nikomatsakis (Apr 28 2020 at 17:27, on Zulip):

I think we should import async-read, async-write, stream, and various utilities into libstd

nikomatsakis (Apr 28 2020 at 17:27, on Zulip):

and at that point, there's not much left there

nikomatsakis (Apr 28 2020 at 17:27, on Zulip):

I mean I'm sure there's tons I'm overlooking, but it seems like that's the "big stuff"

nikomatsakis (Apr 28 2020 at 17:27, on Zulip):

well, ok, there's the utility functions on future etc, but a lot of those are reproduced in executors etc

nikomatsakis (Apr 28 2020 at 17:28, on Zulip):

in any case, I think probably we should bring it under the "polish" umbrella -- update it to keep pace with the library changes we have inmind, but it doesn't have a "story" of its own --

tmandry (Apr 28 2020 at 17:28, on Zulip):

bring utility functions under the polish umbrella you mean?

Lucio Franco (Apr 28 2020 at 17:28, on Zulip):

Can we provide a similar thing to must_use but for things that should not be held as a local across yields? This from my pov would be very helpful for protecting users.

nikomatsakis (Apr 28 2020 at 17:28, on Zulip):

one thing that did come up a lot was talking about "stability expectations" for the futures crate, I think we could probably publish some, given this idea that we don't really expect to be evolving it much

nikomatsakis (Apr 28 2020 at 17:29, on Zulip):

Lucio Franco said:

Can we provide a similar thing to must_use but for things that should not be held as a local across yields? This from my pov would be very helpful for protecting users.

interesting idea! That's something I've not heard floating before. I think it fits under polish, but only barely

nikomatsakis (Apr 28 2020 at 17:29, on Zulip):

tmandry said:

bring utility functions under the polish umbrella you mean?

more or less, yeah. like it's also about maintaining some of the libs

tmandry (Apr 28 2020 at 17:29, on Zulip):

Lucio Franco said:

Can we provide a similar thing to must_use but for things that should not be held as a local across yields? This from my pov would be very helpful for protecting users.

yeah, that's what I was getting at with the AsyncSafe trait. an attribute would be nice since we could introduce it as a lint

nikomatsakis (Apr 28 2020 at 17:29, on Zulip):

do you think that fits into this "async lifecycle" concept?

nikomatsakis (Apr 28 2020 at 17:29, on Zulip):

seems like yes

nikomatsakis (Apr 28 2020 at 17:29, on Zulip):

The reason I cc'd @David Barsky earlier is that they were saying they wanted to write an RFC

tmandry (Apr 28 2020 at 17:30, on Zulip):

related, but somewhat independent in my mind

nikomatsakis (Apr 28 2020 at 17:30, on Zulip):

I think they are technically distinct for sure

nikomatsakis (Apr 28 2020 at 17:30, on Zulip):

but there is definitely some conceptual relationship

nikomatsakis (Apr 28 2020 at 17:30, on Zulip):

I'm a bit nervous about AsyncSafe (presuming that was intended as an auto trait) beacuse dyn Foo

Lucio Franco (Apr 28 2020 at 17:30, on Zulip):

orthogonal imo, one is to protect users and one is to make things nicer. I consider the lifecycle thing as sugar basically. Similar to what async/awati brings vs futures 0.1.

nikomatsakis (Apr 28 2020 at 17:30, on Zulip):

I think something more "lint-y" and "best effort" makes sense

David Barsky (Apr 28 2020 at 17:31, on Zulip):

(I’ve shared it with @tmandry before; they’ve suggested implementing it as a proc macro that transforms .awaits into explicit yields)

nikomatsakis (Apr 28 2020 at 17:31, on Zulip):

the #[must_use] analogy seems good

Lucio Franco (Apr 28 2020 at 17:31, on Zulip):

nikomatsakis said:

I think something more "lint-y" and "best effort" makes sense

#[yield_safe = "THIS WILL DEADLOCK"] :)

tmandry (Apr 28 2020 at 17:31, on Zulip):

David Barsky said:

(I’ve shared it with tmandry before; they’ve suggested implementing it as a proc macro that transforms .awaits into explicit yields)

oh I forgot about that :) that was a way of proving the concept more

David Barsky (Apr 28 2020 at 17:32, on Zulip):

I had the idea—but haven’t explored it—that the yield_safe could be a compile-time error with the lifecycle trait

Lucio Franco (Apr 28 2020 at 17:32, on Zulip):

but yes best effort with a lint that can be ignored. The Sync stuff is helping us now with Send executors but for local ones it won't stop you so I think something along the lines that could warn users and make them aware. That will enable them to then ask the right questions.

tmandry (Apr 28 2020 at 17:32, on Zulip):

though if there are enough people talking about it, maybe that's not necessary

nikomatsakis (Apr 28 2020 at 17:32, on Zulip):

@tmandry so I like your idea of having a "general call" to folks interested --

nikomatsakis (Apr 28 2020 at 17:32, on Zulip):

I am wondering if it makes sense to take the bullet points I listed above as a "rough agenda"

David Barsky (Apr 28 2020 at 17:32, on Zulip):

(such that an on_yield on an async mutex would be a compile error)

nikomatsakis (Apr 28 2020 at 17:32, on Zulip):

and basically try to have a meeting (and maybe some further conversion) about that agenda

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

I anticipate that AsyncREad/AsyncWrite may be somewhat controversial

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

but I think I would want to start this more from the POV of

Lucio Franco (Apr 28 2020 at 17:33, on Zulip):

David Barsky said:

(such that an on_yield on an async mutex would be a compile error)

don't think you want a compile error but we can discuss this later.

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

(a) are there great ideas missing, like the one @Lucio Franco shared

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

i.e., a positive bent where we are gathering concepts

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

and to some extent

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

(b) collecting the major concerns and arguments -- but not in the way of "let's shoot down the idea"

Lucio Franco (Apr 28 2020 at 17:33, on Zulip):

nikomatsakis said:

I anticipate that AsyncREad/AsyncWrite may be somewhat controversial

This will for sure be controversial, tokio is likely to go with different traits. Especially with uring changing a lot. So I think framing here will be very important.

nikomatsakis (Apr 28 2020 at 17:33, on Zulip):

i.e., I think I already know the objections to async-read and async-write, and I would probably start them off

nikomatsakis (Apr 28 2020 at 17:34, on Zulip):

Yes, the major objection I see is the uring interactions

tmandry (Apr 28 2020 at 17:34, on Zulip):

nikomatsakis said:

the #[must_use] analogy seems good

I agree this makes sense from a practical perspective. In the long term I'm _slightly_ worried about async being second-class, compared to concurrency with Send and Sync, but I don't see any real practical concerns there at the moment.

nikomatsakis (Apr 28 2020 at 17:34, on Zulip):

I'm of the mind that async-read and async-write still make sense

nikomatsakis (Apr 28 2020 at 17:34, on Zulip):

but I think I'd want to avoid getting too far down in that discussion

nikomatsakis (Apr 28 2020 at 17:34, on Zulip):

(and I would want to note the concerns up front, for sure)

nikomatsakis (Apr 28 2020 at 17:35, on Zulip):

@tmandry hmm, I guess the question is how much we need to guarantee "async-safety"

tmandry (Apr 28 2020 at 17:35, on Zulip):

right, we're trying to prevent deadlocks, which are already possible in safe Rust

nikomatsakis (Apr 28 2020 at 17:36, on Zulip):

yeah, I suspect we couldn't totally achieve that regardless, anbd we can get like 90% of the practical benefit

tmandry (Apr 28 2020 at 17:36, on Zulip):

anyway, I'm very happy to go with an attribute for now, just noting my thoughts

tmandry (Apr 28 2020 at 17:37, on Zulip):

(and I suspect you're right, and that this is the right way for the long term as well)

Lucio Franco (Apr 28 2020 at 17:38, on Zulip):

I think for me its gotchas that are the issue, right now the fact that MutexGuard is !Sync is saving ALOT of people from weird prod issues. But if you were to use any local executors they'd be screwed. So more so trying to protect that and surface this possibility before it can happen. I don't think it should error but should warn with a link to correct docs explaining what could go wrong. This is just for anything that could be an issue if held as a local. Happy to chat more and ping me if there is something that comes up.

I'd also be somewhat interested in helping draft rfc/implementation, I've been wanting to get involved in the compiler for a while anyways.

tmandry (Apr 28 2020 at 17:38, on Zulip):

data point: we definitely hit this very issue on fuchsia. we've added a clippy lint and are working on integrating clippy to help prevent it in the future.

Lucio Franco (Apr 28 2020 at 17:39, on Zulip):

tmandry said:

data point: we definitely hit this very issue on fuchsia. we've added a clippy lint and are working on integrating clippy to help prevent it in the future.

link to this lint? Would be good to start sharing that if it exists already

tmandry (Apr 28 2020 at 17:40, on Zulip):

nikomatsakis said:

I am wondering if it makes sense to take the bullet points I listed above as a "rough agenda"

it's a lot to talk about. I'd suggest starting with one issue at a time perhaps? or what were you hoping to get out of a more general meeting?

David Barsky (Apr 28 2020 at 17:40, on Zulip):

don't think you want a compile error but we can discuss this later.

@Lucio Franco Sure, yeah. the thinking is that you can make something unyieldable across yield points. If that's useful? ╮ (. ❛ ᴗ ❛.) ╭

tmandry (Apr 28 2020 at 17:41, on Zulip):

@Lucio Franco https://github.com/rust-lang/rust-clippy/pull/5439

David Barsky (Apr 28 2020 at 17:42, on Zulip):

(I'm bad at using Zulip.)

Lucio Franco (Apr 28 2020 at 17:42, on Zulip):

tmandry said:

Lucio Franco https://github.com/rust-lang/rust-clippy/pull/5439

Ah looks like it doesn't let you add the lint like must_use but gets us about halfway there!

Let me know how I can help move this forward though. Should be pingable here.

tmandry (Apr 28 2020 at 17:44, on Zulip):

@Lucio Franco is https://github.com/tokio-rs/tokio/pull/1744 the best source on rationale behind tokio's AsyncRead/AsyncWrite design?

tmandry (Apr 28 2020 at 17:46, on Zulip):

Ah looks like it doesn't let you add the lint like must_use but gets us about halfway there!

indeed, we just targeted existing common mutex types. do you have other types in mind you would want to annotate?

tmandry (Apr 28 2020 at 17:47, on Zulip):

tmandry said:

Lucio Franco is https://github.com/tokio-rs/tokio/pull/1744 the best source on rationale behind tokio's AsyncRead/AsyncWrite design?

I realize I linked a PR that was never merged, but I thought I'd seen it advertised as the right source before, maybe I'm misremembering.

tmandry (Apr 28 2020 at 17:48, on Zulip):

there's certainly a lot of discussion there, it would be good to condense it somehow

tmandry (Apr 28 2020 at 17:48, on Zulip):

maybe this is a good use for the ol' Collaborative Summary Document :slight_smile:

tmandry (Apr 28 2020 at 17:53, on Zulip):

I mean to be clear, it seems likely (but not a foregone conclusion) to me that we'll end up with traits that are symmetric to Read/Write in std, but it would be best for us all to make that decision with clear eyes about what we aren't getting, and what future additions might be possible.

Lucio Franco (Apr 28 2020 at 17:58, on Zulip):

tmandry said:

Ah looks like it doesn't let you add the lint like must_use but gets us about halfway there!

indeed, we just targeted existing common mutex types. do you have other types in mind you would want to annotate?

I could probably get you a better use case but I am thinking things like parking lot and other custom mutexes. I did have a solid use case before but I can't remember it. I can totally get you more info later but I do see it being useful in the general. Guards are implicit and can be confusing so having some way to lint them within rustc would be useful.

Lucio Franco (Apr 28 2020 at 17:59, on Zulip):

tmandry said:

Lucio Franco is https://github.com/tokio-rs/tokio/pull/1744 the best source on rationale behind tokio's AsyncRead/AsyncWrite design?

A lot of this stuff has gone stale, basically we are starting from the ground up now but progress is somewhat slow. I think carl might have a better idea. Happy to start a chat for this. Let me know. We totally want to be involved and want to do what is best for everyone while still trying to push things. Uring has kinda put a wrench in a lot of this :D

Lucio Franco (Apr 28 2020 at 18:00, on Zulip):

tmandry said:

I mean to be clear, it seems likely (but not a foregone conclusion) to me that we'll end up with traits that are symmetric to Read/Write in std, but it would be best for us all to make that decision with clear eyes about what we aren't getting, and what future additions might be possible.

We basically want to explore uring and see how things fit with tls libs etc. So I think in general holding off just seems best. I have yet to really see any issues with the io traits living in a crate vs in std. But this should probably extend to a longer convo.

tmandry (Apr 28 2020 at 18:06, on Zulip):

Hmm. I guess my main desire to standardize a trait is to get rid of the ecosystem split, which seems less likely to actually happen now.

Steven Fackler (Apr 28 2020 at 18:07, on Zulip):

There is some lower-hanging fruit than AsyncRead/AsyncWrite, I think.

tmandry (Apr 28 2020 at 18:07, on Zulip):

(there's definitely also a polish aspect of it, where we have futures-rs as the de facto blessed library, but you have to add it and figure that out on your own)

Steven Fackler (Apr 28 2020 at 18:07, on Zulip):

Stream should be straightforward to pull into core for example, right?

tmandry (Apr 28 2020 at 18:08, on Zulip):

yes, I think so.

Lucio Franco (Apr 28 2020 at 18:09, on Zulip):

tmandry said:

Hmm. I guess my main desire to standardize a trait is to get rid of the ecosystem split, which seems less likely to actually happen now.

fixing io types isn't going to fix ecosystem split. This is not a technical issue imo. I do think a concerted effort from a group pushing futures-rs forward as the "general all-purpose" async team would be helpful. But things like uring are only going to make the split worse.

tmandry (Apr 28 2020 at 18:10, on Zulip):

I should clarify what I meant by that, which is "make more async code interoperate seamlessly"

tmandry (Apr 28 2020 at 18:12, on Zulip):

and pulling io traits into std is less a technical solution and more a signaling one, imo

Lucio Franco (Apr 28 2020 at 18:14, on Zulip):

Another thing I've been thinking of that might be good for polish, is a way to downcast a reactor type from Context?

Lucio Franco (Apr 28 2020 at 18:15, on Zulip):

tmandry said:

and pulling io traits into std is less a technical solution and more a signaling one, imo

Yeah, fair, I am not totally sure if its the right one or not, would be good to think about goals here. What would we be trying to signal?

Lucio Franco (Apr 28 2020 at 18:15, on Zulip):

also happy to chat about this during non-meeting times :) I realize this is a meeting so trying not to get too deep into things.

tmandry (Apr 28 2020 at 18:15, on Zulip):

hehe, we are well past meeting time, I should probably split this out into its own topic :)

tmandry (Apr 28 2020 at 18:17, on Zulip):

we would be signaling that all async libraries should be compatible with these traits (even if they use some kind of shim internally)

Steven Fackler (Apr 28 2020 at 18:17, on Zulip):

I'm honestly pretty suspicious that an interface that is optimal for io_uring will actually be one that people are going to be happy to work with at higher levels

Steven Fackler (Apr 28 2020 at 18:17, on Zulip):

and it'll just be a thing that's reserved for TcpStream/File internals with a memcpy on either end to interface back to standard traits

Florian Gilcher (Apr 28 2020 at 18:18, on Zulip):

(I'm in another meeting, but I hold the opinion that io_uring should not be intermingled with classic IO and should be kept out of the AsyncRead/Write traits as a concern)

Lucio Franco (Apr 28 2020 at 18:19, on Zulip):

This is probably true, but I think its worth it to explore the space first before jumping into something that most likely won't help the current situation much.

Florian Gilcher (Apr 28 2020 at 18:28, on Zulip):

I think the window for that exploration is closing quickly, or is maybe even closed already. All these things aren't new. But it leads to a number of very concrete issues in the ecosystem.

nikomatsakis (Apr 28 2020 at 19:23, on Zulip):

Hey all, sorry, I got pulled AFK when the time slot for this meeting expired, but to give my thoughts:

nikomatsakis (Apr 28 2020 at 19:24, on Zulip):

First off, I think that the top priority should be working to enable interoperable libraries. To that end, I think that both stream and async-read/async-write each enable "classes" of libraries to operate interoperably. I agree with @Steven Fackler that Stream is the lower hanging fruit of the two and it is probably where I would start.

nikomatsakis (Apr 28 2020 at 19:28, on Zulip):

When it comes to async-read/async-write, I know of a few classes of problems, but the three biggest concerns I've heard are

  1. dealing with uninitialized data, which I think is well understood and roughly speaking a "solved problem" at this juncture
  2. how to think about things like integrating with io-uring, which you can see just from this topic still has a ton of unknowns
  3. the fact that read/write can't go into core

I don't want to get too far into point 2 right now, but I think it would be good to talk out more. I guess my current position is that, whatever happens, epoll is a pretty widespread thing that is not goin to go away instantly, and it makes sense for us to offer standard traits that work with it -- we already have sync ones, we should have async ones. Once I/O U-Ring has had time to bake, and we have more experience with it, I would expect that it makes sense to start pulling traits for it into std as well.

Point 3 I hadn't heard before but @Florian Gilcher recently pointed it out to me. I feel like we should try to address this in a way that is uniform across read/write and async-read/async-write.

In general, I think that the principle of having sync/async be "analogous" is very valuable and one that we should stick with. It will not only make things easier to learn, but it gives us a good guideline -- i.e., if we start to diverge, we will make things more confusing, but we also make it much harder to reach consensus, as suddenly there are a great many choices.

nikomatsakis (Apr 28 2020 at 19:28, on Zulip):

Anyway I'd be happy to talk a bit more about I/O u-ring, but I think we should move it to different topic.

nikomatsakis (Apr 28 2020 at 19:30, on Zulip):

tmandry said:

it's a lot to talk about. I'd suggest starting with one issue at a time perhaps? or what were you hoping to get out of a more general meeting?

agreed! This is perhaps what I'd most like to focus on. I think a meeting is maybe not the best structure, but I think what I am interested in is:

The latter can be hard to achieve, perhaps. :)

nikomatsakis (Apr 28 2020 at 19:32, on Zulip):

Beyond looking for things to pull to std, another thing that @Florian Gilcher and I talked about, which I think seems interesting is looking for opportunities to create shared libraries that multiple executors might use, such as e.g. tokio/async-std/bastion/fuchsia sharing a TLS implementation or something

Lucio Franco (Apr 28 2020 at 19:32, on Zulip):

Agree, Stream into std should be easy, I think we are overall very happy with it. I do think the solution for uninit reads will help us with uring but very happy to chat more. These all seem like great ideas :) I am going to try and attend these more.

Lucio Franco (Apr 28 2020 at 19:32, on Zulip):

nikomatsakis said:

Beyond looking for things to pull to std, another thing that Florian Gilcher and I talked about, which I think seems interesting is looking for opportunities to create shared libraries that multiple executors might use, such as e.g. tokio/async-std/bastion/fuchsia sharing a TLS implementation or something

I am happy to get involved on this as well! :) Let me know what makes sense and where to start.

nikomatsakis (Apr 28 2020 at 19:33, on Zulip):

Did you mean specifically TLS, or just the general idea?

nikomatsakis (Apr 28 2020 at 19:34, on Zulip):

I don't really know anything about TLS in particular, but I think for the "general idea" it'd be interesting to try to come up with a set of candidates -- what are concepts that basically every executor is re-inventing, and can they somehow be shared? Some things that come to mind are tracing and async-task (though the latter is more of an impl detail), but I'm sure there's much more.

Lucio Franco (Apr 28 2020 at 19:36, on Zulip):

Both, I think there is benefit in doing it. So mostly just offering that I am going to try and be pingable and should be around more to help communicate for tokio etc.

tmandry (Apr 28 2020 at 19:39, on Zulip):

Can you clarify what you mean by TLS? as in, a unified way to gain access to the reactor?

Florian Gilcher (Apr 28 2020 at 19:40, on Zulip):

TLS, the protocol

tmandry (Apr 28 2020 at 19:41, on Zulip):

..oh, that TLS :)

Lucio Franco (Apr 28 2020 at 19:42, on Zulip):

tmandry said:

Can you clarify what you mean by TLS? as in, a unified way to gain access to the reactor?

on that topic, we should probably explore Context a bit more?

tmandry (Apr 28 2020 at 19:43, on Zulip):

yeah, that seems like a likely candidate to come up as we talk about ways that executors can unify what they're all doing

Steven Fackler (Apr 28 2020 at 19:44, on Zulip):

Speaking of uninitialized buffers, I finally started writing up that RFC. I have adjusted things a little bit since that dropbox paper/discussion with niko. The ReadBuf type now tracks both the initialization cursor and a read cursor, which avoids some weird edge cases around trusted and untrusted values from the impl: https://github.com/sfackler/rfcs/blob/read-buf/text/0000-read-buf.md

nikomatsakis (Apr 28 2020 at 19:56, on Zulip):

Florian Gilcher said:

TLS, the protocol

lol ok ok I thought you meant thread-local storage :)

nikomatsakis (Apr 28 2020 at 19:56, on Zulip):

both probably make sense :)

Steven Fackler (Apr 28 2020 at 19:59, on Zulip):

definitely happy to take feedback/prs on the rfc if people feel like it, btw :)

Matthias247 (May 02 2020 at 03:10, on Zulip):

My biggest concern with the AsyncRead/Write and Stream/Sink traits is that they all enforce the users to write implementations via a state-machine poll_x() again. There is no way to naturally compose those types. E.g. if you want to implement a writer which forwards all data to 2 one is up to implementing things by hand - whereas in other async/await environments (like C#, Kotlin, etc) one can also use async/await syntax in the implementation of an async Stream/..., and then use normal delegation.

And I actually would prefer application-level code in the future no longer having to deal with all the details of pinning, Wakers, polling anymore.

Matthias247 (May 02 2020 at 03:31, on Zulip):

@Steven Fackler Since there was some talk about TLS. I recently found out that some TLS libraries expect that all follow-up writes after an EAGAIN expect exactly the same data to be passed in the next write call. If you don't to it, you get a corrupted stream. That makes them actually also not really an ideal fit for AsyncWrite - which does not make any assumptions on what you pass in the next write call. Unless you add some additional buffering in front of the library (do native-tls or tokio-tls do that?)

Steven Fackler (May 02 2020 at 11:33, on Zulip):

To the best of my knowledge that has never been a problem other than enabling SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default 4 years ago: https://github.com/sfackler/rust-openssl/pull/423

Matthias247 (May 02 2020 at 20:20, on Zulip):

I also had not heard of the issue until very recently. But then I learned about a service which run into a production issue (corrupted TLS stream) due to this behavior. I guess commonly services/protocols do not change the data they want to write. But some protocols will do a re-priorization of data on next writes - they now have more urgent data to send - with the assumption that the old data hasn't been written yet. This will then break if the APIs do not really do what they claim to have done (e.g. if AsyncWrite::poll_write returns 5 you might assume it only copied 5 bytes of your data, but in reality it might have copied more and ignore those bytes from your next poll_write call).

The SSL flag does only allow to reference the same data from another memory address, but the data being passed in consecutive writes must still be the same.

nikomatsakis (May 04 2020 at 18:31, on Zulip):

@Matthias247 when it comes to dealing with hand-written pin impls, I agree with you; it seems like the solution for Stream is to support a generator-based syntax. I'm not sure about async-read and async-write.

Taylor Cramer (May 04 2020 at 18:42, on Zulip):

@nikomatsakis I've talked about this before, but you can also write asyncread and asyncwrite via generator syntax

Taylor Cramer (May 04 2020 at 18:42, on Zulip):

if generators get the full set of abilities I'd like them to have, that is

Taylor Cramer (May 04 2020 at 18:43, on Zulip):

However, they do have a similar issue to the io_uring stuff that the easiest-to-use design is one which offers a buffer which will continue to be available in future polls

Last update: Jul 03 2020 at 16:45UTC