Stream: wg-async-foundations

Topic: sink


Lucio Franco (Dec 12 2019 at 16:51, on Zulip):

@Taylor Cramer hey! not sure if this is the right place or not, but I was curious in your mind what the status of the futures-sink stuff is and what I might be able to do to take it over the finish line aka not in its own crate but into futures-core. I didn't see any issues around this so just wanted to clear some stuff up. Thanks!

nikomatsakis (Dec 12 2019 at 18:19, on Zulip):

I'd like to learn more about what role the sink trait plays

Lucio Franco (Dec 12 2019 at 18:28, on Zulip):

Sure! So a couple of things, high level its used generally in combination with streams to create a chain of async processing. Things like tokio-tower uses both of these interfaces to abstract low level protocols/framing. Currently, lets say we have a stream of items coming in, sink could be used as an interface to forward those items into a buffer for example. Where sink fits in regards to generators and actual language things I am not totally sure. But with futures 0.1 where combinators were heavily used it was really helpful.

For practical use, at least at my work we have used it as a core foundational component https://github.com/timberio/vector/blob/master/src/sinks/mod.rs#L27 for how we create logs. So streams are a lower level piece compared to say the tower Service trait.

Happy to explain more if there is something you're more interested in.

Taylor Cramer (Dec 13 2019 at 00:47, on Zulip):

(deleted)

Taylor Cramer (Dec 13 2019 at 00:48, on Zulip):

What is the motivation for having the Sink trait in futures-core?

Taylor Cramer (Dec 13 2019 at 00:48, on Zulip):

Is there something about having it be independently versioned / in a separate crate that is making it difficult to use?

Taylor Cramer (Dec 13 2019 at 00:49, on Zulip):

I don't know of any open issues that I feel need to be resolved, but I also don't see a reason to take an extra step to tie its version inextricably to that of Stream.

Matthias247 (Dec 13 2019 at 05:49, on Zulip):

What is the motivation for having the Sink trait in futures-core?

Why do you think the motivation is different from including Stream? Just because Stream might get special language support? Even though it's not sure whether it's needed? Imho they are just both counterparts of each other. And if the goal of future-rs is to standardize common traits, then it should contain either both (or none).

Matthias247 (Dec 13 2019 at 05:52, on Zulip):

I'd like to learn more about what role the sink trait plays

As I had written above, to me it is just the write counterpart of the Stream. One reads a value, the other writes a value. E.g. in order to model zero-copy async IO, you would have an object (socket/file/etc) which acts as a Stream<Bytes> for reading from it and a Sink<Bytes> in order to write to it.

nikomatsakis (Dec 13 2019 at 13:13, on Zulip):

I agree they are "tied together", but I'm not convinced that we need a Sink abstraction. We have Iterator (a producer) but we have no "consumer" counterpart, for example.

nikomatsakis (Dec 13 2019 at 13:13, on Zulip):

I'm not saying it's a problem to have one, but I think at minimum there's no reason they have to go into std together or be versioned together.

nikomatsakis (Dec 13 2019 at 13:13, on Zulip):

(I'll try to look at the links @Lucio Franco sent)

nikomatsakis (Dec 13 2019 at 13:14, on Zulip):

I guess one could say that the sync consumer is FnMut

Lucio Franco (Dec 13 2019 at 17:57, on Zulip):

@Taylor Cramer mostly, the question is around confusion for users, if I were new I would expect that sink wasn't ready/I shouldn't use it. So having it in a different crate makes it not as visible as say stream is.

Lucio Franco (Dec 13 2019 at 17:58, on Zulip):

Yeah, maybe it doesn't fit and people need specialized ones but I think we have seen good success with sink in futures 0.1 that it probably doesn't make sense to not include it. Mostly, I just wanted to start a discussion around this.

Matthias247 (Dec 14 2019 at 06:39, on Zulip):

I agree they are "tied together", but I'm not convinced that we need a Sink abstraction. We have Iterator (a producer) but we have no "consumer" counterpart, for example.

It might be helpful to decouple the Stream/Sink view from the Iterator one. Of course a Stream is some kind of iterator, but it is also the primitive interface to read an owned object, which needs a write equivalent.

If we look at byte IO, we have AsyncRead and AsyncWrite for reading and writing values by copy. All seem interesting enough for people that they want them to be standardized. For reading data by move, we have Stream. And Sink would be required for writing data by move.

Same goes e.g. for channels. You can abstract the reading side behind a Stream. But there exists no interface for the writing side without Sink.

Matthias247 (Dec 14 2019 at 06:43, on Zulip):

FnMut is indeed a suitable interface in the synchronous world. In the async world we however also have to wait until the consumer can accept the data, so the interface needs to be somewhat poll-based. It could theoretically be a single method poll_send(item) -> Poll<Result<(), Err<T>> method. But the current implementation decided to split the process into an synchronous reservation/readiness part and a synchronous submission part.

Lucio Franco (Dec 17 2019 at 16:15, on Zulip):

I would like to note, that the ability to poll_ready has been very useful for us in vector and is the basis for how tower works which is basically a non driven submit trait.

Last update: Jul 02 2020 at 19:30UTC