Stream: t-compiler/wg-pipelining

Topic: initial results


Alex Crichton (May 01 2019 at 15:16, on Zulip):

@nnethercote hm so I've started testing locally just to see some smoke test numbers with pipelining all enabled

Alex Crichton (May 01 2019 at 15:17, on Zulip):

If you download and build https://github.com/rust-lang/cargo/pull/6883 that'll give you Cargo (and you'll have to set CARGO_BUILD_PIPELINING=true as an env var)

Alex Crichton (May 01 2019 at 15:17, on Zulip):

and then I built a local rustc with your patch to emit metadata earlier

Alex Crichton (May 01 2019 at 15:17, on Zulip):

and unfortunately at least for Cargo itself it makes almost no difference for compiling from scratch

Alex Crichton (May 01 2019 at 15:17, on Zulip):

turns out Cargo's longest serial portion is a bunch of build scripts that probably shouldn't really be that serial

Alex Crichton (May 01 2019 at 15:17, on Zulip):

I saw a tiny bit of pipelining happening but it was nigh instantaneous

Alex Crichton (May 01 2019 at 15:18, on Zulip):

Cargo is probably a bad example though since it's just one big crate that depends on a huge number of little crates

Alex Crichton (May 01 2019 at 15:18, on Zulip):

I suspect pipelining is only really a win if there's two big crates

Alex Crichton (May 01 2019 at 15:19, on Zulip):

rustc also won't really get any benefit b/c it's all a bunch of dylibs which can't be pipelined

eddyb (May 01 2019 at 15:21, on Zulip):

we should really get rid of those dylibs, ugh

Alex Crichton (May 01 2019 at 15:21, on Zulip):

It's currently being attempted in https://github.com/rust-lang/rust/pull/59800 which is a revival from earlier

Alex Crichton (May 01 2019 at 15:21, on Zulip):

but progress is slow from what it looks like

nnethercote (May 01 2019 at 21:18, on Zulip):

@alexcrichton This definitely needs measurement across a range of workloads. The programs in rustc-perf might be a good place to start. (Note that you can't use the rustc-perf harness because that just measures compilation of the final crate.)

nnethercote (May 05 2019 at 22:48, on Zulip):

@alexcrichton I'm happy to help with taking measurements, if you give me instructions on how to run Cargo with pipelining on

nikomatsakis (May 07 2019 at 18:55, on Zulip):

@Alex Crichton :point_up:

nikomatsakis (May 07 2019 at 18:56, on Zulip):

Sounds like we're primarily blocked on landing this in cargo nightly or something?

nnethercote (May 07 2019 at 23:11, on Zulip):

@nikomatsakis we need someone to make measurements. If it lands in Cargo nightly that will make it easier for people other than @Alex Crichton to perform those measurements.

Alex Crichton (May 08 2019 at 13:58, on Zulip):

Ah yeah sorry I was just discussing my initial results, I still want to land pipelining in nightly Cargo to give others a chance to test.

nikomatsakis (May 08 2019 at 14:12, on Zulip):

OK. @Alex Crichton do you think it's close to landing?

Alex Crichton (May 08 2019 at 14:21, on Zulip):

@nikomatsakis I believe so, yeah, although looks like ehuss found an issue which may take today to fix before it gets in

Alex Crichton (May 08 2019 at 14:21, on Zulip):

it'll take some time to get to rust-lang/rust as well due to the submodule usage

nikomatsakis (May 14 2019 at 18:57, on Zulip):

Looks like https://github.com/rust-lang/cargo/pull/6883 landed, at least -- but we still need a submodule update, @Alex Crichton ?

nikomatsakis (May 14 2019 at 19:01, on Zulip):

roc apparently has some sort of degenerate case -- maybe we can get them to try it out :)

Alex Crichton (May 14 2019 at 19:06, on Zulip):

@nikomatsakis correct yeah, not on nightlies yet until submodule is updated

Alex Crichton (May 14 2019 at 19:06, on Zulip):

I plan to make a post on internals after it's available

Eric Huss (May 14 2019 at 19:18, on Zulip):

fyi, I plan a submodule update soon. There's a long queue of issues (the clippy update needs to land — which looks like it will miss tonight's nightly :sad: , then I need to land an rls PR, then do a combined rls/cargo update). I really hope to make it before next week's beta, but the timing is getting tight.

nnethercote (May 15 2019 at 04:07, on Zulip):

@Alex Crichton Have you managed to fix the issue you mentioned?

Alex Crichton (May 15 2019 at 13:57, on Zulip):

@nnethercote I did yeah, just needs a submodule update now! (although I'm sure there's more bugs lurking)

Last update: Nov 15 2019 at 10:50UTC