@nnethercote hm so I've started testing locally just to see some smoke test numbers with pipelining all enabled
If you download and build https://github.com/rust-lang/cargo/pull/6883 that'll give you Cargo (and you'll have to set
CARGO_BUILD_PIPELINING=true as an env var)
and then I built a local rustc with your patch to emit metadata earlier
and unfortunately at least for Cargo itself it makes almost no difference for compiling from scratch
turns out Cargo's longest serial portion is a bunch of build scripts that probably shouldn't really be that serial
I saw a tiny bit of pipelining happening but it was nigh instantaneous
Cargo is probably a bad example though since it's just one big crate that depends on a huge number of little crates
I suspect pipelining is only really a win if there's two big crates
rustc also won't really get any benefit b/c it's all a bunch of dylibs which can't be pipelined
we should really get rid of those dylibs, ugh
It's currently being attempted in https://github.com/rust-lang/rust/pull/59800 which is a revival from earlier
but progress is slow from what it looks like
@alexcrichton This definitely needs measurement across a range of workloads. The programs in rustc-perf might be a good place to start. (Note that you can't use the rustc-perf harness because that just measures compilation of the final crate.)
@alexcrichton I'm happy to help with taking measurements, if you give me instructions on how to run Cargo with pipelining on
@Alex Crichton :point_up:
Sounds like we're primarily blocked on landing this in cargo nightly or something?
@nikomatsakis we need someone to make measurements. If it lands in Cargo nightly that will make it easier for people other than @Alex Crichton to perform those measurements.
Ah yeah sorry I was just discussing my initial results, I still want to land pipelining in nightly Cargo to give others a chance to test.
OK. @Alex Crichton do you think it's close to landing?
@nikomatsakis I believe so, yeah, although looks like ehuss found an issue which may take today to fix before it gets in
it'll take some time to get to rust-lang/rust as well due to the submodule usage
Looks like https://github.com/rust-lang/cargo/pull/6883 landed, at least -- but we still need a submodule update, @Alex Crichton ?
roc apparently has some sort of degenerate case -- maybe we can get them to try it out :)
@nikomatsakis correct yeah, not on nightlies yet until submodule is updated
I plan to make a post on internals after it's available
fyi, I plan a submodule update soon. There's a long queue of issues (the clippy update needs to land — which looks like it will miss tonight's nightly :sad: , then I need to land an rls PR, then do a combined rls/cargo update). I really hope to make it before next week's beta, but the timing is getting tight.
@Alex Crichton Have you managed to fix the issue you mentioned?
@nnethercote I did yeah, just needs a submodule update now! (although I'm sure there's more bugs lurking)