Stream: t-compiler/wg-pipelining

Topic: DAG model vs header inclusion model


Vadim Petrochenkov (Apr 27 2019 at 22:10, on Zulip):

A fun experimental paper from the recent C++ committee mailing - "Are modules fast?" (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1441r1.pdf)

Vadim Petrochenkov (Apr 27 2019 at 22:12, on Zulip):

The proposed C++ modules are very similar to our packages and they similarly don't scale to massively parallel builds.

Vadim Petrochenkov (Apr 27 2019 at 22:15, on Zulip):

I wonder if Cargo eventually ends up compiling leaf crates multiple times given enough parallelism to compensate deep dependency trees, that would be kind of a reasonable return to the header inclusion model.

Vadim Petrochenkov (Apr 27 2019 at 22:28, on Zulip):

Or maybe that's a nonsense, unless builds span multiple machines.

mw (Apr 29 2019 at 15:06, on Zulip):

interesting

Alex Crichton (Apr 30 2019 at 18:52, on Zulip):

FWIW I've often though that Cargo's model of compilation is basically unsuitable for the "we have infinite parallelism" compilation model (aka distributed compilation)

Alex Crichton (Apr 30 2019 at 18:53, on Zulip):

My gut is that we would have to rethink quite a lot to really take advantage of infinite parallelism, and all of cargo/rustc/parallelism we have today is really heavily optimized for "reasonable parallelism on one machine"

Alex Crichton (Apr 30 2019 at 18:53, on Zulip):

although that is also starting to show wear and tear with lots of cpus and big crate graphs...

nnethercote (Apr 30 2019 at 22:34, on Zulip):

I have a machine with 14 physical cores and the difference between C++ compilation and Rust compilation is huge (e.g. in Firefox). Compilation of Rust code doesn't get close to saturating the cores.

nnethercote (Apr 30 2019 at 22:45, on Zulip):

That's my primary motivation for working on pipelining.

eddyb (May 01 2019 at 13:19, on Zulip):

IMO fixing https://github.com/rust-lang/rust/issues/54712#issuecomment-485656192 and https://github.com/rust-lang/rust/issues/58485#issuecomment-485658943 would provide a better improvement than pipelining

eddyb (May 01 2019 at 13:19, on Zulip):

Rust also saturates all the cores during codegen, and ofc during running tests

eddyb (May 01 2019 at 13:20, on Zulip):

but this is with incremental. for batch compilation, pipelining should be a clear win

eddyb (May 01 2019 at 13:21, on Zulip):

(I assumed you meant "compiling rustc" but that's not all the Rust code out there, heh)

Alex Crichton (May 01 2019 at 15:22, on Zulip):

@eddyb your improvements are rustc-specific though I think, right? They're not generally applicable to crate graphs in the large?

eddyb (May 01 2019 at 15:23, on Zulip):

that last message from me should've started with "oops"

eddyb (May 01 2019 at 15:24, on Zulip):

(I had misread what @nnethercote had said. and now I can't edit my messages to make it even clearer)

Last update: Nov 15 2019 at 10:35UTC