Stream: t-compiler/wg-pipelining

Topic: DAG model vs header inclusion model

Vadim Petrochenkov (Apr 27 2019 at 22:10, on Zulip):

A fun experimental paper from the recent C++ committee mailing - "Are modules fast?" (

Vadim Petrochenkov (Apr 27 2019 at 22:12, on Zulip):

The proposed C++ modules are very similar to our packages and they similarly don't scale to massively parallel builds.

Vadim Petrochenkov (Apr 27 2019 at 22:15, on Zulip):

I wonder if Cargo eventually ends up compiling leaf crates multiple times given enough parallelism to compensate deep dependency trees, that would be kind of a reasonable return to the header inclusion model.

Vadim Petrochenkov (Apr 27 2019 at 22:28, on Zulip):

Or maybe that's a nonsense, unless builds span multiple machines.

mw (Apr 29 2019 at 15:06, on Zulip):


Alex Crichton (Apr 30 2019 at 18:52, on Zulip):

FWIW I've often though that Cargo's model of compilation is basically unsuitable for the "we have infinite parallelism" compilation model (aka distributed compilation)

Alex Crichton (Apr 30 2019 at 18:53, on Zulip):

My gut is that we would have to rethink quite a lot to really take advantage of infinite parallelism, and all of cargo/rustc/parallelism we have today is really heavily optimized for "reasonable parallelism on one machine"

Alex Crichton (Apr 30 2019 at 18:53, on Zulip):

although that is also starting to show wear and tear with lots of cpus and big crate graphs...

nnethercote (Apr 30 2019 at 22:34, on Zulip):

I have a machine with 14 physical cores and the difference between C++ compilation and Rust compilation is huge (e.g. in Firefox). Compilation of Rust code doesn't get close to saturating the cores.

nnethercote (Apr 30 2019 at 22:45, on Zulip):

That's my primary motivation for working on pipelining.

eddyb (May 01 2019 at 13:19, on Zulip):

IMO fixing and would provide a better improvement than pipelining

eddyb (May 01 2019 at 13:19, on Zulip):

Rust also saturates all the cores during codegen, and ofc during running tests

eddyb (May 01 2019 at 13:20, on Zulip):

but this is with incremental. for batch compilation, pipelining should be a clear win

eddyb (May 01 2019 at 13:21, on Zulip):

(I assumed you meant "compiling rustc" but that's not all the Rust code out there, heh)

Alex Crichton (May 01 2019 at 15:22, on Zulip):

@eddyb your improvements are rustc-specific though I think, right? They're not generally applicable to crate graphs in the large?

eddyb (May 01 2019 at 15:23, on Zulip):

that last message from me should've started with "oops"

eddyb (May 01 2019 at 15:24, on Zulip):

(I had misread what @nnethercote had said. and now I can't edit my messages to make it even clearer)

Last update: Apr 06 2020 at 03:15UTC