Stream: t-compiler/wg-pipelining

Topic: Pipelining for Buck


Jeremy Fitzhardinge (May 20 2019 at 19:55, on Zulip):

The pipelining work is very exciting - I've been thinking/talking about it, but hadn't realized that there had been so much substantial work. I'm interested in extending Buck to support it. What should I be looking at? What caveats are there (aside from "all very experimental/unstable, expect it to stabilize in some other form")? Is there anything that's deeply, intrinsically Cargo-specific?

Alex Crichton (May 20 2019 at 22:10, on Zulip):

@Jeremy Fitzhardinge the design currently matches pretty closely with the initial notes, nothing is really Cargo specific in the sense that it's just flags in rustc really

Alex Crichton (May 20 2019 at 22:10, on Zulip):

Buck may or may not architecturally be ready for the model of pipelined compilation we're using though

Alex Crichton (May 20 2019 at 22:10, on Zulip):

which is that halfway through a step's execution it can produce an artifact which is consumed by further steps

Alex Crichton (May 20 2019 at 22:11, on Zulip):

so you basically have to watch a process, wait for a signal via stdout that a file is ready, go spawn more work, and then continue to wait for the original process

Alex Crichton (May 20 2019 at 22:11, on Zulip):

and then also make sure you synchronize correctly when linking and all that (wait entirely for dependencies rather than just their metadata)

Jeremy Fitzhardinge (May 20 2019 at 22:21, on Zulip):

I think I can model it as two separate build actions which internally depend on a single rustc invocation, so each can "finish" when the corresponding artifact is ready.

Jeremy Fitzhardinge (May 20 2019 at 22:23, on Zulip):

One design change I'd like to see: the artifact message should also have which thing is being emitted rather than having it being implied by the filename extension. eg, {"artifact":"/Users/jsgf/git/cargo/target/release/deps/libsemver-46a2b7dea0b8356b.rmeta", "emit": "metadata"}(to match the --emit option).

Jeremy Fitzhardinge (May 20 2019 at 22:25, on Zulip):

The "two action" model will naturally get the deps right, so long as I make a library build depend on the rmeta, and the link depend on the rlib (and the partial ordering of all the rmeta -> rlib dependencies will make everything else work out).

Jeremy Fitzhardinge (May 20 2019 at 22:43, on Zulip):

@Alex Crichton ^

Alex Crichton (May 20 2019 at 23:05, on Zulip):

@Jeremy Fitzhardinge ah yeah this is the original model I implemented in Carog

Alex Crichton (May 20 2019 at 23:06, on Zulip):

we ended up not landing it because it was too invasive and had weird repercussions elsewhere in Cargo

Alex Crichton (May 20 2019 at 23:06, on Zulip):

but Cargo handles a lot more I think than buck probably does (aka build scripts and such)

Alex Crichton (May 20 2019 at 23:06, on Zulip):

we ended up with a model of each action produces some number of outputs

Alex Crichton (May 20 2019 at 23:06, on Zulip):

and you depend on an action + an output

Alex Crichton (May 20 2019 at 23:06, on Zulip):

(e.g. the edges in the dependency graph are annotated with an output they depend on)

Alex Crichton (May 20 2019 at 23:07, on Zulip):

and that way when an output is ready you delete all those edges and then just continuously schedule things that have no dependencies that aren't finished

Jeremy Fitzhardinge (May 20 2019 at 23:27, on Zulip):

@Alex Crichton It would help if build scripts had proper dependencies, then it would be a clean mapping. I'm assuming that Cargo has to maintain a pile of pseudo dependencies to try and sequence build.rs builds and execution correctly.

Alex Crichton (May 20 2019 at 23:27, on Zulip):

Nah it's more just sequencing compiles appropriately that was the issue

Alex Crichton (May 20 2019 at 23:28, on Zulip):

the deps issue is actually quite easy within cargo, this was just making sure we got as much parallelism as we could without tampering with cargo's entire backend

Jeremy Fitzhardinge (May 20 2019 at 23:31, on Zulip):

Yeah, but the build depends on running the buildscript for its side-effects, which means that it can't overlap the crate's build. Nor do you really know when it needs rebuilding.

Last update: Nov 15 2019 at 10:10UTC