Stream: t-compiler/wg-pipelining

Topic: pipelining for distributed compilation


Taylor Cramer (Apr 11 2019 at 18:24, on Zulip):

cc @eddyb

eddyb (Apr 11 2019 at 18:25, on Zulip):

hi! I'm curious what your plans are, so far

eddyb (Apr 11 2019 at 18:25, on Zulip):

you can definitely do codegen "on the second run" right now, by relying on incremental caches

eddyb (Apr 11 2019 at 18:26, on Zulip):

and we want to optimize those, going forward

Taylor Cramer (Apr 11 2019 at 18:27, on Zulip):

@eddyb so far my plan was not to use incremental at all because I didn't want the incremental inputs to dirty the cache entries

eddyb (Apr 11 2019 at 18:27, on Zulip):

once parsing is incremental, unchanged sources would probably mean it goes straight to codegen the second time, so you only have the overhead of loading the caches into memory (which is one of the things we want to optimize)

eddyb (Apr 11 2019 at 18:27, on Zulip):

I'm not sure what you're referring to

eddyb (Apr 11 2019 at 18:27, on Zulip):

what are "incremental inputs" and what "cache entries" would be dirtied?

eddyb (Apr 11 2019 at 18:29, on Zulip):

cc @mw @Zoxc @nikomatsakis

Taylor Cramer (Apr 11 2019 at 18:29, on Zulip):

so the short version of this is that you tell the tool which command to run and what files it needs in order to run that command on either the local machine or another machine. the naiive behavior is to rerun whenever any of those inputs changes.

eddyb (Apr 11 2019 at 18:31, on Zulip):

if you sync incremental state you could be doing cargo check locally and codegen remotely

eddyb (Apr 11 2019 at 18:31, on Zulip):

potentially very efficiently

Taylor Cramer (Apr 11 2019 at 18:31, on Zulip):

@eddyb I want to avoid potentially doing anything at all locally

eddyb (Apr 11 2019 at 18:31, on Zulip):

my kneejerk reaction is "that's an antiquated design, you need more integration", but it might still work

eddyb (Apr 11 2019 at 18:32, on Zulip):

okay, sure, what's the problem then?

eddyb (Apr 11 2019 at 18:32, on Zulip):

naively applying incremental just means some of those commands take less to run

eddyb (Apr 11 2019 at 18:32, on Zulip):

clever use of incremental would result in distributed deduplication of work

eddyb (Apr 11 2019 at 18:33, on Zulip):

kind of like how "parallel rustc" does queries, except cross-machine instead of cross-thread

eddyb (Apr 11 2019 at 18:33, on Zulip):

I still don't know what you mean by "incremental inputs" or "cache entires"

Taylor Cramer (Apr 11 2019 at 18:34, on Zulip):

@eddyb when incremental reruns, it is taking advantage of the results of a previous build

eddyb (Apr 11 2019 at 18:35, on Zulip):

sort of, but less "results" and more "intermediary computations"

Taylor Cramer (Apr 11 2019 at 18:35, on Zulip):

sure

eddyb (Apr 11 2019 at 18:35, on Zulip):

it's really just persisted memoization

eddyb (Apr 11 2019 at 18:36, on Zulip):

almost the same thing as rmeta, but with a newer design

Taylor Cramer (Apr 11 2019 at 18:36, on Zulip):

no, the key difference is that the data is different depending on the last thing that you compiled

Taylor Cramer (Apr 11 2019 at 18:36, on Zulip):

the result of the previous build cannot be an input to the current build process

eddyb (Apr 11 2019 at 18:37, on Zulip):

wait, what is this even for?

Taylor Cramer (Apr 11 2019 at 18:37, on Zulip):

https://chromium.googlesource.com/infra/goma/client/

eddyb (Apr 11 2019 at 18:37, on Zulip):

it seems like you have a different usecase of "distributed compilation" in mind

eddyb (Apr 11 2019 at 18:37, on Zulip):

is this 100% batch builds?

Taylor Cramer (Apr 11 2019 at 18:38, on Zulip):

can you clarify what you mean by "batch builds"?

eddyb (Apr 11 2019 at 18:38, on Zulip):

like, with a strong requirement of starting from scratch every time?

Taylor Cramer (Apr 11 2019 at 18:39, on Zulip):

Yes, previous compilation outputs shouldn't affect the results of the current compilation.

eddyb (Apr 11 2019 at 18:39, on Zulip):

okay then that's not at all what I was thinking of, sorry

eddyb (Apr 11 2019 at 18:40, on Zulip):

so why would you even need split compiles? to run codegen on a different machine?

eddyb (Apr 11 2019 at 18:40, on Zulip):

you can still do what I said, you'd just be starting incremental from scratch and using it only once

eddyb (Apr 11 2019 at 18:41, on Zulip):

basically, run rustc --emit=metadata or w/e, in incremental mode, and copy the incremental cache to another machine

eddyb (Apr 11 2019 at 18:41, on Zulip):

you wouldn't be using it across builds, but within one build

eddyb (Apr 11 2019 at 18:41, on Zulip):

(rmeta doesn't have enough information to do codegen, but the incremental caches do)

Last update: Nov 15 2019 at 09:40UTC