Stream: t-compiler/wg-parallel-rustc

Topic: next few steps?


nikomatsakis (Feb 28 2019 at 16:25, on Zulip):

I know there is a plan, or at least a vague one, but I don't remember precisely what it is .. what are the next few steps we have in mind? @Zoxc if nothing else maybe we can enumerate your pending PRs. =)

Do we have some kind of "criteria" that we want to have established before we stabilize? (Performance and otherwise)

I remember @nagisa in particular raising concerns about using our own fork of rayon. We should probably be forming a plan for how to stop doing that, too. This was something @Josh Stone and I had sort of hoped to have done at the Rust All Hands, but that didn't happen, although I think one of @Josh Stone's recent PRs would help somewhat here.

mw (Feb 28 2019 at 16:45, on Zulip):

There's https://github.com/rust-lang/rust/issues/48685 which is probably rather out-of-date.

mw (Feb 28 2019 at 16:46, on Zulip):

I think setting a goal for what would be an acceptable overhead is a good idea.

mw (Feb 28 2019 at 16:47, on Zulip):

e.g. if the parallel compiler (restricted to a single thread) is within 5% of the current performance, then we can switch, or something like that

mw (Feb 28 2019 at 16:47, on Zulip):

one of the bigger unknowns is also testing.

mw (Feb 28 2019 at 16:48, on Zulip):

I think we kind of agree that the regular tests should be run on a single thread and that they don't stress parallelism enough to be worth much anyway (because it's all small crates).

mw (Feb 28 2019 at 16:49, on Zulip):

we will get some degree of testing out of the normal bootstrapping process

mw (Feb 28 2019 at 16:54, on Zulip):

one question here is what we even want to test for. if it's just crashes then it wouldn't be to hard to have bors make sure that a certain set of large crates can be compiled without crashes or unexpected results.

mw (Feb 28 2019 at 16:55, on Zulip):

we could also do something like we did for incremental compilation: compile crates in both modes (1 and N threads) and then compare their output.

mw (Feb 28 2019 at 16:56, on Zulip):

(this was the tool in question: https://github.com/nikomatsakis/cargo-incremental)

mw (Feb 28 2019 at 17:07, on Zulip):

I wonder if we could do some A/B testing, that is, compile every second nightly with parallel-compiler=true :)

Zoxc (Feb 28 2019 at 17:14, on Zulip):

Comparing crates with parallel-compiler=false and parallel-compiler=true seems like a good idea. Is our output deterministic enough for that?

mw (Feb 28 2019 at 17:15, on Zulip):

maybe sorting the JSON output alphabetically would be enough?

mw (Feb 28 2019 at 17:15, on Zulip):

binary output should be deterministic, I think

cuviper (Feb 28 2019 at 17:15, on Zulip):

I think one of @_Josh Stone's recent PRs would help somewhat here.

Said PR: https://github.com/rayon-rs/rayon/pull/636 -- could replace rustc-rayon's main-handler and scoped threadpool

cuviper (Feb 28 2019 at 17:16, on Zulip):

The other big thing in rustc-rayon was the thread-local stuff, especially the one that follows stolen jobs

Zoxc (Feb 28 2019 at 17:17, on Zulip):

I'm not sure we should get rid of cfg!(parallel_compiler). That is a useful tool to measure the overhead of parallelization, and isn't overly invasive.

mw (Feb 28 2019 at 17:23, on Zulip):

I'm not sure we should get rid of cfg!(parallel_compiler).

We don't need to rush it. In general I'm not a big fan of keeping it around forever because it makes things more complicated.

mw (Feb 28 2019 at 17:24, on Zulip):

But yes, it's not too bad and I think defaulting to parallel-compiler is the actual goal.

mw (Feb 28 2019 at 17:25, on Zulip):

I'll change the wording in the PR to make this clear.

mw (Feb 28 2019 at 17:26, on Zulip):

(done)

Zoxc (Feb 28 2019 at 17:36, on Zulip):

My mental plan was to
- Land https://github.com/rust-lang/rust/pull/58679 and https://github.com/rust-lang/rust/pull/58019
- Run crater, evaluate perf
- Enable by default on nightly

Zoxc (Feb 28 2019 at 17:47, on Zulip):

I also have a local branch which does parallel parsing, there's a race condition in it though

nikomatsakis (Mar 01 2019 at 19:36, on Zulip):

(thanks all, this was enlightening to read)

Last update: Nov 17 2019 at 08:10UTC