I know there is a plan, or at least a vague one, but I don't remember precisely what it is .. what are the next few steps we have in mind? @Zoxc if nothing else maybe we can enumerate your pending PRs. =)
Do we have some kind of "criteria" that we want to have established before we stabilize? (Performance and otherwise)
I remember @nagisa in particular raising concerns about using our own fork of rayon. We should probably be forming a plan for how to stop doing that, too. This was something @Josh Stone and I had sort of hoped to have done at the Rust All Hands, but that didn't happen, although I think one of @Josh Stone's recent PRs would help somewhat here.
There's https://github.com/rust-lang/rust/issues/48685 which is probably rather out-of-date.
I think setting a goal for what would be an acceptable overhead is a good idea.
e.g. if the parallel compiler (restricted to a single thread) is within 5% of the current performance, then we can switch, or something like that
one of the bigger unknowns is also testing.
I think we kind of agree that the regular tests should be run on a single thread and that they don't stress parallelism enough to be worth much anyway (because it's all small crates).
we will get some degree of testing out of the normal bootstrapping process
one question here is what we even want to test for. if it's just crashes then it wouldn't be to hard to have bors make sure that a certain set of large crates can be compiled without crashes or unexpected results.
we could also do something like we did for incremental compilation: compile crates in both modes (1 and N threads) and then compare their output.
(this was the tool in question: https://github.com/nikomatsakis/cargo-incremental)
I wonder if we could do some A/B testing, that is, compile every second nightly with
Comparing crates with
parallel-compiler=true seems like a good idea. Is our output deterministic enough for that?
maybe sorting the JSON output alphabetically would be enough?
binary output should be deterministic, I think
I think one of @_Josh Stone's recent PRs would help somewhat here.
Said PR: https://github.com/rayon-rs/rayon/pull/636 -- could replace rustc-rayon's main-handler and scoped threadpool
The other big thing in rustc-rayon was the thread-local stuff, especially the one that follows stolen jobs
I'm not sure we should get rid of
cfg!(parallel_compiler). That is a useful tool to measure the overhead of parallelization, and isn't overly invasive.
I'm not sure we should get rid of cfg!(parallel_compiler).
We don't need to rush it. In general I'm not a big fan of keeping it around forever because it makes things more complicated.
But yes, it's not too bad and I think defaulting to parallel-compiler is the actual goal.
I'll change the wording in the PR to make this clear.
I also have a local branch which does parallel parsing, there's a race condition in it though
(thanks all, this was enlightening to read)