Stream: t-compiler/wg-parallel-rustc

Topic: Initial benchmarks


Josh Triplett (Dec 18 2019 at 06:50, on Zulip):

I posted some initial benchmarks at https://internals.rust-lang.org/t/plan-to-test-parallel-rustc/11487/15 and https://internals.rust-lang.org/t/plan-to-test-parallel-rustc/11487/16 , and they look good!

Zoxc (Dec 18 2019 at 07:35, on Zulip):

The difference in output sizes are surprising. Do these also happen with -Z threads=1?

Zoxc (Dec 18 2019 at 07:37, on Zulip):

Also you didn't mention how many cores your CPU has. You may want to specify a higher thread count to rustc (it defaults to a maximum of 4).

Alex Crichton (Dec 18 2019 at 18:50, on Zulip):

So simon's question on internals about "what's still serial" led me to investigate a little

Alex Crichton (Dec 18 2019 at 18:50, on Zulip):

Using -Z self-profile it's actually really apparent what's still serial (yay!)

Alex Crichton (Dec 18 2019 at 18:51, on Zulip):

I compiled Cargo itself with -Z self-profile with our parallel nightly, and I'm actually pretty surprised by the results

Alex Crichton (Dec 18 2019 at 18:51, on Zulip):

this is the overall picture: Screen-Shot-2019-12-18-at-12.51.36-PM.png

Alex Crichton (Dec 18 2019 at 18:52, on Zulip):

type checking is suprisingly not really parallel at all Screen-Shot-2019-12-18-at-12.51.59-PM.png

Alex Crichton (Dec 18 2019 at 18:53, on Zulip):

and partitioning is only really barely parallel -- Screen-Shot-2019-12-18-at-12.52.35-PM.png

Alex Crichton (Dec 18 2019 at 18:53, on Zulip):

is a lot of this work sort of just not inherently that parallel? or are things not getting stolen?

Alex Crichton (Dec 18 2019 at 18:54, on Zulip):

this was just a single crate build via cargo +nightly-2019-12-18 rustc --lib -- -Z self-profile after a full build, so it was only building the one library crate

nikomatsakis (Dec 19 2019 at 19:24, on Zulip):

I'm not sure I know how to interpret those graphs

nikomatsakis (Dec 19 2019 at 19:24, on Zulip):

what do "parallel things" look like?

simulacrum (Dec 19 2019 at 19:28, on Zulip):

on the left there are thread ids

simulacrum (Dec 19 2019 at 19:28, on Zulip):

and I think we're only parallel if we have multiple thread ids with simultaneous bars

Wesley Wiser (Dec 19 2019 at 19:43, on Zulip):

Yeah, that's correct. As a concrete example, if you look in the first image "Screen-Shot-2019-12-18-at-12.51.36-Pm.png", there are two purple bars on Thread 3 (see left hand gutter) around 1350 - 1380ms (see top bar for timeline) and a purple bar and a green bar at the same time on Thread 4. The two overlapping purple bars and the overlapping purple and green bars mean threads 3 & 4 were doing work at the same time.

Zoxc (Dec 19 2019 at 23:37, on Zulip):

@Alex Crichton Only wf checking and item-bodies checking is parallel within type_check_crate.

@Wesley Wiser Why doesn't -Z self-profile include time calls?

Wesley Wiser (Dec 20 2019 at 00:45, on Zulip):

@Zoxc No one's asked for it before :slight_smile:

Zoxc (Dec 20 2019 at 01:38, on Zulip):

I'm sure I mentioned it on some PR, since there's locations where there's both a self-profile timer and time call, which seems a bit redundant.

Zoxc (Dec 25 2019 at 19:04, on Zulip):

@Josh Triplett Maybe you could do some benchmarks while bumping the shard count (You can adjust it in sharded.rs). It's only been tuned for my 8-core R7 1700 =P

Last update: Jan 28 2020 at 00:55UTC