Stream: t-compiler/wg-parallel-rustc

Topic: sync 2019.06.06


nikomatsakis (Jun 06 2019 at 18:23, on Zulip):

Hey @lwshang =)

nikomatsakis (Jun 06 2019 at 18:25, on Zulip):

I was just reading the 2019.05.24 topic

nikomatsakis (Jun 06 2019 at 18:25, on Zulip):

I see this comment by @simulacrum:

we basically want to ask for community feedback on a parallel-enabled compiler and to do so would wants somewhat regular updates of a parallel compiler build

that seems like a shift from the previous plan, though

nikomatsakis (Jun 06 2019 at 18:25, on Zulip):

anyway, I'm curious, have we "pulled together" the data that's been gathered so far into one place?

lwshang (Jun 06 2019 at 18:28, on Zulip):

Not in one place yet. In the meeting minutes, I mentioned there are basically three places we have some data gathered. The hackmd page and two Zulip topics: "Gathering parallel data" and "effect on firefox".

nikomatsakis (Jun 06 2019 at 18:29, on Zulip):

OK

nikomatsakis (Jun 06 2019 at 18:29, on Zulip):

It seems like it'd be good to try and pull that a bit together

nikomatsakis (Jun 06 2019 at 18:29, on Zulip):

I also just left a comment with a few other minor things

nikomatsakis (Jun 06 2019 at 18:29, on Zulip):

that seems like a shift from the previous plan, though

in particular, this comment was discussing using "alt builds", which would mean that people "opt-in" to the parallel-enabled build.

nikomatsakis (Jun 06 2019 at 18:29, on Zulip):

But of course we already use alt builds for another purpose

nikomatsakis (Jun 06 2019 at 18:30, on Zulip):

However, that has some appeal, since then they won't experience the seq. overhead

nikomatsakis (Jun 06 2019 at 18:30, on Zulip):

One thing I'd like to see is the "whole crate graph" effect of -Zthreads=1 for common cases

nikomatsakis (Jun 06 2019 at 18:30, on Zulip):

not sure if we've measured that

nikomatsakis (Jun 06 2019 at 18:30, on Zulip):

basically our perf results suggested that seq overhead wasn't that bad, but I'm not sure so sure

nikomatsakis (Jun 06 2019 at 18:30, on Zulip):

based on the numbers I saw elsewhere...?

nikomatsakis (Jun 06 2019 at 18:31, on Zulip):

I'm not sure the best way to organize the data

nikomatsakis (Jun 06 2019 at 18:31, on Zulip):

gotta run now tho :)

nikomatsakis (Jun 06 2019 at 18:31, on Zulip):

I'll try to stay on top of this stream a bit more

lwshang (Jun 06 2019 at 18:33, on Zulip):

That's great! I'll try to organize data we gathered. And "whole crate graph" is not very clear for me. Could you explain briefly?

lwshang (Jun 06 2019 at 18:36, on Zulip):

When you are back, could you help me to find the discussion about documentation?
Just saw your comment in the PR. Will read those sources.

simulacrum (Jun 06 2019 at 18:58, on Zulip):

I think we have whole crate graph at -Zthreads=1

simulacrum (Jun 06 2019 at 18:58, on Zulip):

yeah, the "1" column here https://docs.google.com/spreadsheets/d/1vadQWQQqTODU1_cAENnUjLyXM6cxms-tiCf2kCiNGGM/edit#gid=0

simulacrum (Jun 06 2019 at 18:59, on Zulip):

0 is a sequential compiler for reference

lwshang (Jun 06 2019 at 19:14, on Zulip):

That's great. I just recorded the link and explanation in the hackmd page.

simulacrum (Jun 06 2019 at 19:21, on Zulip):

It's worth noting that data is unique (i.e., not elsewhere on that hackmd)

lwshang (Jun 06 2019 at 19:58, on Zulip):

@simulacrum As quoted in the very beginning of this topic by niko, you said we will ship the parallel-enabled rustc, and gather community feedback on it. I'm wondering what kind of feedback do we want.

Since the user will only have a parallel-enabled rustc in hand, they can't do the comparison between a real single threaded rustc and the parallel-enabled rustc with -Zthreads=1. So that we can't get measurement of the seq overhead from the audience. We may ask the users to compile any rust code with different n in -Zthreads=n and report the difference there. This is currently the only plan in my head. Looking forward to more suggestion.

simulacrum (Jun 06 2019 at 20:00, on Zulip):

so if we want to, we can somewhat easily ship (for a limited set of platforms, e.g. just linux, due to money/CI time budget) ship both parallel and non-parallel compilers to end-users, and make them somewhat readily installable via a shell script or something like that

simulacrum (Jun 06 2019 at 20:00, on Zulip):

(similar to what my script did)

simulacrum (Jun 06 2019 at 20:00, on Zulip):

collect-crategraph-data.sh

simulacrum (Jun 06 2019 at 20:00, on Zulip):

namely the rustup-toolchain-install-master bits

lwshang (Jun 06 2019 at 20:04, on Zulip):

Wow. Glad to see that opportunity. Niko mentions we did such alt build before. So maybe we can go through a similar approach.

simulacrum (Jun 06 2019 at 20:06, on Zulip):

it doesn't even need to be an alt build -- we can run try builds (or something similar) that just have parallel compiler switched on, that just gets us fewer platforms

simulacrum (Jun 06 2019 at 20:07, on Zulip):

or if we're willing to release somewhat rarely (e.g., just once a week or even just once) we can build a full set of compilers by landing on master and then reverting in the next PR to land

lwshang (Jun 06 2019 at 20:14, on Zulip):

Yes, that's what I want. Let me elaborate a little bit to make sure we are on the same page.
To do the comparison which shows overhead, the volunteer should have two versions of rustc in hand. They are all latest master branch (aka nightly channel). The only difference is whether parallel-compiler = true or false.

Does this look good?

simulacrum (Jun 06 2019 at 20:21, on Zulip):

in theory yes but this wouldn't really be "normal" nightly channel I imagine

simulacrum (Jun 06 2019 at 20:21, on Zulip):

but rather we say, have, one "nightly"-like a week with this sort of split

simulacrum (Jun 06 2019 at 20:22, on Zulip):

but it'll be just rustup-toolchain-install-master installable commits

lwshang (Jun 06 2019 at 20:26, on Zulip):

My key point is making sure that the only difference between those two versions is parallel-compiler = true or false. Ship it once a week is fine, but nightly are updating every day. Then we must require the user to use a nightly from days before to do the experiment.

simulacrum (Jun 06 2019 at 20:27, on Zulip):

oh, yeah, that would be the only difference

simulacrum (Jun 06 2019 at 20:28, on Zulip):

It's not feasible for us to ship these on a nightly basis or as part of rustup I think though

simulacrum (Jun 06 2019 at 20:28, on Zulip):

so they'd need to be installed somewhat out-of-band

lwshang (Jun 06 2019 at 20:28, on Zulip):

Will it be pre-compiled?

simulacrum (Jun 06 2019 at 20:29, on Zulip):

yeah

lwshang (Jun 06 2019 at 20:30, on Zulip):

I believe I got all what I want from the side of a general user who may want to join the experiment.

Last update: Nov 17 2019 at 08:15UTC