Stream: t-compiler/wg-parallel-rustc

Topic: planning meeting


Santiago Pastorino (Sep 26 2019 at 19:14, on Zulip):

@WG-parallel-rustc @nikomatsakis @Alex Crichton (unsure if there's somebody else interested not listed here), we have been talking with Niko about doing a planning meeting next week

Santiago Pastorino (Sep 26 2019 at 19:14, on Zulip):

may end sending a doodle or just set a date as people prefer :)

simulacrum (Sep 26 2019 at 19:14, on Zulip):

voice or text? but doodle would probably be best

Santiago Pastorino (Sep 26 2019 at 19:15, on Zulip):

yeah I can make a doodle, text I guess?

Santiago Pastorino (Sep 26 2019 at 19:16, on Zulip):

well, what works better I'd say ... if voice is preferred I think it's fine too

Santiago Pastorino (Sep 26 2019 at 19:16, on Zulip):

let's try to find a date first

Alex Crichton (Sep 26 2019 at 19:17, on Zulip):

doodle would work for me, but I'm also fine if you don't schedule around me

Santiago Pastorino (Sep 26 2019 at 19:21, on Zulip):

https://doodle.com/poll/ccfpdme449bk67xy

nikomatsakis (Sep 27 2019 at 13:00, on Zulip):

I'd prefer zulip

nikomatsakis (Sep 27 2019 at 13:00, on Zulip):

(as a general rule)

nikomatsakis (Sep 27 2019 at 13:00, on Zulip):

anyway i'm filling out the doodle now, sorry

nikomatsakis (Sep 27 2019 at 13:02, on Zulip):

done!

Santiago Pastorino (Sep 27 2019 at 13:54, on Zulip):

I think we already have a date

Santiago Pastorino (Sep 27 2019 at 13:55, on Zulip):

which is Monday 3pm GMT-3 (on my timezone) but doing to add this to the compiler calendar

Alex Crichton (Sep 27 2019 at 15:16, on Zulip):

If there's a calendar event, can you invite me to that?

simulacrum (Sep 27 2019 at 15:23, on Zulip):

(I would also appreciate an invite and do not see one on monday)

Santiago Pastorino (Sep 27 2019 at 17:31, on Zulip):

hey, yeah, I'm adding that right now

Santiago Pastorino (Sep 27 2019 at 17:32, on Zulip):

@Alex Crichton @WG-parallel-rustc the ones interested in getting an invite, hand me a gsuite email

simulacrum (Sep 27 2019 at 17:32, on Zulip):

mark.simulacrum@gmail.com

Alex Crichton (Sep 27 2019 at 17:38, on Zulip):

acrichton@mozilla.com

Santiago Pastorino (Sep 27 2019 at 17:57, on Zulip):

invite sent, you probably got two mails, because I fixed the date afterwards :)

cuviper (Sep 27 2019 at 19:22, on Zulip):

please send to cuviper@gmail.com too, thanks!

Santiago Pastorino (Sep 27 2019 at 20:25, on Zulip):

done

Santiago Pastorino (Sep 27 2019 at 21:28, on Zulip):

@WG-parallel-rustc @Alex Crichton @nikomatsakis I'm leaving a HackMD document here as placeholder, for now, before the meeting we can fill it with ideas and during the meeting we can use it to leave a summary there

Santiago Pastorino (Sep 27 2019 at 21:28, on Zulip):

btw @nikomatsakis and @Alex Crichton are not in @WG-parallel-rustc group, unsure if you guys want to be there :)

simulacrum (Sep 27 2019 at 21:29, on Zulip):

I'll try to fill that doc over the weekend with some agenda items

Santiago Pastorino (Sep 27 2019 at 21:29, on Zulip):

added a link to that document on the calendar event too

Santiago Pastorino (Sep 30 2019 at 12:33, on Zulip):

@WG-parallel-rustc @Alex Crichton @nikomatsakis reminder that we have the planning meeting in 5.5hs, remember that there's this HackMD document with some agenda items and thoughts, feel free to add more stuff there

Alex Crichton (Sep 30 2019 at 14:53, on Zulip):

As a heads up I have an appointment ending at the same time the meeting starts, so I'll need to drive home, so I may be 15-20m late

Alex Crichton (Sep 30 2019 at 14:53, on Zulip):

(I suspect I can be brought up to speed quickly though)

cuviper (Sep 30 2019 at 17:52, on Zulip):

are we doing video chat, or just here?

simulacrum (Sep 30 2019 at 17:53, on Zulip):

I believe just here?

simulacrum (Sep 30 2019 at 17:54, on Zulip):

I'm happy with either though

cuviper (Sep 30 2019 at 17:55, on Zulip):

that's fine, just figuring out whether I need to use my laptop :)

Santiago Pastorino (Sep 30 2019 at 18:00, on Zulip):

@WG-parallel-rustc @Alex Crichton @nikomatsakis meeting time :wave: everyone!

simulacrum (Sep 30 2019 at 18:00, on Zulip):

:wave:

cuviper (Sep 30 2019 at 18:00, on Zulip):

:wave:

cuviper (Sep 30 2019 at 18:00, on Zulip):

is there a way to "officially" join that WG?

Santiago Pastorino (Sep 30 2019 at 18:01, on Zulip):

@cuviper I can add you to the Zulip group

cuviper (Sep 30 2019 at 18:01, on Zulip):

that would be great, thanks

Santiago Pastorino (Sep 30 2019 at 18:01, on Zulip):

I was helping getting some time for this meeting and organizing this a bit but I guess I'd leave @nikomatsakis to handle it?, Niko is that ok?

nikomatsakis (Sep 30 2019 at 18:01, on Zulip):

:wave:

nikomatsakis (Sep 30 2019 at 18:01, on Zulip):

sure

nikomatsakis (Sep 30 2019 at 18:02, on Zulip):

I'm looking over the hackmd document

simulacrum (Sep 30 2019 at 18:03, on Zulip):

(I am not entirely clear on what the goals for this meeting should be -- I mostly threw potential ideas on the doc)

simulacrum (Sep 30 2019 at 18:03, on Zulip):

So feel free to steer in an entirely separate direction :)

Santiago Pastorino (Sep 30 2019 at 18:03, on Zulip):

yeah, I was going to say that exactly

Santiago Pastorino (Sep 30 2019 at 18:03, on Zulip):

we are aiming for a Roadmap, is that right?

nikomatsakis (Sep 30 2019 at 18:03, on Zulip):

I was hoping we'd find some steps to go forward

nikomatsakis (Sep 30 2019 at 18:03, on Zulip):

I don't really agree with this: "Queries and parallelism are a poor fit"

cuviper (Sep 30 2019 at 18:04, on Zulip):

seems like the "Purpose" bullet may need to be first

nikomatsakis (Sep 30 2019 at 18:04, on Zulip):

agreed

nikomatsakis (Sep 30 2019 at 18:04, on Zulip):

So yeah what I had in mind was that we would kind of revisit some of the themes from the last time we chatted and try to push them into more specific steps to go forward

nikomatsakis (Sep 30 2019 at 18:05, on Zulip):

I'm trying to go fish out some links there :)

nikomatsakis (Sep 30 2019 at 18:05, on Zulip):

I guess I am referring to this topic

nikomatsakis (Sep 30 2019 at 18:06, on Zulip):

and I was hoping to continue from roughly these comments

nikomatsakis (Sep 30 2019 at 18:06, on Zulip):

basically, continue to work of pushing this current branch of work into a shippable state

nikomatsakis (Sep 30 2019 at 18:07, on Zulip):

(but I guess it's worth asking if we think we'd need to do something more radical)

nikomatsakis (Sep 30 2019 at 18:08, on Zulip):

Personally my goal would be that we are able to parallelize a few key things:

- running the type/borrow checker across functions in parallel
- translation codegen units to LLVM in parallel

etc, and it seems like the current architecture should suffice to do that.

simulacrum (Sep 30 2019 at 18:09, on Zulip):

I would agree that we should be able to accomplish those two

simulacrum (Sep 30 2019 at 18:09, on Zulip):

even without really any "major" work

nikomatsakis (Sep 30 2019 at 18:11, on Zulip):

I don't really agree with this: "Queries and parallelism are a poor fit"

(to return to this, I think perhaps I am interpreting the term "queries" too generally -- that is, I think the concepts we are working with are pretty amenable to parallelization, as they are basically just pure functions -- but I can definitely imagine that we'll want to consider different impl strategies)

nikomatsakis (Sep 30 2019 at 18:12, on Zulip):

in any case, I was sketching some of what I view as the current concerns with shipping *now*

simulacrum (Sep 30 2019 at 18:12, on Zulip):

Yes -- when they're pure functions they work great, but particularly around the top level we have a lot of "non pure" queries that are using Steal and such in hard-to-define ways

simulacrum (Sep 30 2019 at 18:13, on Zulip):

One concern I had is whether we think this is a "good idea" in general

nikomatsakis (Sep 30 2019 at 18:13, on Zulip):

Not really

cuviper (Sep 30 2019 at 18:13, on Zulip):

Alex filed a couple issues re: thread management too

simulacrum (Sep 30 2019 at 18:13, on Zulip):

Ah, this=parallelism within the compiler, to be clear

nikomatsakis (Sep 30 2019 at 18:13, on Zulip):

What exceptions are there? Steal is fairly local

simulacrum (Sep 30 2019 at 18:14, on Zulip):

https://github.com/rust-lang/rust/blob/master/src/librustc_interface/queries.rs#L213-L224

nikomatsakis (Sep 30 2019 at 18:14, on Zulip):

I don't really think it's a blocking problem, it seems more like a "we should refactor this" project

simulacrum (Sep 30 2019 at 18:14, on Zulip):

but yeah it's probably non blocking

nikomatsakis (Sep 30 2019 at 18:14, on Zulip):

Right, so .. that is "pre-query"

simulacrum (Sep 30 2019 at 18:14, on Zulip):

heh, sure

nikomatsakis (Sep 30 2019 at 18:14, on Zulip):

I guess I still think that parallelism within the compiler makes sense

nikomatsakis (Sep 30 2019 at 18:15, on Zulip):

But it's worth being clear about which scenarios

nikomatsakis (Sep 30 2019 at 18:15, on Zulip):

Which I guess leads back to the overall goals

simulacrum (Sep 30 2019 at 18:15, on Zulip):

So my main concern is that I think it might be good to scale back what I _think_ are our current ambitions, to an extent, to what Niko highlighted earlier

simulacrum (Sep 30 2019 at 18:15, on Zulip):

Specifically
- running the type/borrow checker across functions in parallel
- translation codegen units to LLVM in parallel

simulacrum (Sep 30 2019 at 18:16, on Zulip):

Like, I don't think we want to try to parallelize things like parsing and name resolution and such, i.e., to intertwine them, quite yet

nikomatsakis (Sep 30 2019 at 18:16, on Zulip):

(The reason that I included translation of codegen units is that we identified that as a blocker to experimenting with different strategies around generating LLVM bitcode that might help with reducing duplication)

Alex Crichton (Sep 30 2019 at 18:16, on Zulip):

:wave:

nikomatsakis (Sep 30 2019 at 18:16, on Zulip):

(But also because it seems to fit just fine)

nikomatsakis (Sep 30 2019 at 18:16, on Zulip):

Like, I don't think we want to try to parallelize things like parsing and name resolution and such, i.e., to intertwine them, quite yet

Agreed... are you citing those because they are not part of the query system yet?

nikomatsakis (Sep 30 2019 at 18:16, on Zulip):

(Or because there are pending PRs along those lines?)

simulacrum (Sep 30 2019 at 18:17, on Zulip):

Mostly to try and establish non-goals

simulacrum (Sep 30 2019 at 18:17, on Zulip):

Not because of any particular reason

nikomatsakis (Sep 30 2019 at 18:17, on Zulip):

OK, just checking. In any case, I agree :)

nikomatsakis (Sep 30 2019 at 18:17, on Zulip):

Another non-goal from my POV is very fine-grained parallelism, which is partly why I think Rayon integration is not something we have to have at this stage

nikomatsakis (Sep 30 2019 at 18:18, on Zulip):

e.g., I wouldn't expect to parallelize type-checking within a particular function, or at intra-query granularity

Alex Crichton (Sep 30 2019 at 18:18, on Zulip):

I may have missed this but do we have data to show the high-level passes which need aprallelization the most? (e.g. the slowest rustc passes)

cuviper (Sep 30 2019 at 18:19, on Zulip):

you don't think rayon is useful coarsely? "here are all my functions, go borrowck!"

simulacrum (Sep 30 2019 at 18:19, on Zulip):

typeck_tables and codegen are ~always the top two in most benchmarks, I believe

nikomatsakis (Sep 30 2019 at 18:19, on Zulip):

yes, what @simulacrum said

nikomatsakis (Sep 30 2019 at 18:19, on Zulip):

you don't think rayon is useful coarsely? "here are all my functions, go borrowck!"

I didn't say that :) I just said I don't think it's needed

nikomatsakis (Sep 30 2019 at 18:20, on Zulip):

in particular if it complicates jobserver integration (which I think it very well may)

Zoxc (Sep 30 2019 at 18:20, on Zulip):

Type checking is parallel already. I have code to make codegen parallel too, but I'd rather just get rid of the messy LLVM threads and use a simple for loop for both codegen+LLVM

simulacrum (Sep 30 2019 at 18:20, on Zulip):

@Zoxc do you know if we have data on whether that's showing expected (linear-ish) improvements in time?

cuviper (Sep 30 2019 at 18:20, on Zulip):

ok -- I think the jobserver issue is roughly the same problem whether rayon manages the threads or not

simulacrum (Sep 30 2019 at 18:20, on Zulip):

at least within those specific queries

Alex Crichton (Sep 30 2019 at 18:21, on Zulip):

(FWIW I think it's a fantastic idea to start at a coarse level and then go from there, that's technically what we've already done with parallel codegen)

nikomatsakis (Sep 30 2019 at 18:21, on Zulip):

Type checking is parallel already.

right. I basically want to do the work we need to do to ship this

nikomatsakis (Sep 30 2019 at 18:21, on Zulip):

( and iterate from there )

nikomatsakis (Sep 30 2019 at 18:22, on Zulip):

that said, the LLVM threads and so forth is a bit tricky since we have to control peak memory usage, iirc

Zoxc (Sep 30 2019 at 18:23, on Zulip):

Less threads = less peak memory usage. It's not very hard =P

simulacrum (Sep 30 2019 at 18:23, on Zulip):

Do we consider being confident that we don't deadlock etc (and essentially can't) a prereq for shipping?

Alex Crichton (Sep 30 2019 at 18:23, on Zulip):

It sounds like it would definitely be worthwhile to ship parallel typeck with perhaps a new architecture of managing the jobserver and then slowly port parallel codegen to the new architecture

Zoxc (Sep 30 2019 at 18:24, on Zulip):

We probably need some benchmark to see if switching between processes due to jobserver is actually a big problem

nikomatsakis (Sep 30 2019 at 18:24, on Zulip):

Do we consider being confident that we don't deadlock etc (and essentially can't) a prereq for shipping?

I've been debating about this. We consistently rank "tech debt" amongst our problems and I am concerned about growing that. This is why I listed documentation in the list of concerns. On the other hand, I don't want to block indefinitely on vague concerns.

Zoxc (Sep 30 2019 at 18:25, on Zulip):

There's things we can do to reduce that overhead (make rayon wait longer before going idle), ensuring there's always work to be done in rustc, spreading jobserver tokens better across processes)

simulacrum (Sep 30 2019 at 18:25, on Zulip):

I personally think that eliminating the shared state on Session, e.g., which is pretty much a pre-req for this sort of confidence, will be quite hard -- I guess I could imagine we just don't, and instead document some fields as "be careful when accessing these"

nikomatsakis (Sep 30 2019 at 18:26, on Zulip):

I don't think you need to eliminate shared state entirely to have confidence

nikomatsakis (Sep 30 2019 at 18:26, on Zulip):

Certainly not for dead-lock freedom

Alex Crichton (Sep 30 2019 at 18:26, on Zulip):

FWIW I do think we should strive very hard for high confidence in implementation here, bugs just lead to every tool/crate trying to have its own workaround

Alex Crichton (Sep 30 2019 at 18:26, on Zulip):

people primarily just want their code to compile, and they'd rather it compile correctly than never

nikomatsakis (Sep 30 2019 at 18:27, on Zulip):

One thing worth bringing up: I've not successfully used it yet, but we might experiment with things like loom

Alex Crichton (Sep 30 2019 at 18:27, on Zulip):

yeah so I'm not sure we necessarily need to prove it correct or anything like that

nikomatsakis (Sep 30 2019 at 18:27, on Zulip):

Certainly not for dead-lock freedom

but we do have to be careful about having too many locks, or returning guards so that we can't see how long a lock is held

Alex Crichton (Sep 30 2019 at 18:27, on Zulip):

but I think we need more than "one of us knows how it works and rustc says it's ok"

simulacrum (Sep 30 2019 at 18:28, on Zulip):

@Alex Crichton Can you elaborate perhaps on what sort of docs or other materials you'd want to see?

nikomatsakis (Sep 30 2019 at 18:28, on Zulip):

Yes, I think I would be happy if we had a set of 2-3 people working on stuff, some reasonable amount of docs on the "big picture" and some notes where we use fine-grained locking, and a lot of testing (that said, loom seems like something we could setup, but not as a blocker)

simulacrum (Sep 30 2019 at 18:28, on Zulip):

e.g. how do we increase "one of us knows how it works"

Alex Crichton (Sep 30 2019 at 18:29, on Zulip):

I'm basically stressing how bad it is to have subtle nondeterministic bugs in your compiler

nikomatsakis (Sep 30 2019 at 18:29, on Zulip):

I think I would give incremental as an anti-pattern

nikomatsakis (Sep 30 2019 at 18:29, on Zulip):

We've not had a ton of bugs but

nikomatsakis (Sep 30 2019 at 18:29, on Zulip):

we don't have a lot of people familiar with the details

Alex Crichton (Sep 30 2019 at 18:29, on Zulip):

it's got the highest likelihood of showing up in big projects (which want to turn this on the most), and runs the risk of forcing our hand to, for example, add configuration to cargo to control all the knobs here

nikomatsakis (Sep 30 2019 at 18:29, on Zulip):

there are pretty decent docs though

simulacrum (Sep 30 2019 at 18:31, on Zulip):

I think in some sense my impression is that getting good docs for the highlevel parts (jobserver integration, where the "parallel loops" are) is going to be fairly easy. Getting confidence for the relatively fine-grained locks I think is going to be quite hard, though.

nikomatsakis (Sep 30 2019 at 18:31, on Zulip):

( so time-check, we're 30 minutes in :)

simulacrum (Sep 30 2019 at 18:33, on Zulip):

It seems like we pretty much are agreed that the primary blocker is getting docs/confidence in the implementation, right?

I believe the jobserver integration for example we have "known knobs" to try and tune -- as @Zoxc mentioned

nikomatsakis (Sep 30 2019 at 18:33, on Zulip):

we've got 71 instances of Lock< in librustc* :)

nikomatsakis (Sep 30 2019 at 18:34, on Zulip):

some of which are public (ungreat)

nikomatsakis (Sep 30 2019 at 18:34, on Zulip):

but most of which are not

nikomatsakis (Sep 30 2019 at 18:36, on Zulip):

it seems plausible to me that we can audit them and refactor them to use:

and for the remainder, well, we'll have to see what testing brings I guess.

simulacrum (Sep 30 2019 at 18:37, on Zulip):

Seems reasonable -- do we think a good way to do this might be to just put up some issues to start (maybe 3 or 4 or so) on some public locks and assign some folks in this meeting to them?

nikomatsakis (Sep 30 2019 at 18:37, on Zulip):

maybe. let's table that for one second.

nikomatsakis (Sep 30 2019 at 18:37, on Zulip):

because I'm curious about the other things.

nikomatsakis (Sep 30 2019 at 18:38, on Zulip):

like, what are they? :)

nikomatsakis (Sep 30 2019 at 18:38, on Zulip):

one thing I didn't write down is "rayon fork"

Alex Crichton (Sep 30 2019 at 18:38, on Zulip):

"other things" in terms of what we need for confidence?

nikomatsakis (Sep 30 2019 at 18:39, on Zulip):

like, what else do we want to do before shipping?

nikomatsakis (Sep 30 2019 at 18:39, on Zulip):
nikomatsakis (Sep 30 2019 at 18:39, on Zulip):

that's what I have in the hackmd doc

nikomatsakis (Sep 30 2019 at 18:39, on Zulip):

the reason I'm asking is

Alex Crichton (Sep 30 2019 at 18:40, on Zulip):
nikomatsakis (Sep 30 2019 at 18:40, on Zulip):

I'd sort of like to identify "steps" we can take for each of them

cuviper (Sep 30 2019 at 18:40, on Zulip):

let's _update_ the rayon fork, at least...

cuviper (Sep 30 2019 at 18:41, on Zulip):

I dropped one patch in favor of build_scoped

cuviper (Sep 30 2019 at 18:41, on Zulip):

we can review the others

nikomatsakis (Sep 30 2019 at 18:41, on Zulip):

yep, I think I'd like to do a review of each of them

nikomatsakis (Sep 30 2019 at 18:41, on Zulip):

and talk out why we have it

nikomatsakis (Sep 30 2019 at 18:41, on Zulip):

( and if we can get rid of it )

nikomatsakis (Sep 30 2019 at 18:41, on Zulip):

maybe we can just do that over a few weeks?

nikomatsakis (Sep 30 2019 at 18:42, on Zulip):

(or all at once, maybe, not sure how long it'll take, I guess 1 meeting is likely enough)

nikomatsakis (Sep 30 2019 at 18:42, on Zulip):

I have this under documentation:

* Why do we have the Rayon fork?
Alex Crichton (Sep 30 2019 at 18:42, on Zulip):

I think one thing we want on a plan is having an idea of how transition parallel codegen/typeck to same parallelization framework

nikomatsakis (Sep 30 2019 at 18:42, on Zulip):

I'm sort of seeing something here -- I feel like the frame of "documentaiton" is helpful here

Zoxc (Sep 30 2019 at 18:43, on Zulip):

Don't forget about my Rayon fiber fork. That would also help prevent threads from falling asleep =P

nikomatsakis (Sep 30 2019 at 18:44, on Zulip):

I think one thing we want on a plan is having an idea of how transition parallel codegen/typeck to same parallelization framework

e.g., in this case, I feel like it'd be useful to have somebody start by doing a quick review of how things are working today (maybe over Zoom), and then we can talk about how they could be unified and improved? And the result is most of the way to docs

Zoxc (Sep 30 2019 at 18:44, on Zulip):

@Alex Crichton I don't think that's important until the parallel compiler is the default, so we can actually remove the LLVM threads without regressing performance

nikomatsakis (Sep 30 2019 at 18:44, on Zulip):

(I was proposing something similar for the rayon fork)

Alex Crichton (Sep 30 2019 at 18:45, on Zulip):

If mw isn't available I can probably give a review of how parallel codegen works

Alex Crichton (Sep 30 2019 at 18:45, on Zulip):

(later on a call)

nikomatsakis (Sep 30 2019 at 18:46, on Zulip):

so some bullets where I don't have anything listed include:

nikomatsakis (Sep 30 2019 at 18:47, on Zulip):

the latter feels like something we could do the same "explain, discuss, experiment/document" pattern on, if we want

nikomatsakis (Sep 30 2019 at 18:47, on Zulip):

ah well I guess @Zoxc listed some solutions

nikomatsakis (Sep 30 2019 at 18:47, on Zulip):

I personally wouldn't mind talking them over though :)

nikomatsakis (Sep 30 2019 at 18:47, on Zulip):

(and better understanding what we're doing now)

nikomatsakis (Sep 30 2019 at 18:48, on Zulip):

but for sequential overhead, I think the real next steps are to identify cases we think are too high and just profile/optimize them?

nikomatsakis (Sep 30 2019 at 18:48, on Zulip):

Maybe @nnethercote might be "nerd-swipable" to help with this kind of optimization effort :)

Alex Crichton (Sep 30 2019 at 18:48, on Zulip):

I though we were largely only worried about that for shipping off-by-default-but-possible-to-enable parallel support?

Alex Crichton (Sep 30 2019 at 18:49, on Zulip):

in that we expect parallelism to easily regain any sequential regressions

nikomatsakis (Sep 30 2019 at 18:49, on Zulip):

Well, wouldn't you like to do that? =)

nikomatsakis (Sep 30 2019 at 18:49, on Zulip):

I would...

nikomatsakis (Sep 30 2019 at 18:49, on Zulip):

But also I'm not sure that's true

Alex Crichton (Sep 30 2019 at 18:49, on Zulip):

lol true

nikomatsakis (Sep 30 2019 at 18:49, on Zulip):

I mean if you have a big crate graph

Alex Crichton (Sep 30 2019 at 18:49, on Zulip):

ah I see what you mean

Alex Crichton (Sep 30 2019 at 18:49, on Zulip):

if you don't actually have parallelism at one point what's the slowdown

nikomatsakis (Sep 30 2019 at 18:49, on Zulip):

Right, it'll definitely eat away at any wins

nikomatsakis (Sep 30 2019 at 18:50, on Zulip):

I sort of thought we had concluded that the timings were "pretty good" but then @simulacrum seemed to have concerns

nikomatsakis (Sep 30 2019 at 18:50, on Zulip):

I haven't reviewed the latest measurements, though

Alex Crichton (Sep 30 2019 at 18:50, on Zulip):

although given the cpu usage graphs I've seen it probably won't eat that much, but I do agree we should have a baseline understanding of what the expected regression is

Alex Crichton (Sep 30 2019 at 18:50, on Zulip):

I would say that to start off we should just be confident is "what is the perf of -j1 rustc"

Alex Crichton (Sep 30 2019 at 18:50, on Zulip):

and we can evaluate from there about whether optimization is necessary

nikomatsakis (Sep 30 2019 at 18:51, on Zulip):

I am wondering if it makes sense to pick 3-5 "real-world" crates and try to use them as our benchmarks?

simulacrum (Sep 30 2019 at 18:51, on Zulip):

I think we need to determine exactly what we want to collect data on and recollect (both because things have changed -- e.g. pipelining) and because it is not clear that the data I collected was what we wanted to

nikomatsakis (Sep 30 2019 at 18:51, on Zulip):

This has always been hard for us :)

nikomatsakis (Sep 30 2019 at 18:52, on Zulip):

really I mean more than crates

Alex Crichton (Sep 30 2019 at 18:52, on Zulip):

oh I'm assuming we run the current perf suite at -j1

nikomatsakis (Sep 30 2019 at 18:52, on Zulip):

I mean like "scenarios"

Alex Crichton (Sep 30 2019 at 18:52, on Zulip):

(but "enable parallel support")

nikomatsakis (Sep 30 2019 at 18:52, on Zulip):

Yes, we should do that too

Alex Crichton (Sep 30 2019 at 18:52, on Zulip):

and that's a good enough test for what the expected regression would be

nikomatsakis (Sep 30 2019 at 18:52, on Zulip):

well anyway start with that

simulacrum (Sep 30 2019 at 18:52, on Zulip):

-j1 has the unfortunate problem of limiting parallel codegen which is terrible

nikomatsakis (Sep 30 2019 at 18:52, on Zulip):

we had some metric before like a "% threshold" and an "ms threshold"

simulacrum (Sep 30 2019 at 18:52, on Zulip):

but I can imagine some workaround for that? Presuming we want one.

Alex Crichton (Sep 30 2019 at 18:53, on Zulip):

we could only look at the "check" benchmarks, but @nikomatsakis I think it's probably fine to just look at the numbers and evaluate from there

nikomatsakis (Sep 30 2019 at 18:53, on Zulip):

another concern I remember being raised was bootstrap time

Alex Crichton (Sep 30 2019 at 18:53, on Zulip):

any number throw out is probably just a gut feeling for now so it seems fine to develop the gut feeling after we have data

nikomatsakis (Sep 30 2019 at 18:54, on Zulip):

yes I agree

Santiago Pastorino (Sep 30 2019 at 18:54, on Zulip):

although given the cpu usage graphs I've seen it probably won't eat that much, but I do agree we should have a baseline understanding of what the expected regression is

are there some graphs or benchmarks that were already ran?

nikomatsakis (Sep 30 2019 at 18:54, on Zulip):

I just think it'd be useful for us to start collecting some good set of benchmarks we want to focus on; let's just start with a perf run in any case

Alex Crichton (Sep 30 2019 at 18:54, on Zulip):

@Santiago Pastorino oh I'm thinking of whole crate graph builds using Cargo's new -Z timings flag which shows cpu usage and compile time information

simulacrum (Sep 30 2019 at 18:54, on Zulip):

I can take on getting a try build with parallel enabled and running perf on that

Alex Crichton (Sep 30 2019 at 18:55, on Zulip):

@nikomatsakis oh true, yeah, so I definitely agree on that

Alex Crichton (Sep 30 2019 at 18:55, on Zulip):

do you think we should list those off here?

nikomatsakis (Sep 30 2019 at 18:55, on Zulip):

I don't know that we have to do it "live" right now, but maybe leave notes in the hackmd doc?

nikomatsakis (Sep 30 2019 at 18:55, on Zulip):

hackmd doc

nikomatsakis (Sep 30 2019 at 18:56, on Zulip):

I just think it'd be useful for us to start collecting some good set of benchmarks we want to focus on; let's just start with a perf run in any case

one thing in particular: I don't want to get to the point where we are trying to announce things without us being able to say "here are some awesome graphs to look at!"

Alex Crichton (Sep 30 2019 at 18:56, on Zulip):

for sure

nikomatsakis (Sep 30 2019 at 18:56, on Zulip):

I think we'll all feel better if we're like "we're not gonna ship until cargo check on regex is 2x" or something

nikomatsakis (Sep 30 2019 at 18:56, on Zulip):

if we have to adjust later, fine

cuviper (Sep 30 2019 at 18:57, on Zulip):

tentatively fearless concurrency

cuviper (Sep 30 2019 at 18:57, on Zulip):

(sorry)

nikomatsakis (Sep 30 2019 at 18:57, on Zulip):

ok we're 57 minutes in

nikomatsakis (Sep 30 2019 at 18:59, on Zulip):

I'm going to try to type in some "next steps" in a separate section of the doc and see if we can put this together

nikomatsakis (Sep 30 2019 at 19:02, on Zulip):

this is what I wrote

nikomatsakis (Sep 30 2019 at 19:02, on Zulip):

curse you zulip, no nested lists?

nikomatsakis (Sep 30 2019 at 19:02, on Zulip):

first they took my - (edit: admittedly, I'm getting vaguely used to *)

nikomatsakis (Sep 30 2019 at 19:02, on Zulip):

/me grumbles

Alex Crichton (Sep 30 2019 at 19:02, on Zulip):

that sounds like a great start yeah, although I do think we want more confidence in the current implementation before we get "random rust users" involved

nikomatsakis (Sep 30 2019 at 19:02, on Zulip):

agreed, I should move that down the list

nikomatsakis (Sep 30 2019 at 19:02, on Zulip):

it seems reasonably low in priority

Alex Crichton (Sep 30 2019 at 19:03, on Zulip):

I will replace it with something else though, we should get alt builds so we can test easily

nikomatsakis (Sep 30 2019 at 19:03, on Zulip):

I was just going to say that...

nikomatsakis (Sep 30 2019 at 19:03, on Zulip):

so @simulacrum on the topic of audits:

I think what would make sense is to setup a standing time to move through things, and try to spin out tasks. I think it probably makes sense for this to be reasonably experienced folks.

nikomatsakis (Sep 30 2019 at 19:03, on Zulip):

(since I tabled when you asked that question before)

nikomatsakis (Sep 30 2019 at 19:04, on Zulip):

maybe we can spin that out into a distinct topic, but I think I'd be happy to try to add an hour on my calendar for that if we could find a good one

simulacrum (Sep 30 2019 at 19:04, on Zulip):

So we think something like "Zoom call every monday" so to speak and 2-3 of us spend an hour trying to audit and then document findings (maybe by just recording zoom output?)

nikomatsakis (Sep 30 2019 at 19:04, on Zulip):

unlike the last time we did it, I think we'd just try to go deep into each particular item, coming up with a final design and/or things to document

nikomatsakis (Sep 30 2019 at 19:04, on Zulip):

So we think something like "Zoom call every monday" so to speak and 2-3 of us spend an hour trying to audit and then document findings (maybe by just recording zoom output?)

yep, this

nikomatsakis (Sep 30 2019 at 19:05, on Zulip):

I'd hope afterwards that we volunteer someone to try and polish up the results

nikomatsakis (Sep 30 2019 at 19:05, on Zulip):

but yeah

nikomatsakis (Sep 30 2019 at 19:05, on Zulip):

I think the zoom calls alone are better docs than we often have :P

nikomatsakis (Sep 30 2019 at 19:05, on Zulip):

we should get a better list than my rg 'Lock<' src/librust*

simulacrum (Sep 30 2019 at 19:06, on Zulip):

I am personally up for getting that list put together -- maybe we should try to pencil in a meeting slot this week or next to trial it out, though.

nikomatsakis (Sep 30 2019 at 19:06, on Zulip):

yep.

simulacrum (Sep 30 2019 at 19:07, on Zulip):

(Should I put a Doodle together?)

nikomatsakis (Sep 30 2019 at 19:08, on Zulip):

/me sighs

nikomatsakis (Sep 30 2019 at 19:08, on Zulip):

yeah, I guess so

nikomatsakis (Sep 30 2019 at 19:08, on Zulip):

:)

Alex Crichton (Sep 30 2019 at 19:08, on Zulip):

alternatively we could use this time?

Alex Crichton (Sep 30 2019 at 19:08, on Zulip):

if it still works for folks?

nikomatsakis (Sep 30 2019 at 19:08, on Zulip):

yeah, I could do this time

simulacrum (Sep 30 2019 at 19:09, on Zulip):

I could too. Certainly gives enough time to make some headway on a list of things for us to dig into

nikomatsakis (Sep 30 2019 at 19:09, on Zulip):

do we want to try and do something this week, or wait until next week?

nikomatsakis (Sep 30 2019 at 19:09, on Zulip):

I'm ok either way

nikomatsakis (Sep 30 2019 at 19:10, on Zulip):

proabbly we start with the audits and try that out before attempting the "explain/discuss/document" on bigger questions?

nikomatsakis (Sep 30 2019 at 19:10, on Zulip):

I don't want to get too overly ambitious

Alex Crichton (Sep 30 2019 at 19:10, on Zulip):

it sounds like we should tackle the "rayon or not" problem earliest perhaps?

Alex Crichton (Sep 30 2019 at 19:10, on Zulip):

that may help inform what sort of shared state we need at all?

Alex Crichton (Sep 30 2019 at 19:11, on Zulip):

like basically finalizing how we want concurrency to look like at all will inform quite a few other decisions

simulacrum (Sep 30 2019 at 19:11, on Zulip):

hm, I think we definitely have some shared state in e.g. Session which is shared no matter what

simulacrum (Sep 30 2019 at 19:11, on Zulip):

but maybe not too interesting, too

simulacrum (Sep 30 2019 at 19:12, on Zulip):

Regardless of rayon or not -- I think that just means less / more work to do something like this, right?

functions.par_iter().for_each(|function| tcx.typeck(function));
nikomatsakis (Sep 30 2019 at 19:12, on Zulip):

it sounds like we should tackle the "rayon or not" problem earliest perhaps?

that's a good choic

Alex Crichton (Sep 30 2019 at 19:12, on Zulip):

oh sure, but I'm basically saying that where we apply parallelism feels like a more core decision than what shared state we have

nikomatsakis (Sep 30 2019 at 19:13, on Zulip):

what I'm wondering is

nikomatsakis (Sep 30 2019 at 19:13, on Zulip):

do we want to do separate meetings on those

nikomatsakis (Sep 30 2019 at 19:13, on Zulip):

or try to re-use that same slot

nikomatsakis (Sep 30 2019 at 19:13, on Zulip):

I'm inclined to the former I guess

Alex Crichton (Sep 30 2019 at 19:13, on Zulip):

oh sorry, misunderstood

nikomatsakis (Sep 30 2019 at 19:13, on Zulip):

we could start not by an audit but by one of the questions

nikomatsakis (Sep 30 2019 at 19:14, on Zulip):

also, I should go look at that old list of "things to document" and see if I'm missing anything

nikomatsakis (Sep 30 2019 at 19:14, on Zulip):

but overall I'm pretty happy with this plan

nikomatsakis (Sep 30 2019 at 19:14, on Zulip):

and I kind of think a weekly slot to either audit or discuss some aspect of design or deep dive into whatever else seems relevant that week is good

simulacrum (Sep 30 2019 at 19:15, on Zulip):

So we'd perhaps have the meeting this time next week (or maybe something this week, to sort of get things moving?) with a goal of tackling one of the things to document quetions?

nikomatsakis (Sep 30 2019 at 19:15, on Zulip):

I am debating what to do first, but yes -- that or audit. I could go either way.

nikomatsakis (Sep 30 2019 at 19:17, on Zulip):

if we were to do this week, I'd favor Thu at 1pm eastern :)

simulacrum (Sep 30 2019 at 19:17, on Zulip):

hm, do we have a link to the questions doc handy?

nikomatsakis (Sep 30 2019 at 19:17, on Zulip):

Parallel documentation questions -- fixed

nikomatsakis (Sep 30 2019 at 19:18, on Zulip):

looks like we covered a lot of them

simulacrum (Sep 30 2019 at 19:20, on Zulip):

I can't do 1pm ET on Thursday (another meeting) unfortunately -- 1:30 would work though.

I guess maybe easiest is to just wait till Monday next week

simulacrum (Sep 30 2019 at 19:20, on Zulip):

since we all seem free

nikomatsakis (Sep 30 2019 at 19:20, on Zulip):

yeah, seems ok.

nikomatsakis (Sep 30 2019 at 19:20, on Zulip):

@simulacrum you'll work on the perf run tho?

simulacrum (Sep 30 2019 at 19:21, on Zulip):

yeah, I can get that done -- at least the basic bit of "just perf.rlo"

simulacrum (Sep 30 2019 at 19:21, on Zulip):

the scenarios on the doc are possibly a bit harder but should be feasible as well, plus, we have -Ztiming now on Cargo which is great

simulacrum (Sep 30 2019 at 19:22, on Zulip):

Maybe next week we spend ~5 minutes of the meeting on Zoom discussing what we think is best for the meeting depending on perf results

simulacrum (Sep 30 2019 at 19:22, on Zulip):

e.g. if things are mostly looking good then I think audit makes more sense

simulacrum (Sep 30 2019 at 19:23, on Zulip):

but if things are looking pretty not great, then it might make sense to looking into jobserver (since that's probably "easiest"/"biggest" wins) first

nikomatsakis (Sep 30 2019 at 19:24, on Zulip):

ideally a bit before, but that sounds good

Zoxc (Sep 30 2019 at 19:24, on Zulip):

Note that incremental is expected to be slower. I have some PRs open which will help there.

nikomatsakis (Sep 30 2019 at 19:24, on Zulip):

( i.e., I'd prefer to know before hand is we're going to discuss jobserver )

simulacrum (Sep 30 2019 at 19:24, on Zulip):

So agenda for myself is :

nikomatsakis (Sep 30 2019 at 19:24, on Zulip):

@Zoxc ps I think that if you have PRs that are blocked on landing because they do a lot of things, and you can make the meeting, I'd like to use it to talk over what the PR does

nikomatsakis (Sep 30 2019 at 19:25, on Zulip):

@simulacrum it seems like the librustc_data_structures::sync imports are probably a good start too?

nikomatsakis (Sep 30 2019 at 19:25, on Zulip):

anyway that sounds great

nikomatsakis (Sep 30 2019 at 19:25, on Zulip):

I could also imagine starting to do audits or attempt simpler refactorings, if you're feeling bored, as there are bound to be "easy" cases, and we can reserve the more complex cases ...

simulacrum (Sep 30 2019 at 19:25, on Zulip):

Ah, yes, we could talk through the primitives but that seems not as useful

nikomatsakis (Sep 30 2019 at 19:25, on Zulip):

no I meant

nikomatsakis (Sep 30 2019 at 19:26, on Zulip):

searching for imports/uses to "seed" the audit process

simulacrum (Sep 30 2019 at 19:27, on Zulip):

Sure -- I was intending to just grep for Lock< as you did, but that works too.

I might attempt some audits/refactorings, though I think particularly for the audit bit it'll be good to have some meeting to base what the results of such an audit should be on

simulacrum (Sep 30 2019 at 19:27, on Zulip):

the use sites being "get a lock field and figure out what uses it"

Zoxc (Sep 30 2019 at 19:34, on Zulip):

@nikomatsakis https://github.com/rust-lang/rust/pull/61845 does only 3 things =P
The first thing is split into it's own PR - https://github.com/rust-lang/rust/pull/63756 , which I think @mw basically approved?

Santiago Pastorino (Oct 02 2019 at 17:48, on Zulip):

@cuviper you requested to be added to @WG-parallel-rustc, just did so. Also added Niko and Alex, so it's easier to ping everyone by using the group, if someone don't want to be there it's easy to remove yourself from settings :)

simulacrum (Oct 03 2019 at 01:55, on Zulip):

@nikomatsakis could you create a compiler team invite recurring for mondays and populate the zoom link and such?

nikomatsakis (Oct 03 2019 at 13:27, on Zulip):

yes

nikomatsakis (Oct 03 2019 at 13:29, on Zulip):

@Zoxc I'll take a look at #63756 -- the description sounds simple enough, at least.

Santiago Pastorino (Oct 03 2019 at 15:43, on Zulip):

@simulacrum what's the best way to catch up with what you've done?

Santiago Pastorino (Oct 03 2019 at 15:43, on Zulip):

saw that you were doing some stuff and wanted to catch up with that

Santiago Pastorino (Oct 03 2019 at 15:45, on Zulip):

(deleted)

simulacrum (Oct 03 2019 at 15:46, on Zulip):

@Santiago Pastorino wrong channel/stream perhaps?

Santiago Pastorino (Oct 03 2019 at 15:48, on Zulip):

well I wasn't talking specifically about the planning per se, I was talking in general about the things we planned on parallel

Santiago Pastorino (Oct 03 2019 at 15:48, on Zulip):

saw that you ran benches

Santiago Pastorino (Oct 03 2019 at 15:48, on Zulip):

and were doing a bunch of cool stuff

simulacrum (Oct 03 2019 at 15:48, on Zulip):

(You just linked to bisect so was confused)

Santiago Pastorino (Oct 03 2019 at 15:48, on Zulip):

ohh

Santiago Pastorino (Oct 03 2019 at 15:48, on Zulip):

that was for the other thread

simulacrum (Oct 03 2019 at 15:49, on Zulip):

okay yeah I can give an update on what I've done for parallel so far

simulacrum (Oct 03 2019 at 15:52, on Zulip):
simulacrum (Oct 03 2019 at 15:53, on Zulip):

we also will have https://github.com/rust-lang/rust/commit/f0b7b0a9327d3b43aa45a89e90d9785a06059b5a which is a -Zthreads=(num_virtual_cpus) run on perf

simulacrum (Oct 03 2019 at 15:54, on Zulip):

and I plan on putting several hours at least into getting some initial audit reports put together for us to possibly dig into on Monday

simulacrum (Oct 03 2019 at 15:54, on Zulip):

I'm not entirely sure what those will look like quite yet though

simulacrum (Oct 03 2019 at 18:15, on Zulip):

okay performance collection on my end is mostly done, still need to wait for a couple of perf.rlo builds but that's mostly just a waiting game

simulacrum (Oct 04 2019 at 02:32, on Zulip):

Tomorrow I plan to try and dig in and establish an agenda for monday's triage/audit session

simulacrum (Oct 04 2019 at 19:18, on Zulip):

@nikomatsakis write up of potential audit topic for monday: https://hackmd.io/@simulacrum/HJv68ZHuH

I plan to put together a few more of these, though maybe with less detail, over the weekend

nikomatsakis (Oct 04 2019 at 19:32, on Zulip):

ok -- another good one might be the dep-graph code being modified in https://github.com/rust-lang/rust/pull/63756

simulacrum (Oct 04 2019 at 19:36, on Zulip):

okay, I might try and write something up

simulacrum (Oct 04 2019 at 19:36, on Zulip):

that does sound like a good target

nikomatsakis (Oct 07 2019 at 17:54, on Zulip):

hey all, we'll meet today?

Santiago Pastorino (Oct 07 2019 at 17:55, on Zulip):

:+1:

Santiago Pastorino (Oct 07 2019 at 17:56, on Zulip):

@WG-parallel-rustc :point_up:

nikomatsakis (Oct 07 2019 at 17:56, on Zulip):

I'm not sure @simulacrum if you had thoughts on what to discuss in detail

nikomatsakis (Oct 07 2019 at 17:56, on Zulip):

we tossed out a few options looks like

simulacrum (Oct 07 2019 at 17:56, on Zulip):

yeah I was thinking we could take the first couple minutes

simulacrum (Oct 07 2019 at 17:57, on Zulip):

I'm about to hop on Zoom if you want to prediscuss

nikomatsakis (Oct 07 2019 at 17:57, on Zulip):

yep ok

simulacrum (Oct 07 2019 at 17:58, on Zulip):

https://hackmd.io/_1S8_ChMSa2N8mRw6EsGPA

simulacrum (Oct 07 2019 at 18:01, on Zulip):

https://hackmd.io/@simulacrum/r1jsI5U_r

Alex Crichton (Oct 07 2019 at 19:29, on Zulip):

Hey sorry wasn't able to make it today, forgot to mention earlier too

Alex Crichton (Oct 07 2019 at 19:29, on Zulip):

I'll review materials later today and respond to stuff

nikomatsakis (Oct 07 2019 at 19:31, on Zulip):

@Alex Crichton np -- but for next time, we asked if @cuviper can do a bit of digging into our current jobserver setup, and maybe ping you with questions

nikomatsakis (Oct 07 2019 at 19:48, on Zulip):

Video uploading to youtube now

nikomatsakis (Oct 09 2019 at 14:59, on Zulip):

Hey @WG-parallel-rustc, can we maybe change the meeting time? I'd be good with either an hour earlier or later, or perhaps the same time (2pm Eastern) but on wednesday

Santiago Pastorino (Oct 09 2019 at 15:00, on Zulip):

I'm fine with all the options better if it's not wednesday that time though

simulacrum (Oct 09 2019 at 15:04, on Zulip):

Hour later on Monday would work even better for me, earlier isn't ideal. Wednesday is probably not great either, though I could perhaps make it work.

nikomatsakis (Oct 09 2019 at 15:18, on Zulip):

ok let's go to 3pm Eastern on wed?

Alex Crichton (Oct 09 2019 at 15:20, on Zulip):

Isn't 3pm the project sync?

Alex Crichton (Oct 09 2019 at 15:21, on Zulip):

I'm open all Monday though

nikomatsakis (Oct 09 2019 at 15:21, on Zulip):

ok let's go to 3pm Eastern on wed?

oops I meant to say "on monday"

Alex Crichton (Oct 09 2019 at 15:22, on Zulip):

:+1: to 3pm on monday

Santiago Pastorino (Oct 09 2019 at 15:26, on Zulip):

:+1: too

nikomatsakis (Oct 09 2019 at 15:28, on Zulip):

adjusted calendar entry

Alex Crichton (Oct 14 2019 at 17:45, on Zulip):

Hey sorry I won't be able to make today's meeting either, I really do want to help out though! I'll be around for the rest of the week for follow ups (unlike last week)

cuviper (Oct 14 2019 at 18:11, on Zulip):

sorry, I have to miss today too

nikomatsakis (Oct 14 2019 at 18:55, on Zulip):

Hmm

nikomatsakis (Oct 14 2019 at 18:55, on Zulip):

maybe we shoudl cancel this week

nikomatsakis (Oct 14 2019 at 18:55, on Zulip):

I'm dealing with various things right now

nikomatsakis (Oct 14 2019 at 18:55, on Zulip):

Not sure @Santiago Pastorino or @simulacrum what you're up to today?

simulacrum (Oct 14 2019 at 18:56, on Zulip):

I am free, but we can cancel; I think we don't have too much to discuss

Santiago Pastorino (Oct 14 2019 at 18:56, on Zulip):

:+1:

simulacrum (Oct 14 2019 at 18:56, on Zulip):

I believe the main thing is @Santiago Pastorino's doc but that can probably wait a week and would be good to get input from multiple people on I think

simulacrum (Oct 14 2019 at 19:01, on Zulip):

@nikomatsakis let's cancel then

nikomatsakis (Oct 14 2019 at 19:01, on Zulip):

OK

nikomatsakis (Oct 18 2019 at 20:53, on Zulip):

HEy @simulacrum are you around by any chance?

simulacrum (Oct 18 2019 at 20:54, on Zulip):

Yeah, sort of - definitely in ~5 minutes

simulacrum (Oct 18 2019 at 20:54, on Zulip):

About to get in an elevator

nikomatsakis (Oct 18 2019 at 20:54, on Zulip):

@Santiago Pastorino and I are sitting here at RBR and we were just looking over his doc. My impression from random ripgrep'ing around is that there aren't all that many locks to look through

simulacrum (Oct 18 2019 at 20:55, on Zulip):

Yes, this matches my view.

nikomatsakis (Oct 18 2019 at 20:56, on Zulip):

I was thinking that some "juicy" topics for discussion might be:

As an aside, one thing I was saying to him is that, in my mind, the value of this work is not necessarily finding bugs (I don't really expect that many) but rather (a) improving confidence, (b) finding a list of things to refactor, and (c) increasing the bus factor on the code generally =)

nikomatsakis (Oct 18 2019 at 20:57, on Zulip):

anyway I am basically wondering if we want to pick one of those for monday meeting and -- if so -- what prep we want to do

nikomatsakis (Oct 18 2019 at 20:57, on Zulip):

seems good to decide now-ish :)

simulacrum (Oct 18 2019 at 20:58, on Zulip):

I would personally lean towards parallel LLVM stuff -- I think Session, CrateMetadata, etc. are good topics but have less "lead time" needed

simulacrum (Oct 18 2019 at 20:59, on Zulip):

but we would need to make sure @Alex Crichton and maybe @cuviper can make it I think for that

nikomatsakis (Oct 18 2019 at 20:59, on Zulip):

I agree

nikomatsakis (Oct 18 2019 at 20:59, on Zulip):

That's the most "juicy" area I think

simulacrum (Oct 18 2019 at 21:00, on Zulip):

I am uncertain _what_ there is to discuss, beyond just scoping out what we do today, though. In some sense I think this area is perhaps the most ripe for improvements but in conversation with Alex my impression was that it's also essentially blocked on parallel-by-default

nikomatsakis (Oct 18 2019 at 21:00, on Zulip):

well, I at least would enjoy learning more about how it even works today :)

nikomatsakis (Oct 18 2019 at 21:00, on Zulip):

just plain old "knowledge transfer"...

nikomatsakis (Oct 18 2019 at 21:01, on Zulip):

I don't quite understand, for example, why it would be blocked on par by default?

nikomatsakis (Oct 18 2019 at 21:01, on Zulip):

anyway, we're getting kicked out of this room so I'm going to go :)

simulacrum (Oct 18 2019 at 21:01, on Zulip):

okay -- I think that topic is a good fit for monday though

simulacrum (Oct 18 2019 at 21:01, on Zulip):

and if Alex isn't around we can easily fallback on diving into session or so

simulacrum (Oct 18 2019 at 21:01, on Zulip):

that seems 'not hard'

Alex Crichton (Oct 18 2019 at 21:04, on Zulip):

I'd be down for preparing some notes about how llvm/codegen works in parallel by monday

Alex Crichton (Oct 18 2019 at 21:04, on Zulip):

I should be around on monday for sure this time too!

Vadim Petrochenkov (Oct 18 2019 at 21:17, on Zulip):

CrateMetadata type which has random public fields and locks (but maybe @Vadim Petrochenkov's PR does something here?)

The crate store PR doesn't do anything there.
#65026 and #65170 removed some fields from CrateMetadata and documented the remaining ones, nothing lock-related though.

Vadim Petrochenkov (Oct 20 2019 at 14:23, on Zulip):

CrateMetadata type which has random public fields and locks

cc https://github.com/rust-lang/rust/pull/65625#issuecomment-544256417

simulacrum (Oct 21 2019 at 14:33, on Zulip):

I'd be down for preparing some notes about how llvm/codegen works in parallel by monday

@Alex Crichton did you get a chance to prepare these notes? I'd like to look over and maybe come up with some questions

simulacrum (Oct 21 2019 at 14:34, on Zulip):

if not that's fine, we can fallback to @Santiago Pastorino's work on the sync module I think :)

Alex Crichton (Oct 21 2019 at 14:48, on Zulip):

oh heh sure

Alex Crichton (Oct 21 2019 at 14:48, on Zulip):

let me write up some notes

Alex Crichton (Oct 21 2019 at 14:48, on Zulip):

I'll start writing some notes at https://hackmd.io/DnuWl7-PRb6t15GBduGLZw

simulacrum (Oct 21 2019 at 15:02, on Zulip):

Thanks!

simulacrum (Oct 21 2019 at 15:02, on Zulip):

I've added that link to the main planning doc (https://hackmd.io/_1S8_ChMSa2N8mRw6EsGPA?edit)

Zoxc (Oct 21 2019 at 15:06, on Zulip):

@Alex Crichton Step 1 is a bit parallel already

Zoxc (Oct 21 2019 at 15:19, on Zulip):

I also have a branch which makes translation to LLVM parallel. It is quite messy though and a simple parallel loop over CGUs would be much simpler. It does however allow us to use more threads for LLVM optimization. This seems like a useful property since LLVM optimization will scale to higher thread counts than the rest of the compiler. This would only help high core count CPUs and a distributed rustc would perhaps even things out a bit.

Zoxc (Oct 21 2019 at 15:23, on Zulip):

Is there a simpler scheme that would allow us to scale LLVM optimization to more threads than we have in the Rayon thread-pool?
I guess we could spawn another thread-pool and just have a global list of CGUs, we inject a job which just steals from this list unit complete?

Alex Crichton (Oct 21 2019 at 15:36, on Zulip):

ok I've finished writing up the doc -- https://hackmd.io/DnuWl7-PRb6t15GBduGLZw

Alex Crichton (Oct 21 2019 at 15:36, on Zulip):

it may have a bunch of info that's either already known or not of interest to this gropu though

simulacrum (Oct 21 2019 at 17:05, on Zulip):

thanks! I'll take a look in a bit.

simulacrum (Oct 21 2019 at 19:03, on Zulip):

@Alex Crichton meeting?

Alex Crichton (Oct 21 2019 at 19:03, on Zulip):

er my bad

simulacrum (Oct 21 2019 at 19:03, on Zulip):

462 096 851 is the zoom code

simulacrum (Oct 21 2019 at 19:37, on Zulip):

@nikomatsakis or @Alex Crichton https://github.com/rust-lang/rust/blob/857a55b8f0a692cb2304ab7c062d9cc74122e516/src/librustc_interface/util.rs#L209-L214

nikomatsakis (Oct 21 2019 at 20:02, on Zulip):

I also have a branch which makes translation to LLVM parallel. It is quite messy though and a simple parallel loop over CGUs would be much simpler. It does however allow us to use more threads for LLVM optimization. This seems like a useful property since LLVM optimization will scale to higher thread counts than the rest of the compiler. This would only help high core count CPUs and a distributed rustc would perhaps even things out a bit.

@Zoxc I'm not sure what you mean by this actually?

nikomatsakis (Oct 21 2019 at 20:03, on Zulip):

I guess you just meant that we know that each CGU can be executed in parallel? (Whereas, e.g., executing the type-checker for two distinct functions in parallel may wind up with a common dependency they both block on?)

Zoxc (Oct 21 2019 at 20:05, on Zulip):

There's no communication between threads with LLVM optimizing CGUs, and optimizing CGUs takes a while. So no fine grained parallelism like elsewhere in the compiler (queries, interners, etc.)

nikomatsakis (Oct 21 2019 at 21:33, on Zulip):

makes sense

Paul Faria (Oct 28 2019 at 00:06, on Zulip):

I updated the shared doc with a link to my current task's doc. I might be able to attend tomorrow's meeting.

Paul Faria (Oct 28 2019 at 00:08, on Zulip):

Should I also put any of my work under current action items?

nikomatsakis (Oct 28 2019 at 18:12, on Zulip):

I was just going to ask, @WG-parallel-rustc, whether we planned to meet today? I've not had any time to follow up on the things we discussed last time

cuviper (Oct 28 2019 at 18:13, on Zulip):

I'm available to meet, but I don't have any updates to bring

Santiago Pastorino (Oct 28 2019 at 18:14, on Zulip):

I'm not sure what we were going to do exactly, I've scheduled a "pairing" session with @Paul Faria for tonight

Santiago Pastorino (Oct 28 2019 at 18:14, on Zulip):

we need to define what to do exactly but I have some ideas

Santiago Pastorino (Oct 28 2019 at 18:15, on Zulip):

no updates for today rather than the doc I've shared already with sync parallel constructions

Paul Faria (Oct 28 2019 at 18:18, on Zulip):

I've already added my updates to the doc. Nothing that needs to be discussed in a synchronous meeting I think

Santiago Pastorino (Oct 28 2019 at 18:57, on Zulip):

@nikomatsakis is the meeting happening? it ended not being clear to me

simulacrum (Oct 28 2019 at 18:57, on Zulip):

I think we should meet briefly to go over the sync conventions

simulacrum (Oct 28 2019 at 18:57, on Zulip):

It'll probably not take us the full hour but I think it'd be good to at least make sure we're all on the same page

Santiago Pastorino (Oct 28 2019 at 18:58, on Zulip):

seems good to me

simulacrum (Oct 28 2019 at 18:59, on Zulip):

I'd also like to touch on how moving things from Session to TyCtxt went -- not sure if it _needs_ to be sync, but I think it could benefit from that. (i.e., do we want to document this pattern / add it to the guidelines)

simulacrum (Oct 28 2019 at 19:01, on Zulip):

@nikomatsakis ^

nikomatsakis (Oct 28 2019 at 19:04, on Zulip):

hey all

nikomatsakis (Oct 28 2019 at 19:04, on Zulip):

coming

simulacrum (Oct 28 2019 at 19:04, on Zulip):

https://hackmd.io/_1S8_ChMSa2N8mRw6EsGPA

Last update: Nov 17 2019 at 08:20UTC