Stream: t-compiler/wg-parallel-rustc

Topic: refcell-vs-lock


Aaron Turon (Jul 08 2019 at 17:47, on Zulip):

hey @Zoxc! @nikomatsakis asked me to get up to speed on the (prolific!) work you've been doing on parallelizing rustc, so that i can start reviewing your current PRs, and help write more docs etc

Aaron Turon (Jul 08 2019 at 17:47, on Zulip):

i did have a few questions though

Aaron Turon (Jul 08 2019 at 17:48, on Zulip):

in particular, i was wondering whether you put much thought into atomicity and lock order concerns -- i was noticing that a lot of key structures have relatively "fine-grained" locks, in the sense of having lots of fields with individual locks around them

Aaron Turon (Jul 08 2019 at 17:49, on Zulip):

i was thinking that, compared to when these were RefCells, there could be a couple of new problems to worry about:

Aaron Turon (Jul 08 2019 at 17:50, on Zulip):
Aaron Turon (Jul 08 2019 at 17:51, on Zulip):

it's been a bit hard to assess this, though, because the Lock API closely matches the RefCell API, so the diffs aren't showing all the places that reading or mutation occurs

Aaron Turon (Jul 08 2019 at 17:51, on Zulip):
Aaron Turon (Jul 08 2019 at 17:52, on Zulip):

i'm less worried about the latter, because it's easier to audit for, though even in cases that are "fine" right now it seems helpful to have a documented order to avoid deadlocks creeping in

Aaron Turon (Jul 08 2019 at 17:53, on Zulip):

overall this makes me a bit worried that we may hit heisenbugs and other hard-to-track-down issues

nikomatsakis (Jul 08 2019 at 17:53, on Zulip):

(I moved this conversation to its own topic)

Aaron Turon (Jul 08 2019 at 17:53, on Zulip):

thanks :heart:

Aaron Turon (Jul 08 2019 at 17:55, on Zulip):

so, anyway, was wondering what your thinking was on these topics. my own inclination would be to use a combination of audit/documentation (which i've gotten a start on), and perhaps refactoring to reduce the need for locks in the first place (as I've been reading through PR history, there's definitely some low-hanging fruit there). i'm more than happy to take on that work, fwiw

Zoxc (Jul 08 2019 at 18:26, on Zulip):

Changing the borrow_mut/borrow calls to lock would probably be a good idea now, so things look like lock accesses.

Zoxc (Jul 08 2019 at 18:48, on Zulip):

I did look for these issues when I converted the code to using locks (looking at all the accesses for each lock). Documentation won't help much besides possibly prevent regressions here. Testing with rr in chaos mode or with many core CPU might be useful, but testing realistic incremental workloads is tricky. We'd probably need to record actual user workflows for that.

nikomatsakis (Jul 08 2019 at 19:31, on Zulip):

I'm confusing by what you mean by "documentation won't help much" -- like, I don't know what documentation you are referring to, or what you mean by help :)

nikomatsakis (Jul 08 2019 at 23:49, on Zulip):

Hey @Aaron Turon I was talking to @Santiago Pastorino and he'd be interested I think in helping out with refactoring things

Santiago Pastorino (Jul 08 2019 at 23:56, on Zulip):

@Aaron Turon @nikomatsakis definitely

Aaron Turon (Jul 09 2019 at 17:36, on Zulip):

fantastic! and yeah, it seems like the best of all worlds is pushing to eliminate (or simplify) shared state wherever we can

Aaron Turon (Jul 09 2019 at 17:38, on Zulip):

i'll work toward some kind of checklist we can parallelize

centril (Jul 09 2019 at 17:45, on Zulip):

@Aaron Turon :wave:


I think one thing that's missing from the data structures / sync code in the compiler is comments on the unsafe code; would be good to change that as part of documenting

Zoxc (Jul 09 2019 at 17:56, on Zulip):

I wouldn't bother with larger refactorings for code which does not yet use the query system, unless they can be brought into it

Aaron Turon (Jul 09 2019 at 17:58, on Zulip):

Aaron Turon :wave:


I think one thing that's missing from the data structures / sync code in the compiler is comments on the unsafe code; would be good to change that as part of documenting

yeah -- in general this seems like a good excuse to make some improvements not even related to parallelism

Santiago Pastorino (Jul 10 2019 at 16:36, on Zulip):

i'll work toward some kind of checklist we can parallelize

@Aaron Turon yeah, please let me know and we can start talking about things to do :)

Last update: Nov 17 2019 at 07:40UTC