Stream: t-compiler

Topic: steering meeting 2018-11-16


nikomatsakis (Nov 14 2018 at 14:14, on Zulip):

Reminder: Steering meeting planned this Friday. I just posted a comment on internals laying out some thoughts for an agenda. Feedback desired!

cc @T-compiler

varkor (Nov 16 2018 at 13:34, on Zulip):

I can't make this meeting :( I'll try to catch up on everything afterwards!

nikomatsakis (Nov 16 2018 at 14:57, on Zulip):

@T-compiler meeting in a few minutes

nagisa (Nov 16 2018 at 15:00, on Zulip):

/me makes a sandwich first

nikomatsakis (Nov 16 2018 at 15:03, on Zulip):

Now you've made me hungry, @nagisa

nikomatsakis (Nov 16 2018 at 15:04, on Zulip):

Let's get started at 10:05, seems like a nice round number

mw (Nov 16 2018 at 15:04, on Zulip):

o/

davidtwco (Nov 16 2018 at 15:05, on Zulip):

o/

Esteban Küber (Nov 16 2018 at 15:05, on Zulip):

o/

nikomatsakis (Nov 16 2018 at 15:06, on Zulip):

So, I had proposed that we work more on "future planning"

nikomatsakis (Nov 16 2018 at 15:07, on Zulip):

Specifically I wrote

I still hope we can talk about our priorities and planning for the next year. I'd like to do a few things:

however, I don't think we necessarily need to focus just on refactorings of the compiler. I guess the question is sort of "what do we think our major priorities should be and how can we structure ourselves to achieve them"...?

nikomatsakis (Nov 16 2018 at 15:08, on Zulip):

I've been thinking a bit about conway's law (“… organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations")

nikomatsakis (Nov 16 2018 at 15:09, on Zulip):

the flip side I guess is that we should look at what we want and try to organize for it :)

mw (Nov 16 2018 at 15:09, on Zulip):

I think the answer depends (among other things) on what projects we can find interested people for

nikomatsakis (Nov 16 2018 at 15:09, on Zulip):

maybe. I feel like people will come to most projects, if we have available leadership, but that is the trick

nikomatsakis (Nov 16 2018 at 15:10, on Zulip):

@Alex Crichton had an interesting post talking about another angle on compiler performance as well

mw (Nov 16 2018 at 15:10, on Zulip):

ok, replace "people" with "leaders" in the above :)

nikomatsakis (Nov 16 2018 at 15:10, on Zulip):

the idea of building more distributed caching

nikomatsakis (Nov 16 2018 at 15:10, on Zulip):

ok, replace "people" with "leaders" in the above :)

I think the trick here is also that we can't expect to just figure this out up front, to some extent. It's kind of a cycle. But I guess we can find a few things and hope to expand from there

nikomatsakis (Nov 16 2018 at 15:11, on Zulip):

(We talked before about having working groups with 2 leaders, an idea which I like)

nikomatsakis (Nov 16 2018 at 15:11, on Zulip):

but it seems like starting from "people" is tricky, since we're an open organization etc, so it seems better to somehow explore the "tech side" a bit first?

nikomatsakis (Nov 16 2018 at 15:12, on Zulip):

anyway, one thought I had when it comes to compiler perf was trying to talk about (a) if we have big ideas that could be transformative and (b) how we can evaluate their potential

mw (Nov 16 2018 at 15:12, on Zulip):

sure, we need some details on the individual projects before people can know if they'd want to lead them

nagisa (Nov 16 2018 at 15:13, on Zulip):

I have things I would love to work on/lead, but I’m worried about my time

nikomatsakis (Nov 16 2018 at 15:13, on Zulip):

also a factor

nagisa (Nov 16 2018 at 15:13, on Zulip):

(I’ve already talked about this with @nikomatsakis last all-hands)

mw (Nov 16 2018 at 15:14, on Zulip):

this would be somewhat mitigated by having more than one person in the lead

nikomatsakis (Nov 16 2018 at 15:14, on Zulip):

at the highest level, I think our ideas for "radically transforming" total compilation time are roughly this:

nikomatsakis (Nov 16 2018 at 15:16, on Zulip):

I'm not sure if the second "do what we do better" category is useful

nikomatsakis (Nov 16 2018 at 15:16, on Zulip):

I guess maybe parallel compilation is along that theme too :)

mw (Nov 16 2018 at 15:16, on Zulip):

there might be another category, specific to incr. comp.: cache work that we don't cache now

nikomatsakis (Nov 16 2018 at 15:17, on Zulip):

interesting

mw (Nov 16 2018 at 15:17, on Zulip):

e.g. when a compilation session fails

nikomatsakis (Nov 16 2018 at 15:17, on Zulip):

I was thinking that RLS etc is sort of an orthogonal category

nagisa (Nov 16 2018 at 15:17, on Zulip):

Just thought of it, but a nice way to visualise our query engine and time everything takes would be nice. I have some inspired ideas on it…

nikomatsakis (Nov 16 2018 at 15:17, on Zulip):

(though incremental is not specific to rLS)

nagisa (Nov 16 2018 at 15:18, on Zulip):

At the very least to more easily deduce where bottlenecks are, etc.

Alex Crichton (Nov 16 2018 at 15:18, on Zulip):

FWIW I've found a useful rule of thumb to be "if this thing I'm optimizing takes zero time, will that radically improve things?" -- where an example is that if type checking took zero time in rustc I feel like it wouldn't be that radically different an experience than we have today

mw (Nov 16 2018 at 15:18, on Zulip):

in my mental venn diagram RLS and batch-compilation intersect at the query engine

nikomatsakis (Nov 16 2018 at 15:19, on Zulip):

FWIW I've found a useful rule of thumb to be "if this thing I'm optimizing takes zero time, will that radically improve things?" -- where an example is that if type checking took zero time in rustc I feel like it wouldn't be that radically different an experience than we have today

I .. am not sure

nikomatsakis (Nov 16 2018 at 15:19, on Zulip):

I also find that useful

Alex Crichton (Nov 16 2018 at 15:19, on Zulip):

(something to measure at least)

nikomatsakis (Nov 16 2018 at 15:19, on Zulip):

I think in general this comes back to having a better picture of our profiles

nikomatsakis (Nov 16 2018 at 15:19, on Zulip):

@Wesley Wiser did great work here but it hasn't made it full circle into perf yet :(

nikomatsakis (Nov 16 2018 at 15:19, on Zulip):

in particular, I think that for some "use cases"/"scenarios" type-check etc can be quite significant

nikomatsakis (Nov 16 2018 at 15:20, on Zulip):

I also suspect we won't find a "silver bullet", but more we'll do 2 or 3 things that are high impact

nikomatsakis (Nov 16 2018 at 15:20, on Zulip):

but together are better

Alex Crichton (Nov 16 2018 at 15:20, on Zulip):

true yeah

nikomatsakis (Nov 16 2018 at 15:20, on Zulip):

that said, I do agree that type-checking alone is not going to be enough

nikomatsakis (Nov 16 2018 at 15:20, on Zulip):

in my mental venn diagram RLS and batch-compilation intersect at the query engine

I'm not 100% sure what this means, @mw

nikomatsakis (Nov 16 2018 at 15:21, on Zulip):

I guess just that they both build on it?

mw (Nov 16 2018 at 15:21, on Zulip):

I'm not 100% sure what this means, @mw

that the central evaluation model must support both, but the goals are a bit at a tension

nikomatsakis (Nov 16 2018 at 15:21, on Zulip):

ps @nagisa totally agree about having more ability to visualize queries etc

nagisa (Nov 16 2018 at 15:22, on Zulip):

That would be https://github.com/rust-lang/rust/issues/53631, and related issues. Turns out I had already documented some of my ideas here.

nikomatsakis (Nov 16 2018 at 15:22, on Zulip):

yeah, I guess I think that if we were going to engineer for a radically better RLS experience, the priorities would be very different

Zack M. Davis (Nov 16 2018 at 15:22, on Zulip):

I think there was speculation that killing off NodeId would help with incremental (but I don't know the details/have the measurements to know whether it was a priority)? https://github.com/rust-lang/rust/issues/50928

nikomatsakis (Nov 16 2018 at 15:22, on Zulip):

interesting

nikomatsakis (Nov 16 2018 at 15:22, on Zulip):

it's true, NodeId is not incremental friendly

mw (Nov 16 2018 at 15:23, on Zulip):

that's mostly clean up at this point though

mw (Nov 16 2018 at 15:23, on Zulip):

i.e. it won't change performance, just code quality

nikomatsakis (Nov 16 2018 at 15:23, on Zulip):

c'mon everybody loves convering from nodeid to hirid to defid to hirid and back

nikomatsakis (Nov 16 2018 at 15:24, on Zulip):

(improving code quality is also a way to improve contributor experience and improve our overall velocity)

mw (Nov 16 2018 at 15:24, on Zulip):

it's what I do for fun on the weekend

nikomatsakis (Nov 16 2018 at 15:24, on Zulip):

well so in terms of my bullet list above, I guess my first question is: does anybody have any ideas that don't fit those categories?

mw (Nov 16 2018 at 15:24, on Zulip):

I'm not saying we shouldn't do it :)

nikomatsakis (Nov 16 2018 at 15:24, on Zulip):

perhaps they are so broad it's not possible :)

nikomatsakis (Nov 16 2018 at 15:24, on Zulip):

if not, maybe we should drill a bit more into one of them?

mw (Nov 16 2018 at 15:25, on Zulip):

incremental compilation wouldn't fit

nikomatsakis (Nov 16 2018 at 15:25, on Zulip):

I propose for the purposes of this meeting, we focus not on RLS btw, but rather "batch performance"

nikomatsakis (Nov 16 2018 at 15:25, on Zulip):

yeah =) I meant the list to be specific to batch compilation

eddyb (Nov 16 2018 at 15:25, on Zulip):

aaaah hiii (/me was paying attention to rebase and missed the notification)

mw (Nov 16 2018 at 15:25, on Zulip):

well, maybe into "avoid work"

nikomatsakis (Nov 16 2018 at 15:25, on Zulip):

I'd like to circle back to RLS at some point

nikomatsakis (Nov 16 2018 at 15:26, on Zulip):

but it seems like we're already 25 min in and we have to start somewhere :)

nagisa (Nov 16 2018 at 15:26, on Zulip):

@nikomatsakis there could be ways to improve time in the backend (linking, LTO) as well, but I haven’t looked too much at it.

nagisa (Nov 16 2018 at 15:26, on Zulip):

at the very least it seems to me that clang’s LTO is doing ThinLTO much better than we, somehow

Esteban Küber (Nov 16 2018 at 15:26, on Zulip):

I think there's one thing that isn't explicitly covered, which affects UX. We currently are average in continuing past some points for errors. We continue after some parse errors, and we continue after some type check errors, but the experience of making the type checker happy and then being inundated with borrow checker errors is still the norm.

nikomatsakis (Nov 16 2018 at 15:27, on Zulip):

interesting @nagisa ; dsymutil is something that Anthony Jones at Mozilla always points out as a surprising time consumer during linking on mac

nikomatsakis (Nov 16 2018 at 15:27, on Zulip):

@Esteban Küber very good point, that's particularly problemetic on RLS, I think

mw (Nov 16 2018 at 15:27, on Zulip):

also defaulting to LLD would help with linking heavy codebases

nikomatsakis (Nov 16 2018 at 15:27, on Zulip):

also, the new lint ordering is bad

nikomatsakis (Nov 16 2018 at 15:27, on Zulip):

that is, I often get warnings about e.g. unused imports

Esteban Küber (Nov 16 2018 at 15:27, on Zulip):

I think that is tied to parallelizing work, if the unit of work are smaller it'd give us more ergonomic handling of those cases.

nikomatsakis (Nov 16 2018 at 15:27, on Zulip):

that are only unused because of typos or type-check errors later

eddyb (Nov 16 2018 at 15:27, on Zulip):

I have previously said we should look for all the fatal calls and just remove them

nikomatsakis (Nov 16 2018 at 15:27, on Zulip):

so that's good but I feel like another category of goal

nikomatsakis (Nov 16 2018 at 15:28, on Zulip):

more in the RLS camp, though perhaps distinct

nikomatsakis (Nov 16 2018 at 15:28, on Zulip):

still, definitely worth noting

nikomatsakis (Nov 16 2018 at 15:28, on Zulip):

(I also wanted to talk more about error messages at some point, perhaps a "UX" pillar is a good thing to look at)

nikomatsakis (Nov 16 2018 at 15:28, on Zulip):

interesting

eddyb (Nov 16 2018 at 15:28, on Zulip):

there's no real need to stop the compiler if there's anything else it could be doing, even across crates (but we don't have the infra for that yet)

nikomatsakis (Nov 16 2018 at 15:28, on Zulip):

(I guess I'll come back afterwards and make minutes and note this stuff somewhere)

nikomatsakis (Nov 16 2018 at 15:29, on Zulip):

so yeah I was going to propose we talk a bit more about this bullet point:

eddyb (Nov 16 2018 at 15:29, on Zulip):

if a part of the compiler cannot handle incomplete/erroneous information, it should react to that itself and skip itself

nikomatsakis (Nov 16 2018 at 15:29, on Zulip):

(agreed)

eddyb (Nov 16 2018 at 15:30, on Zulip):

(as opposed to preemptively stopping everything to avoid that situation)

nagisa (Nov 16 2018 at 15:30, on Zulip):

multi-crate compilation model

as long as we make it work better than C’s multi-file compilation, we win by default!

Esteban Küber (Nov 16 2018 at 15:30, on Zulip):

That's the direction I feel we've been going in, it's just a lot of work :-/

nikomatsakis (Nov 16 2018 at 15:30, on Zulip):

I could see that being a "working group"

nikomatsakis (Nov 16 2018 at 15:31, on Zulip):

i.e., come up with a design, which probably involves more use of sentinel values etc, and push it through

eddyb (Nov 16 2018 at 15:31, on Zulip):

AFAIK the most work would be updating all the tests that would cause all sorts of warnings/errors

mw (Nov 16 2018 at 15:31, on Zulip):

the working group that never stops working

nikomatsakis (Nov 16 2018 at 15:31, on Zulip):

it .. feels like it has an end

nikomatsakis (Nov 16 2018 at 15:31, on Zulip):

at some point, there are no more fatal calls :)

eddyb (Nov 16 2018 at 15:31, on Zulip):

there aren't even that many today!

mw (Nov 16 2018 at 15:31, on Zulip):

the working group that doesn't stop working even if it fails

eddyb (Nov 16 2018 at 15:31, on Zulip):

most of it is the driver stopping if any errors were reported

nikomatsakis (Nov 16 2018 at 15:31, on Zulip):

I...get the pun now

nikomatsakis (Nov 16 2018 at 15:32, on Zulip):

/me groans

mw (Nov 16 2018 at 15:32, on Zulip):

so I think polymorphization is pretty straightforward

nikomatsakis (Nov 16 2018 at 15:32, on Zulip):

anyway, so, we have some estimates of mir-only rlibs from @mw 's work -- one thing I'd really like to see us do is to try and estimate how much opportunity there is for what I'm calling "polymorphization" -- basically identifying cases where we can avoid instantiating MIR for each value of its type parameters.

nikomatsakis (Nov 16 2018 at 15:32, on Zulip):

we've seen some stabs here

mw (Nov 16 2018 at 15:32, on Zulip):

i.e. it doesn't depend on other things

nikomatsakis (Nov 16 2018 at 15:32, on Zulip):

there are many knobs to turn

nagisa (Nov 16 2018 at 15:33, on Zulip):

"polymorphization" to coallesce monomorphized instances that have the same code

is this referring to the same thing as LLVM’s merge-functions pass?

nikomatsakis (Nov 16 2018 at 15:33, on Zulip):

yes, but better I would think

mw (Nov 16 2018 at 15:33, on Zulip):

multi-crate + mir-only rlibs interact, I think

eddyb (Nov 16 2018 at 15:33, on Zulip):

I think the best thing to do is to do what we do in miri

nikomatsakis (Nov 16 2018 at 15:33, on Zulip):

well, that was one of my questions actually

eddyb (Nov 16 2018 at 15:33, on Zulip):

miri can evaluate still-polymorphic code

nikomatsakis (Nov 16 2018 at 15:33, on Zulip):

I don't know whether there are places that could be mergerd, but merge-functions doesn't see it

nikomatsakis (Nov 16 2018 at 15:33, on Zulip):

if I understand you, @eddyb, that's on the far end of the spectrum,

eddyb (Nov 16 2018 at 15:34, on Zulip):

(it will error if e.g. layout needs to know a type parameter, but in the absence of that, it will keep working)

nikomatsakis (Nov 16 2018 at 15:34, on Zulip):

i.e., we identify not only cases where the static code is the same, but cases where we could thread some dictionaries etc

eddyb (Nov 16 2018 at 15:34, on Zulip):

no, it's trivial: just make sure you don't choke on params

nikomatsakis (Nov 16 2018 at 15:34, on Zulip):

that..is not trivial when generating code

eddyb (Nov 16 2018 at 15:34, on Zulip):

miri doesn't do any dictionary passing

nagisa (Nov 16 2018 at 15:34, on Zulip):

well doing it in MIR would help in two respects: we could merge functions while they are still polymorphic (avoiding redoing some work) and we could then merge the monomorphisations as well (avoiding extra work of translating to LLVM IR and waiting for LLVM to optimize)

nikomatsakis (Nov 16 2018 at 15:34, on Zulip):

maybe we're talking past each other

eddyb (Nov 16 2018 at 15:34, on Zulip):

it just doesn't die on parametrized code if the evaluation doesn't depend on parameters

nikomatsakis (Nov 16 2018 at 15:34, on Zulip):

ok, the if the tricky bit

nikomatsakis (Nov 16 2018 at 15:34, on Zulip):

that is, I thnk the first level is:

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

figuring out when you depend on the parameters and how

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

so that you can share the same LLVM code

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

the next level is:

eddyb (Nov 16 2018 at 15:35, on Zulip):

e.g. miri can evaluate Vec::new() polymorphically

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

figuring out when you could make the LLVM code take dictionaries

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

sure, that's the first (and easier) level

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

I would presume we will shoot for the first level to start

eddyb (Nov 16 2018 at 15:35, on Zulip):

we can fix codegen to do the same, then have a simple analysis

nikomatsakis (Nov 16 2018 at 15:35, on Zulip):

yes, that is what I am proposing :)

eddyb (Nov 16 2018 at 15:35, on Zulip):

previous attempts failed because they introduced a much more complex mechanism

eddyb (Nov 16 2018 at 15:36, on Zulip):

than making codegen not choke on still-polymorphic code

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

I was further going to propose

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

that we start by trying to do the analysis

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

so we can estimate how effective it is

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

(that is of course most of the work)

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

but that's ok

eddyb (Nov 16 2018 at 15:36, on Zulip):

+100 for more statistics

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

also, @nagisa, something I was wondering about

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

(catching up with what you said)

eddyb (Nov 16 2018 at 15:36, on Zulip):

the 'tcx/'gcx split arose from statistics :P

eddyb (Nov 16 2018 at 15:36, on Zulip):

which I'm not even sure hold anymore

nikomatsakis (Nov 16 2018 at 15:36, on Zulip):

in LLVM, are there cases where we generate nominal types that prevent merge func ?

nikomatsakis (Nov 16 2018 at 15:37, on Zulip):

e.g. if you have two structs that have the same layout

nikomatsakis (Nov 16 2018 at 15:37, on Zulip):

will they get merged?

nagisa (Nov 16 2018 at 15:37, on Zulip):

I never checked, but I doubt it would be an issue.

eddyb (Nov 16 2018 at 15:37, on Zulip):

we generate nominal types based on the Rust type identity and I doubt LLVM merges them

nikomatsakis (Nov 16 2018 at 15:37, on Zulip):

we generate nominal types based on the Rust type identity and I doubt LLVM merges them

this was roughly my assumption, but I am not sure

nikomatsakis (Nov 16 2018 at 15:38, on Zulip):

basically I'm trying to figure out

eddyb (Nov 16 2018 at 15:38, on Zulip):

especially since LLVM still has those accursed pointee types

nikomatsakis (Nov 16 2018 at 15:38, on Zulip):

how much more merging we would be able to do than LLVM can do -- obviously doing it earlier is also just better

mw (Nov 16 2018 at 15:38, on Zulip):

debuginfo is a bit of a problem here since it needs monomorphic code generally

nikomatsakis (Nov 16 2018 at 15:38, on Zulip):

hmm yes -- also debuginfo is a good thing to cover in the "UX" area

nagisa (Nov 16 2018 at 15:38, on Zulip):

@mw debuginfo usually doesn’t work with merged functions either way

eddyb (Nov 16 2018 at 15:39, on Zulip):

@mw we can assume that if the code doesn't e.g. access through pointers, then debuginfo can be opaque about it

nikomatsakis (Nov 16 2018 at 15:39, on Zulip):

PS: For the minutes, I am going to pay special attention to any message that has :point_up: on it

nikomatsakis (Nov 16 2018 at 15:39, on Zulip):

so if you see something you think we should definitely remember to note, add that emoji :)

eddyb (Nov 16 2018 at 15:39, on Zulip):

oh this reminds me

nagisa (Nov 16 2018 at 15:40, on Zulip):

@eddyb could we preemptively get rid of pointee types by making every pointer a *{} and then ptrcasting before each load/store?

eddyb (Nov 16 2018 at 15:40, on Zulip):

@nagisa sort of, but you're slowing down LLVM

nikomatsakis (Nov 16 2018 at 15:40, on Zulip):

we can assume that if the code doesn't e.g. access through pointers, then debuginfo can be opaque about it

@eddyb are you sure? it seems like when running gdb, you might want to print things the code doesn't directly touch

nikomatsakis (Nov 16 2018 at 15:40, on Zulip):

I feel like we should talk to @Tom Tromey about this

eddyb (Nov 16 2018 at 15:40, on Zulip):

https://github.com/rust-lang-nursery/rls/issues/6#issuecomment-436942412

nikomatsakis (Nov 16 2018 at 15:40, on Zulip):

I mean we are literally editing gdb/lldb to make them understand Rust

mw (Nov 16 2018 at 15:40, on Zulip):

it's at least not a trivial yes or no

nikomatsakis (Nov 16 2018 at 15:40, on Zulip):

perhaps they can understand this too

eddyb (Nov 16 2018 at 15:41, on Zulip):

so I won't make anyone read that, but, it vaguely mentions that we wanted to do a bunch for 2018

eddyb (Nov 16 2018 at 15:41, on Zulip):

more specifically, I was supposed to look into things like moving incremental quieres earlier than HIR

eddyb (Nov 16 2018 at 15:42, on Zulip):

do we have any plans for making sure these things... happen?

eddyb (Nov 16 2018 at 15:42, on Zulip):

and how we should prioritize low-hanging-fruit vs large-scale refactors?

nikomatsakis (Nov 16 2018 at 15:42, on Zulip):

(note for later: one other thing I think we could measure-- I hypothesize that when you have a lot of dependencies from crates.io, much of the code within is not used by your crate, but we still force you type-check and build it, at least the first time. It might be nice if we didn't, or at least if we did it very late in the cycle. We could measure how much that would matter.)

nikomatsakis (Nov 16 2018 at 15:43, on Zulip):

and how we should prioritize low-hanging-fruit vs large-scale refactors?

that is precisely what I want us to be talking about :)

nikomatsakis (Nov 16 2018 at 15:43, on Zulip):

basically I think the answer is "yes" we have plans -- and we are trying to make them

nikomatsakis (Nov 16 2018 at 15:44, on Zulip):

stepping back one bit, we are trying in general within Rust to encourage the teams to set goals for themselves and organize to achieve them (and then track and report on our progress)

mw (Nov 16 2018 at 15:44, on Zulip):

having well-thought out plans will certainly help with convincing whomever is paying you

eddyb (Nov 16 2018 at 15:44, on Zulip):

bandwidth hasn't gotten much better, and it's also hard to keep track of everything happening with incremental (at various scales)

nikomatsakis (Nov 16 2018 at 15:44, on Zulip):

one thing I would love to have @eddyb is a more complete diagram listing the kinds of big refactors we think we need and what each one enables, how they interact

eddyb (Nov 16 2018 at 15:44, on Zulip):

right, that would be very useful

nikomatsakis (Nov 16 2018 at 15:44, on Zulip):

it seems like "end-to-end query" is an important step

nikomatsakis (Nov 16 2018 at 15:45, on Zulip):

e.g., it will enable more parallelization

nikomatsakis (Nov 16 2018 at 15:45, on Zulip):

more incremental

nikomatsakis (Nov 16 2018 at 15:45, on Zulip):

maybe not right away, as we also have to make the queries "smaller"

nikomatsakis (Nov 16 2018 at 15:45, on Zulip):

but eventually :)

nikomatsakis (Nov 16 2018 at 15:45, on Zulip):

I also believe we have piled on more tech debt than is good for us

mw (Nov 16 2018 at 15:45, on Zulip):

I'm not entirely convinced. we spend almost no time before the query system

nikomatsakis (Nov 16 2018 at 15:46, on Zulip):

so while I want us to focus on results, I think we need to also do cleanup

nagisa (Nov 16 2018 at 15:46, on Zulip):

well from what I can see some cleanup is happening all the time

eddyb (Nov 16 2018 at 15:46, on Zulip):

wasn't e.g. librustc or libsyntax spending like 5 seconds in parsing and expansion?

nagisa (Nov 16 2018 at 15:46, on Zulip):

e.g. the LLVM traitification thing.

nikomatsakis (Nov 16 2018 at 15:46, on Zulip):

I'm not entirely convinced. we spend almost no time before the query system

fair point

nikomatsakis (Nov 16 2018 at 15:47, on Zulip):

well, @mw, I guess it depends on our goals again -- e.g., if we ever want to get to faster updates for RLS, it's clearly key

eddyb (Nov 16 2018 at 15:47, on Zulip):

and RLS wants, ideally, milliseconds

nagisa (Nov 16 2018 at 15:47, on Zulip):

also we are very likely to increase in time in early frontend if we make our parsing more complex (GLL) etc.

mw (Nov 16 2018 at 15:47, on Zulip):

or put more constructively: I support any initiative that collects numbers on where we should optimize

nagisa (Nov 16 2018 at 15:47, on Zulip):

which seems to be the general direction the language is heading towards

nikomatsakis (Nov 16 2018 at 15:47, on Zulip):

well from what I can see some cleanup is happening all the time

some, yes. but it's never been an explicit goal for us.

eddyb (Nov 16 2018 at 15:47, on Zulip):

@nagisa I wouldn't want to introduce GLL unless it helps with parse times

eddyb (Nov 16 2018 at 15:47, on Zulip):

specifically, GLL is one of the ways we can get fine-grained incremental parsing

nikomatsakis (Nov 16 2018 at 15:47, on Zulip):

or put more constructively: I support any initiative that collects numbers on where we should optimize

ok so maybe this should be one our top priorities for next steering meeting

nagisa (Nov 16 2018 at 15:47, on Zulip):

@eddyb you might end up having no choice but to introduce it regardless :slight_smile:

eddyb (Nov 16 2018 at 15:47, on Zulip):

should I assume that's not a goal for 2019 :)?

nikomatsakis (Nov 16 2018 at 15:48, on Zulip):

maybe we can stop here, actually, and try to collect our thoughts in last 10 mionutes

nikomatsakis (Nov 16 2018 at 15:48, on Zulip):

specifically:

nikomatsakis (Nov 16 2018 at 15:48, on Zulip):

what are some action items and information we can gather to help us?

Esteban Küber (Nov 16 2018 at 15:48, on Zulip):

@eddyb Ideally, the RLS could have a way to do local type checking on the current file, while at the same time kicking off a global incremental compilation. That way you can type/borrow check with information you have, while letting the full check happen later once the actual compilation finishes.

Esteban Küber (Nov 16 2018 at 15:48, on Zulip):

I don't think that's something we're geared to do.

nikomatsakis (Nov 16 2018 at 15:48, on Zulip):

one I see is:

eddyb (Nov 16 2018 at 15:48, on Zulip):

@Esteban Küber it actually is

nikomatsakis (Nov 16 2018 at 15:48, on Zulip):

another I see is:

eddyb (Nov 16 2018 at 15:49, on Zulip):

"local type-checking" just means "run the type-checking query for all the items in this module"

eddyb (Nov 16 2018 at 15:49, on Zulip):

there's no reason not to do something more local, even, depending on the user's viewport

nikomatsakis (Nov 16 2018 at 15:49, on Zulip):

perhaps:

mw (Nov 16 2018 at 15:50, on Zulip):

I volunteer for collecting some numbers on how parallel queries performs these days

nikomatsakis (Nov 16 2018 at 15:50, on Zulip):

should I assume that's not a goal for 2019 :)?

in all seriousness, I think it is a legit question whether "fine-grained incremental parsing" should be a goal for 2019 or not -- it seems to depend on whether we are prioritizing RLS or batch

eddyb (Nov 16 2018 at 15:50, on Zulip):

@nikomatsakis and someone needs to draw the graph of refactor dependencies

nikomatsakis (Nov 16 2018 at 15:50, on Zulip):

I volunteer for collecting some numbers on how parallel queries performs these days

ah great

Esteban Küber (Nov 16 2018 at 15:50, on Zulip):

@eddyb glad to hear that, and it makes sense. I guess it'd be a factor of exposing that to the RLS.

nikomatsakis (Nov 16 2018 at 15:50, on Zulip):

so another:

eddyb (Nov 16 2018 at 15:51, on Zulip):

@Esteban Küber TBH RLS could do it today if it was properly integrated and not siphoning data through a side-channel

eddyb (Nov 16 2018 at 15:51, on Zulip):

so it's more of an RLS design problem

nikomatsakis (Nov 16 2018 at 15:51, on Zulip):

ok, @eddyb, nice

(fwiw I've taken a few stabs at this but will try again)

nikomatsakis (Nov 16 2018 at 15:51, on Zulip):

let me collect those actions items into a hackmd doc

nikomatsakis (Nov 16 2018 at 15:51, on Zulip):

should I just make one for each steering meeting? and a master one? is hackmd our "etherpad-like thing of choice"?

nikomatsakis (Nov 16 2018 at 15:51, on Zulip):

maybe we should make a github repository

nikomatsakis (Nov 16 2018 at 15:51, on Zulip):

it seems to be the hip thing to do these days

eddyb (Nov 16 2018 at 15:52, on Zulip):

frankly, technical debt should be work items just as important as new features

nikomatsakis (Nov 16 2018 at 15:52, on Zulip):

it could host the rustc-guide etc

nikomatsakis (Nov 16 2018 at 15:52, on Zulip):

frankly, technical debt should be work items just as important as new features

I agree

eddyb (Nov 16 2018 at 15:52, on Zulip):

having an official plan that we could get people working on full-time would be great

Esteban Küber (Nov 16 2018 at 15:53, on Zulip):

One thing that I've hit my head against, most recently trying to add an integrated lint in the libsyntax, is that the crate dependency graph is a bit coarse. When doing refactorings I often end up with circular dependencies that would ideally be fixed by making those crates smaller. Is that something that we should be doing proactively (smaller loosely tied crates) rather than reactively?

Esteban Küber (Nov 16 2018 at 15:53, on Zulip):

It doesn't happen often, but I feel this is something that will come up with the proposed refactors.

Esteban Küber (Nov 16 2018 at 15:53, on Zulip):

And ties back to a separate conversation about moving some crates off repo.

eddyb (Nov 16 2018 at 15:53, on Zulip):

as in, being able to say, at the level of rust-lang/compiler, that you can't have these X, Y, Z nice things, without certain prerequisites, and have e.g. managers not involved with the development get behind that

eddyb (Nov 16 2018 at 15:54, on Zulip):

@Esteban Küber the solution is almost never to split crates up

eddyb (Nov 16 2018 at 15:54, on Zulip):

in fact we can probably unsplit several crates nowadays, the split is no longer meaningful

eddyb (Nov 16 2018 at 15:55, on Zulip):

reusable components, sure, but part of the compiler mainly need an "API root" crate, e.g. librustc and its query engine

eddyb (Nov 16 2018 at 15:55, on Zulip):

(speaking of reusable components, https://github.com/rust-lang/rust/issues/55993)

nikomatsakis (Nov 16 2018 at 15:56, on Zulip):

One thing that I've hit my head against, most recently trying to add an integrated lint in the libsyntax, is that the crate dependency graph is a bit coarse. When doing refactorings I often end up with circular dependencies that would ideally be fixed by making those crates smaller. Is that something that we should be doing proactively (smaller loosely tied crates) rather than reactively?

I would like to start refactoring the compiler into smaller crates, yes. I used to feel differently but my opinions have changed. I'd also like us to pay careful attention to the interfaces between those crates, and see if we can do better.

nikomatsakis (Nov 16 2018 at 15:56, on Zulip):

but I think this is something we should talk about

nikomatsakis (Nov 16 2018 at 15:56, on Zulip):

the current divisions are fairly random

eddyb (Nov 16 2018 at 15:57, on Zulip):

I think we should have less crates, but also more structured

nikomatsakis (Nov 16 2018 at 15:58, on Zulip):

reusable components, sure, but part of the compiler mainly need an "API root" crate, e.g. librustc and its query engine

(note that the salsa design explicitly avoids having a root place where all queries are declared -- rather, each crate declares traits that are composed at the very end)

eddyb (Nov 16 2018 at 15:58, on Zulip):

(e.g. librustc_syntax / librustc_front)

nikomatsakis (Nov 16 2018 at 15:58, on Zulip):

which is an alternative

eddyb (Nov 16 2018 at 15:58, on Zulip):

@nikomatsakis how much overhead does that have?

nikomatsakis (Nov 16 2018 at 15:58, on Zulip):

but I've got to run, I'm going to try and draw up the bullet points though

nikomatsakis (Nov 16 2018 at 15:58, on Zulip):

@nikomatsakis how much overhead does that have?

zero?

nikomatsakis (Nov 16 2018 at 15:58, on Zulip):

not sure what you mean by overhead though, I assume runtime overhead?

nikomatsakis (Nov 16 2018 at 15:58, on Zulip):

everything is built against impl Trait, so statically dispatched

mw (Nov 16 2018 at 15:58, on Zulip):

it will cause a lot of the LLVM work to be deferred to the final crate

nikomatsakis (Nov 16 2018 at 15:59, on Zulip):

yes

nikomatsakis (Nov 16 2018 at 15:59, on Zulip):

but .. you're going to compile anyway

eddyb (Nov 16 2018 at 15:59, on Zulip):

is everything generic over traits?

nikomatsakis (Nov 16 2018 at 15:59, on Zulip):

everything that needs to interact with the query system, yes

eddyb (Nov 16 2018 at 15:59, on Zulip):

that... won't work in rustc

nikomatsakis (Nov 16 2018 at 15:59, on Zulip):

I'd like though to work on making those traits dyn capable, that'd be nice

nikomatsakis (Nov 16 2018 at 15:59, on Zulip):

they're not now, I think they could be

eddyb (Nov 16 2018 at 15:59, on Zulip):

not without more work than everything we've already thought about doing

nikomatsakis (Nov 16 2018 at 16:00, on Zulip):

(as I said, I don't see why they couldn't be dyn capable traits, but it would take some refactorings to do it)

nikomatsakis (Nov 16 2018 at 16:00, on Zulip):

we already dispatch queries in rustc through fn pointers anyway, so no difference there

nikomatsakis (Nov 16 2018 at 16:00, on Zulip):

anyway, I'm not proposing that, just saying that it's worth considering

eddyb (Nov 16 2018 at 16:00, on Zulip):

the important bit is that the set of fn pointers is fixed

eddyb (Nov 16 2018 at 16:00, on Zulip):

you need extra indirection if you want it to be pluggable

eddyb (Nov 16 2018 at 16:01, on Zulip):

at best an array, at worst an AnyMap

nikomatsakis (Nov 16 2018 at 16:01, on Zulip):

that is one direction; there are others. but let's defer this, as we are out of time, and I have to run :)

mw (Nov 16 2018 at 16:01, on Zulip):

do we need runtime pluggability?

eddyb (Nov 16 2018 at 16:01, on Zulip):

@nikomatsakis oh if you're making a list, you can throw in there "type-check on MIR instead of HIR" (i.e. don't run inference separate to the NLL type-checking), that might be an interesting aspect. and in general, moving away from doing things on the HIR

mw (Nov 16 2018 at 16:01, on Zulip):

:wave:

eddyb (Nov 16 2018 at 16:02, on Zulip):

@mw we need cross-crate pluggability

nikomatsakis (Nov 16 2018 at 16:02, on Zulip):

I'll try to draw up some notes. I think I might make a rust-compiler-team repository, as well, to start hosting these minutes, design documents, and other things

davidtwco (Nov 16 2018 at 16:02, on Zulip):

I've not had much to contribute to this discussion, but I really enjoyed reading it, thanks everyone.

nikomatsakis (Nov 16 2018 at 16:03, on Zulip):

@eddyb do you mean do all type inference on MIR? I'm not sure how that would work, as we need to do type-inference to even construct the MIR...? If you meant region checking and borrow checking, that's already on there.

eddyb (Nov 16 2018 at 16:03, on Zulip):

I think I wrote to you about this... elsewhere, idk where

eddyb (Nov 16 2018 at 16:03, on Zulip):

you can encode uninferred HIR into MIR without much effort

eddyb (Nov 16 2018 at 16:04, on Zulip):

most of the control-flow doesn't even depend on type inference

nagisa (Nov 16 2018 at 16:05, on Zulip):

Making inference lazy would make it work all the time without necessarily requiring full inference :slight_smile:

nagisa (Nov 16 2018 at 16:06, on Zulip):

Didn’t we have an item on making more things lazy?

eddyb (Nov 16 2018 at 16:06, on Zulip):

I don't mean that :P

nagisa (Nov 16 2018 at 16:07, on Zulip):

Sure, but I believe there are some language constructs that could make cfg depend on inference/types

nagisa (Nov 16 2018 at 16:07, on Zulip):

(associated constants come to mind)

eddyb (Nov 16 2018 at 16:08, on Zulip):

it's only the unwind path that can change, last I checked

Last update: Nov 16 2019 at 01:00UTC