Stream: t-compiler

Topic: ide design meeting


nikomatsakis (Jan 18 2019 at 15:46, on Zulip):

@Igor Matuszewski, @Aleksey Kladov -- when should we discuss? on Zulip or over some video thing? Also, let's discuss how to coordinate "current design review"

matklad (Jan 18 2019 at 15:46, on Zulip):

zulip SGTM

Igor Matuszewski (Jan 18 2019 at 15:48, on Zulip):

I'm fine with whichever

Igor Matuszewski (Jan 18 2019 at 15:48, on Zulip):

we'd do that before AH, right?

Igor Matuszewski (Jan 18 2019 at 15:48, on Zulip):

@nikomatsakis do we know how many slots (and which specifically) we aim for during the AH?

matklad (Jan 18 2019 at 15:48, on Zulip):

I must say I am slightly confused atm about what we should discuss when.

I am personally ready to discuss anything at any time, of course :-)

My current understanding is that we

Igor Matuszewski (Jan 18 2019 at 15:49, on Zulip):

I saw "WG: RLS (now IDEs?), "rustc <-> RLS" - anything else?

Igor Matuszewski (Jan 18 2019 at 15:49, on Zulip):

@Aleksey Kladov that sounds good, yeah

nikomatsakis (Jan 18 2019 at 15:55, on Zulip):

yes so I specifically meant: I want to figure out what are the "crucial bits of the design" that we should discuss in detail at the all hands

nikomatsakis (Jan 18 2019 at 15:55, on Zulip):

see also this most recent comment :)

nikomatsakis (Jan 18 2019 at 15:55, on Zulip):

does that sound reasonable?

matklad (Jan 18 2019 at 15:58, on Zulip):

somewhat :) "crucial bits" sounds to vague. What I think crucial bits are "quries from the ground up", "analyzing interdependent crates together", "full-fidelity syntax trees", "divorce from batch compilation model in favor of here's the static set of crates with dependencies, analyze ".

nikomatsakis (Jan 18 2019 at 15:59, on Zulip):

well, making it less vague is exactly what I wanted to discuss

nikomatsakis (Jan 18 2019 at 15:59, on Zulip):

I'm happy to talk now, I don't envision this being a huge conversation :)

nikomatsakis (Jan 18 2019 at 16:00, on Zulip):

I guess I would say first off what are some "key challenges" in terms of features to support:

matklad (Jan 18 2019 at 16:00, on Zulip):

Oh, one more curcial bit: "pure (as in, absolutelly no IO) analyzis"

nikomatsakis (Jan 18 2019 at 16:00, on Zulip):

those feel like two important-ish cases to me, completions requiring us to bring all the type-checker etc to bear in a local fashion, find all references being the key to a lot of other random stuff (though not very hard)

nikomatsakis (Jan 18 2019 at 16:01, on Zulip):

but some of the key questions then might be:

matklad (Jan 18 2019 at 16:01, on Zulip):

I'll add "fixes support". Modifing source code is a separate bin of requirenments

matklad (Jan 18 2019 at 16:02, on Zulip):

how will we represent source text of input files?

I'd start more broadly, "what are the inputs to analysis"?

nikomatsakis (Jan 18 2019 at 16:02, on Zulip):

yeah

nikomatsakis (Jan 18 2019 at 16:03, on Zulip):

that feels connected to the cargo question, too

nikomatsakis (Jan 18 2019 at 16:03, on Zulip):

one of the things that jntrnr has been saying to me is that we may want to rethink the cargo-rustc layering

nikomatsakis (Jan 18 2019 at 16:03, on Zulip):

though I feel like that's potentially out of scope for 2021

matklad (Jan 18 2019 at 16:04, on Zulip):

yeah, and this also a point where I disagree most significantly with what rls is doing :)

nikomatsakis (Jan 18 2019 at 16:04, on Zulip):

yes, I wanted to dig more into that very thing -- though prob not this second

nikomatsakis (Jan 18 2019 at 16:04, on Zulip):

though I feel like that's potentially out of scope for 2021

mostly I mean we've got a lot of rewriting on our hands, I want to see where we can limit things :)

matklad (Jan 18 2019 at 16:05, on Zulip):

coming back to important questions:

I am not super sure that "find all refs" is super imoportant. In intelliJ, this basically works as "for each ref, resolve and check that the result matches the input declaration"

nikomatsakis (Jan 18 2019 at 16:05, on Zulip):

@Igor Matuszewski what do you think of the above lists? are there key questions that determine the end design that seem to be missing to you?

nikomatsakis (Jan 18 2019 at 16:05, on Zulip):

coming back to important questions:

I am not super sure that "find all refs" is super imoportant. In intelliJ, this basically works as "for each ref, resolve and check that the result matches the input declaration"

yeah, maybe not

nikomatsakis (Jan 18 2019 at 16:06, on Zulip):

I feel like it's one of the foundational operations, but it's also a relatively easy one

nikomatsakis (Jan 18 2019 at 16:06, on Zulip):

I guess if we want to dig a bit deeper, some things that come to mind are:

nikomatsakis (Jan 18 2019 at 16:06, on Zulip):
matklad (Jan 18 2019 at 16:06, on Zulip):

And I'd love to add

"find all symbols whose name match regex" -- again, at least in Intellij/rust analyzer it's a separate indexing infra

nikomatsakis (Jan 18 2019 at 16:07, on Zulip):

maybe

though it certainly touches on chalk, and I'm not sure ultimately how important it is

matklad (Jan 18 2019 at 16:07, on Zulip):

A not so-high level, but very important thing is

"how to map from internal data structures back to raw syntax".

nikomatsakis (Jan 18 2019 at 16:07, on Zulip):

actually yes

nikomatsakis (Jan 18 2019 at 16:07, on Zulip):

that seems very important

matklad (Jan 18 2019 at 16:08, on Zulip):

Yep, trait resolution is super interesting, because it's non-local

nikomatsakis (Jan 18 2019 at 16:08, on Zulip):

in particular, one of the big topics we've gone around and around on is maintaining the link between various IRs

nikomatsakis (Jan 18 2019 at 16:08, on Zulip):

Yep, trait resolution is super interesting, because it's non-local

right, so that part of things (e.g., finding potentially interesting impls) is sort of orthogonal to chalk and something I've not though esp hard about

nikomatsakis (Jan 18 2019 at 16:08, on Zulip):

in particular, one of the big topics we've gone around and around on is maintaining the link between various IRs

the general consensus here seems to be spans

matklad (Jan 18 2019 at 16:08, on Zulip):

Yeah, I've actually crystalized a consept of SourceMap for rust-analyzer basicaly today

nikomatsakis (Jan 18 2019 at 16:09, on Zulip):

I guess for that matter, how the type-checker should work is sort of on that list

nikomatsakis (Jan 18 2019 at 16:09, on Zulip):

it's a big topic

matklad (Jan 18 2019 at 16:09, on Zulip):

Not sure actually...

matklad (Jan 18 2019 at 16:09, on Zulip):

typechecking is local

Igor Matuszewski (Jan 18 2019 at 16:09, on Zulip):

Yeah, sorry for being so 'remote', just got through most of the stuff

matklad (Jan 18 2019 at 16:10, on Zulip):

so it can be almost arbitrary slow, etc, and still work fine with IDE

nikomatsakis (Jan 18 2019 at 16:10, on Zulip):

yah, ok, I'd rathe rkeep it off

nikomatsakis (Jan 18 2019 at 16:10, on Zulip):

well

nikomatsakis (Jan 18 2019 at 16:10, on Zulip):

it is needed for completions potentially

nikomatsakis (Jan 18 2019 at 16:10, on Zulip):

or maybe that's a false assumption on my part :)

Igor Matuszewski (Jan 18 2019 at 16:10, on Zulip):

The 'what's the input' is a rather important question, yeah

matklad (Jan 18 2019 at 16:10, on Zulip):

It is needed, yes, but you need to typechek only a single function body

nikomatsakis (Jan 18 2019 at 16:10, on Zulip):

but regardless, I don't know that it's foundational, I do think it's a potentially important thing to talk out for what "Rust 2021" looks like maybe, but perhaps a second order concern

Igor Matuszewski (Jan 18 2019 at 16:10, on Zulip):

Are we talking about incrementality of type chekcing now?

nikomatsakis (Jan 18 2019 at 16:11, on Zulip):

I am assuming that we will not try to be incremental when doing type-checking within a fn per se

nikomatsakis (Jan 18 2019 at 16:11, on Zulip):

though there are designs that allow for that

Igor Matuszewski (Jan 18 2019 at 16:11, on Zulip):

I meant establishing a boundary, like per-fn-boundary :)

nikomatsakis (Jan 18 2019 at 16:11, on Zulip):

ah. yes.

nikomatsakis (Jan 18 2019 at 16:11, on Zulip):

we have that now, even.

Igor Matuszewski (Jan 18 2019 at 16:11, on Zulip):

In any case, I believe it'd be good to settle on what are the inputs

Igor Matuszewski (Jan 18 2019 at 16:12, on Zulip):

and, albeit more philosophical, what's a 'crate'

matklad (Jan 18 2019 at 16:12, on Zulip):

I can describe briefly the inputs to rust-analyzer. I think they more or less makes sense

Igor Matuszewski (Jan 18 2019 at 16:12, on Zulip):

As a side note it's confusing since Cargo has packages, which it calls crates, which actually can contain "targets" which are "crates" per rustc's definition (compilation unit)

nikomatsakis (Jan 18 2019 at 16:12, on Zulip):

so to reiterate our list of things it was sort of like

but some of the key questions then might be:

nikomatsakis (Jan 18 2019 at 16:13, on Zulip):

starting to feel like a long-ish list but I guess what we're grasping for here is kind of the "whiteboard design sketch"

Igor Matuszewski (Jan 18 2019 at 16:13, on Zulip):

It'd be good to talk about rustc-as-a-deamon interface/approach

nikomatsakis (Jan 18 2019 at 16:13, on Zulip):

yeah, I was wondering if we should discuss e.g. "where are the processes"

matklad (Jan 18 2019 at 16:13, on Zulip):

@nikomatsakis : add "what are incrementality boundaries?"

Igor Matuszewski (Jan 18 2019 at 16:13, on Zulip):

rather than one-off 'do your job' batch approach it takes now

matklad (Jan 18 2019 at 16:14, on Zulip):

I mean, should we try to make "fixed point import resolution and macro expansion" incremental?

nikomatsakis (Jan 18 2019 at 16:14, on Zulip):

I have a specific question on that :)

matklad (Jan 18 2019 at 16:14, on Zulip):

I have so many questions about that....

nikomatsakis (Jan 18 2019 at 16:14, on Zulip):

to do so may require a different approach from salsa

nikomatsakis (Jan 18 2019 at 16:14, on Zulip):

though..hmm

nikomatsakis (Jan 18 2019 at 16:14, on Zulip):

differential dataflow could probably model it

nikomatsakis (Jan 18 2019 at 16:14, on Zulip):

I guess if we knew what it did, that would help :P

matklad (Jan 18 2019 at 16:15, on Zulip):

I hope that it'll be "fast enough"

nikomatsakis (Jan 18 2019 at 16:15, on Zulip):

this may be an interesting use case for differential-dataflow, I should return to that

matklad (Jan 18 2019 at 16:15, on Zulip):

"what it did"

it == "differentail dataflow" or it == "rust name resolution"?

matklad (Jan 18 2019 at 16:15, on Zulip):

/s

nikomatsakis (Jan 18 2019 at 16:17, on Zulip):

the latter

nikomatsakis (Jan 18 2019 at 16:17, on Zulip):

I mean it's not magic

Igor Matuszewski (Jan 18 2019 at 16:17, on Zulip):

you never know...

nikomatsakis (Jan 18 2019 at 16:17, on Zulip):

heh

nikomatsakis (Jan 18 2019 at 16:19, on Zulip):

well, ok, this seems like a great starting point

Igor Matuszewski (Jan 18 2019 at 16:19, on Zulip):

I'm slowly approaching this but boy it's hard to dig into a 5k LOC lib.rs full of edge cases :crying_cat:

nikomatsakis (Jan 18 2019 at 16:19, on Zulip):

though I have to collect our latest additions

matklad (Jan 18 2019 at 16:19, on Zulip):

My current hope is that, if we cache macro expansions (which should be easy, key is (token set, macro def), value is (token set)), then just running the whole fixed point thing for the whole crate will be fast enough, and needs to be run only rarely (when adding new top-level items).

nikomatsakis (Jan 18 2019 at 16:19, on Zulip):

(yes, I sort of hope that too)

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

so let's just briefly talk about the "reading material"

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

do you all plan to produce a design doc describing rust-analyzer / RLS?

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

video?

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

we could do a recorded, interactive session

Igor Matuszewski (Jan 18 2019 at 16:20, on Zulip):

I'm fine with a design doc/architectural overview

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

but really whatever you want is fine

matklad (Jan 18 2019 at 16:20, on Zulip):

I plan to do a doc + (if I am on a roll) a screencast

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

I don't think it has to be like mega thorough, just hit the high-level picture

nikomatsakis (Jan 18 2019 at 16:20, on Zulip):

if it's too long nobody will read anyway :P

Igor Matuszewski (Jan 18 2019 at 16:21, on Zulip):

That's the design talk, but - do we plan to talk about merging/overlapping the compiler and IDE team?

Igor Matuszewski (Jan 18 2019 at 16:22, on Zulip):

Since I feel we're kind of unnecessarily split too much, if that makes sense

nikomatsakis (Jan 18 2019 at 16:22, on Zulip):

I also agree

nikomatsakis (Jan 18 2019 at 16:22, on Zulip):

honestly, I would be happy to just mergre them completely

nikomatsakis (Jan 18 2019 at 16:23, on Zulip):

I don't want to step on any toes

nikomatsakis (Jan 18 2019 at 16:23, on Zulip):

I'm also happy to just have a lot of overlap and maybe distinct teams :)

nikomatsakis (Jan 18 2019 at 16:23, on Zulip):

but I feel like the trend is going to be that RLS + rustc are more and more indistinguishable

Igor Matuszewski (Jan 18 2019 at 16:23, on Zulip):

FWIW I think Nick also was for the merge

nikomatsakis (Jan 18 2019 at 16:23, on Zulip):

and I think we should all be thinking of IDE as the primary mode of execution, and batch secondary

nikomatsakis (Jan 18 2019 at 16:23, on Zulip):

ok, honestly my main concern was upsetting you, so if you are in favor =)

nikomatsakis (Jan 18 2019 at 16:24, on Zulip):

well you and other RLS folks

Igor Matuszewski (Jan 18 2019 at 16:24, on Zulip):

Yup! Sounds reasonable

matklad (Jan 18 2019 at 16:24, on Zulip):

I also wonder how I fit into all of this organizationally...

I've been building "my own IDE thing" for the last n years, and that certainly feels suboptimal.

nikomatsakis (Jan 18 2019 at 16:24, on Zulip):

so in that case, we had talked about an "organizational meeting"

nikomatsakis (Jan 18 2019 at 16:24, on Zulip):

this maybe fits in there

nikomatsakis (Jan 18 2019 at 16:24, on Zulip):

but maybe it deserves its own topic

nikomatsakis (Jan 18 2019 at 16:25, on Zulip):

that is, the "RLS"-proper probably winds up being not dissimilar from other "crates in the rustc orbit" that we have to maintain well

nikomatsakis (Jan 18 2019 at 16:25, on Zulip):

but we have to figure out how to make our triage process etc accommodate that

Igor Matuszewski (Jan 18 2019 at 16:27, on Zulip):

Right now RLS is just about the orchestration - calls and inspects Cargo with appropriate options, then collects the rustc-emitted data and it serves that

Igor Matuszewski (Jan 18 2019 at 16:27, on Zulip):

where the collection and indexing is mostly about being able to cross-reference definitions across multiple different crate 'compilation sessions', so maybe if we could revisit that then it's totally doable to make it act as a bare server serving the compiler knowledge

Igor Matuszewski (Jan 18 2019 at 16:28, on Zulip):

(which basically means I'm all for moving most of the actual bits into rustc, yeah :p)

nikomatsakis (Jan 18 2019 at 16:28, on Zulip):

=)

Igor Matuszewski (Jan 18 2019 at 16:29, on Zulip):

but I think it doesn't mean pruning librustc_save_analysis - it'd be good to think about how we can support dumping all the knowledge data for code indexers such as sourcegraph or searchfox

matklad (Jan 18 2019 at 16:30, on Zulip):

+1 here. "rust code index file format" is probably a good question to ask

Igor Matuszewski (Jan 18 2019 at 16:30, on Zulip):

(I guess that overlaps very much with rustdoc as well?)

matklad (Jan 18 2019 at 16:31, on Zulip):

batch processing a crate and dumping info is useful for two scenarios

nikomatsakis (Jan 18 2019 at 16:31, on Zulip):

actually, I've been talking with @Vytautas Astrauskas about a related concept

nikomatsakis (Jan 18 2019 at 16:31, on Zulip):

I owe them an email back

nikomatsakis (Jan 18 2019 at 16:31, on Zulip):

they've got a kind of "rust-query-language" project they've been tinkering with at ETH

matklad (Jan 18 2019 at 16:31, on Zulip):
nikomatsakis (Jan 18 2019 at 16:31, on Zulip):

similar to the idea from this blog post of mine

QuietMisdreavus (Jan 18 2019 at 16:31, on Zulip):

(semi-related, i've proposed at some point extending save-analysis or something like it to act like a rustdoc JSON output)

nikomatsakis (Jan 18 2019 at 16:31, on Zulip):

I suspect it could serve as the basis for that sort of thing

Igor Matuszewski (Jan 18 2019 at 16:32, on Zulip):

It seems we have more to talk about that we'll probably wind up getting time to do so =) That's good, though.

nikomatsakis (Jan 18 2019 at 16:33, on Zulip):

always the case

matklad (Jan 18 2019 at 16:34, on Zulip):

Yeah. Are there actually any specific bits we want to discuss in this meeting?

I guess:

Do want to talk about smt else?

matklad (Jan 18 2019 at 16:35, on Zulip):

Perhaps we can dig into a couple of "questions", just to talk them through a bit?

I think "what are the inputs" and "how to do name resolution \w macros" are discussion worthy

matklad (Jan 18 2019 at 16:35, on Zulip):

and we've talked about the second one ("hope the stupid solution will be just fast enough :D")

nikomatsakis (Jan 18 2019 at 16:36, on Zulip):

I've got to run in a bit

nikomatsakis (Jan 18 2019 at 16:36, on Zulip):

but feel free to log thoughts here and I will read them async

Igor Matuszewski (Jan 18 2019 at 16:36, on Zulip):

Hm, how about we put off now, do the doc overview and come back for another meeting before AH but after looking at the designs?

matklad (Jan 18 2019 at 16:36, on Zulip):

excellent idea!

Igor Matuszewski (Jan 18 2019 at 16:36, on Zulip):

Maybe that'll make us come up with more questions or crystallize some concepts further

matklad (Jan 18 2019 at 16:38, on Zulip):

:wave: then I guess? :) I feel today's meetings were super-productive. I am excited :-)

nikomatsakis (Jan 18 2019 at 16:38, on Zulip):

@Aleksey Kladov btw, I was just talking to @Jonathan Turner about maybe preparing a bit of a discussion about some of the things we did in Lark (some of which are relevant and probably slightly different from what rust-analyzer is doing), and they suggested that maybe I should do a salsa presentation, kind of discussing what it is, how it works, etc. I think I may do that, which would perhaps be useful to you so you can just kind of "reference it" (I imagine it's a bit below the abstraction level anyway)

nikomatsakis (Jan 18 2019 at 16:38, on Zulip):

then I guess? :) I feel today's meetings were super-productive. I am excited :-)

me too!

matklad (Jan 18 2019 at 16:39, on Zulip):

Yep, having a "salsa book/video" to point to would be useful :)

Igor Matuszewski (Jan 18 2019 at 16:39, on Zulip):

Sounds interesting!

Igor Matuszewski (Jan 18 2019 at 16:40, on Zulip):

Maybe this year will be the year of Rust IDE :laughing:

matklad (Jan 18 2019 at 16:41, on Zulip):

It's definitely more than a year's worth of work =) A "пятилетка" of Rust IDEs?

Igor Matuszewski (Jan 18 2019 at 16:41, on Zulip):

"pljatiletka"?

QuietMisdreavus (Jan 18 2019 at 16:42, on Zulip):

a five year plan? :thinking:

matklad (Jan 18 2019 at 16:42, on Zulip):

https://en.wikipedia.org/wiki/Five-year_plans_for_the_national_economy_of_the_Soviet_Union

Igor Matuszewski (Jan 18 2019 at 16:42, on Zulip):

FWIW I'm also super excited :muscle:

Jonathan Turner (Jan 18 2019 at 16:43, on Zulip):

--A wild @Jonathan Turner appears--

Igor Matuszewski (Jan 18 2019 at 16:43, on Zulip):

aah, right

Igor Matuszewski (Jan 18 2019 at 16:43, on Zulip):

I guess I'm slowly exchanging history (or anything, really) with software development knowledge :cry:

Jonathan Turner (Jan 18 2019 at 16:43, on Zulip):

a five year plan? :thinking:

I think that's a really good idea... at least have a feeling for where things should be in 3-5 years

Igor Matuszewski (Jan 18 2019 at 16:45, on Zulip):

Gotta run, thanks for the meeting and I guess we'll be in touch with the docs and a quick second meeting before AH :wave:

nikomatsakis (Jan 18 2019 at 16:46, on Zulip):

@Jonathan Turner you may also be interested in the steering meeting notes from today; I plan to write a summary eventually of course

Vytautas Astrauskas (Jan 18 2019 at 17:02, on Zulip):

they've got a kind of "rust-query-language" project they've been tinkering with at ETH

@Igor Matuszewski That is the same thing I mentioned to you in Rust Fest Rome. The current prototype cannot do much yet, but we are looking for a master student to work on this. If you would like to know more or have any ideas how this could be useful for you, feel free to ping me.

matklad (Jan 20 2019 at 13:15, on Zulip):

A draft of rust-analyzer guide is up: https://github.com/rust-analyzer/rust-analyzer/pull/578

matklad (Jan 20 2019 at 16:55, on Zulip):

and a video version: https://www.youtube.com/watch?v=ANKBNiSWyfc

matklad (Jan 21 2019 at 05:50, on Zulip):

Was rereading rls IDE and came across a very interesting comment: https://github.com/rust-lang/rfcs/pull/1317#issuecomment-150965895

That’s the same things I’ve learned when building IntelliJ Rust, including the algorithm for find references

Igor Matuszewski (Jan 24 2019 at 10:38, on Zulip):

Also a draft for the current RLS architecture doc: https://github.com/rust-lang/rls/pull/1257

Last update: Nov 22 2019 at 04:35UTC