Stream: t-compiler

Topic: pre-design meeting 2019-09-13


nikomatsakis (Sep 13 2019 at 09:29, on Zulip):

Hey @matklad, you around? So I had hoped to chat with you more in advance of proposed meeting today but .. yeah, life. Are you around at the moment? I have a bit of time. Anyway so we created this hackmd capturing a lot of our thinking. It seems useful to try and put that before folks and ask them about. But what I'm wondering is... is there some central question we would prefer to extract? That doc obviously covers a lot of ground.

matklad (Sep 13 2019 at 09:29, on Zulip):

yep, I am around

matklad (Sep 13 2019 at 09:30, on Zulip):

rewriting our name resolution again :D

matklad (Sep 13 2019 at 09:30, on Zulip):

So, the central question I think is "what are immediate action items?"

matklad (Sep 13 2019 at 09:31, on Zulip):

The hypothesis is that it's "complete library-ification of chalk and surrounding infra"

matklad (Sep 13 2019 at 09:32, on Zulip):

but it would be great to get more feedback on this

nikomatsakis (Sep 13 2019 at 09:33, on Zulip):

Yeah so

nikomatsakis (Sep 13 2019 at 09:33, on Zulip):

I guess I think it depends a bit on your scope

matklad (Sep 13 2019 at 09:33, on Zulip):

Like, I feel we need a social/ideologial question, and not necessary a technical one: are we ready to do BIG CHANGES to make ide GO FAST?

nikomatsakis (Sep 13 2019 at 09:34, on Zulip):

I remember now that I had hoped to lay out some kind of chalk timeline, and I can try to edit the doc to at least give a loose one. I think my current thinking would be that this might be a good first step:

matklad (Sep 13 2019 at 09:35, on Zulip):

cc @Florian Diebold , I've seen you around

nikomatsakis (Sep 13 2019 at 09:36, on Zulip):

other steps, maybe part of that:

nikomatsakis (Sep 13 2019 at 09:36, on Zulip):

I guess the point is, to sort of prototype the proposed libraries with rust-analyzer as first consumer

nikomatsakis (Sep 13 2019 at 09:36, on Zulip):

but keeping rustc in mind

nikomatsakis (Sep 13 2019 at 09:36, on Zulip):

and where possible trying to integrate with rustc and/or maybe have follow-up meetings to describe how things are looking and what that might mean for rustc

nikomatsakis (Sep 13 2019 at 09:37, on Zulip):

well meetings and/or blog posts whatever

nikomatsakis (Sep 13 2019 at 09:37, on Zulip):

Like, I feel we need a social/ideologial question, and not necessary a technical one: are we ready to do BIG CHANGES to make ide GO FAST?

I do agree that ultimately what we need most of all is a sort of ... consensus around this point. I worry that talking this abstractly will not get something useful.

matklad (Sep 13 2019 at 09:38, on Zulip):

and where possible trying to integrate with rustc

I feel like finding an integration point with rustc is one specific question I'd love to know the answer to

nikomatsakis (Sep 13 2019 at 09:38, on Zulip):

but keeping rustc in mind

( my main concern here is that rustc has immediate needs too :) I think that we need to be exploring those in tandem, and I'm worried about spreading too thin. )

matklad (Sep 13 2019 at 09:39, on Zulip):

Continuing experiments feels good, but I fear that, unless it is integrated in production rustc, there's always a risk of loosing momentum and just not finishing stuff

nikomatsakis (Sep 13 2019 at 09:39, on Zulip):

/me thinks

nikomatsakis (Sep 13 2019 at 09:40, on Zulip):

there's also the question of what integration .. means

nikomatsakis (Sep 13 2019 at 09:40, on Zulip):

e.g., chalk is integrated into rustc

centril (Sep 13 2019 at 09:40, on Zulip):

I think if we can do "libraryfication by refactoring" that would be good

nikomatsakis (Sep 13 2019 at 09:40, on Zulip):

but there's room for improvement

centril (Sep 13 2019 at 09:40, on Zulip):

the overall aim being cleaner code than the messes that we sometimes have in the compiler today

nikomatsakis (Sep 13 2019 at 09:41, on Zulip):

anyway, maybe this is the conversation we want to be having in said meeting :)

matklad (Sep 13 2019 at 09:41, on Zulip):

I think there's a natural contradiction between "immediate goals" and "we badly need to refactor this". Can we agree on the subset of immediate goals that we do, and, more importantnly, on the subset we don't do, until the foundation is in the better shape?

centril (Sep 13 2019 at 09:42, on Zulip):

@matklad not necessarily -- you refactored the lexer and extracted it, I did some of the same with the parser and it is more amenable to library-fication now?

centril (Sep 13 2019 at 09:42, on Zulip):

(thus satisfying the immediate goals...?)

nikomatsakis (Sep 13 2019 at 09:42, on Zulip):

I agree it is good to refactor in place where possible

matklad (Sep 13 2019 at 09:43, on Zulip):

I was able to refactor lexer only because no one else was touching it, and I feel like I've tanked at least one "let's improve error recovering in raw string literals" PR with my effort.

centril (Sep 13 2019 at 09:43, on Zulip):

@matklad ah well... folks touch the parser very often and I was able to push splitting a 8k line file apart in the bors queue nonetheless ;)

nikomatsakis (Sep 13 2019 at 09:47, on Zulip):

Continuing experiments feels good, but I fear that, unless it is integrated in production rustc, there's always a risk of loosing momentum and just not finishing stuff

I'm thinking about this @matklad

centril (Sep 13 2019 at 09:47, on Zulip):

@nikomatsakis Aside: I found https://www.youtube.com/watch?v=x3evzO8O9e8 a most excellent watch.

nikomatsakis (Sep 13 2019 at 09:47, on Zulip):

I'm trying to make it more concrete

nikomatsakis (Sep 13 2019 at 09:48, on Zulip):

I guess this is not a question so much as a statement from "our" side

nikomatsakis (Sep 13 2019 at 09:48, on Zulip):

That is, the proposal is basically that we want to work more aggressively on creating libraries that are integrated into rustc, and these are a few candidates

nikomatsakis (Sep 13 2019 at 09:49, on Zulip):

I worry simultaneously that we'll get lost in technical details on those libraries or that we'll be so high-level it's meaningless, so maybe we should just see where it goes :) 50/50 nobody even shows up :P

nikomatsakis (Sep 13 2019 at 09:50, on Zulip):

But yeah it seems like a reasonably good plan for meeting to say: here are some ideas for those libraries, let's discuss some of what it might mean to try and extract from rustc in detail..? I think that chalk is actually not the most interesting bit here, since a lot of that is understood, but rather trying to go a bit more outside current rustc-chalk-integration, notably to include the lowering code and/or def'n of types.

nikomatsakis (Sep 13 2019 at 09:50, on Zulip):

so maybe I'll refocus that part of document to talk a bit about that

matklad (Sep 13 2019 at 09:50, on Zulip):

Yeah... I sort of feel that the md doc we've created covers technical side of things pretty well, so I see most value in dispersing info and maybe exciting some folks to help out :)

nikomatsakis (Sep 13 2019 at 09:50, on Zulip):

and to give rust-analyzer as an example of where we have gone a bit further, though not as far as I'd like

nikomatsakis (Sep 13 2019 at 09:51, on Zulip):

ok, this is helpful, i'm going to try and tweak some of the content in/around chalk

centril (Sep 13 2019 at 09:55, on Zulip):

@matklad to add to the technical side of things, I think you noted that using visibility to have clear public interfaces to lotsa private details is helpful for library-ifiction. I basically agree -- that's what I was going for in e.g. typeck/lowering/parser

centril (Sep 13 2019 at 09:55, on Zulip):

I think we can do a lot more of that

matklad (Sep 13 2019 at 09:58, on Zulip):

Yeah, reducing visibility is a low-hanging fruit. It doesn't help with cutting dependenies per se, but it helps with identifying dependency edges that shouldn't be there.

Another good mechanical thing is removing state.

before

after

centril (Sep 13 2019 at 10:04, on Zulip):

after

Oh that's just beautiful!

matklad (Sep 13 2019 at 10:11, on Zulip):

@nikomatsakis question about chalk-rustc integration: is chalk-engine behind a feature flag, or is it the actual thing that's being used for trait solving?

nikomatsakis (Sep 13 2019 at 10:38, on Zulip):

@matklad it's a flag (-Zchalk)-- also, I'll be gone for ~1.5 hours or so, I'll try to wrap up the additions. I was trying to create a top section like "main questions" to try and guide overall discusison, feel free to edit if you like

pnkfelix (Sep 13 2019 at 12:26, on Zulip):

Another good mechanical thing is removing state.

(where "removing" may be removal from code base entirely, or may be factoring into a different struct tailored to certain purposes. I note this mainly because I was curious about where e.g. the open_braces field went.

nikomatsakis (Sep 13 2019 at 12:53, on Zulip):

(it occurs to me @matklad we're looking over another obvious candidate, which is merging rustc's query system with salsa)

nikomatsakis (Sep 13 2019 at 12:54, on Zulip):

but right now the gulf there is large enough I dont' want to go there :)

centril (Sep 13 2019 at 12:55, on Zulip):

@nikomatsakis I want to encourage that thinking :P rustc's query system feels very inscrutable to use

matklad (Sep 13 2019 at 12:55, on Zulip):

@pnkfelix yeah! Another great example of state factoring is https://github.com/rust-lang/rust/pull/63400

centril (Sep 13 2019 at 12:56, on Zulip):

@matklad might be possible to do some of that for the Parser

nikomatsakis (Sep 13 2019 at 12:59, on Zulip):

(it occurs to me matklad we're looking over another obvious candidate, which is merging rustc's query system with salsa)

(I'm reminded that I have 25 messages sitting in the salsa zulip I have yet follow up on, dang it -- I've been spending my "free morning hours" on rayon lately)

simulacrum (Sep 13 2019 at 13:04, on Zulip):

matklad might be possible to do some of that for the Parser

In particular, I think a great improvement would be to remove ParseSess almost entirely, if we have Parser it's a bit weird to have something that persists across multiple parses

simulacrum (Sep 13 2019 at 13:04, on Zulip):

and we often use it as sort of "implicit output" from parser which means locks and such

centril (Sep 13 2019 at 13:11, on Zulip):

pre-expansion gating is such an implicit ouput but it's probably going to be hard to ditch

centril (Sep 13 2019 at 13:11, on Zulip):

(and I want to add more pre-exp gating...)

matklad (Sep 13 2019 at 13:15, on Zulip):

@nikomatsakis another huge, but mostly orthogonal thing, is the overall strategy of rls vs rls2.0

matklad (Sep 13 2019 at 13:15, on Zulip):

Do we support both? Are we switching to one?

simulacrum (Sep 13 2019 at 13:58, on Zulip):

Pre-expansion gating can just generate errors on a passed in handler I'd imagine?

centril (Sep 13 2019 at 13:58, on Zulip):

@simulacrum yea I suppose; -- not exactly sure why we use state for this atm

centril (Sep 13 2019 at 13:59, on Zulip):

oh wait... I do know why... @simulacrum we haven't done cfg expansion during parsing

centril (Sep 13 2019 at 13:59, on Zulip):

so it's time travel to emit those errors

simulacrum (Sep 13 2019 at 13:59, on Zulip):

I'm fine with time travel :)

centril (Sep 13 2019 at 13:59, on Zulip):

there could be a #![cfg_attr(feature "unstable", feature(or_patterns)]

centril (Sep 13 2019 at 14:00, on Zulip):

@simulacrum so am I, in theory ;)

centril (Sep 13 2019 at 14:01, on Zulip):

we'd need to do some amount of cfg-expansion before full parsing to fix it

centril (Sep 13 2019 at 14:01, on Zulip):

which means we'd need to do some amount of parsing first

simulacrum (Sep 13 2019 at 14:02, on Zulip):

Seems like expanding crate level attributes makes sense. This also interestingly intersects with why we made editions CLI flag only

centril (Sep 13 2019 at 14:02, on Zulip):

@simulacrum right; but then you need to parse those crate level attrs first that is

simulacrum (Sep 13 2019 at 14:02, on Zulip):

In theory we could do the same for features though it would be quite painful

simulacrum (Sep 13 2019 at 14:03, on Zulip):

Seems plausible to parse them first.

centril (Sep 13 2019 at 14:03, on Zulip):

yea, probably not very difficult to do

centril (Sep 13 2019 at 14:03, on Zulip):

mainly plumbing changes

nikomatsakis (Sep 13 2019 at 14:49, on Zulip):

nikomatsakis another huge, but mostly orthogonal thing, is the overall strategy of rls vs rls2.0

omg yes we need to make a separate meeting on this, @matklad

Last update: Nov 22 2019 at 04:35UTC