Stream: t-compiler

Topic: design meeting 2019-09-13


nikomatsakis (Sep 13 2019 at 13:56, on Zulip):

Hey @T-compiler/meeting -- design meeting starting in 5 minutes. The topic will be "rust-analyzer and library-ification". You can view the hackmd document here that collects some thoughts.

To start, let's have

Announcements

nikomatsakis (Sep 13 2019 at 13:59, on Zulip):

cc @matklad and @Florian Diebold, not sure if you are part of t-compiler/meeting (let me know if you want to be added...)

matklad (Sep 13 2019 at 14:04, on Zulip):

:wave:

nikomatsakis (Sep 13 2019 at 14:04, on Zulip):

OK, let's see, maybe we can make a kind of "agenda-ish" to guide our time

nikomatsakis (Sep 13 2019 at 14:04, on Zulip):

(Also, I realized that I should probably -- after each planning meeting -- be sending an e-mail to the compiler mailing list with upcoming dates. Going to start doing that after this meeting.)

centril (Sep 13 2019 at 14:05, on Zulip):

(I don't think I'm on that list... but interested)

nikomatsakis (Sep 13 2019 at 14:05, on Zulip):

actually the right place to post these sorts of updates is the team blog

nikomatsakis (Sep 13 2019 at 14:05, on Zulip):

that we're "about to setup"

nikomatsakis (Sep 13 2019 at 14:06, on Zulip):

(the canonical place for now is the compiler team calendar, of course)

nikomatsakis (Sep 13 2019 at 14:07, on Zulip):

anyway, shall we get started?

nikomatsakis (Sep 13 2019 at 14:07, on Zulip):

@matklad and I had kind of identified these questions to help guide us

anyway, shall we get started?

but it seems like we should start maybe by "summarizing" a bit about what's been happening around rust-analyzer + what library-ification means

eddyb (Sep 13 2019 at 14:08, on Zulip):

I don't get pinged by T-compiler/meeting

nikomatsakis (Sep 13 2019 at 14:08, on Zulip):

curious, you are on the list

nikomatsakis (Sep 13 2019 at 14:09, on Zulip):

(just checked)

varkor (Sep 13 2019 at 14:09, on Zulip):

I have been in the past, but I didn't get pinged just then

nikomatsakis (Sep 13 2019 at 14:09, on Zulip):

cc @T-compiler/meeting

nikomatsakis (Sep 13 2019 at 14:09, on Zulip):

maybe because I edited the message? :)

nikomatsakis (Sep 13 2019 at 14:09, on Zulip):

weird

eddyb (Sep 13 2019 at 14:09, on Zulip):

OH

varkor (Sep 13 2019 at 14:09, on Zulip):

ah, that worked :+1:

eddyb (Sep 13 2019 at 14:10, on Zulip):

yeah that would explain why Zulip had a ghost ping that didn't really show up but counted up the mentions thing

centril (Sep 13 2019 at 14:10, on Zulip):

let's start =)

nikomatsakis (Sep 13 2019 at 14:10, on Zulip):

anyway, re: rust-analyzer and its current status, I don't want to talk too much about it, but it seemed worth mentioning some details. Basically, work there has been proceeding pretty well and rust-analyzer now incuds:

and, notably, a shared lexer with rustc (right, @matklad?)

matklad (Sep 13 2019 at 14:11, on Zulip):

Right!

nikomatsakis (Sep 13 2019 at 14:11, on Zulip):

previously when we had talked about, we had discussed the idea of trying to identify key libraries for parts of rustc that we can extract and share

nikomatsakis (Sep 13 2019 at 14:11, on Zulip):

and essentially this meeting is to discuss that process in a bit more detail, and talk through some details of possible candidates --

nikomatsakis (Sep 13 2019 at 14:11, on Zulip):

though I think there's also room for people to object if they think the overall direction isn't great

centril (Sep 13 2019 at 14:12, on Zulip):

what does "extract" mean then; refactor a bit and publish to crates.io in a "totally unstable" manner?

nikomatsakis (Sep 13 2019 at 14:12, on Zulip):

that is a good question, and one of the things we should decide :)

nikomatsakis (Sep 13 2019 at 14:12, on Zulip):

I think that we are clearly a long way from "stable APIs" that we expect to support long term

eddyb (Sep 13 2019 at 14:12, on Zulip):

I frankly have failed to prove anything because the past year (or two?) have been a mess, so any past objections I may have had are like "eh whatever" now

simulacrum (Sep 13 2019 at 14:12, on Zulip):

I think another key element is "does extract mean no 'path' dependencies"

eddyb (Sep 13 2019 at 14:13, on Zulip):

what I do want to say is that we could use an "abstract MIR"

simulacrum (Sep 13 2019 at 14:13, on Zulip):

which includes the sort of weird rustc-ap- deps

eddyb (Sep 13 2019 at 14:13, on Zulip):

that uses the chalk infrastructure to talk about types and whatnot and is otherwise relatively typesystem-agnostic

centril (Sep 13 2019 at 14:13, on Zulip):

I wouldn't want "extract" to mean "only stable features" since that removes the possibility to dogfood features in the compiler

mw (Sep 13 2019 at 14:13, on Zulip):

it would be nice to come up with a list (tree) of libraries that we think the compiler should be comprised of

Igor Matuszewski (Sep 13 2019 at 14:13, on Zulip):

would extrating a stable parser to crates.io would mean commiting to a stable AST-like interface?

nikomatsakis (Sep 13 2019 at 14:13, on Zulip):

I think that if rust-analyzer is to use the libraries, then a requirement would be that they are ultiamtely dependent on std and work on stable

matklad (Sep 13 2019 at 14:13, on Zulip):

@simulacrum I'd define "extract" = builds on stabeli-ish compiler without bootstrap process.

centril (Sep 13 2019 at 14:14, on Zulip):

why is "works on stable" a requirement here?

eddyb (Sep 13 2019 at 14:14, on Zulip):

@simulacrum I wish the -ap- crates were never a thing and we just did in-tree builds :(

matklad (Sep 13 2019 at 14:14, on Zulip):

@centril working on stable make the project MASSIVELY easier to contribute

nikomatsakis (Sep 13 2019 at 14:15, on Zulip):

Yeah. I think that works on stable is ok, but also something we could defer

nikomatsakis (Sep 13 2019 at 14:15, on Zulip):

it doesn't seem like the central point

nikomatsakis (Sep 13 2019 at 14:15, on Zulip):

the main thing is: trying to build up libraries that can be used as "normal" code

matklad (Sep 13 2019 at 14:15, on Zulip):

True, it's only a means to get "easy build process", it's not an end

nikomatsakis (Sep 13 2019 at 14:15, on Zulip):

that is, through the normal "cargo dev process"

centril (Sep 13 2019 at 14:15, on Zulip):

(I think cargo check is valuable, but that's different than "uses stable compiler")

nikomatsakis (Sep 13 2019 at 14:16, on Zulip):

indeed

nikomatsakis (Sep 13 2019 at 14:16, on Zulip):

one of the things we discuss in the document, which also seems relevant, is the question of "monorepo vs poly-repo"

centril (Sep 13 2019 at 14:16, on Zulip):

(and e.g. once we have if let chains I'd like to simplify code in the compiler with it and dogfood it through the compiler)

nikomatsakis (Sep 13 2019 at 14:16, on Zulip):

I thkn we've examples of both today --

matklad (Sep 13 2019 at 14:17, on Zulip):

Note that there's a failure in feature-design process, if it requires doggdoffing the compiler: compiler's are only a tiny slice of rust' domain

pnkfelix (Sep 13 2019 at 14:17, on Zulip):

one of the things we discuss in the document, which also seems relevant, is the question of "monorepo vs poly-repo"

also known as: Will we reap the benefits if we continue to gate on bors/homu

nikomatsakis (Sep 13 2019 at 14:17, on Zulip):

and x.py

matklad (Sep 13 2019 at 14:17, on Zulip):

@pnkfelix could you elaborate on this?

simulacrum (Sep 13 2019 at 14:18, on Zulip):

well, in some sense, no, because we can gate on bors sort of "individually" -- I would put it more so as "gate on src/test/{ui,etc}"

pnkfelix (Sep 13 2019 at 14:18, on Zulip):

pnkfelix could you elaborate on this?

I take monorepo to mean: we continue gating landing any PR through the usual r+ on bors

oli (Sep 13 2019 at 14:18, on Zulip):

sorry, internet troubles, ignore my messages

simulacrum (Sep 13 2019 at 14:18, on Zulip):

e.g. I would say that rustc_lexer's unit tests (i.e., cargo test) is actually much weaker than full src/test/ui

centril (Sep 13 2019 at 14:18, on Zulip):

Note that there's a failure in feature-design process, if it requires doggdoffing the compiler: compiler's are only a tiny slice of rust' domain

I disagree; rustc is a huge project and a good place to test features that are sometimes too minor or mostly convenience to expect anyone to use outside

mw (Sep 13 2019 at 14:19, on Zulip):

having a poly-repo would mean that the individual repos would really have to be rather separate standalone projects that are loosely integrated with the rest of the compiler

pnkfelix (Sep 13 2019 at 14:19, on Zulip):

I take monorepo to mean: we continue gating landing any PR through the usual r+ on bors

then you each PR needs to wait its turn

centril (Sep 13 2019 at 14:19, on Zulip):

then you each PR needs to wait its turn

or rollups :slight_smile:

nikomatsakis (Sep 13 2019 at 14:19, on Zulip):

e.g. I would say that rustc_lexer's unit tests (i.e., cargo test) is actually much weaker than full src/test/ui

yes, I think this is a core point. One of the questions that arises around having many libaries is figuring out a good testing strategy

nikomatsakis (Sep 13 2019 at 14:19, on Zulip):

in general, one thing that @matklad and I agreed on is that overly narrow unit tests can be "future hostile"

pnkfelix (Sep 13 2019 at 14:19, on Zulip):

rollups are a symptom of something wrong

pnkfelix (Sep 13 2019 at 14:20, on Zulip):

not an answer.

nikomatsakis (Sep 13 2019 at 14:20, on Zulip):

i.e., if you change how the library works in some fundamental way, it becomes really hard to port those tests forward, whereas e.g. src/test/ui tests are relatively easy

qmx (Sep 13 2019 at 14:20, on Zulip):

isn't this the standard separation between unit tests and integration tests? you kinda want both?

nikomatsakis (Sep 13 2019 at 14:20, on Zulip):

yeah it is, nothing magic

eddyb (Sep 13 2019 at 14:20, on Zulip):

this reminds me, one crate that's not necessarily relevant to rust-analyzer but could be useful to other projects is rustc_apfloat - see https://github.com/rust-lang/rust/issues/55993

nikomatsakis (Sep 13 2019 at 14:20, on Zulip):

but I think it also informs the boundaries that make a good library

eddyb (Sep 13 2019 at 14:20, on Zulip):

so it could be a test subject

centril (Sep 13 2019 at 14:20, on Zulip):

@pnkfelix eh well... I think rolling up small docs PRs are not a symptom of a problem... and the problem where it exists is mostly "we need better machines"

oli (Sep 13 2019 at 14:20, on Zulip):

wouldn't the gating be on "bumping rustc version dependency on crate", so that only happens every now and then, and the crate can be developed independently

centril (Sep 13 2019 at 14:21, on Zulip):

I'm strongly in favor of a monorepo; I don't want to audit what a Cargo.lock file meant in terms of changes to the language from a lang team POV.

nikomatsakis (Sep 13 2019 at 14:21, on Zulip):

wouldn't the gating be on "bumping rustc version dependency on crate", so that only happens every now and then, and the crate can be developed independently

some challenges I see here:

qmx (Sep 13 2019 at 14:22, on Zulip):

I meant that niko, having the integration tests living where the crate is used

varkor (Sep 13 2019 at 14:22, on Zulip):

so that only happens every now and then

maybe this could be done automatically: build a version of rust with all updated dependencies every day (or so), so we identify problems that have snuck into the libraries early

oli (Sep 13 2019 at 14:22, on Zulip):

sure

nikomatsakis (Sep 13 2019 at 14:23, on Zulip):

So, i'm not hearing a lot of people arguing about whether we should try to extract libraries with stronger boundaries, mostly about some details of how we should do it?

pnkfelix (Sep 13 2019 at 14:23, on Zulip):

I think petrochenkov may have expressed concern in the past

mw (Sep 13 2019 at 14:23, on Zulip):

I'd like to see something like LLVM's model, where everything is a transformation from data to data

pnkfelix (Sep 13 2019 at 14:23, on Zulip):

but I'm not sure if that was more about mono-repo vs poly-repo Q

varkor (Sep 13 2019 at 14:23, on Zulip):

if we did automatic updating as well, then reviewing bumps wouldn't be so problematic

mw (Sep 13 2019 at 14:23, on Zulip):

that would enable having rather stable unit tests (because the data definition is stable)

pnkfelix (Sep 13 2019 at 14:23, on Zulip):

(oh and I think @Vadim Petrochenkov was also concerned about inefficiency from (increased) breaking into multiple cratees)

simulacrum (Sep 13 2019 at 14:24, on Zulip):

One concern that I've had is that due to coherence it's a bit harder if we split up into crates

nikomatsakis (Sep 13 2019 at 14:24, on Zulip):

I'm wondering if the best way to resolve these details around mono-vs-poliy-repo and workflow is to do a bit more experimentation first and revisit with more experience. At present we have some projects trying to inhabit both worlds.

mw (Sep 13 2019 at 14:24, on Zulip):

you could then have a human readable encoding of the data for writing tests

nikomatsakis (Sep 13 2019 at 14:24, on Zulip):

e.g., I'm pretty reluctant to move chalk or polonius into the monorepo at present

nikomatsakis (Sep 13 2019 at 14:24, on Zulip):

that would enable having rather stable unit tests (because the data definition is stable)

yes, this I think is an interesting question too. Chalk kind of steers a middle course here

centril (Sep 13 2019 at 14:24, on Zulip):

@mw /me thinks of the Clojure "it's just data" joke ;)

nikomatsakis (Sep 13 2019 at 14:25, on Zulip):

(the input is defined as queries that we demand from some environment, but we have a "driver" that reads data input in a canonical text format for testing)

mw (Sep 13 2019 at 14:25, on Zulip):

yes

mw (Sep 13 2019 at 14:25, on Zulip):

it would require quite a bit of initial setup, I guess

nikomatsakis (Sep 13 2019 at 14:25, on Zulip):

(and one of the things I'd like to do is to make it easy to extract "traces" from rust-analyzer, @Florian Diebold did some first steps towards this and its very useful)

nikomatsakis (Sep 13 2019 at 14:25, on Zulip):

i.e., so we can make unit tests from problematic scenarios

matklad (Sep 13 2019 at 14:25, on Zulip):

in rust-analyzer, all tests are data-in / data-out. The problem is that data-in is rust surface syntax, so it depends on the lower parts of the compiler's stack

nikomatsakis (Sep 13 2019 at 14:25, on Zulip):

(also, that text format is based on rust syntax, so it should be "long lived")

nikomatsakis (Sep 13 2019 at 14:26, on Zulip):

in any case, if we have general agreement around library-ification, maybe it makes sense to turn to some specifics of possible libraries? and maybe someone (prob me) can go back over the transcript later and try to pick out and summarize the pros/cons from various setups for future discussion

eddyb (Sep 13 2019 at 14:26, on Zulip):

FWIW the grammars we want to work with in wg-grammar are meant to be extensible so chalk could eventually be built on that

nikomatsakis (Sep 13 2019 at 14:27, on Zulip):

in the hackmd document we listed out

eddyb (Sep 13 2019 at 14:27, on Zulip):

(or you can even replace some syntax, like WhereClause, with Chalk's equivalent)

nikomatsakis (Sep 13 2019 at 14:27, on Zulip):

probably in order of "well to least undersztood"? maybe not, kind of a toss-up between name res and traits

nikomatsakis (Sep 13 2019 at 14:27, on Zulip):

but each has interesting characteristics

eddyb (Sep 13 2019 at 14:28, on Zulip):

name resolution is worse than traits IME, but maybe less code in total

centril (Sep 13 2019 at 14:28, on Zulip):

(I think I still don't really understand what library-ification means)

eddyb (Sep 13 2019 at 14:28, on Zulip):

it's less pure than trait solving, that's the sad part

mw (Sep 13 2019 at 14:28, on Zulip):

@centril yeah, I don't think we clarified that :)

varkor (Sep 13 2019 at 14:28, on Zulip):

does the diagnostics extraction fit this theme too?

nikomatsakis (Sep 13 2019 at 14:29, on Zulip):

(I think I still don't really understand what library-ification means)

is it worth discussing more about this then

centril (Sep 13 2019 at 14:29, on Zulip):

I think best-understood is probably parser, lowering, and typeck?

nikomatsakis (Sep 13 2019 at 14:29, on Zulip):

I think the most important question is probably "does it just mean making a new crate" (in my view, no)

pnkfelix (Sep 13 2019 at 14:29, on Zulip):

"library-ification" : add entry point that isn't the rustc binary, for testing?

nikomatsakis (Sep 13 2019 at 14:30, on Zulip):

it's really about producing something that is relatively disentangled from the specifics of rustc and reusable in more contexts, right?

nikomatsakis (Sep 13 2019 at 14:30, on Zulip):

does the diagnostics extraction fit this theme too?

that's a good question :) I think yes but it is distinct from the other examples we've given

eddyb (Sep 13 2019 at 14:30, on Zulip):

@pnkfelix I mean that's rustc_interface isn't it?

nikomatsakis (Sep 13 2019 at 14:30, on Zulip):

but it fits my "disentangled" above?

eddyb (Sep 13 2019 at 14:30, on Zulip):

you could build a tiny LSP (like, a subset of RLS) on top of rustc_interface and a bit of glue code, for example

simulacrum (Sep 13 2019 at 14:30, on Zulip):

I think in some sense it means decoupling things "until" rustc_interface or some similar crate ties things together

nikomatsakis (Sep 13 2019 at 14:31, on Zulip):

right

matklad (Sep 13 2019 at 14:31, on Zulip):

Yeah, I think we want to detangle not from the binary, but from the types and state. Like, a library-ified parser shouldn't have a ParseSess

centril (Sep 13 2019 at 14:31, on Zulip):

@nikomatsakis generally, it sorta sounds like "simpler boundaries with less implicit state between libraries"

simulacrum (Sep 13 2019 at 14:31, on Zulip):

so e.g. one could imagine that library-ification of the parser means that librustc does not depend on a parser

nikomatsakis (Sep 13 2019 at 14:31, on Zulip):

and probably there is a bit of a "ladder" -- so e.g. we might (eventually) have some crate that defines type representations, and presumably other steps will use that

nikomatsakis (Sep 13 2019 at 14:32, on Zulip):

so e.g. one could imagine that library-ification of the parser means that librustc does not depend on a parser

I think this is not as imporant -- librustc might link directly to the parser crate, but it's more about "other people can drive the parser in a relatively straight-forward way"

nikomatsakis (Sep 13 2019 at 14:32, on Zulip):

e.g., the lexer was extracted into a library

centril (Sep 13 2019 at 14:32, on Zulip):

@nikomatsakis that feels very clean -- we have crates defining data representation, and crates defining transforms between those representations

nikomatsakis (Sep 13 2019 at 14:32, on Zulip):

it doesn't mean that rustc doesn't know which lexer it's using

nikomatsakis (Sep 13 2019 at 14:32, on Zulip):

(this is kind of what I was getting at with this "ladder" discussion above)

varkor (Sep 13 2019 at 14:32, on Zulip):

it's the sort of process that would make it easier to write an independent compiler, reusing some of the same components

nikomatsakis (Sep 13 2019 at 14:33, on Zulip):

(that said, we would want to reduce coupling where we can)

mw (Sep 13 2019 at 14:33, on Zulip):

I'd like to remove steps from the ladder :)

centril (Sep 13 2019 at 14:33, on Zulip):

(https://www.youtube.com/watch?v=jlPaby7suOc ^,-)

matklad (Sep 13 2019 at 14:33, on Zulip):

@nikomatsakis although it would be lovely to install a compilation firewall (a-la C++'s pimpl) between the lexer and the rest of the compiler, just to avoid rebuilds. But I think rust doesn't have good idioms for this yet.

nikomatsakis (Sep 13 2019 at 14:33, on Zulip):

say more, @mw? one question I've been wondering about is what makes sense to think of a "unit". I make a case in the hackmd doc, for example, that "chalk" should encompass more things -- e.g. definition of types, unification, and constraint solving

nikomatsakis (Sep 13 2019 at 14:34, on Zulip):

although it would be lovely to install a compilation firewall (a-la C++'s pimpl) between the lexer and the rest of the compiler, just to avoid rebuilds. But I think rust doesn't have good idioms for this yet.

yes, I just think that's mildly orthogonal.

centril (Sep 13 2019 at 14:34, on Zulip):

But I think rust doesn't have good idioms for this yet.

How do you mean?

centril (Sep 13 2019 at 14:34, on Zulip):

(re. coherence and traits, one can just... not use traits for this... ^^)

eddyb (Sep 13 2019 at 14:34, on Zulip):

Servo uses crates that only have data types and traits in them

eddyb (Sep 13 2019 at 14:34, on Zulip):

a lot

eddyb (Sep 13 2019 at 14:34, on Zulip):

specifically to avoid recompilation when changing the implementation

nikomatsakis (Sep 13 2019 at 14:35, on Zulip):

it seems like finding the "right" place to draw boundaries is important on this point

mw (Sep 13 2019 at 14:35, on Zulip):

yeah, I think it would be good to have a shallow graph of crates

nikomatsakis (Sep 13 2019 at 14:35, on Zulip):

but also that doing better with incremental may make it a moot point

nikomatsakis (Sep 13 2019 at 14:35, on Zulip):

(if compilaton time is the sole concern)

mw (Sep 13 2019 at 14:36, on Zulip):

having a split between type definition crates and logic crates seems good

centril (Sep 13 2019 at 14:36, on Zulip):

@eddyb those traits being internal to the crate & then free functions between?

eddyb (Sep 13 2019 at 14:36, on Zulip):

no it's just a set of data and behaviors

mw (Sep 13 2019 at 14:36, on Zulip):

and having infrastructure crates, like salsa

eddyb (Sep 13 2019 at 14:36, on Zulip):

that another crate implements

centril (Sep 13 2019 at 14:36, on Zulip):

so libast, libparser, libfeature_gate, etc. then?

nikomatsakis (Sep 13 2019 at 14:36, on Zulip):

having a split between type definition crates and logic crates seems good

(I agree with this, incidentally, I do like the idea of factoring out "the IR" from the rest)

mw (Sep 13 2019 at 14:36, on Zulip):

and diagnostics would also be infrastructure

eddyb (Sep 13 2019 at 14:37, on Zulip):

@centril not that fine-grained

eddyb (Sep 13 2019 at 14:37, on Zulip):

more like libsyntax_traits and libsyntax

nikomatsakis (Sep 13 2019 at 14:37, on Zulip):

do we now feel like we have a good idea what "library-ification" is? maybe we should try to summarize. I think we captured the spirit of our goal, but we also came up with various guidelines

nikomatsakis (Sep 13 2019 at 14:37, on Zulip):

the spirit being: a set of reusable components that could be combined to build new tools, new compilers, etc.

nikomatsakis (Sep 13 2019 at 14:37, on Zulip):

as distinct from just "lots of crates"

centril (Sep 13 2019 at 14:38, on Zulip):

@eddyb hmm... maybe libfeature_gate is excessive, (maybe not...) but libast & libparser seems like what @mw is saying

nikomatsakis (Sep 13 2019 at 14:38, on Zulip):

some guidelines might be:

nikomatsakis (Sep 13 2019 at 14:38, on Zulip):

(did I miss things?)

centril (Sep 13 2019 at 14:38, on Zulip):

so we have parser + lowering communicating via libast and they don't know each other otherwise

eddyb (Sep 13 2019 at 14:39, on Zulip):

yeah and you can substitute libparser if you wanted to

nikomatsakis (Sep 13 2019 at 14:39, on Zulip):

certainly a lot of this structure is "inherent" in the compiler already, albeit in a messy form

mw (Sep 13 2019 at 14:39, on Zulip):

yeah

nikomatsakis (Sep 13 2019 at 14:39, on Zulip):

so we're trying to tease that out

simulacrum (Sep 13 2019 at 14:39, on Zulip):

Though replacing libparser would be sort of "hard" i.e., maybe you'd need to patch it out via Cargo.toml, we don't want to get into dylibs here or w/e

simulacrum (Sep 13 2019 at 14:40, on Zulip):

(right?)

eddyb (Sep 13 2019 at 14:40, on Zulip):

yeah I meant source-wise

nikomatsakis (Sep 13 2019 at 14:40, on Zulip):

I'd lke to keep "hot swapping" components as a "tabled" topic :)

eddyb (Sep 13 2019 at 14:40, on Zulip):

you could build two impls and then switch them in the Cargo.toml

matklad (Sep 13 2019 at 14:40, on Zulip):

@nikomatsakis I think one missing bit is "making sure that libraries work for IDE"

mw (Sep 13 2019 at 14:40, on Zulip):

I'd like it if things like specific string interning strategies would not need to be known throughout the compiler

matklad (Sep 13 2019 at 14:40, on Zulip):

We could extract libast from current libsyntax, but that would be pretty useless for rust-analyzer

nikomatsakis (Sep 13 2019 at 14:40, on Zulip):

I think one missing bit is "making sure that libraries work for IDE"

that's a good point -- the design of the interface needs to accommodate "on demand" processing

eddyb (Sep 13 2019 at 14:41, on Zulip):

@mw if we had a trait for interning and a trait for accessing the data, that'd be better right?

simulacrum (Sep 13 2019 at 14:41, on Zulip):

do we see that as a first order bit though? It seems like it might be something we'd want to wait on, right?

nikomatsakis (Sep 13 2019 at 14:41, on Zulip):

(I left some notes in the doc of places where chalk's queries, for example, were suboptimal and we need to rework them a bit)

mw (Sep 13 2019 at 14:41, on Zulip):

@eddyb yeah, something like that.

Igor Matuszewski (Sep 13 2019 at 14:41, on Zulip):

also such libast should be possible to encode possibly partially missing/invalid AST?

nikomatsakis (Sep 13 2019 at 14:41, on Zulip):

do we see that as a first order bit though? It seems like it might be something we'd want to wait on, right?

mm I think it's pretty imporant

eddyb (Sep 13 2019 at 14:41, on Zulip):

I actually use Index<IStr, Output = str> in grammer

Igor Matuszewski (Sep 13 2019 at 14:41, on Zulip):

as is often the case for the IDEs

centril (Sep 13 2019 at 14:41, on Zulip):

@Igor Matuszewski can have sentinels for that

nikomatsakis (Sep 13 2019 at 14:41, on Zulip):

yes, that's important

centril (Sep 13 2019 at 14:41, on Zulip):

feels like typed holes

mw (Sep 13 2019 at 14:41, on Zulip):

@Igor Matuszewski that's an interesting question

nikomatsakis (Sep 13 2019 at 14:42, on Zulip):

this is another thing that came up around chalk + rust-analyzer, wnating e.g. an error type

simulacrum (Sep 13 2019 at 14:42, on Zulip):

I think it's important, yes, but it's not clear to me that we don't get a lot of benefit from doing this library-ification internally without bringing IDEs into it :)

mw (Sep 13 2019 at 14:42, on Zulip):

i.e. there are probably things were the reqs for a batch compiler and IDEs diverge

nikomatsakis (Sep 13 2019 at 14:42, on Zulip):

well, I think we get benefit, but it seems like if we're going through the trouble to design an interface

simulacrum (Sep 13 2019 at 14:42, on Zulip):

I also want to avoid putting two things together and then never actually doing it

nikomatsakis (Sep 13 2019 at 14:42, on Zulip):

we shoulld take the time to think about it

centril (Sep 13 2019 at 14:43, on Zulip):

we don't have to do all at once

centril (Sep 13 2019 at 14:43, on Zulip):

can just do libast, parser, lowering as one step for example

simulacrum (Sep 13 2019 at 14:43, on Zulip):

But I think a viable outcome of said thinking is "we can start before finishing our thinking"

pnkfelix (Sep 13 2019 at 14:43, on Zulip):

sure, but improving the IDE experience is an important task for both short- and long-term

nikomatsakis (Sep 13 2019 at 14:43, on Zulip):

we don't, but I think we should prioritize things that can help with the IDE experience -- and yes, I think we would do things in stages, and maybe we just start by trying to sever links

nikomatsakis (Sep 13 2019 at 14:44, on Zulip):

but e.g. we're not aiming to develop some "pristine" interface that doesn't have to worry about error reporting

nikomatsakis (Sep 13 2019 at 14:44, on Zulip):

or messy inputs

nikomatsakis (Sep 13 2019 at 14:44, on Zulip):

and those things are often important to think about early on

nikomatsakis (Sep 13 2019 at 14:44, on Zulip):

( chalk e.g. hasn't thought about them, but I think it should; to be fair, the existing trait solver in rust only barely does :P )

centril (Sep 13 2019 at 14:45, on Zulip):

@nikomatsakis sure, but that messiness already exists in e.g. libparser & friends so it will be noticed when we refactor?

nikomatsakis (Sep 13 2019 at 14:45, on Zulip):

anyway, is there a "disagreement" here? if so, I guess it might be about how to prioritize effort

nikomatsakis (Sep 13 2019 at 14:45, on Zulip):

but I think the answer is probably "not"

simulacrum (Sep 13 2019 at 14:45, on Zulip):

Yes, I think this is purely about prioritization/ordering

nikomatsakis (Sep 13 2019 at 14:45, on Zulip):

so we have 15 minutes left

centril (Sep 13 2019 at 14:46, on Zulip):

I identified two potential disagreements: 1) stable/nightly features & dogfooding, 2) mono/poly repo.
Otherwise it seems we have consensus for @mw's philosophy?

mw (Sep 13 2019 at 14:46, on Zulip):

anything that keeps rustc and rust-analyzer pulling in the same direction is good for now, I'd say

mw (Sep 13 2019 at 14:46, on Zulip):

I think a poly repo should only be done when everything is in really good shape for doing so

mw (Sep 13 2019 at 14:47, on Zulip):

it's the more difficult option

mw (Sep 13 2019 at 14:47, on Zulip):

which needs things to be in a clean, decoupled state

nikomatsakis (Sep 13 2019 at 14:47, on Zulip):

things I think we could dig into

mw (Sep 13 2019 at 14:47, on Zulip):

it might be a better option in the future though

nikomatsakis (Sep 13 2019 at 14:47, on Zulip):

I think if we are extracting something from rustc, doing it in repo makes a lot of sense

centril (Sep 13 2019 at 14:48, on Zulip):

:+1: -- I'd personally not do poly-repos ever so not doing it for a long time is good from that POV :slight_smile:

nikomatsakis (Sep 13 2019 at 14:48, on Zulip):

(and potentially moving out at some point)

nikomatsakis (Sep 13 2019 at 14:49, on Zulip):

maybe I should rephrase this -- can we do more to help support this effort? are there folks who'd like to express an interest in collaborating on some of these projects?

centril (Sep 13 2019 at 14:49, on Zulip):

(with polyrepos, implementing a new language change, e.g. or-patterns, that touches many parts of the compiler becomes really hard)

nikomatsakis (Sep 13 2019 at 14:50, on Zulip):

maybe name resolution is an area where we could think about getting more interaction?

pnkfelix (Sep 13 2019 at 14:51, on Zulip):

(I was supposed to help with nameres but compiler stuff, as well as life, was major distraction )

centril (Sep 13 2019 at 14:51, on Zulip):

I'm personally mostly interested in helping out with refactorings (I've been doing some internal ones in parser, lowering, and typeck)

nikomatsakis (Sep 13 2019 at 14:51, on Zulip):

I intend to be pushing hard in/around traits personally, though I'm trying to figure out how much time that means, particularly as I think we need to keep moving on bugs in/around rustc's trait system

nikomatsakis (Sep 13 2019 at 14:51, on Zulip):

I also think we shouldn't try to do everything at once

matklad (Sep 13 2019 at 14:51, on Zulip):

Specifically, name-resolution would be the most valuable thing for rust-analyzer, but it's also one of the thorniest bits

centril (Sep 13 2019 at 14:52, on Zulip):

@matklad libast,nameres,parser,lowering seem natural as a first pass then?

nikomatsakis (Sep 13 2019 at 14:52, on Zulip):

when you say you'd like more interaction from t-compiler, @matklad, what are you thinking of more specifically?

matklad (Sep 13 2019 at 14:52, on Zulip):

I personally am planning to keep the whole rust-analyzer afloat, as well as chipping bits of the rustc's parser here and there

nikomatsakis (Sep 13 2019 at 14:52, on Zulip):

e.g. I could imagine a few things

nikomatsakis (Sep 13 2019 at 14:53, on Zulip):

and more no doubt :)

centril (Sep 13 2019 at 14:53, on Zulip):

perhaps we can "let the types tell us what the phases of library-ification are"

nikomatsakis (Sep 13 2019 at 14:53, on Zulip):

I'm thinking that trying to create shared libraries is obviously a "forcing function" for coming to agreement around interfaces

nikomatsakis (Sep 13 2019 at 14:53, on Zulip):

it seems like we've been sort of side-stepping the ast/hir in the things we've chosen so far

nikomatsakis (Sep 13 2019 at 14:54, on Zulip):

which .. is probably good to start, but also putting off some of the most important work?

matklad (Sep 13 2019 at 14:54, on Zulip):

@centril I belive libast in the current state would be useful for rust-analyzer. Designing the ast that works for both rustc and rust-analyzer seems like a really major needs-design things. I'd prefer to clean up things around AST first. In particular, I think it's possible to make a parser that doens't know a specific AST type

nikomatsakis (Sep 13 2019 at 14:54, on Zulip):

(I guess it depends on how the parser works)

matklad (Sep 13 2019 at 14:54, on Zulip):

s/useful/useless :)

centril (Sep 13 2019 at 14:55, on Zulip):

@matklad are you thinking of some sort of "tagless final interpreter" scheme?

nikomatsakis (Sep 13 2019 at 14:55, on Zulip):

maybe we should use last 5 minutes or so to talk about follow-up actions? do we want to try to have follow-up meetings, for example, to talk about specific library-ification details?

nikomatsakis (Sep 13 2019 at 14:55, on Zulip):

(or not meetings, just splinter out :)

matklad (Sep 13 2019 at 14:55, on Zulip):

I am thinking about untyped event-based parser :)

nikomatsakis (Sep 13 2019 at 14:55, on Zulip):

(e.g. I'd like to folow along with how a parser library-ification will work)

matklad (Sep 13 2019 at 14:55, on Zulip):

+1, I thin the parser plans needs a separate meeting

nikomatsakis (Sep 13 2019 at 14:56, on Zulip):

it sounds like a small core of folks could get together

nikomatsakis (Sep 13 2019 at 14:56, on Zulip):

talk over zulip and whatever

nikomatsakis (Sep 13 2019 at 14:56, on Zulip):

and then talk to the rest of us about it? :)

centril (Sep 13 2019 at 14:56, on Zulip):

I'd be game for that since I've done a lot of hacking on the parser lately

nikomatsakis (Sep 13 2019 at 14:56, on Zulip):

(I was hoping to do something similar around traits/types -- try to make more concrete plans and come back with them, say in a month or so)

matklad (Sep 13 2019 at 14:57, on Zulip):

SGTM, though, for parser specifically, I feel like "moving to proc-macros token model" is prerequisite, and that is a no-small chunk of work

nikomatsakis (Sep 13 2019 at 14:57, on Zulip):

say a bit more?

nikomatsakis (Sep 13 2019 at 14:57, on Zulip):

do you mean doing that in rustc's parser?

matklad (Sep 13 2019 at 14:57, on Zulip):

https://github.com/rust-lang/rust/issues/63689

eddyb (Sep 13 2019 at 14:57, on Zulip):

I am thinking about untyped event-based parser :)

untyped: sure, fine, grammer is like that
event-based: huh, I thought that only made sense for LISP and XML, and maybe our token trees :P

nikomatsakis (Sep 13 2019 at 14:58, on Zulip):

do you feel like you can't talk through what the parser interface would look like after that?

nikomatsakis (Sep 13 2019 at 14:58, on Zulip):

or just that it's not worth the effort?

eddyb (Sep 13 2019 at 14:58, on Zulip):

it's mostly internal though isn't it?

nikomatsakis (Sep 13 2019 at 14:58, on Zulip):

it seems like it'd be good to get agreement (if possible) on the overall direction, probably in parallel with refactoring and working on #63689?

eddyb (Sep 13 2019 at 14:58, on Zulip):

the interface already takes TokenStream

eddyb (Sep 13 2019 at 14:58, on Zulip):

it's just that we have two token representations and convert on the fly between them

nikomatsakis (Sep 13 2019 at 14:59, on Zulip):

ok; what I meant was, it seems like there's some questions around how to get the output from the parser

nikomatsakis (Sep 13 2019 at 14:59, on Zulip):

anyway either way is ok for me :)

centril (Sep 13 2019 at 14:59, on Zulip):

@matklad tagless final would mean that we use traits to encode "constructor" rather than using enums to have specific ones

centril (Sep 13 2019 at 14:59, on Zulip):

then you'll need "is X variant" in some cases as well

matklad (Sep 13 2019 at 15:00, on Zulip):

I am a little confused, but my point is: #63689 seems like a prerequsite for anything else, so I don't think there's a ton of value in discussing specifics until iti s fixed

centril (Sep 13 2019 at 15:00, on Zulip):

@matklad I think there's value for rustc tho in terms of splitting parser/ast/lowering

eddyb (Sep 13 2019 at 15:00, on Zulip):

@centril ah, the mad scientist approach to data

nikomatsakis (Sep 13 2019 at 15:01, on Zulip):

if I may summarize the meeting, in broad strokes:

sound right?

eddyb (Sep 13 2019 at 15:01, on Zulip):

@centril I actually wanted something like that for the gll crate's autogenerated types, so you can easily make an AST/have actions/etc. but now that I'm moving to a more dynamic model I care less

nikomatsakis (Sep 13 2019 at 15:01, on Zulip):

and you all fork this parser-specific discussion into a different topic :P

nikomatsakis (Sep 13 2019 at 15:01, on Zulip):

/me wishes Zulip had a way to tag individual messages

centril (Sep 13 2019 at 15:01, on Zulip):

@eddyb If by "mad scientist" you mean Oleg Kiselyov

eddyb (Sep 13 2019 at 15:02, on Zulip):

(but I could still help someone deliver that ridiculous vision. it uhh involves type HRTB stuff)

matklad (Sep 13 2019 at 15:02, on Zulip):

And if some folks want to spearhead macros/nameres, that would be awesome :D

eddyb (Sep 13 2019 at 15:02, on Zulip):

I want to but I doubt I have the time

eddyb (Sep 13 2019 at 15:02, on Zulip):

(and I want to do it in a way none of you would like except maybe if it did work)

nikomatsakis (Sep 13 2019 at 15:02, on Zulip):

Yeah, I totally want to say "we'll do that!" but I'm feeling like the right answer is "and that's next on our queue"

eddyb (Sep 13 2019 at 15:03, on Zulip):

@matklad something that might not be obvious: we tests things through compiler errors most of the time. would be nice to actually dumb the resolutions of paths somehow and make sure those make sense, especially for the complex macro stuff

nikomatsakis (Sep 13 2019 at 15:04, on Zulip):

OK thanks :heart: we're out of time, this was awesome as ever, y'all are wonderful. :heart_eyes:

eddyb (Sep 13 2019 at 15:04, on Zulip):

maybe fix the pretty-printer so it uses modern indentation rules :P

nikomatsakis (Sep 13 2019 at 15:05, on Zulip):

something that might not be obvious: we tests things through compiler errors most of the time. would be nice to actually dumb the resolutions of paths somehow and make sure those make sense, especially for the complex macro stuff

I've wondered about this -- I sometimes add funky things to the compiler to make errors for internal state like this that wouldn't otherwise be visible. I really like this approach. But it's only "sort of" an integration test -- still, I think it's a good approach, as long as that state that's being dumped is something that would likely be "forwards portable" to a new system

centril (Sep 13 2019 at 15:06, on Zulip):

(I made a new topic for frontend ast/parser/macros/nameres/lowering stuff)

matklad (Sep 13 2019 at 15:06, on Zulip):

In rust-analyzer, nameres/macro expansion tests dump, for each module, the set of visible names (with namespaeces)

matklad (Sep 14 2019 at 21:50, on Zulip):

@centril expanded a bit on "rust doesn't have good idioms for pimpl": https://internals.rust-lang.org/t/request-for-an-rfc-pimpl-for-rust/10947

Last update: Nov 22 2019 at 05:40UTC