Stream: t-compiler/wg-rls-2.0

Topic: weekly sync-up


matklad (Sep 03 2019 at 12:20, on Zulip):

:wave: @nikomatsakis

nikomatsakis (Sep 03 2019 at 12:21, on Zulip):

So, yeah, last Friday I spent some time reading into Chalk and (somewhat) into the RLS

nikomatsakis (Sep 03 2019 at 12:21, on Zulip):

Trying to remember the lay of the land

nikomatsakis (Sep 03 2019 at 12:22, on Zulip):

I came up with a number of improvements and steps that we could take in/around Chalk, but they're kind of low-level specifics.

nikomatsakis (Sep 03 2019 at 12:23, on Zulip):

I guess it's worth stepping back a bit to review our overall goals over this week etc

matklad (Sep 03 2019 at 12:23, on Zulip):

I'd say the high order bit right now is fixing the "chalk spins CPU at 100%" problem

nikomatsakis (Sep 03 2019 at 12:24, on Zulip):

Well, depends I guess ;)

matklad (Sep 03 2019 at 12:24, on Zulip):

Yeah, reviewing goals seems right, but let me quickly note that happend on the ra side recently

nikomatsakis (Sep 03 2019 at 12:24, on Zulip):

go for it

matklad (Sep 03 2019 at 12:25, on Zulip):
matklad (Sep 03 2019 at 12:26, on Zulip):

The last bit is interesting, in that I think starts are aligned perfectly to make us reconsider file watching and general interractions with file system. I'd love to transition to lazy file loading

matklad (Sep 03 2019 at 12:26, on Zulip):

(hence that question in salsa's zulip)

nikomatsakis (Sep 03 2019 at 12:26, on Zulip):

Yeah, that is interesting

matklad (Sep 03 2019 at 12:27, on Zulip):

oh, and, on rustc side, I do first small steps to make rustc parser use the same token model as proc-macros use (with jointness)

matklad (Sep 03 2019 at 12:27, on Zulip):

that's all I can remember I guess?

nikomatsakis (Sep 03 2019 at 12:28, on Zulip):

So I guess one question is -- we had planned to have a compiler team meeting. One of my immediate goals was to .. figure out what to be discussing there.

nikomatsakis (Sep 03 2019 at 12:29, on Zulip):

I guess I can see a few things to do

matklad (Sep 03 2019 at 12:30, on Zulip):

Say more?

nikomatsakis (Sep 03 2019 at 12:30, on Zulip):

Well I don't think we're going to be in a position to propose a detailed technical propsal around chalk -- but we could focus on improving the experience in rust-analyzer and try to give an overview of where it is and discuss the idea of trying to extract chalk/type-checker into shared library, and maybe float out some of the challenges there.

nikomatsakis (Sep 03 2019 at 12:31, on Zulip):

I was vacillating about how much it makes sense to try and propose specific technical designs

nikomatsakis (Sep 03 2019 at 12:31, on Zulip):

But I think the answer is probably not at all relatively little, at least not yet

matklad (Sep 03 2019 at 12:33, on Zulip):

I am not sure: I think, at least in general terms, everyone is on board with the idea. So, I think it would be valuable to:

nikomatsakis (Sep 03 2019 at 12:33, on Zulip):

I'd say the high order bit right now is fixing the "chalk spins CPU at 100%" problem

On this topic, one thing I was thinking about was that it'd be nice to have a mode for RA (or chalk, perhaps) where it can basically dump out a self-contained foo.chalk file that tracks the goal(s) it is solving, so that we can easily reproduce these problems and isolate them

matklad (Sep 03 2019 at 12:34, on Zulip):

Would that really be more useful than dumping the rust source-code?

nikomatsakis (Sep 03 2019 at 12:35, on Zulip):

seems like yes

nikomatsakis (Sep 03 2019 at 12:35, on Zulip):

it could be minimzed more readily, I think, and it's smaller than the rust source code

matklad (Sep 03 2019 at 12:35, on Zulip):

Like, I think it doesn't really help with isolation: a small source file and a small goal look pretty equivalent to me? You can dbg!(goal) at the entry to chalk to see the actual goal.

nikomatsakis (Sep 03 2019 at 12:36, on Zulip):

it's more about the "environment" around the goal

matklad (Sep 03 2019 at 12:36, on Zulip):

And for minimization the biggest problem seems to the the envirionment with a lot of traits, and minimizing that seems not easy either way?

nikomatsakis (Sep 03 2019 at 12:36, on Zulip):

right now you have to debug that from within rust-analyzer

nikomatsakis (Sep 03 2019 at 12:37, on Zulip):

I'd like to be able to create stronger boundaries, so that you can debug and test chalk problems without really knowing much about rust-analyzer

nikomatsakis (Sep 03 2019 at 12:37, on Zulip):

I feel like that's some of the promise of library-ification is to enable these sorts of boundaries

nikomatsakis (Sep 03 2019 at 12:37, on Zulip):

(speaking as someone who feels "uncomfortable" in the r-a code base, I can attest it'd be nice if there were some standalone file I could load up into chalk)

matklad (Sep 03 2019 at 12:37, on Zulip):

that's true :)

nikomatsakis (Sep 03 2019 at 12:38, on Zulip):

that said I agree it's maybe more of a "side goal"

Florian Diebold (Sep 03 2019 at 12:38, on Zulip):

One thing to note about the Chalk-spinning-problem is that we're currently still producing a lot of 'broken' clauses (because of name resolution bugs or missing features), so in the cases I've looked into the problem seemed to me to be that it was enumerating solutions for some subgoal where in the end the final goal had no solution. My point being, maybe these problems aren't really representative of 'real' situations, where usually only a few clauses/impls are broken like that :)

nikomatsakis (Sep 03 2019 at 12:38, on Zulip):

I am not sure: I think, at least in general terms, everyone is on board with the idea. So, I think it would be valuable to:

in any case you're probably right about this

matklad (Sep 03 2019 at 12:38, on Zulip):

Actually, I think figuring out such things should be the primary goal !

matklad (Sep 03 2019 at 12:39, on Zulip):

It seems like retrofit such inspectability would be hard after the fact

matklad (Sep 03 2019 at 12:39, on Zulip):

the main problem I see here is that we want pull-based interface from chalk: there isn't a "world" that we can just serde_json::to_string, we have a bunch of callbacks...

nikomatsakis (Sep 03 2019 at 12:40, on Zulip):

well, chalk pulls info from r-a;

nikomatsakis (Sep 03 2019 at 12:40, on Zulip):

I was imagining that a given engine incarnation would gradually dump the info it pulls as it pulls it into a file

nikomatsakis (Sep 03 2019 at 12:40, on Zulip):

along with goals

nikomatsakis (Sep 03 2019 at 12:40, on Zulip):

so you could kind of load that up and "replay"

nikomatsakis (Sep 03 2019 at 12:40, on Zulip):

so, a very relevant question -- I think recently you were asking @matklad about integration vs unit tests

nikomatsakis (Sep 03 2019 at 12:41, on Zulip):

I guess a real question is

nikomatsakis (Sep 03 2019 at 12:41, on Zulip):

suppose we encountered some bug in chalk via rust-analyzer, would we want to (as I'm vagualy proposing) extract out a .chalk file and add that as a unit test to chalk,

nikomatsakis (Sep 03 2019 at 12:41, on Zulip):

or try to reduce it to Rust source

nikomatsakis (Sep 03 2019 at 12:41, on Zulip):

I think in general I tend towards the latter because it's more "canonical"

nikomatsakis (Sep 03 2019 at 12:42, on Zulip):

but I think in some ideal world we'd have relatively robust, Rust-like interfaces

nikomatsakis (Sep 03 2019 at 12:43, on Zulip):

anyway, that's kind of a bigger scope than I was initially imagining :) beacuse you wouldn't want like serialized chalk data structures, but some Rust-subset or something that captures the problem setup I guess?

nikomatsakis (Sep 03 2019 at 12:43, on Zulip):

plausible, just more complex

matklad (Sep 03 2019 at 12:44, on Zulip):

So, I think what we really want for tests is "data in, data out" setup

matklad (Sep 03 2019 at 12:44, on Zulip):

Boundary which is not data, but is an interface (callbacks, etc) is much harder to keep stable

nikomatsakis (Sep 03 2019 at 12:45, on Zulip):

One thing to note about the Chalk-spinning-problem is that we're currently still producing a lot of 'broken' clauses ...

interesting, btw, I hadn't realize that.

nikomatsakis (Sep 03 2019 at 12:45, on Zulip):

So, I think what we really want for tests is "data in, data out" setup

Yes for sure this

nikomatsakis (Sep 03 2019 at 12:45, on Zulip):

And I think also that the "data in" has to be pretty high-level

matklad (Sep 03 2019 at 12:45, on Zulip):

So, for thinkgs like lexing or parsing where the input and output data has simple format, unit-tests and this boundary seem OK

nikomatsakis (Sep 03 2019 at 12:45, on Zulip):

Something like a .chalk file is actually pretty plausible (i.e., some subset of Rust syntax)

matklad (Sep 03 2019 at 12:46, on Zulip):

Seems so

nikomatsakis (Sep 03 2019 at 12:47, on Zulip):

It seems like this should be part of the "things we figure out" when deciding on a good library boundary

matklad (Sep 03 2019 at 12:47, on Zulip):

(BTW, I like how Swift .SIL is a real well-defined intermediate langauge which you can read from string)

nikomatsakis (Sep 03 2019 at 12:47, on Zulip):

I'd like to see MIR becoming comparably stable (and spec'd)

nikomatsakis (Sep 03 2019 at 12:47, on Zulip):

In any case, to circle back a bit, re: chalk spinning and being slow

nikomatsakis (Sep 03 2019 at 12:48, on Zulip):

So one thing that @Florian Diebold pointed out to me was issues around normalization; I have specific thoughts on those, but I'll try to write them out in chalk issues / on that channel

nikomatsakis (Sep 03 2019 at 12:48, on Zulip):

But more deeply I am wondering about two options:

nikomatsakis (Sep 03 2019 at 12:48, on Zulip):
nikomatsakis (Sep 03 2019 at 12:49, on Zulip):

I think for fuel (as I wrote somewhere) what I'd want to do is to refactor chalk's loop to not do looping, so that fuel can be enforced more at the r-a level (stop pulling for answers after enough iterations)

nikomatsakis (Sep 03 2019 at 12:49, on Zulip):

(though @Florian Diebold had some concerns, we can discuss that a bit separately)

nikomatsakis (Sep 03 2019 at 12:50, on Zulip):

the main question is: prioritize the general mechanism first? or solve the specific bugs?

nikomatsakis (Sep 03 2019 at 12:50, on Zulip):

it feels like the general mechanism will be needed at the end of the day no matter what

matklad (Sep 03 2019 at 12:50, on Zulip):

Specific bugs should have priority

matklad (Sep 03 2019 at 12:50, on Zulip):

fuel can too easily mask bugs

nikomatsakis (Sep 03 2019 at 12:50, on Zulip):

Interesting

nikomatsakis (Sep 03 2019 at 12:51, on Zulip):

fuel can too easily mask bugs

yeah so this is what I was wrestling back and forth with :)

Florian Diebold (Sep 03 2019 at 12:54, on Zulip):

(though Florian Diebold had some concerns, we can discuss that a bit separately)

not so much concerns as, I couldn't figure out a good way to do it ;)

matklad (Sep 03 2019 at 12:54, on Zulip):

My point being, maybe these problems aren't really representative of 'real' situations, where usually only a few clauses/impls are broken like that :)

It still would be nice if chalk degraded gracefully in such cases. Like, if you mistyped a dependency name in Cargo.toml you still can get a lot of broken code even for "normal" projects

Florian Diebold (Sep 03 2019 at 12:55, on Zulip):

yeah, it would probably be fine if that was caught by fuel though

matklad (Sep 03 2019 at 12:58, on Zulip):

fine, yeah, but not perfect. I think ideally fuel should be required only for explicitelly adversary code, like "look at my type-level Turing machine!". For random garbage, I intuitively feel that arriving an "there's no answer" should be fast

nikomatsakis (Sep 03 2019 at 12:58, on Zulip):

I'm trying to summarize up some of our thoughts so far

1. For meeting, we should be thinking about identifying key technical challenges and maybe proposing specific boundaries to explore
- interesting question: what do unit tests look like? How can we enable people to be experts on one side but not the other?
2. For chalk and r-a etc, let's look at r-a pain points and try to diagnose them
- fuel is a somewhat lower priority
3. experience from the second point kind of feeds into the first

nikomatsakis (Sep 03 2019 at 12:59, on Zulip):

seems like zulip's markdown parser doesn't like my mixed bullet/numbered/indented lists but hopefully can you read that :)

matklad (Sep 03 2019 at 13:00, on Zulip):

4. file IO is another acute pain point at the moment, which needs attention

nikomatsakis (Sep 03 2019 at 13:01, on Zulip):

As far as specific boundaries for libraries, I think that there is an obvious one that encompasses:

and then there's another level up that includes the type checker. The distinction I am drawing is basically that the type checker requires us to model the HIR and the full set of Rust expressions.

matklad (Sep 03 2019 at 13:01, on Zulip):

How can we enable people to be experts on one side but not the other?

Unit tests should also help tremendously with rebuild times: i guess building only chalk is much faster than building ra + chalk (and we really should be splitting our ra_hir which does everything into crates )

nikomatsakis (Sep 03 2019 at 13:01, on Zulip):

4. file IO is another acute pain point at the moment, which needs attention

yes and this is partially a salsa problem, right? We can discuss a bit more about how to model "lazy inputs" on the salsa side, it doesn't seem all that hard to me

Florian Diebold (Sep 03 2019 at 13:02, on Zulip):

@nikomatsakis what do you mean by "type inference" separate from type checking there?

matklad (Sep 03 2019 at 13:02, on Zulip):

Yeah, salsa's part seems easy, what is hard is cross-platform file watching

nikomatsakis (Sep 03 2019 at 13:03, on Zulip):

@Florian Diebold I mean encompassing operations like T1 <: T2 (make T1 and subtype of T2) and "coerce T1 to T2"

nikomatsakis (Sep 03 2019 at 13:03, on Zulip):

basically, you can think of type-checking as "generating a bunch of constraints" and "solving a bunch of constraints" -- I am separating out the solving part

nikomatsakis (Sep 03 2019 at 13:03, on Zulip):

the generation comes from walking the HIR

Florian Diebold (Sep 03 2019 at 13:03, on Zulip):

ah ok

nikomatsakis (Sep 03 2019 at 13:04, on Zulip):

OK. So I think I'll have some more time to devote to this late this afternoon or tomorrow. I'm going to try to convert some of my current thoughts into a vague paper doc and send it to y'all for feedback -- and maybe hack a bit on chalk because I can't help myself :P

nikomatsakis (Sep 03 2019 at 13:05, on Zulip):

Yeah, salsa's part seems easy, what is hard is cross-platform file watching

I feel I personally don't have much to contribute to that latter bit :)

nikomatsakis (Sep 03 2019 at 13:05, on Zulip):

That's a good example of the kind of thing I endeavor not to learn :P

matklad (Sep 03 2019 at 13:06, on Zulip):

I'd rather not to as well, but I fear I had to :D

nikomatsakis (Sep 03 2019 at 13:06, on Zulip):

that and linkers

nikomatsakis (Sep 03 2019 at 13:06, on Zulip):

In all seriousness, though, and I guess we want to be sure to keep in mind use cases like FB, where there are too many files -- but I guess that's somewhat addressed just by laziness.

matklad (Sep 03 2019 at 13:07, on Zulip):

Yeah, that's another recent conversation that convinced me that the current eager model is no right

matklad (Sep 03 2019 at 13:07, on Zulip):

(I suggest we table file-watching discussion though, for the lack of relevant expertise among participants :) )

matklad (Sep 03 2019 at 13:11, on Zulip):

and maybe hack a bit on chalk because I can't help myself :P

@nikomatsakis I've noticed that "trivial" prs like "update deps" are not merged into chalk on a timely basis. Do you think we should try to actively seek chalks co-maintainer?

Florian Diebold (Sep 03 2019 at 13:11, on Zulip):

@nikomatsakis so concerning using Chalk's Ty from RA, there are a few points that come to mind:
- I think we need a Ty::Error in Chalk (I currently handle this in a not-great way that may actually be contributing to the hangs)
- it would be nice if we could get rid of the lalrpop_intern::InternedString in UnselectedProjectionTy, maybe we could just introduce a new Id type for associated type names? I'd be up for trying that out

nikomatsakis (Sep 03 2019 at 21:24, on Zulip):

we could and probably should remove the "unselected" stuff for now

matklad (Sep 07 2019 at 18:24, on Zulip):

In term of reproducing a problem, I've spend the last hour looking at the cycling error (in macro expansion) on CrateId(52) somewhere in rustc. I guess a nice first step would be to add crate.dbg(db) method, which prints actually useful information. I am somewhat proud by the fact that so little information is exposed to our HIR, but debugging is a pain in the back due to this

nikomatsakis (Sep 09 2019 at 17:24, on Zulip):

Hey @matklad so I'm around now-ish at last

nikomatsakis (Sep 09 2019 at 17:24, on Zulip):

You still around?

matklad (Sep 09 2019 at 17:24, on Zulip):

yep!

matklad (Sep 09 2019 at 17:25, on Zulip):

So, some things that has happend:

matklad (Sep 09 2019 at 17:26, on Zulip):
nikomatsakis (Sep 09 2019 at 17:26, on Zulip):

what is "watchman"?

nikomatsakis (Sep 09 2019 at 17:26, on Zulip):

i.e., is that some pre-existing library?

matklad (Sep 09 2019 at 17:27, on Zulip):
matklad (Sep 09 2019 at 17:27, on Zulip):

Watchman is Facebooks daemon for file watching

matklad (Sep 09 2019 at 17:28, on Zulip):

Ie, it is sort of a "library" for file watching, but it exists in a separate process, because OS APIs are horrible and it's hard to make a library

matklad (Sep 09 2019 at 17:29, on Zulip):
matklad (Sep 09 2019 at 17:30, on Zulip):
matklad (Sep 09 2019 at 17:31, on Zulip):
matklad (Sep 09 2019 at 17:31, on Zulip):

And I think that's all bigger things that have happend since last week

nikomatsakis (Sep 09 2019 at 17:31, on Zulip):

ok. On my part not much has happened :) but I'm thinking about this upcoming Fri meeting

nikomatsakis (Sep 09 2019 at 17:32, on Zulip):

On Friday I was mostly focused on looking into some soundness related problems unrelated to rust-analyzer :)

nikomatsakis (Sep 09 2019 at 17:32, on Zulip):

That and reviewing various PRs

nikomatsakis (Sep 09 2019 at 17:35, on Zulip):

Anyway I'm going to try and draw up a start of a plan today sketching out how the type checker could be separated out, and also give some feedback in/around rust-analyzer/chalk problems.

matklad (Sep 09 2019 at 17:36, on Zulip):

@nikomatsakis what's the prep I need to do for Friday meeting? Short update about rust-analyzer + rough plan for librarification (are doing/can do/should do/must do table)?

matklad (Sep 09 2019 at 17:36, on Zulip):

Anything else?

nikomatsakis (Sep 09 2019 at 17:36, on Zulip):

That's what I'm trying to figure out

nikomatsakis (Sep 09 2019 at 17:36, on Zulip):

Let's start creating some kind of rough sketch

nikomatsakis (Sep 09 2019 at 17:36, on Zulip):

https://hackmd.io/_ixQrAr5TKafV9m8eSU_VA

nikomatsakis (Sep 09 2019 at 17:36, on Zulip):

I think we've probably said a fair amount of things

nikomatsakis (Sep 09 2019 at 17:37, on Zulip):

in our overall approach

nikomatsakis (Sep 09 2019 at 17:37, on Zulip):

that would be good to document and try to get feedback on

matklad (Sep 09 2019 at 17:38, on Zulip):

ok, I'll jump to the hack.md document then!

nikomatsakis (Sep 09 2019 at 17:39, on Zulip):

great; I'm trying to figure out how to organize it, so feel free to edit

matklad (Sep 09 2019 at 17:42, on Zulip):

Oh, I've also noticed that there's a GSOC happening in Swift, whose goal is to move swift compiler over to Swift's libsyntax. Basically, the same which we'll need to do in rustc. I've contacted the person who does GSOC and asked them to blog about their experience :)

nikomatsakis (Sep 09 2019 at 17:44, on Zulip):

Nice

nikomatsakis (Sep 09 2019 at 18:02, on Zulip):

@matklad one of the things I am wondering about

nikomatsakis (Sep 09 2019 at 18:03, on Zulip):

to what extent does it make sense to try and code libraries "generically", so that they are not tied to a specific representation of types, HIR, etc

nikomatsakis (Sep 09 2019 at 18:03, on Zulip):

and to what extent should we just be sharing the definitions

matklad (Sep 09 2019 at 18:03, on Zulip):

I'd love to be concrete if we can

nikomatsakis (Sep 09 2019 at 18:03, on Zulip):

I agree

nikomatsakis (Sep 09 2019 at 18:03, on Zulip):

That suggests that an area of focus might be on figuring out how to share the actual definitions

matklad (Sep 09 2019 at 18:04, on Zulip):

And I think with index-based structure you can acutlally do that: you can always key additional info by index

nikomatsakis (Sep 09 2019 at 18:04, on Zulip):

I think this is also one of the harder problems though

matklad (Sep 09 2019 at 18:04, on Zulip):

But I am not sure how interning and lifetimes should work for indices

nikomatsakis (Sep 09 2019 at 18:04, on Zulip):

right

nikomatsakis (Sep 09 2019 at 18:04, on Zulip):

I was thinking of calling this out as a "challenge"

matklad (Sep 09 2019 at 18:05, on Zulip):

Especially, how index assignment works across modifications

matklad (Sep 09 2019 at 18:06, on Zulip):

I guess, we can just try to move Ty out of ra_hir to ra_ty, which doesn't have access to DB, and see what breaks?

nikomatsakis (Sep 09 2019 at 18:12, on Zulip):

seems like a first step

nikomatsakis (Sep 09 2019 at 18:12, on Zulip):

I added a few more thoughts in the "managing code" section

nikomatsakis (Sep 09 2019 at 18:12, on Zulip):

see if you think they make sense :)

matklad (Sep 09 2019 at 18:17, on Zulip):

Makes sense, though I feel like we miss the word we want to use instead of "unit tests". We want IR data-driven tests: input is a serialized IR, output is also a piece of data we match against, and the actual #[test] function just pipies input IR via component

matklad (Sep 09 2019 at 18:18, on Zulip):

THat is, thests are not really unit, they test subsytem as a whole, and critically, the are data based, though input data is not necessary rust syntax (to avoid dependency on the parser)

nikomatsakis (Sep 09 2019 at 18:19, on Zulip):

yeah, that sounds right

nikomatsakis (Sep 09 2019 at 18:19, on Zulip):

I think a lot of things are pointing to

nikomatsakis (Sep 09 2019 at 18:19, on Zulip):

the correct selection of the component boundary is key

nikomatsakis (Sep 09 2019 at 18:19, on Zulip):

I'm thinking about e.g. @centril's comments about wanting to audit correctness from a language perspective

nikomatsakis (Sep 09 2019 at 18:20, on Zulip):

I feel like if the component boundaries are well-chosen, that actually becomes potentially easier

nikomatsakis (Sep 09 2019 at 18:21, on Zulip):

the correct selection of the component boundary is key

an example of an ungreat boundary, I think, is probably "type unification"

nikomatsakis (Sep 09 2019 at 18:21, on Zulip):

like, it makes sense to collect the full set of things together I listed

nikomatsakis (Sep 09 2019 at 18:22, on Zulip):

because they are approaching the level of concepts that might appear in a language spec

nikomatsakis (Sep 09 2019 at 18:22, on Zulip):

(at least, to my mind.)

nikomatsakis (Sep 09 2019 at 18:24, on Zulip):

the more I think about it, though, the more I think it makes sense to dive a bit into what sharing types would mean

matklad (Sep 09 2019 at 18:24, on Zulip):

The open question is "do such boundaries exist?" :) Compiler as a whole is obv a great bounary: text on the input, binary on the output, a perfect black-box inside

matklad (Sep 09 2019 at 18:25, on Zulip):

I am not altogether sure that extracting bits out of the middle would be net positive. I hope it would be

nikomatsakis (Sep 09 2019 at 18:26, on Zulip):

Yes, my hypothesis is that such boundaries exist, but they are fairly high-level

nikomatsakis (Sep 09 2019 at 18:26, on Zulip):

And further that exploiting them will be super useful :)

nikomatsakis (Sep 09 2019 at 18:26, on Zulip):

But I suppose there is a degenerate plan in which we use such boundaries only internally, in the code

nikomatsakis (Sep 09 2019 at 18:26, on Zulip):

I think this plan argues for a mono-repo and for integration tests only

matklad (Sep 09 2019 at 18:27, on Zulip):

Specifically, there's a lot of truth to https://www.tedinski.com/2018/02/06/system-boundaries.html#boundaries-arent-free

nikomatsakis (Sep 09 2019 at 18:34, on Zulip):

@matklad ok I'm finding this to be a somewhat useful brain dump. Something I think would help -- maybe a list of questions we would like help to answer. Or maybe we should figure out the question and a proposed answer and look for feedback.

nikomatsakis (Sep 09 2019 at 18:35, on Zulip):

I think from the traits/types side of thing, I would like to try and lay out a "trail" of steps to take, maybe a dep graph. I'm going to try and sketch out what that means :)

matklad (Sep 09 2019 at 18:35, on Zulip):

"Who wants to help out? :)" is a good question

nikomatsakis (Sep 09 2019 at 18:35, on Zulip):

lol

nikomatsakis (Sep 09 2019 at 18:35, on Zulip):

fair enough!

nikomatsakis (Sep 09 2019 at 18:35, on Zulip):

one question I had in mind was something like

nikomatsakis (Sep 09 2019 at 18:36, on Zulip):

is there some .. "time limit"? Are there bits of work we can frontload that would help us to figure out if we're going in the right direction?

nikomatsakis (Sep 09 2019 at 18:36, on Zulip):

but that question is not "well formed" yet I think

nikomatsakis (Sep 09 2019 at 18:37, on Zulip):

I'm basically experiencing a measure of self doubt around "is this a good use of time vs trying to soup up rust's existing systems in place", I suspect.

matklad (Sep 09 2019 at 18:38, on Zulip):

soup up: stir the soup until it nicely separates itself into original components? :)

matklad (Sep 09 2019 at 18:39, on Zulip):

But yeah, I also has "if this even the right direction" feeling: adding boundaries has costs

nikomatsakis (Sep 09 2019 at 18:39, on Zulip):

I think part of the way to answer that question is choosing the order to attack things

nikomatsakis (Sep 09 2019 at 18:40, on Zulip):

and then re-evaluating

matklad (Sep 09 2019 at 18:40, on Zulip):

uguh. I think that means "go for the easiest boundary first"

nikomatsakis (Sep 09 2019 at 18:53, on Zulip):

that's what I'm wondering, indeed

matklad (Sep 16 2019 at 14:01, on Zulip):

@WG-rls2.0 I've realised that we've never actually announced this, but we are trying to have a weekly sync-up in this channel at this time

matklad (Sep 16 2019 at 14:01, on Zulip):

@nikomatsakis are you around?

nikomatsakis (Sep 16 2019 at 14:01, on Zulip):

yep

nikomatsakis (Sep 16 2019 at 14:01, on Zulip):

although it may be wise to make a fresh topic for each week :)

Florian Diebold (Sep 16 2019 at 14:01, on Zulip):

I'm kind of around

nikomatsakis (Sep 16 2019 at 14:02, on Zulip):

I'm a bit distracted because I'm trying to get to the bottom of an async-await thing right now

nikomatsakis (Sep 16 2019 at 14:02, on Zulip):

but I would like to follow up on the friday meeting

nikomatsakis (Sep 16 2019 at 14:02, on Zulip):

I guess makes sense to start with general updates, if there are any since last time we talked

matklad (Sep 16 2019 at 14:02, on Zulip):

@nikomatsakis I think that the same topic would actually be nicer, now? It's easier to caught up with prev meetings and do historical digging

nikomatsakis (Sep 16 2019 at 14:03, on Zulip):

no strong opinion

matklad (Sep 16 2019 at 14:03, on Zulip):

ok, let's stick with status quo, unless anyone objects

matklad (Sep 16 2019 at 14:03, on Zulip):

updates:

matklad (Sep 16 2019 at 14:03, on Zulip):
matklad (Sep 16 2019 at 14:04, on Zulip):
matklad (Sep 16 2019 at 14:04, on Zulip):
matklad (Sep 16 2019 at 14:05, on Zulip):
matklad (Sep 16 2019 at 14:05, on Zulip):
matklad (Sep 16 2019 at 14:05, on Zulip):

Specifically, we have an enum Adt { Struct(Struct), Enum(Enum), Union(Union) }

matklad (Sep 16 2019 at 14:06, on Zulip):

and enum ModuleDef {Adt(Adt), Function(Function), ...}

matklad (Sep 16 2019 at 14:06, on Zulip):

and transitive Intoimpls: Struct -> Adt, Adt -> ModuleDef, Struct -> ModuleDef.

Florian Diebold (Sep 16 2019 at 14:07, on Zulip):

but we also have e.g. TypableDef which includes a subset of those, so it's kind of multiple inheritance ;)

matklad (Sep 16 2019 at 14:07, on Zulip):

ohhh, https://github.com/uHOOCCOOHu added completion for macros, so there's no need to type out unimplemented any more :tada:

nikomatsakis (Sep 16 2019 at 14:08, on Zulip):

I was anticipating that this would happen, if we had "strongly typed ids"

nikomatsakis (Sep 16 2019 at 14:09, on Zulip):

So on my side, I started doing some work on chalk, specifically looking at what it would take to alter the interface to be more "selective" when it asks for impls

Florian Diebold (Sep 16 2019 at 14:09, on Zulip):

oh, I worked on closure types, but that's now a bit blocked on (at least a small subset of) https://github.com/rust-lang/chalk/issues/241 @nikomatsakis

nikomatsakis (Sep 16 2019 at 14:09, on Zulip):

Yeah, I was just going to say, I opened https://github.com/rust-lang/chalk/issues/241 -- and I see @Florian Diebold left some comments I've not yet read :)

nikomatsakis (Sep 16 2019 at 14:10, on Zulip):

I have to look more closely at my initial results. I made some preliminary changes in that direction and I'm getting some failing tests, but I don't know that the failures are significant.

nikomatsakis (Sep 16 2019 at 14:10, on Zulip):

(They have to do with how we model coherence specifically, and there is probably a way to circumvent them)

nikomatsakis (Sep 16 2019 at 14:12, on Zulip):

I think @Florian Diebold we should be able to land some changes that, while not perfect, help a lot in practice relatively soon. My hope for today was that I would sketch out a small series of steps around chalk, prioritizing (a) support for dyn and impl types, which is started, (b) the RustIrDatabase traits, (c) and steps to improve how we are handling lazy norm (although I think there are some open questions, as I wrote) -- but I'm not sure if those are the best priorities.

nikomatsakis (Sep 16 2019 at 14:12, on Zulip):

One thing I was wondering is whether my change to flounder when self type is unknown helped with the "chalk never terminates" bugs

Florian Diebold (Sep 16 2019 at 14:13, on Zulip):

I haven't seen any "chalk never terminates" bugs since the change that makes all traits non-enumerable

matklad (Sep 16 2019 at 14:13, on Zulip):

It did, yeah: at least, there's a clean improvement in runtime from oo to 10s for that PR

Florian Diebold (Sep 16 2019 at 14:14, on Zulip):

there are some reports of crashes inside Chalk, which I haven't looked into in detail yet

Florian Diebold (Sep 16 2019 at 14:14, on Zulip):

https://github.com/rust-analyzer/rust-analyzer/issues/1805

Florian Diebold (Sep 16 2019 at 14:15, on Zulip):

it's very possible that this is some error in the Chalk lowering code in RA ;)

nikomatsakis (Sep 16 2019 at 14:15, on Zulip):

ah, interesting

nikomatsakis (Sep 16 2019 at 14:15, on Zulip):

one thing i'd like to do for chalk that's been on my radar for some time

nikomatsakis (Sep 16 2019 at 14:15, on Zulip):

is to try and make some "Benchmarks"

nikomatsakis (Sep 16 2019 at 14:16, on Zulip):

the code was never really written with perf in mind

nikomatsakis (Sep 16 2019 at 14:16, on Zulip):

it'd be nice to have e.g. some realistic-ish setup with some number of traits and a real query that we can run perf on and use to measure changes

nikomatsakis (Sep 16 2019 at 14:16, on Zulip):

just something to keep in mind if you come across some queries that seem plausible

nikomatsakis (Sep 16 2019 at 14:17, on Zulip):

that said, this is exactly the sort of thing that I would think a "chalk extractor" could do a great job on (and I'd still like to add that at some point, on the chalk side)

nikomatsakis (Sep 16 2019 at 14:17, on Zulip):

anyway, do my priorities sound plausible?

(a) support for dyn and impl types, which is started, (b) the RustIrDatabase traits, (c) and steps to improve how we are handling lazy norm

matklad (Sep 16 2019 at 14:17, on Zulip):

I think ra_cli analys-bench is a pretty realistic example (but it is bound to rust-analyzer of course)

matklad (Sep 16 2019 at 14:18, on Zulip):

It would be interesting to tie "dumping chalk IR as we go" idea from prev meeting into analysis-bench, so that you can record a real sample, and then use it to bench chalk, without ra

nikomatsakis (Sep 16 2019 at 14:19, on Zulip):

yeah; I think the chalk side is the right side to do it on

nikomatsakis (Sep 16 2019 at 14:19, on Zulip):

it'd be nice to be able to do it for rustc too, for example

nikomatsakis (Sep 16 2019 at 14:20, on Zulip):

anyway, @matklad, what is status arond parser work?

nikomatsakis (Sep 16 2019 at 14:20, on Zulip):

I've yet to catch up with that discussion from friday

matklad (Sep 16 2019 at 14:20, on Zulip):

Remembered two more things:

nikomatsakis (Sep 16 2019 at 14:20, on Zulip):

Yes. In particular, I would expect to change this at a rust edition boundary

nikomatsakis (Sep 16 2019 at 14:21, on Zulip):

(I was thinking about that earlier, that we ought to be preparing for such a change well in advance)

matklad (Sep 16 2019 at 14:21, on Zulip):

re parser: nothing new on the rustc side, but rust-analyzer now handles composed tokens in the way I want rustc to handle them, so I know have the model to change rustc

nikomatsakis (Sep 16 2019 at 14:21, on Zulip):

I don't quite understand what you mean about Source<T> yet

matklad (Sep 16 2019 at 14:21, on Zulip):

Let me try to talk this through, as it seems important.

matklad (Sep 16 2019 at 14:22, on Zulip):

So, syntax trees in rust-analyzer are by designg value types. 1 + 1 in foo.rs looks exactly like 1 + 1 in bar.rs, and given that AST is reference counted, it's ok to intern syntax trees

matklad (Sep 16 2019 at 14:23, on Zulip):

(and this even happens within a single file already)

matklad (Sep 16 2019 at 14:24, on Zulip):

For analysis, you need to add some form of identity to the syntax trees.

matklad (Sep 16 2019 at 14:24, on Zulip):

for example, if you have #[path = "foo.rs"] mod x; #[path = "foo.rs"] mod y; , then the same syntax from foo.rs would represent two different modules

nikomatsakis (Sep 16 2019 at 14:25, on Zulip):

editor's note: responses about Source<T> broken out into a separate topic

matklad (Sep 23 2019 at 14:01, on Zulip):

@nikomatsakis, @WG-rls2.0 are you around for a weekly sync-up ? :)

nikomatsakis (Sep 23 2019 at 14:02, on Zulip):

I'm around

matklad (Sep 23 2019 at 14:03, on Zulip):

Stuff that happened this week:

nikomatsakis (Sep 23 2019 at 14:03, on Zulip):

Nice

nikomatsakis (Sep 23 2019 at 14:03, on Zulip):

"web version" of rust-analyzer?

matklad (Sep 23 2019 at 14:03, on Zulip):

https://rust-analyzer.github.io/wasm-demo.html

matklad (Sep 23 2019 at 14:04, on Zulip):

(warning: 5 MB download)

nikomatsakis (Sep 23 2019 at 14:04, on Zulip):

That's very cool :)

matklad (Sep 23 2019 at 14:05, on Zulip):
matklad (Sep 23 2019 at 14:06, on Zulip):
matklad (Sep 23 2019 at 14:07, on Zulip):
matklad (Sep 23 2019 at 14:07, on Zulip):

We also had a pretty interesting discussion about joint rls/rust-analyzer strategy. We didn't come up with a strategy, but made this nice markdown doc: https://hackmd.io/S5wub4mfToSnnLGvK3I-ew

matklad (Sep 23 2019 at 14:08, on Zulip):

I think that's it?

nikomatsakis (Sep 23 2019 at 14:08, on Zulip):

OK. My update is shorter, I guess. I landed a small update to chalk's IR, removing the all_structs callback, and I plan to work a bit more today to refine the "request impl" callback, and also to help push the dyn/impl integration forward.

matklad (Sep 23 2019 at 14:09, on Zulip):

Oh, one more thing: while working on mbe, I've realized that this could be another candidate for rather painless librarification. Maybe

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

Ah, interesting

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

I've thought about that before

matklad (Sep 23 2019 at 14:10, on Zulip):

In theory the interface for mbe are token trees, which is a relatively thing data-structure

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

But I had kind of forgotten it

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

Yeah; there is some interaction with name resolution, but there is also a part that can be factored out from that I suppose

matklad (Sep 23 2019 at 14:10, on Zulip):

In practice, the token trees in rustc contain AST, and that throws quite a wrench in the works

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

Indeed

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

I have also disliked that

nikomatsakis (Sep 23 2019 at 14:10, on Zulip):

Particularly since it's visible (in edge cases)

nikomatsakis (Sep 23 2019 at 14:11, on Zulip):

It's basically an efficiency optimization, which I think we could/should do in other ways

matklad (Sep 23 2019 at 14:11, on Zulip):

Is it visible in non-backwards compatible way?

nikomatsakis (Sep 23 2019 at 14:11, on Zulip):

Hmm so I think something else I should prioritize is doing a bit of work in/around rust-analyzer's type checker. I guess I can start by reviewing what @Florian Diebold did around associated types.

matklad (Sep 23 2019 at 14:12, on Zulip):

Like, we can't today re-parse $var:expr as something else, but it seems like if we allow reparsing, that would only increase the set of accepted programs

nikomatsakis (Sep 23 2019 at 14:12, on Zulip):

Is it visible in non-backwards compatible way?

depends what you mean by "non backawrds compatible" -- I guess the question is "compatible with what"

matklad (Sep 23 2019 at 14:12, on Zulip):

Compatible with current stable behavior

nikomatsakis (Sep 23 2019 at 14:12, on Zulip):

if you had some stream of tokens that could be parsed as either an expression or a type

nikomatsakis (Sep 23 2019 at 14:12, on Zulip):

no, I mean, what future behavior are you imagining

nikomatsakis (Sep 23 2019 at 14:13, on Zulip):

I think that you could observe e.g. macro_rules! foo { ($x:expr) => { .. } ($x:ty) => { .. } }

matklad (Sep 23 2019 at 14:13, on Zulip):

Ah. So, I imagine that we store $var:expr as a token tree with virtual delimiters

nikomatsakis (Sep 23 2019 at 14:13, on Zulip):

right, and I think that might not be enough to be fully compatible

nikomatsakis (Sep 23 2019 at 14:13, on Zulip):

though it is arguably a bug-fix

nikomatsakis (Sep 23 2019 at 14:14, on Zulip):

it's the kind of change where the behavior is sort of loosely documented :)

matklad (Sep 23 2019 at 14:14, on Zulip):

we definitelly should make sure that macro thing doesn't use the AST

matklad (Sep 23 2019 at 14:14, on Zulip):

macro 2.0 that is

nikomatsakis (Sep 23 2019 at 14:14, on Zulip):

Yeah.

nikomatsakis (Sep 23 2019 at 14:14, on Zulip):

( I have to run a bit early today btw -- but I'll be back around hacking on traits and rust-analyzer this afternoon )

nikomatsakis (Sep 23 2019 at 14:15, on Zulip):

So, are you still actively investigating parsing extraction?

nikomatsakis (Sep 23 2019 at 14:15, on Zulip):

I'm trying to understand the "status" of that

matklad (Sep 23 2019 at 14:15, on Zulip):

sort-of

matklad (Sep 23 2019 at 14:15, on Zulip):

I am actively trying to move the parser to the token-tree modelN

matklad (Sep 23 2019 at 14:15, on Zulip):

(which is a part of that)

matklad (Sep 23 2019 at 14:16, on Zulip):

I haven't looked further than that

nikomatsakis (Sep 23 2019 at 14:16, on Zulip):

the rustc parser?

nikomatsakis (Sep 23 2019 at 14:16, on Zulip):

or r-a parser?

nikomatsakis (Sep 23 2019 at 14:16, on Zulip):

that does seem like a good first step

matklad (Sep 23 2019 at 14:16, on Zulip):

rustc parser

nikomatsakis (Sep 23 2019 at 14:16, on Zulip):

Ah. So, I imagine that we store $var:expr as a token tree with virtual delimiters

to be clear: I think I'd prefer to try this model first,

matklad (Sep 23 2019 at 14:16, on Zulip):

r-a parser uses token tree model like the one is proc_macro

nikomatsakis (Sep 23 2019 at 14:17, on Zulip):

because I think it's more what users would expect

nikomatsakis (Sep 23 2019 at 14:17, on Zulip):

but I think the "fix" to match rustc is also not that terrible

nikomatsakis (Sep 23 2019 at 14:17, on Zulip):

it'd kind of be that the "virtual delimeters" also encode the fragment

nikomatsakis (Sep 23 2019 at 14:17, on Zulip):

i.e., there are virtual "type" delimeters, virtual "expr" delimeters

nikomatsakis (Sep 23 2019 at 14:17, on Zulip):

I should probably make some proof of concept test case for what I'm talking about

matklad (Sep 23 2019 at 14:18, on Zulip):

Yeah, I think I undestand what you are talking about

matklad (Sep 23 2019 at 14:18, on Zulip):

It wouldn't be hard to implement indeed

matklad (Sep 23 2019 at 14:19, on Zulip):

but it would mean that mbe model is not quite proc_macro2 tokens, which is a rather big problem, even if the difference is small

matklad (Sep 23 2019 at 14:19, on Zulip):

Though, perhaps we can store this info on the side

nikomatsakis (Sep 23 2019 at 14:20, on Zulip):

not very realistic, but here is an example

nikomatsakis (Sep 23 2019 at 14:21, on Zulip):

oh, maybe not, I guess that doesn't parse as an expr for us

nikomatsakis (Sep 23 2019 at 14:21, on Zulip):

well I think I can make one but it takes some work :P

matklad (Sep 23 2019 at 14:21, on Zulip):

yeah, I did the same: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=e9a29816fa71e6abb09fb349f4053d5a

nikomatsakis (Sep 23 2019 at 14:21, on Zulip):

ah yeah duh there you go

nikomatsakis (Sep 23 2019 at 14:21, on Zulip):

great

nikomatsakis (Sep 23 2019 at 14:21, on Zulip):

ok, I gotta run though, bbiab

matklad (Sep 23 2019 at 14:21, on Zulip):

heah, yeh, mine is better :D

matklad (Sep 23 2019 at 14:22, on Zulip):

:wave:

matklad (Sep 30 2019 at 14:01, on Zulip):

hey, @WG-rls2.0, it's time for another weekly sink up. I have to run really soon, so I'll give just a brife update, and try to follow up later

nikomatsakis (Sep 30 2019 at 14:04, on Zulip):

:wave:

matklad (Sep 30 2019 at 14:05, on Zulip):

Things that happened since last week:

nikomatsakis (Sep 30 2019 at 14:05, on Zulip):

Can I get some links to the closures/coercions PRs etc?

nikomatsakis (Sep 30 2019 at 14:05, on Zulip):

Or is it best to read the source

nikomatsakis (Sep 30 2019 at 14:05, on Zulip):

I guess I can find them on the repo

matklad (Sep 30 2019 at 14:05, on Zulip):

Also, I will be traveling this week: giving a "Rust as a High Level Langauge" talk in Toulouse, so I might be slow to repond.

matklad (Sep 30 2019 at 14:06, on Zulip):

@nikomatsakis commit hash is c7420ddaaa76741d1eebe393406b38ba5596e54a

matklad (Sep 30 2019 at 14:07, on Zulip):

a better link: https://github.com/rust-analyzer/rust-analyzer/commit/c7420ddaaa76741d1eebe393406b38ba5596e54a

nikomatsakis (Sep 30 2019 at 14:08, on Zulip):

thanks

nikomatsakis (Sep 30 2019 at 14:10, on Zulip):

on my side not a lot to tell, I plan to work more on the chalk changes that @Florian Diebold alluded to today, along with some other things. I also started thinking about what it might look like to extract types into a library that many could share.

nikomatsakis (Sep 30 2019 at 14:10, on Zulip):

seems like one of the hardest questions to answer will come up with interning and other such details, and how generic we want to be about that

nikomatsakis (Sep 30 2019 at 14:11, on Zulip):

rust doesn't make it super easy to extract over these sorts of things, though I think in this case it could plausibly be done.

nikomatsakis (Sep 30 2019 at 14:11, on Zulip):

though mostly I've been pondering how to cleanup chalk to address the normalization and "known self type" cases

matklad (Oct 07 2019 at 14:00, on Zulip):

Hello @WG-rls2.0 , time for another update.

matklad (Oct 07 2019 at 14:01, on Zulip):

Not a lot of stuff happened this week, but the highlight is the support for #[cfg()] attributes for conditional compilation by https://github.com/oxalica: https://github.com/rust-analyzer/rust-analyzer/pull/1928

matklad (Oct 07 2019 at 14:05, on Zulip):

Some other things:

matklad (Oct 07 2019 at 14:07, on Zulip):

I think I want to look at supporting dynamic reloading of the crate graph next:

matklad (Oct 07 2019 at 14:08, on Zulip):

@nikomatsakis , any updates on your side?

nikomatsakis (Oct 07 2019 at 14:09, on Zulip):

Hey @matklad sorry -- a neighbor came by with an emergency

nikomatsakis (Oct 07 2019 at 14:09, on Zulip):

was afk

nikomatsakis (Oct 07 2019 at 14:09, on Zulip):

just got back, let me catch up

nikomatsakis (Oct 07 2019 at 14:11, on Zulip):

OK, so, updates:

nikomatsakis (Oct 07 2019 at 14:11, on Zulip):

I spent most of my time last week exploring some non-RLS-related trait things (e.g., upcasts in rustc and what it takes to unblock const generics)

nikomatsakis (Oct 07 2019 at 14:11, on Zulip):

(The good news being that I think I see the path, but we'll see)

matklad (Oct 07 2019 at 14:13, on Zulip):

That reminds me... At the all hands, when we were discussing traits, there was an idea that, to unblock immediate progress on rustc side, we (well, you I presume :) ) should pause chalk work and instead augment the current trait solver.

I am curious what's the final plan here: are we doing GATs and stuff before chalkification?

nikomatsakis (Oct 07 2019 at 14:14, on Zulip):

I'm still kind of trying to figure that out. My current plan is to pursue two main things:

nikomatsakis (Oct 07 2019 at 14:15, on Zulip):

Once we have lazy norm going, the next step would probably be figuring out some details of the "universe story" on rustc side; I did some of that work already. That might then enable GATs on the rustc side, but I'm not entirely sure. I remember the last time I did work on this I felt like it would be hard

nikomatsakis (Oct 07 2019 at 14:16, on Zulip):

(I also opened a lot of issues around refactoring chalk last week, incidentally, but I need to prioritize them a bit)

matklad (Oct 14 2019 at 14:33, on Zulip):

Belated sync for this week:

matklad (Oct 21 2019 at 14:00, on Zulip):

Next week, next syncup!

nikomatsakis (Oct 21 2019 at 14:03, on Zulip):

:wave:

nikomatsakis (Oct 21 2019 at 14:05, on Zulip):

Update from last week:

matklad (Oct 21 2019 at 14:05, on Zulip):

I unfortunately don't have a lot of too report this week: I've been trying different thing, and they didn't quite work out.

In particular, I've made all preliminary refactoings for a lazy VFS, only to realise that we probably don't actually want a fully lazy VFS. We want to load file texts lazily, but the set of files should be loaded eagarly. This is how watchman works, and this I think is the only reliable way to get file watching right.

I've also looked into splitting hir database of rust-analyzer into several crates. The results are mixed:

nikomatsakis (Oct 21 2019 at 14:06, on Zulip):
nikomatsakis (Oct 21 2019 at 14:07, on Zulip):

Hmm, ok, I will try to catch up a bit on that discussion

nikomatsakis (Oct 21 2019 at 14:07, on Zulip):

(the splitting of the crate)

nikomatsakis (Oct 21 2019 at 14:07, on Zulip):

Oh, there is another thing, which is that we landed some support for dyn Trait and impl Trait, but it's quite broken

matklad (Oct 21 2019 at 14:07, on Zulip):

Indeed we ought to figure this particular elephant in the room out

nikomatsakis (Oct 21 2019 at 14:07, on Zulip):

Yeah. I was finding it useful to try and explore various "scenarios"

matklad (Oct 21 2019 at 14:08, on Zulip):

@nikomatsakis yeah, @Florian Diebold already has a WIP PR which integrates that with rust-analyzer, which doens't quite work yet

nikomatsakis (Oct 21 2019 at 14:08, on Zulip):

e.g., if we go "all in" on rust-analyzer, what does that look like? which of the things we might want to do on rustc conflict with that? What is the opposite extreme? What's in the middle?

nikomatsakis (Oct 21 2019 at 14:08, on Zulip):

OK, I see a topic that looks like I need to catch up on it

nikomatsakis (Oct 21 2019 at 14:08, on Zulip):

on this topic, not sure if you followed that at all @matklad?

matklad (Oct 21 2019 at 14:09, on Zulip):

Nope, I think I've missed that

nikomatsakis (Oct 21 2019 at 14:09, on Zulip):

the relevant thing to rust-analyzer is that -- at least to start and maybe permanently -- I want to have traits that let you interact with types generically, without regard to their concrete representation

matklad (Oct 21 2019 at 14:09, on Zulip):

Like, I think I wasn't pinged, do you have a link to the zulip thread?

nikomatsakis (Oct 21 2019 at 14:09, on Zulip):

one sec

nikomatsakis (Oct 21 2019 at 14:09, on Zulip):

here are the notes on the meeting

nikomatsakis (Oct 21 2019 at 14:10, on Zulip):

there are links to the topic but also significant points

nikomatsakis (Oct 21 2019 at 14:11, on Zulip):

the relevant thing to rust-analyzer is that -- at least to start and maybe permanently -- I want to have traits that let you interact with types generically, without regard to their concrete representation

there are two reasons to do this:

nikomatsakis (Oct 21 2019 at 14:11, on Zulip):

I'm exploring this

nikomatsakis (Oct 21 2019 at 14:11, on Zulip):

(I expect to put more time into that today)

nikomatsakis (Oct 21 2019 at 14:12, on Zulip):

If that makes more progress, I am definitely interested in trying to work with @Florian Diebold to factor out the type checker from RLS into something more independent (maybe even integrate it into the chalk repository, not sure -- it depends on what we decide the "scope" of the chalk repo should be)

matklad (Oct 21 2019 at 14:12, on Zulip):

Aha, the typefamaly pattern seems like a hammer big enough to handle every librarification

nikomatsakis (Oct 21 2019 at 14:12, on Zulip):

Yes, I've found it a very powerful pattern in the past

nikomatsakis (Oct 21 2019 at 14:12, on Zulip):

Right now though what I'm trying to do

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

is refactor some of the parts of chalk that are quite messy anyway

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

my goal is that the code reads nicer with the generics than without

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

and helps to avoid some kinds of bugs

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

we'll see if I can achieve that ;)

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

my main concern is undue complexity; otoh I think that people who hack on chalk should probably be pretty comfortable with traits :P

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

one of the nice things though of the "one true underlying type" thing

nikomatsakis (Oct 21 2019 at 14:13, on Zulip):

is that you can kind of "cut off" the generics at any point

nikomatsakis (Oct 21 2019 at 14:14, on Zulip):

i.e., in my chalk branch right now, the low-level code is generic, but at some point everybody just hardcodes on "type family"

nikomatsakis (Oct 21 2019 at 14:14, on Zulip):

(this lets me convert crate by crate)

matklad (Oct 21 2019 at 14:14, on Zulip):

I would also be concerned about separate compilation. But yeah, if, in the end, we have some concrete representation, that fixes this problem.

nikomatsakis (Oct 21 2019 at 14:15, on Zulip):

there is certainly the issue of compilation time and monomorphization, but -- yes -- keeping the number of concrete representations minimal means there aren't really multiple copies in the end

nikomatsakis (Oct 21 2019 at 14:15, on Zulip):

(though you do defer some of that compilation to the "leaf crate")

nikomatsakis (Oct 21 2019 at 14:16, on Zulip):

(hopefully incremental helps with this, if not, we should figure out why not)

nikomatsakis (Oct 21 2019 at 14:16, on Zulip):

this reminds me -- I've really failed to keep up with salsa, as you no doubt saw

nikomatsakis (Oct 21 2019 at 14:16, on Zulip):

but you had mentioned something about &dyn Foo not working for rust-analyzer?

matklad (Oct 21 2019 at 14:16, on Zulip):

&(dyn AstDatabase + DefDatabase) not working, yeah

nikomatsakis (Oct 21 2019 at 14:16, on Zulip):

yeah that certainly won't work

matklad (Oct 21 2019 at 14:17, on Zulip):

And there's a separate issue of creating mock databses for tests

nikomatsakis (Oct 21 2019 at 14:17, on Zulip):

actually the work that @Alexander Regueiro has been doing on trait upcast could be quite relevant here

nikomatsakis (Oct 21 2019 at 14:17, on Zulip):

I remember that when I was pushing &dyn changes through Lark

nikomatsakis (Oct 21 2019 at 14:17, on Zulip):

not having trait upcast was a horrible pain

matklad (Oct 21 2019 at 14:17, on Zulip):

basically, you need concrete db not only in the leaf trait, but also in the tests for intermediate crates, and creating that mock everytime is boilerplaty

nikomatsakis (Oct 21 2019 at 14:17, on Zulip):

maybe because we were using a large hierarchy of salsa traits

nikomatsakis (Oct 21 2019 at 14:18, on Zulip):

basically, you need concrete db not only in the leaf trait, but also in the tests for intermediate crates, and creating that mock everytime is boilerplaty

yeah I think in Lark I moved the tests by and large to the leaf crate

nikomatsakis (Oct 21 2019 at 14:18, on Zulip):

so as to have a single "mock db"

nikomatsakis (Oct 21 2019 at 14:18, on Zulip):

I certainly remember encountering this problem

matklad (Oct 21 2019 at 14:18, on Zulip):

Yeah, I also feel that salsa atm pushes me towards this direction, but I think that's horrible long-term

nikomatsakis (Oct 21 2019 at 14:19, on Zulip):

I'm not actually sure that's true

matklad (Oct 21 2019 at 14:19, on Zulip):

basically, if you fix a bug in a low-level crate, you need to recompile the whole tower to test it

nikomatsakis (Oct 21 2019 at 14:19, on Zulip):

But I guess I agree I'd prefer to make the choice for other reasons

nikomatsakis (Oct 21 2019 at 14:19, on Zulip):

Yeah, that part is not great I guess

matklad (Oct 21 2019 at 14:20, on Zulip):

I am actually not sure if it's a good idea to split databases over several crates.....

nikomatsakis (Oct 21 2019 at 14:20, on Zulip):

the other option

nikomatsakis (Oct 21 2019 at 14:20, on Zulip):

which I've experimented with in chalk

nikomatsakis (Oct 21 2019 at 14:20, on Zulip):

is just having regular traits basically

matklad (Oct 21 2019 at 14:20, on Zulip):

I can see a design where there's only a single database crate that does everytihng, and all other code lives in separate crates, and uses traits to interface to DB without knowning about salsa at all

nikomatsakis (Oct 21 2019 at 14:20, on Zulip):

yeah, that

nikomatsakis (Oct 21 2019 at 14:21, on Zulip):

I thnk this is a better fit (at minimum) for library-ification

matklad (Oct 21 2019 at 14:22, on Zulip):

For librarification, definitelly yes. For rust-analyzer itself, not sure

detrumi (Oct 21 2019 at 14:22, on Zulip):

What's the reason to use a DB above using traits directly in the first place?

matklad (Oct 21 2019 at 14:22, on Zulip):

currently, hir is pretty slow to compile and pretty big. I'd love to slice it into at least "types" and "not types"

nikomatsakis (Oct 21 2019 at 14:23, on Zulip):

What's the reason to use a DB above using traits directly in the first place?

there isn't really a very good one that I can see, except that it's slightly less repetitive maybe

nikomatsakis (Oct 21 2019 at 14:24, on Zulip):

I could definitely be persuaded that it makes more sense to just use plain traits all the way down

matklad (Oct 21 2019 at 14:24, on Zulip):

@detrumi

matklad (Oct 21 2019 at 14:24, on Zulip):

but yeah, maybe just using traits is the way to go....

nikomatsakis (Oct 21 2019 at 14:24, on Zulip):

@matklad regarding testing, I guess another way to go about reducing boilerplate would be to have macros or things that help you to generate the impls

nikomatsakis (Oct 21 2019 at 14:24, on Zulip):

I haven't thought much about it

nikomatsakis (Oct 21 2019 at 14:24, on Zulip):

it doesn't really feel like salsa's fault to me

nikomatsakis (Oct 21 2019 at 14:25, on Zulip):

I think Rust in general is not very good at this kind of re-use

nikomatsakis (Oct 21 2019 at 14:25, on Zulip):

It's kind of an OO strength and -- lacking specialization and a few other things -- we don't quite have the tools you want

nikomatsakis (Oct 21 2019 at 14:25, on Zulip):

But regardless it is a limitation

nikomatsakis (Oct 21 2019 at 14:25, on Zulip):

I think the main reason I was thinking that pulling tests to the "root" is ok is the old tension of integration vs unit testing.

nikomatsakis (Oct 21 2019 at 14:26, on Zulip):

To some extent it's nice for the tests to be kind of independent of the precise nature of your division into crates -- but to some extent that's also not nice, esp. for practical compilation time reasons.

matklad (Oct 21 2019 at 14:26, on Zulip):

Hm, I am not sure. I think, if salsa created a concrete DB (which are composed together in single inheritance style), I would be able to write

fn mock_db(fixture: &str) -> TheDbStruct

in the intermediate crate, and then use it in the final crate

nikomatsakis (Oct 21 2019 at 14:26, on Zulip):

if your crates are sufficiently coarse, and thus change infrequently, it's probably better to have the tests be linked to them

nikomatsakis (Oct 21 2019 at 14:27, on Zulip):

hmm, ok, I see, maybe there is some way for us to make the databases more composable in this way

nikomatsakis (Oct 21 2019 at 14:28, on Zulip):

this reminds me that you also had some other comments about state that I never fully understood

nikomatsakis (Oct 21 2019 at 14:28, on Zulip):

we probably need to create some kind of hackmd to help us track these concerns a bit

matklad (Oct 21 2019 at 14:28, on Zulip):

@nikomatsakis it seems like we need a meeting/design document for "physical architecture of salsa-based compilers"?

nikomatsakis (Oct 21 2019 at 14:28, on Zulip):

yes, exactly

matklad (Oct 21 2019 at 14:29, on Zulip):

Heh, great minds think alike

nikomatsakis (Oct 21 2019 at 14:29, on Zulip):

I'd like to go now and think about other things but I'd like to not forget this, and to maybe schedule an hour to talk it out

matklad (Oct 21 2019 at 14:29, on Zulip):

or, as they say in Russia, у дураков мысли сходятся

matklad (Oct 21 2019 at 14:31, on Zulip):

I think I'll open a thread on the salsa repository to just write down some thougts: that way, we won't forget this

nikomatsakis (Oct 21 2019 at 14:32, on Zulip):

good starting point

nikomatsakis (Oct 21 2019 at 14:32, on Zulip):

ok, great chat :) ttyl

matklad (Oct 28 2019 at 14:01, on Zulip):

Hey @WG-rls2.0 , it's time for weekly sync-up

nikomatsakis (Oct 28 2019 at 14:02, on Zulip):

Hey @matklad :wave:

matklad (Oct 28 2019 at 14:02, on Zulip):

A bunch off things happened recently

matklad (Oct 28 2019 at 14:03, on Zulip):

Most importantly, we've finally merged find usages, implemented by https://github.com/viorina

matklad (Oct 28 2019 at 14:03, on Zulip):

It does the classic IDE trick of using text-search to over-approximate the set of matchings, and then confirming the matches using the real analysis

matklad (Oct 28 2019 at 14:04, on Zulip):

We don't have a trigram index yet, so the text-search phase is brute force

matklad (Oct 28 2019 at 14:05, on Zulip):

As a consequence, it's now possible to invoke "rename" for functions, and it mostly works :)

matklad (Oct 28 2019 at 14:06, on Zulip):

Another biggish thing is that we now generate assists docs and tests from "doc comments". And in general I did some cleanup around assists. We have a whole bunch of them now:

https://github.com/rust-analyzer/rust-analyzer/blob/master/docs/user/assists.md

(and a couple new were added this week)

matklad (Oct 28 2019 at 14:08, on Zulip):

Find usages also was a stress-test for our type inference, which uncovered that infinite allocation in chalk problem. I don't think the root cause if fixed yet, but I've improved our profiling infrastructure a bit:

matklad (Oct 28 2019 at 14:10, on Zulip):

Finally, I've put the VFS work on pause without actually finishing it: it seems like I am stuck a bit, as I don't see a design that feels right, and, if we speak about various tradeoff-based designs, the one we have already seems OK. I want to come back to this later and just try to write direct bindings to watchman, but I have sunk way to many days into this, and I think I need to swticht to something else to make progress

matklad (Oct 28 2019 at 14:11, on Zulip):

Oh, and @nikomatsakis and I had a productive discussion about splitting salsa-database across the crates, I hope to do something concrete in this area this week

matklad (Oct 28 2019 at 14:11, on Zulip):

That's all for me I guess?

nikomatsakis (Oct 28 2019 at 14:13, on Zulip):

Cool --

matklad (Oct 28 2019 at 14:13, on Zulip):

Ahhh, one more thing: @detrumi implemented client-side shortening of type hints, which helps a lot with "as long as a screen" infered types :)

nikomatsakis (Oct 28 2019 at 14:13, on Zulip):

I've been working on some changes to chalk, some of which it seems like @Florian Diebold has integrated

nikomatsakis (Oct 28 2019 at 14:14, on Zulip):

the overall goal is to rework how we handle associated types so that we never ask "open-ended" requests for "all impls of a trait"

nikomatsakis (Oct 28 2019 at 14:14, on Zulip):

though we're not there yet

nikomatsakis (Oct 28 2019 at 14:14, on Zulip):

it's also I think laying the foundations for a "shared type" library that both rustc / rust-analyzer can employ

nikomatsakis (Oct 28 2019 at 14:14, on Zulip):

though I want to do a bit more digging in that direction

nikomatsakis (Oct 28 2019 at 14:14, on Zulip):

I also think it's related to that bug you uncovered

detrumi (Oct 28 2019 at 14:14, on Zulip):

Ahhh, one more thing: detrumi implemented client-side shortening of type hints, which helps a lot with "as long as a screen" infered types :slight_smile:

Good to hear it's useful :+1:

nikomatsakis (Oct 28 2019 at 14:14, on Zulip):

but I've not 100% proven that

nikomatsakis (Oct 28 2019 at 14:15, on Zulip):

(that is, I think that bug is caused by the current logic in unification -- which would be removed in the new scheme -- causing a kind of "infinite set of goals to prove")

nikomatsakis (Oct 28 2019 at 14:16, on Zulip):

there has been some good PRs in this direction, with @detrumi ( :tada:) implementing a derive for chalk's Fold trait, and @Jack Huey working on simplifying some of rust's context traits

nikomatsakis (Oct 28 2019 at 14:16, on Zulip):

(as an aside, I am thinking about trying to rip out the chalk integration that's in rustc now and rework it so that it hooks into the chalk-solve crate, like rust-analyzer does)

nikomatsakis (Oct 28 2019 at 14:17, on Zulip):

anyway, we still didn't fix the bugs around dyn Trait but hopefully soon

nikomatsakis (Oct 28 2019 at 14:17, on Zulip):

so that we can combine that logic with rust-analyzer

matklad (Oct 28 2019 at 14:18, on Zulip):

Exciting, given that we want to use dyn Trait inside rust-analyzer more, when splitting stuff across crates :)

nikomatsakis (Oct 28 2019 at 14:18, on Zulip):

the overall goal is to rework how we handle associated types so that we never ask "open-ended" requests for "all impls of a trait"

on this topic, I was thinking that it hopefully soon I can turn to looking more into rust-analyzer's type checker and how to best integrate it with check (and ultimately create a shared library), because I think it's going to take some experimentation to get that truly working

nikomatsakis (Oct 28 2019 at 14:18, on Zulip):

Exciting, given that we want to use dyn Trait inside rust-analyzer more, when splitting stuff across crates :)

Yay =) I think there's not that much work to do, I'm not sure if @Keith Yeung has had a chance to look into it yet

nikomatsakis (Oct 28 2019 at 14:19, on Zulip):

(I'll check if they're busy, because I think it wouldn't take me long to prep the PR anyhow, but I don't want to steal it from them -- and it might be more work than I think)

matklad (Oct 28 2019 at 14:20, on Zulip):

Oh, one more thing: we are receiving more "rust-analyzer eats 8GB of ram after two hours of usage" issues recently. I haven't seen this myself (as I pretty-much rebuild rust-analyzer once in ten minutes anyway), but it might be a symptom of something wrong with GC

nikomatsakis (Oct 28 2019 at 14:21, on Zulip):

hmm

matklad (Oct 28 2019 at 14:21, on Zulip):

Would be cool to be able "print, how much space each table occupies" to debug this issues, but I don't think we can esily implement this at the moment

nikomatsakis (Oct 28 2019 at 14:23, on Zulip):

What's blocking is the "heapsize" support?

matklad (Oct 28 2019 at 14:24, on Zulip):

the fact that heapsize on crates.io is pretty much dead, and the new one in Servo is not public

matklad (Oct 28 2019 at 14:24, on Zulip):

Although perhaps we should look at just the number of keys/values?

detrumi (Oct 28 2019 at 14:25, on Zulip):

the fact that heapsize on crates.io is pretty much dead, and the new one in Servo is not public

I'm guessing they can't just open-source it?

matklad (Oct 28 2019 at 14:26, on Zulip):

"just" means making sure that the API actually makes sense outside of Servo

detrumi (Oct 28 2019 at 14:26, on Zulip):

Ah, of course

matklad (Oct 28 2019 at 14:27, on Zulip):

Like, it seems like this is a pretty solvable problem, but someone needs to apply necessary labor (and a good API/macro design taste) to actually provide a production-ready solution

matklad (Oct 28 2019 at 14:27, on Zulip):

IIRC, HeapSize was just dumped to crates.io, and it's not maintained because it turned out to be not that great

detrumi (Oct 28 2019 at 14:28, on Zulip):

But yeah, approximating it should be fine too as long as you identify the cause

matklad (Oct 28 2019 at 14:28, on Zulip):

(for posterity, the new heapsizeof is here https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs)

Jeremy Kolb (Oct 28 2019 at 14:31, on Zulip):

Is there a plan to defeat compile times? It's getting pretty bad with running tests.

matklad (Oct 28 2019 at 14:34, on Zulip):

Sort of. The immediate things I want to do are:

Long term, I'd love to see if we can make salsa-based API less generic, but I think it makes sense to look into that after by-reference values (@nikomatsakis wants to change salsa sot hat queries return &Thing and not Arc<Thing>).

detrumi (Oct 28 2019 at 14:39, on Zulip):

Large crates are really having a big impact on compile times, it seems. Hopefully that can be improved on rustc's side in the future, as splitting things up each time is a lot of work

detrumi (Oct 28 2019 at 14:41, on Zulip):

@matklad By less generic, do you mean the &impl Database parameters?

matklad (Oct 28 2019 at 14:42, on Zulip):

Yeah, and the fact that test and read database are different types

matklad (Nov 04 2019 at 15:06, on Zulip):

@WG-rls2.0, time for another sync up!

matklad (Nov 04 2019 at 15:07, on Zulip):

I think the two biggest changes this week are:

matklad (Nov 04 2019 at 15:07, on Zulip):

I can probably say a bit about the second one

matklad (Nov 04 2019 at 15:09, on Zulip):

First of all, the split is not complete, so the code is really not great at places. FOr example, we have DefDatabase and DefDatabase2...

matklad (Nov 04 2019 at 15:11, on Zulip):

What we do have, however, is three crates instead of one:

hir_expand deals with macro expansion. Specifically, it defines the core datatypes required to talk about all Rust source code (some of which might be generated from macros). I originally didn't intend to split this part out, but I really really like how it looks now.

It's interesting how macro expansion does not depend on name-resolution even in terms of dependencies between crates: this is beccause we encode resovled macro definition into the id of a macro call

matklad (Nov 04 2019 at 15:13, on Zulip):

hir_def is a work-in-progress -- it should contain everything srictly between macro expansion and types. At the moment, it only has module-level name resolution. I want to move expression lowering to hir_def as well. hir_def is written with ECS API, where one operates with raw IDs

matklad (Nov 04 2019 at 15:14, on Zulip):

hir currently contains everything else. I think that, after hir_def is done, we should split hir into hir_ty and hir, and make hir a strict facade over hir_x crates.

matklad (Nov 04 2019 at 15:15, on Zulip):

One thing we've lost from the split is AstDb / DefDb separation, which we used to enforce via require salsa clause. That is, we put some effort to make sure you can't accidently invoke a "brittle" query form a robust query previously, but this work is undone by the split.

matklad (Nov 04 2019 at 15:16, on Zulip):

For testing, the idea is to have a #[cfg(test)] MockDatabse for each hir crate (renamed TestDb, as it is not really a mock), which share most of the fixture-related code via a WithFixture trait, defined in the root ra_db crate

matklad (Nov 04 2019 at 15:17, on Zulip):

the observation here is that various hir layers never add new inputs, so you only need to define fixtures at the bottom layer

matklad (Nov 04 2019 at 15:20, on Zulip):

Some of the less groundshaking changes:

nikomatsakis (Nov 04 2019 at 15:21, on Zulip):

@matklad sorry I'm late, I should've written, I had to run at the last minute (I've also been heads down on something this morning)

nikomatsakis (Nov 04 2019 at 15:21, on Zulip):

I'm going to catch up in a bit though

matklad (Nov 04 2019 at 15:21, on Zulip):

oh, @Christopher Durham is also doing some crazy unsafe experiments with rowan, with an effort to cut down the number of allocations even more. I haven't had a chance to review it properly, but the latest approach with boxed and interned tokens seems really promising

matklad (Nov 04 2019 at 15:22, on Zulip):

And I am excited to see @Edwin Cheng back! Looks like we are about to see some progress on macro front soon :)

Jeremy Kolb (Nov 04 2019 at 15:31, on Zulip):

I think I can piggyback off @Edwin Cheng 's work for macro signature help

matklad (Nov 11 2019 at 14:55, on Zulip):

I need to run soon, so let me give a short async update:

matklad (Nov 11 2019 at 14:58, on Zulip):

The most interesting bit today is probably the first point. The main problem there was to identify tokens in the expended code as coming from either the call or the definition. My understanding that in rustc this is handled by each token having a global unique span.

In rust-analyzer, tokens are locally numbered with small interegers, and, when you substitute captures into a macro template, you shift the numbers of the tokens in captures, sort-of how you do subst in lambda-calculus with de-Bruijn indices

matklad (Nov 11 2019 at 15:03, on Zulip):

Ok, I am leaving, will catch up later :) :wave:

nikomatsakis (Nov 11 2019 at 15:03, on Zulip):

@matklad thanks for the update! On the chalk side, @Jack Huey has been doing great work refactoring and fixing various bugs. I just remembered that I need to investigate that infinite loop. One good thing is that the work he's been doing is helping to pave way for fuel.

Last update: Nov 12 2019 at 16:35UTC