Stream: t-compiler/rust-analyzer

Topic: Eagerly parsing


kev (Dec 13 2020 at 16:15, on Zulip):

@matklad Was wondering would there be any benefits, if the analyser eagerly parsed files that have been declared with mod?

For example; I start off at lib.rs and it's content is:

mod foo;
mod foo2;
mod foo3;

So before analysis, I parse the files related to foo, foo2, and foo3 eagerly

kev (Dec 13 2020 at 16:16, on Zulip):

The parser would not need to hold any state to do this I think, just check for mod declarations in the parsed file

matklad (Dec 13 2020 at 16:17, on Zulip):

Parsing is done strictly on file basis

matklad (Dec 13 2020 at 16:18, on Zulip):

parser doesn't even have access to a file system to load child files

matklad (Dec 13 2020 at 16:18, on Zulip):

But, as long as you do anything with a crate, we parse all of it's files. Due to the nature of rust's name resolution algorithm, it's not really possible to postopne parsing within the crate

kev (Dec 13 2020 at 16:21, on Zulip):

Hm I see, before I discard this idea, the parser can still have no access to the file system. A component who does would check the CST for mod declarations and call the parser on /foo.

The aim was to do parsing in parallel and have all necessary files parsed before resolution , but I think it introduces a bit more complexity for not that great a benefit

matklad (Dec 13 2020 at 16:24, on Zulip):

That's a great idea actually, and something we'd want to implement

matklad (Dec 13 2020 at 16:24, on Zulip):

although, it should be pushed a little bit further up the stack.

matklad (Dec 13 2020 at 16:25, on Zulip):

Specifically, it's the DefCollector who should done parallelization. When it parsed the file and build an item tree out of it, it should spawn rayon jobs to parse and item-tree-ize foo1, foo2 and foo3.

matklad (Dec 13 2020 at 16:26, on Zulip):

But salsa currently doesn't support within-query parallelism

kev (Dec 13 2020 at 16:27, on Zulip):

matklad said:

Specifically, it's the DefCollector who should done parallelization. When it parsed the file and build an item tree out of it, it should spawn rayon jobs to parse and item-tree-ize foo1, foo2 and foo3.

Oh that makes sense, thanks for clarifying and correcting

Jonas Schievink [he/him] (Dec 13 2020 at 16:29, on Zulip):

Before we go down that route, we can also try parallelizing prime_caches. Last time I tried that I was hitting limitations of rayon, which could also cause issues when salsa supports this natively.

Jonas Schievink [he/him] (Dec 13 2020 at 16:30, on Zulip):

Specifically rayon's behavior around propagating panics caused cancellation to take a lot longer than it should, resulting in poor UX and sporadic test failures

Last update: Jul 29 2021 at 07:45UTC