Stream: t-compiler/wg-rls-2.0

Topic: Improving VFS to support dynamically loading files/workspace


Pyry Kontio (May 10 2020 at 19:35, on Zulip):

This is a continuation from thread "Improving the experience on Sublime Text". ( https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/Improving.20the.20experience.20.20on.20Sublime.20Text )

I want to make supporting textDocument/didOpen with no workspaces already open, happen. For this, it was said that some improvements to VFS need to happen. I read through the source code of the ra_vfs crate. As a next step, I'd like to know if there are any work planned or already being work to improve VFS? Should I try and start tinkering locally, thinking about the changes needed and send some strawman pull requests or do you think some centralized discussion about how to go with it are beneficial?

Pyry Kontio (May 10 2020 at 19:37, on Zulip):

For reference, this issue tracks the work. https://github.com/rust-analyzer/rust-analyzer/issues/3715 Watchman client was mentioned, is there a plan to migrate from the notify crate?

Pyry Kontio (May 10 2020 at 19:38, on Zulip):

(Hmm, this bullet list seems relevant https://github.com/rust-analyzer/rust-analyzer/issues/1968)

Pyry Kontio (May 10 2020 at 19:39, on Zulip):

Notify stuff doesn't perhaps directly have to do with adding roots, but if there's an upcoming significant re-design, there's gonna be churn, I guess.

Pyry Kontio (May 10 2020 at 19:41, on Zulip):

Does "lazy VFS" mean that the watching isn't done eagerly and recursively in the future? Is there any planned changes to the "root" model?

Florian Diebold (May 10 2020 at 21:19, on Zulip):

yeah, I think the idea is that we shouldn't eagerly read into memory and watch every (rust) file in the workspace, and instead only access them and start watching when we actually need them

matklad (May 10 2020 at 22:38, on Zulip):

yeah, I think the idea is that we shouldn't eagerly read into memory and watch every (rust) file in the workspace, and instead only access them and start watching when we actually need them

Not exactly -- I believe that watching needs to be done eagarly, but loading lazliy

matklad (May 10 2020 at 22:39, on Zulip):

It's impossible to correctly lazy watch the files, because the way to guarantee lost update is to first setup watchers and then do a scan of the whole dir, to not miss files created when setting up watchers was in progress

Laurențiu Nicola (May 11 2020 at 06:41, on Zulip):

Stupid question, but do we care about changes to directories we haven't look at yet? E.g. if foo.rs is added to disk but there's not mod foo anywhere, do we care about it? We might need to to support cases like files outside of a project, but can't we add them to the VFS when needed?

GolDDranks / Pyry Kontio (May 11 2020 at 15:32, on Zulip):

Is there a way to build in debug mode with cargo xtask install? I'm using stack traces to try to figure out what the code is doing from time to time, but it seems to skip many stack frames, which I presume, is because optimizations.

Laurențiu Nicola (May 11 2020 at 15:33, on Zulip):

Set debug = 2 under [profile.release]

GolDDranks / Pyry Kontio (May 11 2020 at 15:47, on Zulip):

Thanks!

GolDDranks / Pyry Kontio (May 11 2020 at 16:22, on Zulip):

Hmm, I'm still unable to get full backtraces... For example it's saying that rust_analyzer::main_loop::loop_turn calls rust_analyzer::world::WorldSnapshot::uri_to_file_id, which isn't the case, there's at least two stack frames in between. Plus, it doesn't show line numbers.

GolDDranks / Pyry Kontio (May 11 2020 at 16:22, on Zulip):

I wonder what's wrong :(

Edwin Cheng (May 11 2020 at 16:37, on Zulip):

The method @Laurențiu Nicola mentioned is on release mode, so compiler is still able to inline some functions. And currently, as I know cargo xtask install doesn't support debug build. Is it possible to use
cargo run --debug -p rust-analyzer -- analysis-stats .
on your case ?

GolDDranks / Pyry Kontio (May 11 2020 at 16:52, on Zulip):

@Edwin Cheng Thanks, that helped!

Florian Diebold (May 18 2020 at 14:07, on Zulip):

matklad said:

It's impossible to correctly lazy watch the files, because the way to guarantee lost update is to first setup watchers and then do a scan of the whole dir, to not miss files created when setting up watchers was in progress

I still don't quite understand the reasoning here -- to me, it seems like it should be possible to, whenever we actually start caring about the contents of a file, 1. start a watcher on it and then 2. read the content

Florian Diebold (May 18 2020 at 14:08, on Zulip):

in particular, rust project structure never depends on files just existing somewhere, right? I'm not sure about this, maybe I'm missing some case

matklad (May 18 2020 at 14:08, on Zulip):

It sort-of depends

matklad (May 18 2020 at 14:09, on Zulip):

mod foo; gives you an error if you have both foo.rs and foo/mod.rs

matklad (May 18 2020 at 14:09, on Zulip):

So, you also have to watch the dir

matklad (May 18 2020 at 14:09, on Zulip):

I am also not entirely sure that what I wrote is true, but I think watchman works that way.

matklad (May 18 2020 at 14:10, on Zulip):

Ie, when you watch a directory, it first emits a bunch of "created" events for all the files

Florian Diebold (May 18 2020 at 14:18, on Zulip):

matklad said:

mod foo; gives you an error if you have both foo.rs and foo/mod.rs

ok, but that's still just two files. I guess what I meant was that we never need to have the full list of files in any directory. There's a finite set of files that we care about because the editor mentioned/opened them, and then from them we have to go looking for a still-finite set of related files

matklad (May 18 2020 at 14:20, on Zulip):

On the one hand, it's true. On the other hand, it's good to be aware about the files not linked to the module tree at all....

Florian Diebold (May 18 2020 at 14:21, on Zulip):

yeah, but even if the user doesn't open them?

matklad (May 18 2020 at 14:22, on Zulip):

Maybe? For our current indexing for sure

Florian Diebold (May 18 2020 at 14:24, on Zulip):

hmm. ok, the indexing is a problem. maybe we shouldn't be doing that anyway if we want to support big monorepos?

Wez Furlong (May 18 2020 at 20:15, on Zulip):

Greetings! I'm deeply involved with watchman/EdenFS/big monorepos and helping folks that do build/indexing at FB work well in that environment. I'd be happy to answer questions that you might have about watchman specifically and general thoughts/insights from working in monorepos over the past few years. I'm also happy to do eg: a video chat for a higher bandwidth "note comparison" session. I'm based in California but am a relatively early riser so can accommodate more European friendly times for that.

matklad (May 19 2020 at 19:15, on Zulip):

Hello @Wez Furlong !

I believe a video chat would be very useful! What about 18:00 CET / 9 am PDT some day this week (any day works for me)?

I'll also catch up with comments on the issue and will write in more detail about rust-analyzer requirenments (or at least things I believe to be requirements) and clear up the "consistent snapshots" confusion!

GolDDranks / Pyry Kontio (May 20 2020 at 23:22, on Zulip):

I'm also interested to join

GolDDranks / Pyry Kontio (May 20 2020 at 23:46, on Zulip):

(18:00 CET is possible for me but a few hours earlier would be even better, as my timezone is JST (UTC+9). I'm a night owl so I can make it work though.)

Wez Furlong (May 21 2020 at 16:01, on Zulip):

@matklad hey; sorry for the really short notice; it's turning into a bit of a weird week! Do you have time to chat now for 30 mins? If not, then early next week?

Wez Furlong (May 21 2020 at 16:04, on Zulip):

also, do you have a preferred VC? It's pretty easy for me to host a Blue Jeans

Wez Furlong (May 21 2020 at 16:11, on Zulip):

Ah, I just saw @GolDDranks / Pyry Kontio comment. How about something like 7:30AM pacific time on May 26 for the three of us? I can set up a bluejeans session and email you the details (you can email me wez@fb.com if you want to share those details with me out of band from zulip)

matklad (May 21 2020 at 16:50, on Zulip):

May 26 and bluejeans work for me! Send an email!

Coenen Benjamin (May 22 2020 at 07:41, on Zulip):

Is it going to be a recorded session ? I'm interested by the topic but I'm more comfortable to watch the replay. And I think it could be interesting to have this kind of recorded resources :D

Wez Furlong (May 26 2020 at 14:28, on Zulip):

No plans to record this time around!

Last update: May 29 2020 at 16:20UTC