We are at the point where parsing takes substantional amount of time. I think the main thing we need a parse tree for is the item tree.
I wonder if we should think about producing it without parse tree at all... It seems approximately feasible to use evented parsing for this purpose, instead of a true syntax tree
Can the MBE stuff be disabled? I wonder how much faster parsing would get.
I think only by commenting it out
But where? :D
That I don't know without looking :)
Yeah, it's 10s vs. 18s, so not that much of a win
OTOH, we don't have that many macros (besides
match_ast!), so it's still a considerable speedup
(I stubbed out
It's almost twice as fast?
Yeah, something like that
that feels like very much of a win :D
Meh, not even an order of magnitude :-)
On one hand, according to the argument you often make, skipping bodies would make us less likely to optimize the parser itself
On the other hand,
analysis-stats still has to parse the bodies, so it's not like we'd no longer know how slow it is
On the third hand, I think you wanted diagnostics support for files that aren't open, so we'd still need the bodies for that
THe number of files in the current crate is far fewer than the total number of files
Like, bodies in libstd are completely uninteresting
Filed https://github.com/rust-analyzer/rust-analyzer/issues/5426 for a small thing, if it makes sense
Does the parser have an event-based interface already or does it construct the syntax tree directly?
It does have an event-based interface
(see TreeSink trait)
I'm doing planning for specific support for delayed parsing in rowan/sorbus, so here's a quick question:
Does the syntax tree need to hold on to the string contents of the skipped body (to facilitate parsing it later, on-demand), or is sticking a thunk of "skipped N bytes" enough?
I'd love (at a mem-usage level, less so the implementation level :stuck_out_tongue:) to be able to skip storing these huge unparsed strings, if we can always just refer back to a "whole file" view to do the full parse later
Or if there are situations where we know with high probability that we won't need to parse the item bodies later
(actually, no, the implementation of text-free thunks was a lot simpler than I thought it was going to be.)
It does have cost, though, so I definitely don't want to merge it unless r-a has a chance of using it.
I think "tree is a value" is a very valuable design choice.
So, I'd say sorbus should store just
Box<FnOnce() -> GreenNode> as a thunk
THat way, client can store text, store tokens, remmeber the context like edition, etc
That does seem optimal, it's just that sticking a
Box<dyn FnOnce> into the tree is difficult since its not erasable, and then synchronization issues as well
As far as I can tell so far it's not possible for sorbus (as currently architectured) to do an
ArcSwap-like trick to mostly transparently swap out a thunk for a real value
Why? Because we're doing pointer un/erasure (on unsized types) and pointer mangling, which is definitely _not_ synchronized
We can just stick in an extra layer of indirection, but
Box<Box<dyn Fn>> feels wrong for a reasonable reason