Stream: t-compiler/rust-analyzer

Topic: tree-agnostic parsing

matklad (Feb 21 2019 at 10:47, on Zulip):

I've went ahead and extracted rust-analyzer's parser to a separate crate, which depends neither on syntax trees nor on tokens:

matklad (Feb 21 2019 at 10:48, on Zulip):

The interface is these two traits:

pub trait TokenSource {
    fn token_kind(&self, pos: usize) -> SyntaxKind;
    fn is_token_joint_to_next(&self, pos: usize) -> bool;
    fn is_keyword(&self, pos: usize, kw: &str) -> bool;
pub trait TreeSink {
    fn leaf(&mut self, kind: SyntaxKind, n_tokens: u8);
    fn start_branch(&mut self, kind: SyntaxKind, root: bool);
    fn finish_branch(&mut self, root: bool);
    fn error(&mut self, error: ParseError);
matklad (Feb 21 2019 at 10:50, on Zulip):

That is kinda cool, because parser does not know about the underlying text, comments and whitespace (it doesn't know even token's offsets!), but the tree is still full-fidelity, and parsing is still error-tolerant.

matklad (Feb 23 2019 at 14:53, on Zulip):

This seems to actually work: macros can parse token-trees withoug going through toString now:

Last update: Jul 29 2021 at 10:00UTC