Stream: t-compiler/wg-rls-2.0

Topic: MBE: iterator interface for TokenSource


matklad (Apr 11 2019 at 10:07, on Zulip):

@Edwin Cheng split into the new stream

matklad (Apr 11 2019 at 10:08, on Zulip):

SO yeah, I feel like making TokenSource an iterator-ish thing with explicit bump method would work

matklad (Apr 11 2019 at 10:09, on Zulip):

The two probels we need to sovle are

matklad (Apr 11 2019 at 10:10, on Zulip):

I think we always do a constant amount of lookahead, and that lookahead should never look inside of a token tree, so

fn lookahead(&self, n: usize /* n <= 3*/) -> SyntaxKind would work

matklad (Apr 11 2019 at 10:11, on Zulip):

For composite tokens, we never lookahread them, IIRC, so having an

fn at(&self, composite: SyntaxKind) and fn bump_composite(&self, kind: SyntaxKind) should work

Edwin Cheng (Apr 11 2019 at 10:12, on Zulip):

And fn lookahead will not read the parent / child level, right ?

matklad (Apr 11 2019 at 10:13, on Zulip):

right, lookahead is stricttly within one level

Edwin Cheng (Apr 11 2019 at 10:16, on Zulip):

If so, i think it can be simplify a lot of current implementation of tree-traversal.

matklad (Apr 11 2019 at 10:16, on Zulip):

yeah, that's the hope!

Edwin Cheng (Apr 11 2019 at 10:18, on Zulip):

Just reiterate :

trait TokenSource {
   fn current(&self) -> SyntaxKind;   // Can composite
   fn is_at_composite(&self, kind: SyntaxKind);
   fn bump(&mut self); // Can not composite
   fn bump_composite(&mut sekf, SyntaxKind); // Can composite
   fn lookahead(&self, n: usize /* n <= 3*/) -> SyntaxKind; // Can composite ??!
}

right

Edwin Cheng (Apr 11 2019 at 10:18, on Zulip):

?

matklad (Apr 11 2019 at 10:19, on Zulip):

I guess, both current and lookahead should not composite

matklad (Apr 11 2019 at 10:20, on Zulip):

that is, one should just call is_at_composite explicitelly instead

matklad (Apr 11 2019 at 10:20, on Zulip):

so, if parser is at <<, current returns <, lookahead on 0 and 1 returns <, but is_at_composite('<<') returns true

Edwin Cheng (Apr 11 2019 at 10:21, on Zulip):

That's great !!!

Edwin Cheng (Apr 11 2019 at 10:23, on Zulip):

If so, we maybe even do not need ttToken struct, we could compute the value on the fly.

matklad (Apr 11 2019 at 10:24, on Zulip):

yeah, I guess that is true!

Edwin Cheng (Apr 11 2019 at 10:24, on Zulip):

Okay , i think the remain problem is TreeSink trait ?

matklad (Apr 11 2019 at 10:26, on Zulip):

why is it problematic?

Edwin Cheng (Apr 11 2019 at 10:29, on Zulip):

the current TreeSink::token is defined as following:

fn token::(&mut self, kind: SyntaxKind, n_tokens: u8)

So we need to roll back the cursor and traversal it again to collect the tokens, do you think there is a better solution ?

matklad (Apr 11 2019 at 10:30, on Zulip):

Hm, I wonder why we need n_tokens there at all

matklad (Apr 11 2019 at 10:30, on Zulip):

like SyntaxKind should determine n_tokens exactly?

matklad (Apr 11 2019 at 10:32, on Zulip):

I gues, if TokenSource is a mutable iterator, for MBE token-sink can be 100% no-op?

matklad (Apr 11 2019 at 10:32, on Zulip):

like we just see how far the iterator is advanced

Edwin Cheng (Apr 11 2019 at 10:35, on Zulip):

Why do we need treesink of TokenSource is mutable iterator ?

Edwin Cheng (Apr 11 2019 at 10:35, on Zulip):

Why do we need treesink if TokenSource is mutable iterator ?

matklad (Apr 11 2019 at 10:35, on Zulip):

for macros, we probably don't need it at all

matklad (Apr 11 2019 at 10:35, on Zulip):

but for actual parser, where we have a flat list of tokens and want a syntax tree in the end, we do need tree sink

Edwin Cheng (Apr 11 2019 at 10:38, on Zulip):

Okay, make sense.

Edwin Cheng (Apr 11 2019 at 10:40, on Zulip):

Thats enough for me to try implement a PR for this change, after i finish the add LR_DOLLOR PR :)

Last update: Nov 19 2019 at 17:35UTC