Stream: t-compiler

Topic: Switchig parser from splitting tokens to gluing tokens


matklad (Aug 07 2019 at 19:24, on Zulip):

My understanding is that we want to switch the parser from "composite tokens are always glued" to "composite token are always split" model, to mesh better with proc_macro token trees. @Vadim Petrochenkov , @eddyb is there any existing issue for it? Should I just create a new one? I am not sure what's the best way to proccede here: seems like I should just change everything in one go?

matklad (Aug 07 2019 at 19:24, on Zulip):

Also, do we want to kill the glued tokens themselves?

Vadim Petrochenkov (Aug 07 2019 at 19:48, on Zulip):

is there any existing issue for it?

I don't think so.

Vadim Petrochenkov (Aug 07 2019 at 19:49, on Zulip):

seems like I should just change everything in one go?

Vadim Petrochenkov (Aug 07 2019 at 19:50, on Zulip):

Perhaps it can be split.
E.g. the primary parsing (Parser methods) can glue tokens on bump and still work with old-style tokens for some time, while everything else is converted.

Vadim Petrochenkov (Aug 07 2019 at 19:50, on Zulip):

Also, do we want to kill the glued tokens themselves?

Vadim Petrochenkov (Aug 07 2019 at 19:50, on Zulip):

The knowledge about glued tokens have to exist at least in some form because the tt matcher accepts a glued token.

Last update: Nov 20 2019 at 01:15UTC