Stream: t-compiler

Topic: Nt tokens in the parser


matklad (Aug 31 2019 at 19:34, on Zulip):

I would like to move rust’s parser to the same token model that proc macros use. However, the current model contains various Nt tokens which store parsed fragments of AST. My understanding is that they exist solely for macro-by-example macros.

Are such token fundamental, or are we hoping to replace them with token trees with none delimiters in the future?

Vadim Petrochenkov (Sep 01 2019 at 00:43, on Zulip):

Not fundamental, but we may need some hacks for compatibility.

Vadim Petrochenkov (Sep 01 2019 at 00:44, on Zulip):

$expr:expr is a TokenTree::Group with an implicit delimiter in the token model.

Vadim Petrochenkov (Sep 01 2019 at 00:45, on Zulip):

I think this model covers most of the existing behavior, but not all.

Vadim Petrochenkov (Sep 01 2019 at 00:45, on Zulip):

See https://github.com/rust-lang/rust/issues/55414, https://github.com/rust-lang/rust/issues/47358 for exceptions.

Vadim Petrochenkov (Sep 01 2019 at 00:47, on Zulip):

The basic example is

macro_rules! doc_from_file {
    ($expr:expr) => {
        #[doc = $expr]
        struct S;
    };
}

doc_from_file!(include_str!("doc.md"));
Vadim Petrochenkov (Sep 01 2019 at 00:48, on Zulip):

$expr is a nonterminal token with an ast::Expr fragment inside it.

Vadim Petrochenkov (Sep 01 2019 at 00:49, on Zulip):

And that ast::Expr is visited by the visitor expanding macros as any other expression in less exotic position.

Vadim Petrochenkov (Sep 01 2019 at 00:49, on Zulip):

So we get #[doc = "contents of doc.md"] after macro expansion.

Vadim Petrochenkov (Sep 01 2019 at 00:50, on Zulip):

This is entirely incompatible with the token model where include_str!("doc.md") is not an ast::Expr, but just some token stream that fits into expression grammar.

Vadim Petrochenkov (Sep 01 2019 at 00:52, on Zulip):

(I'm currently preparing an experiment for checking whether this behavior can be limited to e.g. only values in key-value attributes, rather than to arbitrary non-terminal tokens.)

Vadim Petrochenkov (Sep 01 2019 at 00:53, on Zulip):

There may be other cases where nonterminals are fundamentally treated as AST pieces and not tokens streams, but I'm not aware of them right now.

Last update: Nov 16 2019 at 02:10UTC