Stream: t-compiler

Topic: Eager expansion of built-in macros


matklad (Nov 11 2019 at 12:50, on Zulip):

Certain built-in macros expand arguments before expanding the macro itself, which is very different from the way macros usually work. The most common use for this feature is probably

 include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

pattern, where both concat! and include! "evaluate" arguments before doing their thing.

Is there any documentation/interesting PR about this behavior? How it interacts with name resolution?

cc @Vadim Petrochenkov, @Edwin Cheng

Edwin Cheng (Nov 11 2019 at 13:01, on Zulip):

Maybe off-topic : this is a RFC for general eager expansion https://github.com/rust-lang/rfcs/pull/2320

Vadim Petrochenkov (Nov 11 2019 at 18:32, on Zulip):

The interesting PR happened in 2013 to support a specific use case - https://github.com/rust-lang/rust/pull/9740.

Vadim Petrochenkov (Nov 11 2019 at 18:33, on Zulip):

jbclements was a hygiene expert back then, and here's his reaction to that PR - https://github.com/rust-lang/rust/pull/9740#issuecomment-28265862.

Vadim Petrochenkov (Nov 11 2019 at 18:34, on Zulip):

Since then and until 1.0 a bunch of code depended on this in practice and this is how eager expansion got into stable Rust.

Vadim Petrochenkov (Nov 11 2019 at 18:37, on Zulip):

How it interacts with name resolution?

Eagerly expanded macros (and also macros eagerly expanded by eagerly expanded macros, which actually happens in practice too!) are resolved at the location of the "root" macro that performs the eager expansion on its arguments.

Vadim Petrochenkov (Nov 11 2019 at 18:39, on Zulip):

If some name cannot be resolved at the eager expansion time it's considered unresolved, even if becomes available later (e.g. from a glob import or other macro).

Vadim Petrochenkov (Nov 11 2019 at 18:40, on Zulip):

Eagerly expanded macros don't add anything to the module structure of the crate and don't build any speculative module structures, i.e. they are expanded in a "flat" way even if tokens in them look like modules.

Vadim Petrochenkov (Nov 11 2019 at 18:42, on Zulip):

In other words, it kinda works for simple cases for which it was originally intended, and we need to live with it because it's available on stable and widely relied upon.

matklad (Nov 15 2019 at 13:37, on Zulip):

Eagerly expanded macros don't add anything to the module structure

To clarify, this means that it's impossible to define items with eager macros? I.e, there's no way to sneak in an impl via concat_idents!?

matklad (Nov 15 2019 at 13:56, on Zulip):

In general, it seems like proper eager expansion throws a wrench into the incremental name resolution. At the moment, we incrementalise nameres by making each macro expansion a query, whose arguments are a macro def id and a token tree. By resolving the macro, we condense all resolver context into a single integer. For eager expansion, however, we would need the full set of visible names as a query argument.

Cc @nikomatsakis just in case, as they suggested the way to handle non-eager macros incrementally.

matklad (Nov 15 2019 at 14:12, on Zulip):

Or perhaps it's not that bad, if we just use a function to expand an eager-macro, and then just intern the resulting token tree? Ie, we will be interning really large and complex values. Ie, using interning not like traditional space-saving interning, but just like a key-value store

matklad (Nov 15 2019 at 15:59, on Zulip):

oh, cc @Vadim Petrochenkov :)

Vadim Petrochenkov (Nov 16 2019 at 14:58, on Zulip):

To clarify, this means that it's impossible to define items with eager macros? I.e, there's no way to sneak in an impl via concat_idents!?

Correct. This also means it's not possible to create temporary items (e.g. additional macros, or modules, or imports) being used only during the expansion.
The eager expansion RFC has a bunch of examples like this - https://github.com/pierzchalski/rfcs/blob/macro-expansion-api/text/0000-eager-macro-expansion.md#appendix-a-corner-cases.

Vadim Petrochenkov (Nov 16 2019 at 14:59, on Zulip):

In general, it seems like proper eager expansion throws a wrench into the incremental name resolution.

Non-proper eager expansion seems to break incremental as well.
Since the macro needs to resolve names in its output, the current resolver state becomes the macro's input.

Vadim Petrochenkov (Nov 16 2019 at 15:16, on Zulip):

If some name cannot be resolved at the eager expansion time it's considered unresolved, even if becomes available later (e.g. from a glob import or other macro).

I think this part is fixable without too much complications.
The macro's output can be changed from TokenStream to Result<TokenStream, Indeterminate> (internally), and if the macro returns Indeterminate it just returns to the queue to be tried again later.

Eagerly expanded macros don't add anything to the module structure of the crate and don't build any speculative module structures, i.e. they are expanded in a "flat" way even if tokens in them look like modules.

And this is something we may want to leave as is - explicitly specify eager expansion as being "flat" / "token-only" and tell users to deal with it.
Otherwise implementing speculative module structures is a big serious work, and work like this very rarely gets done, see #43081 (unless perhaps some payed person/people are focusing on it full-time).

nikomatsakis (Nov 19 2019 at 19:00, on Zulip):

@matklad I'm not quite clear on whether this got "resolved" here

nikomatsakis (Nov 19 2019 at 19:00, on Zulip):

Let me resummarize to be sure I understand

nikomatsakis (Nov 19 2019 at 19:00, on Zulip):

the core problem here is that expanding certain macros (e.g., include!) "eagerly evaluate" their arguments

nikomatsakis (Nov 19 2019 at 19:01, on Zulip):

right now, in rust-analyzer, expanding macros in general is a query

nikomatsakis (Nov 19 2019 at 19:01, on Zulip):

and that query doesn't have access to enough state to do that eager evaluation

nikomatsakis (Nov 19 2019 at 19:01, on Zulip):

correct?

matklad (Nov 19 2019 at 19:28, on Zulip):

Correct

Last update: Dec 12 2019 at 00:45UTC