Stream: t-compiler

Topic: eager expansion


Alexander Regueiro (Feb 06 2019 at 10:16, on Zulip):

This is to discuss macro eager expansion, in particular w.r.t. @Edward Pierzchalski's recent RFC and @nrc's older one.

Alexander Regueiro (Feb 06 2019 at 10:17, on Zulip):

Ah... I don't think @nrc is on Zulip

Edward Pierzchalski (Feb 06 2019 at 10:31, on Zulip):

For future reference, here are links to the eRFC by me and the RFC by nrc.

Alexander Regueiro (Feb 06 2019 at 18:29, on Zulip):

Yep, good.

Alexander Regueiro (Feb 06 2019 at 18:30, on Zulip):

So... apparently @nrc is leaving Rust or at least hugely decreasing participation.

Alexander Regueiro (Feb 06 2019 at 18:30, on Zulip):

@Edward Pierzchalski

Alexander Regueiro (Feb 06 2019 at 18:31, on Zulip):

We could have a chat with @Vadim Petrochenkov about this though, who is sort of the macros guru.

Alexander Regueiro (Feb 06 2019 at 18:31, on Zulip):

Are you free for a meeting some time next week? :-)

Edward Pierzchalski (Feb 06 2019 at 21:11, on Zulip):

I should be pretty free next week for a meeting. However, if I recall correctly petrochenkov and jseyfried were too busy or otherwise not keen on participate much on this. Should we pull in @centril or llogic?

Edward Pierzchalski (Mar 11 2019 at 23:10, on Zulip):

For anyone interested, after discussion between @Alexander Regueiro and myself, the RFC has been polished and updated (and might enter FCP soonish?), so if you have any feedback now's the time!

pnkfelix (Oct 02 2019 at 12:47, on Zulip):

Hi @Edward Pierzchalski , I have some questions to pose while I write my suggested edits to the RFC

pnkfelix (Oct 02 2019 at 12:49, on Zulip):

(but I wanted to double-check that you are active here before writing them out in this topic)

Edward Pierzchalski (Oct 02 2019 at 13:35, on Zulip):

Sure, go ahead (but I'm just about to go to bed, so I'll respond later)

pnkfelix (Oct 02 2019 at 13:36, on Zulip):

okay, first question (and any of these may reflect gross ignorance on my part)

pnkfelix (Oct 02 2019 at 13:36, on Zulip):

:question: Do you expect anyone using the API to not immediately call .await?

pnkfelix (Oct 02 2019 at 13:37, on Zulip):

I'm basically trying to imagine what my code would look like, and I don't understand what the value is of exposing the Future by returning it, versus calling .await yourself within the implementation of fn expand(self)

pnkfelix (Oct 02 2019 at 13:40, on Zulip):

:question: ExpansionBuilder::from_tokens(tokens: TokenStream) -> Result<Self, ParseError> takes tokens by value, and tries to parse it. What is it going to parse it as: An expression? Or just as a token-tree?

pnkfelix (Oct 02 2019 at 13:40, on Zulip):

(I'm guessing "token-tree", but I wanted to double-check.)

pnkfelix (Oct 02 2019 at 13:42, on Zulip):

(or maybe its expecting a forest of token trees? namely, I don't yet know how it handles trailing input after the first token tree it parses)

pnkfelix (Oct 02 2019 at 13:43, on Zulip):

One goal I have is to try to revise the text so that we can illustrate the API by showing the definition of the declarative expand! in an appendix.

pnkfelix (Oct 02 2019 at 13:45, on Zulip):

That is, instead of saying "here's some useful functionality that is specified by this RFC" (and leaving it unclear whether this is some fundamental new feature being supplied), I'm anticipating it could say "the ExpansionBuilder API is all you need. For example, check out the definition of the declarative expand!, given in Appendix Z."

Edward Pierzchalski (Oct 02 2019 at 23:27, on Zulip):

:question: Do you expect anyone using the API to not immediately call .await after invoking eb.expand()?

That's a fair point. I suppose the other way of making it clear to the caller that expand is a potentially blocking, long-running operation is to... put it in the docs? One situation where that might backfire is when your proc macro does IO - then you want to be very clear about where such long-running things occur.

Edward Pierzchalski (Oct 02 2019 at 23:44, on Zulip):

:question: ExpansionBuilder::from_tokens(tokens: TokenStream) -> Result<Self, ParseError> takes tokens by value, and tries to parse it. What is it going to parse it as: An expression? Or just as a token-tree?

The compiler needs to identify where in the input there might be a macro invocation, which means it needs to do 'real' parsing at some point (we could push that back to when the user calls expand though, and just do token tree parsing here like you say). If we do eager parsing, I'm unsure as to whether we need grammar-context-aware constructors (from_expr_tokens, from_item_tokens, etc).

One alternative is to force from_tokens to only accept a single 'direct and complete' invocation (accepting foo!() but rejecting let x = 5; foo!()) and leaving it up to the caller to find and extract any macros in the input, but this causes issues if we ever want to support 'location aware' macro expansion (e.g. handling super when expanding mod a { super::foo!{} }).

Edward Pierzchalski (Oct 02 2019 at 23:48, on Zulip):

One goal I have is to try to revise the text so that we can illustrate the API by showing the definition of the declarative expand! in an appendix.

That's a good idea, and I hoped to do so in an earlier draft, but I didn't feel like dealing with the details of input parsing and interpolation at the time. I might try and take a crack at it this weekend.

pnkfelix (Oct 03 2019 at 08:55, on Zulip):

If we do eager parsing, I'm unsure as to whether we need grammar-context-aware constructors (from_expr_tokens, from_item_tokens, etc).

Yes, this is exactly the problem I was anticipating.

pnkfelix (Oct 03 2019 at 08:56, on Zulip):

The good news is that a Builder interface like this lets you add new constructors later

pnkfelix (Oct 03 2019 at 08:58, on Zulip):

So I suspect the "right" answer will be delay the identification of macro invocations to the point where the user calls expand. (I'm trying to guess the value one would get from doing the identification of the invocations more upfront, and to be honest I'm drawing a blank, since its not like the ExpansionBuilder API allows one to inspect those identified invocations, right?)

Edward Pierzchalski (Oct 03 2019 at 22:26, on Zulip):

Right, so maybe there's no harm in going all the way in the other direction and making from_tokens infallible, moving all the parsing logic and error reporting up to expand.

Edward Pierzchalski (Oct 03 2019 at 22:30, on Zulip):

I suppose it's worth thinking a bit more about how the compiler will look for invocations in a token stream, see if we need those _expr variants for an MVP anyway

Last update: Nov 20 2019 at 02:10UTC