Stream: t-lang/pub-macro-rules

Topic: recursive macros and shadowing


view this post on Zulip nikomatsakis (Feb 12 2021 at 14:10):

This seems to be the major unknown, so I'm opening this topic to discuss. I'd like to at least have an alternative pattern to recommend for people. I'm not sure we need to supply an automatic migration.

view this post on Zulip nikomatsakis (Feb 12 2021 at 14:10):

One option might be having some way to shadow items or have like an "anonymous scope" you can create.

view this post on Zulip rylev (Feb 18 2021 at 10:22):

@nikomatsakis just to be clear are you talking about recursive macros (i.e., macros that call themselves) or "private" macros (macros that are defined inside of another macro?

view this post on Zulip rylev (Feb 18 2021 at 10:25):

I think these are two separate questions. I've opened a topic on on "private" macros, and I'd like to use this to discuss recursive macros in the sense of macros that call themselves.

view this post on Zulip nikomatsakis (Feb 18 2021 at 10:27):

I see

view this post on Zulip nikomatsakis (Feb 18 2021 at 10:27):

I meant the pattern of a macro defining multiple versions of a helper, which I guess is the other topic

view this post on Zulip nikomatsakis (Feb 18 2021 at 10:27):

I'm not aware of any problems with "recursive" macros

view this post on Zulip nikomatsakis (Feb 18 2021 at 10:28):

Are there problems?

view this post on Zulip nikomatsakis (Feb 18 2021 at 10:28):

I've been a bit sloppy in my terminology...

view this post on Zulip rylev (Feb 18 2021 at 10:33):

I don't believe there's a problem with the transition but writing recursive macros might be harder - Macros usually assume that when they are called they are available in scope with an unqualified name or available at $crate::macro. That might no longer be true as macros can be called through paths (i.e., foo::bar::macro!). If foo::bar::macro! recursively calls itself by calling just macro! without the modules, it won't be in scope.

view this post on Zulip Daniel Henry-Mantilla (Feb 18 2021 at 13:50):

This would kind of require that suggestion of $self paths in macros, or the match_module_path! workaround I posted on GH

view this post on Zulip rylev (Feb 23 2021 at 17:00):

So I just confirmed that this is the case in the current implementation.

view this post on Zulip rylev (Feb 23 2021 at 17:01):

I think adding a $self path makes the most sense. Not having support for this severely limits the usability of macros.

view this post on Zulip rylev (Feb 23 2021 at 17:31):

I'm not familiar with expansion code, but a quick look makes me think that implementing $self won't be trivial.

view this post on Zulip nikomatsakis (Feb 23 2021 at 17:50):

Hmm, this feels like a problem :)

view this post on Zulip rylev (Feb 23 2021 at 17:51):

It is, though I don't think of $self as being necessarily a terrible solution

view this post on Zulip rylev (Feb 23 2021 at 17:51):

It's directly analogous to $crate.

view this post on Zulip nikomatsakis (Feb 23 2021 at 17:56):

it's not .. hmm .. terrible

view this post on Zulip nikomatsakis (Feb 23 2021 at 17:56):

it's just .. more complex

view this post on Zulip rylev (Feb 23 2021 at 17:57):

Certainly. But path based scoping is more complicated than crate based global namespaces.

view this post on Zulip rylev (Feb 23 2021 at 17:59):

$self has its own problems - it would magically have to allow access to the module where the macro is defined but that macro may not be directly available to the caller.

view this post on Zulip nikomatsakis (Feb 23 2021 at 20:03):

as we said in the meeting, definitely starts to feel like hygiene

view this post on Zulip Vadim Petrochenkov (Feb 24 2021 at 08:04):

I thought the goal was to simplify things, not introduce new features.
If some code is difficult to migrate and you think it can be addressed by a new feature, then it is always simpler to provide an option to use macro_rules scoped in the old way instead, because they already implemented and their support won't go away from the compiler any time soon.

view this post on Zulip rylev (Feb 24 2021 at 09:17):

@Vadim Petrochenkov This seemed to be the feeling in the lang team meeting yesterday (i.e., your old plan of just providing an opt-in way to path based scoping). I do think it's a shame there isn't a way to easily do recursive macros in the new system as this is an extremely common pattern.

view this post on Zulip nikomatsakis (Feb 24 2021 at 11:31):

Vadim Petrochenkov said:

I thought the goal was to simplify things, not introduce new features.

I agree with this, but I mildly disagree that having some way to "opt back in" to the old system is simpler

view this post on Zulip nikomatsakis (Feb 24 2021 at 11:31):

In particular, an attribute feels very wrong to me

view this post on Zulip nikomatsakis (Feb 24 2021 at 11:32):

It's like now we are saying: we invented a new system, but it has this weird escape hatch that you have to use sometimes that puts you in a whole new world

view this post on Zulip nikomatsakis (Feb 24 2021 at 11:33):

there is a difference to me between an attribute and having pub "opt-in" to a more normal system-- I think I prefer the latter. It's still "macros are different" but it's not "they're the same; but not always"

view this post on Zulip nikomatsakis (Feb 24 2021 at 11:33):

I realize this is squishy though :)

view this post on Zulip rylev (Feb 24 2021 at 12:19):

I think perhaps we're using "simple" for two different things. There's simple to implement and there's simple to use/explain in the language. Obviously not introducing new concepts like $self is simpler to implement (it's strictly less work), but this doesn't mean it's simpler to use. "Use path based scoping for macros unless your macro is recursive in which case use #[macro_export] is not as simple as "always use path based scoping unless you have some very complex use case"

view this post on Zulip rylev (Feb 24 2021 at 12:19):

I still think we need to discuss recursive macros and the idea of $self. Ideally almost everything should be expressible in the path based system.

view this post on Zulip nikomatsakis (Feb 24 2021 at 17:33):

Yes, that is a trick. I am definitely think of "simple" for end-users, but even there I feel there are various dimensions.

view this post on Zulip Vadim Petrochenkov (Feb 24 2021 at 21:06):

Seems like I confused this case with the "private macro" case from the other thread.

view this post on Zulip Vadim Petrochenkov (Feb 24 2021 at 21:11):

macro macros can just use def-site hygiene to refer to themselves - https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=752b6922a8326de84ee9d0697bcaf56f
So this is more of a "migration problem" rather than a "macro 2.0 problem".

view this post on Zulip Vadim Petrochenkov (Feb 24 2021 at 21:16):

For pub macro_rules! $self can always be replaced with $crate::full::path when necessary - https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=ca18d12267289e0d58865ee0f6f82f58

view this post on Zulip Daniel Henry-Mantilla (Feb 24 2021 at 22:22):

I think the case @rylev had in mind was of somebody defining a recursive macro that gets re-exported out of the private module it was defined in: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=82276ba137cf3fcd7f532bab756a8307

But the fact that macro macros def_site hygiene already handles this well does suggest that implementing $self may be easier than expected

view this post on Zulip rylev (Feb 24 2021 at 22:52):

Indeed the case I meant was when the $crate::full::path is not visible to the caller as could be the case when reexporting.

I looked into implementing $self but I would need @Vadim Petrochenkov’s help to understand how to reuse the logic from macro def site hygiene

view this post on Zulip Daniel Henry-Mantilla (Feb 24 2021 at 22:57):

Here was my attempt at doing it with library code. Oh well :shrug: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=68238880b110b50a5c757d6de96d900d

view this post on Zulip Vadim Petrochenkov (Feb 24 2021 at 23:03):

rylev said:

Indeed the case I meant was when the $crate::full::path is not visible to the caller as could be the case when reexporting.

If the macro is public, then there's at lest one fully visible path to it (possibly involving reexports), which can be used as a $crate::full::path.

view this post on Zulip Vadim Petrochenkov (Feb 24 2021 at 23:07):

Daniel Henry-Mantilla said:

But the fact that macro macros def_site hygiene already handles this well does suggest that implementing $self may be easier than expected

Def-site hygiene is unstable (for multiple reasons), I doubt $self will be useful for migration if it's going to be unstable.
If $self is going to be implemented, which I would prefer to not happen, then it should use stable "mixed-site" hygiene like $crate.

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:45):

Vadim Petrochenkov said:

If the macro is public, then there's at lest one fully visible path to it (possibly involving reexports), which can be used as a $crate::full::path.

one problem is that the macro may not know the path

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:46):

but perhaps you meant something like $self where the macro uses a shorthand that is expanded at the point of use, @Vadim Petrochenkov ?

view this post on Zulip Vadim Petrochenkov (Feb 25 2021 at 09:56):

one problem is that the macro may not know the path

The person who writes the macro certainly knows the path.

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:57):

I don't think that's true in a cross-crate re-export scenario

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:58):

// What happens here?

// crate A
pub mod foo {
  pub macro_rules! m { $crate::foo::m!() }
}

// crate B
pub use crate_a::foo::m;

// crate C
crate_b::m();

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:58):

(from lang team minutes, which I just realized I never published)

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:58):

does that work?

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:58):

it depends on what $crate does, we weren't sure

view this post on Zulip nikomatsakis (Feb 25 2021 at 09:59):

and in particular whether $crate acts "hygienically" -- i.e., maybe crate C doesn't directly depend on crate A

view this post on Zulip rylev (Feb 25 2021 at 10:05):

Just tested $crate and it can "peak through" visibility.

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:06):

you tested it with @Vadim Petrochenkov's PR, @rylev ?

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:06):

what exactly did you test, I guess is my question :)

view this post on Zulip rylev (Feb 25 2021 at 10:06):

No I tested using $crate on the stable compiler

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:07):

how does visibility play in then?

view this post on Zulip rylev (Feb 25 2021 at 10:07):

$crate can see through visbility. I.e., Your example above works even though crate C does not have access to cratea::foo

view this post on Zulip rylev (Feb 25 2021 at 10:08):

Which means @Vadim Petrochenkov is correct that there is always a public path that the macro can refer to (using $crate) that will work no matter who is calling the macro

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:08):

my example above, but translated into #[macro_export] ?

view this post on Zulip rylev (Feb 25 2021 at 10:08):

Ah yes sorry

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:09):

ok, so $crate expands to "the crate where the macro was defined, even if the user couldn't name it", in other words

view this post on Zulip rylev (Feb 25 2021 at 10:09):

We can't yet test it against pub since this doesn't yet work across crates. @Santiago Pastorino is looking into fixing this (there's a problem with metadata currently)

view this post on Zulip rylev (Feb 25 2021 at 10:10):

nikomatsakis said:

ok, so $crate expands to "the crate where the macro was defined, even if the user couldn't name it", in other words

Yes it seems so

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:10):

yes, this is why I was trying to figure out what you meant

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:10):

but if you kept the definition of the macro the same--

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:10):

i.e., it still used $crate::foo::m!...

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:10):

...I don't think I realized that was stable

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:10):

can you just paste exactly what you tested? :)

view this post on Zulip rylev (Feb 25 2021 at 10:14):

crate foo (depends on crate bar):

fn main() {
    bar::my_macro!();
    println!("{}", CONST);
}

crate bar (depends on crate baz):

pub use baz::my_macro;

crate baz:

#[macro_export]
macro_rules! my_macro {
    () => {
       pub use $crate::foo::bar::CONST;
    };
}

pub mod foo {
    pub mod bar {
        pub const CONST: usize = 100;
    }
}

view this post on Zulip rylev (Feb 25 2021 at 10:14):

The above compiles without error ^^

view this post on Zulip rylev (Feb 25 2021 at 10:15):

even though foo cannot directly name baz::foo::bar::CONST

view this post on Zulip rylev (Feb 25 2021 at 10:18):

I'm slightly surprised by this, but it does seem consistent, and means (most likely) that we can create recursive macros in the new system.

view this post on Zulip rylev (Feb 25 2021 at 10:31):

Of course, this doesn't help with local only recursive macros that are not available from the crate root.

view this post on Zulip nikomatsakis (Feb 25 2021 at 10:40):

cool

view this post on Zulip rylev (Feb 25 2021 at 12:21):

So we have a way of allowing cross crate recursive macros but still need to think about the fact that crate private macros cannot be used in locations where the macro cannot be directly accessed from the crate root.

view this post on Zulip rylev (Feb 25 2021 at 15:20):

@Vadim Petrochenkov I've not been able to think of a way that allows crate local macros to always work (outside of a theoretical $self). Do you have any thoughts here?

view this post on Zulip Vadim Petrochenkov (Feb 25 2021 at 19:39):

I think this shouldn't be a problem in practice because in a local crate everything is under control of the macro's author and modules/imports can always be restructured to make the macro's full path (or full path of one of its imports/reexports) accessible from all points in which the macro is used.


Last updated: Jan 26 2022 at 08:46 UTC