Stream: t-compiler/help

Topic: simplifying macro repros


davidtwco (Jul 19 2019 at 13:55, on Zulip):

I've managed to minimize #61693 to something pretty small, but depends on syn, quote and proc_macro2 and I've not managed to strip those out yet. Does anyone have any experience with crafting compiler test cases from code like this?

davidtwco (Jul 19 2019 at 14:09, on Zulip):

My current progress w/ the repro is linked in this comment.

lqd (Jul 19 2019 at 16:59, on Zulip):

@davidtwco haha

lqd (Jul 19 2019 at 16:59, on Zulip):

I tried removing sync/quote etc from the attribute crate

lqd (Jul 19 2019 at 17:00, on Zulip):

this works but note the super important space

lqd (Jul 19 2019 at 17:00, on Zulip):
// note: qux: Qux<Qux<Baz> >,
// the space              ^ here is important
davidtwco (Jul 19 2019 at 17:01, on Zulip):

Oh, wow. I wouldn’t have thought of that.

lqd (Jul 19 2019 at 17:01, on Zulip):

without the space it builds, with it there's your ^^^^^^^^^^^^^ help: use dyn: dyn #[dom_struct]`

davidtwco (Jul 19 2019 at 17:02, on Zulip):

That’s mad.

lqd (Jul 19 2019 at 17:02, on Zulip):

I haven't looked at the custom derive yet but I assume we can use the same strategy: output a static TokenStream

davidtwco (Jul 19 2019 at 17:02, on Zulip):

I’ve not looked much into what’s happening in the compiler yet, just focused on getting a test case that works.

davidtwco (Jul 19 2019 at 17:03, on Zulip):

Yeah, that’s probably a good idea. Way easier than what I had been trying.

davidtwco (Jul 19 2019 at 17:03, on Zulip):

Thanks!

lqd (Jul 19 2019 at 17:03, on Zulip):

this is what I tried in the first crate but couldn't get it to work, so I built both tokenstreams with proc_macro::quote and syn::quote

lqd (Jul 19 2019 at 17:03, on Zulip):

and asserted they were equal, and they were not

lqd (Jul 19 2019 at 17:03, on Zulip):

so this is where the space comes from

davidtwco (Jul 19 2019 at 17:04, on Zulip):

Could probably look at -Z unpretty output.

lqd (Jul 19 2019 at 17:04, on Zulip):

I tried to get the pretty=expanded but I wasn't to easily get the expanded proc macros token

lqd (Jul 19 2019 at 17:05, on Zulip):

do you know if it works with proc macros or only macros ?

davidtwco (Jul 19 2019 at 17:05, on Zulip):

I’m not sure. I’ve not got all that much experience with either.

lqd (Jul 19 2019 at 17:07, on Zulip):

ah yes it does work

lqd (Jul 19 2019 at 17:08, on Zulip):

to get the errors shown on the attributes etc we might indeed need multiple crates tho

davidtwco (Jul 19 2019 at 17:08, on Zulip):

That’s fine, can do that in compiletest.

lqd (Jul 19 2019 at 17:08, on Zulip):

true (I was mostly responding to the comment about unpretty output)

lqd (Jul 19 2019 at 17:08, on Zulip):

I'll try to simplify the derive if you're not working on it ?

davidtwco (Jul 19 2019 at 17:09, on Zulip):

I’ve stepped away from the computer for a bit. Feel free to, I’d appreciate it.

lqd (Jul 19 2019 at 17:09, on Zulip):

sure

lqd (Jul 19 2019 at 17:35, on Zulip):

it's a bit tougher in the second case, it's related to the spans I think, they look weird with syn/quote

davidtwco (Jul 19 2019 at 18:22, on Zulip):

That’s unfortunate.

davidtwco (Jul 19 2019 at 18:22, on Zulip):

I was wondering if it was something like that.

lqd (Jul 19 2019 at 18:36, on Zulip):

it appears we can't really create/mutate spans

lqd (Jul 19 2019 at 18:36, on Zulip):

here they might be invalid, I'll show you the output I get

lqd (Jul 19 2019 at 18:36, on Zulip):

but I'm not familiar with these so I wouldn't know

lqd (Jul 19 2019 at 18:37, on Zulip):

the test and the output

lqd (Jul 19 2019 at 18:38, on Zulip):

I'm not even sure the problem is related to them, but there's some #0 bytes(0..0) spans, which seem like an interesting difference

lqd (Jul 19 2019 at 18:40, on Zulip):

if I modify the way to create the items with syn, juste quote! statically, the error disappears as well

lqd (Jul 19 2019 at 18:40, on Zulip):

so I'm a bit stumped @davidtwco

davidtwco (Jul 19 2019 at 18:41, on Zulip):

Hmm...

davidtwco (Jul 19 2019 at 18:42, on Zulip):

Can you index into the input stream (using .into_iter().nth(..)) and then use that to get the spans?

davidtwco (Jul 19 2019 at 18:42, on Zulip):

Nevermind, I was mixing up which spans were which.

lqd (Jul 19 2019 at 18:42, on Zulip):

I'll try to see how the spans look in the quoted static input just in case

lqd (Jul 19 2019 at 18:44, on Zulip):

they are identical to what proc_macro generates then

lqd (Jul 19 2019 at 18:44, on Zulip):

and the error disappears

davidtwco (Jul 19 2019 at 18:45, on Zulip):

So, we think syn/quote is making spans bytes(0..0) and that's what's causing the issue?

davidtwco (Jul 19 2019 at 18:45, on Zulip):

Since that's the only difference?

lqd (Jul 19 2019 at 18:45, on Zulip):

it's a bit hard to tell honestly :)

lqd (Jul 19 2019 at 18:46, on Zulip):

it's the only difference I was able to see in the time I looked at it

lqd (Jul 19 2019 at 18:46, on Zulip):

I'll makesure again the output is identical

davidtwco (Jul 19 2019 at 18:46, on Zulip):

Thanks for taking a look, I'll see what I can dig up when I get back to a machine.

lqd (Jul 19 2019 at 18:51, on Zulip):

the expanded output is indeed identical

lqd (Jul 19 2019 at 18:52, on Zulip):

why isn't the bare trait giving a warning in the other case ......

lqd (Jul 19 2019 at 18:54, on Zulip):

I really gtg though, I'll update the derive test so that you can see what I'm talking about :)

davidtwco (Jul 19 2019 at 18:54, on Zulip):

Thanks, appreciate the help.

lqd (Jul 19 2019 at 18:54, on Zulip):

updated here

lqd (Jul 19 2019 at 18:54, on Zulip):

no problem :)

lqd (Jul 19 2019 at 18:55, on Zulip):

a weird one for sure

lqd (Jul 19 2019 at 18:55, on Zulip):

and that weird space.... :)

davidtwco (Jul 19 2019 at 19:32, on Zulip):

I think the bug here is probably that there's a way to get spans that are bytes(0..0) at all - I can't figure out how I'd construct those.

davidtwco (Jul 19 2019 at 19:34, on Zulip):

@Vadim Petrochenkov I'm not familiar at all with how proc-macro crates are intended to work, it there supposed to be a way to get what are effectively dummy spans in a #[proc_macro_derive]?

lqd (Jul 19 2019 at 19:34, on Zulip):

I'm also looking into how the quote crate does it

davidtwco (Jul 19 2019 at 19:34, on Zulip):

I'm able to reproduce the results you were getting.

davidtwco (Jul 19 2019 at 19:35, on Zulip):

I've been having a look at quote and syn and nothing is jumping out to me from a few GitHub searches.

lqd (Jul 19 2019 at 19:37, on Zulip):

it seems like an unlucky collision of a bunch of different factors

davidtwco (Jul 19 2019 at 19:38, on Zulip):

Probably from a use of Span::def_site() or Span::call_site()?

lqd (Jul 19 2019 at 19:41, on Zulip):

yeah I don't see another way to create them :)

lqd (Jul 19 2019 at 19:42, on Zulip):

if that's the case then in this context rustc is giving the weird spans to quote/proc_macro2 ?

davidtwco (Jul 19 2019 at 19:43, on Zulip):

Maybe? I have no idea how to craft a test for that.

lqd (Jul 19 2019 at 19:44, on Zulip):

or if it's even the real root cause of the problem

davidtwco (Jul 19 2019 at 19:47, on Zulip):

I can get Span to have a #0 by calling .parent() twice.

davidtwco (Jul 19 2019 at 19:48, on Zulip):

But it doesn't change the range.

davidtwco (Jul 19 2019 at 19:48, on Zulip):

(that is, the def_span and call_site spans)

davidtwco (Jul 19 2019 at 19:50, on Zulip):

Literal::subspan might also be useful if we could get a big enough initial Span.

lqd (Jul 19 2019 at 19:50, on Zulip):

maybe there's a way to get the proc_macro bridge server/client to create the weird range span :/

davidtwco (Jul 19 2019 at 19:51, on Zulip):

:shrug:

davidtwco (Jul 19 2019 at 19:53, on Zulip):

Next thing would be to step through with a debugger everywhere that quote or syn gets a Span and find out where it gets the dummy span?

lqd (Jul 19 2019 at 19:53, on Zulip):

yeah interesting thought, we're a bit in the dark still

davidtwco (Jul 19 2019 at 19:53, on Zulip):

That's assuming that it is some set_span call in those crates that do it and not some other function that manipulates tokens or token streams.

lqd (Jul 19 2019 at 19:54, on Zulip):

it might also be Span::call_site/def_site but from inside proc_macro2

davidtwco (Jul 19 2019 at 19:55, on Zulip):

Yeah.

lqd (Jul 19 2019 at 20:29, on Zulip):

the contents sure look like the syntax DUMMY_SP but for proc_macro::Span

lqd (Jul 19 2019 at 20:49, on Zulip):

@davidtwco https://github.com/rust-lang/rust/blob/9ebf47851a357faa4cd97f4b1dc7835f6376e639/src/libsyntax_ext/proc_macro_server.rs#L454-L465

lqd (Jul 19 2019 at 20:49, on Zulip):

interesting

davidtwco (Jul 19 2019 at 20:50, on Zulip):

Yeah, that is interesting.

davidtwco (Jul 19 2019 at 20:51, on Zulip):

Implies that dummy spans can be valid?

lqd (Jul 19 2019 at 20:52, on Zulip):

maybe ? at the very least known to happen in some cases let's say

davidtwco (Jul 19 2019 at 20:52, on Zulip):

Yeah, that’s reasonable.

lqd (Jul 19 2019 at 20:58, on Zulip):

how hard it is to actually turn DUMMY_SP into a proc_macro::Span is a testament to rustc's crates privacy design, good job :)

Vadim Petrochenkov (Jul 20 2019 at 00:19, on Zulip):

I didn't look into the details, but DUMMY_SP in proc macro tokens means that the tokens were obtained through pretty-printing and then re-parsing the input.

Vadim Petrochenkov (Jul 20 2019 at 00:21, on Zulip):

This happens as a fallback when the original tokens are unavailable or invalidated in any way (https://github.com/rust-lang/rust/issues/43081).

lqd (Jul 20 2019 at 00:46, on Zulip):

it took a while but I figured it out @davidtwco

lqd (Jul 20 2019 at 00:47, on Zulip):

after trying a lot of things including setting up a rustc driver to force the dummy_sp, I saw the light of day

lqd (Jul 20 2019 at 00:48, on Zulip):

the dummy_sp was in the input along, PR here

lqd (Jul 20 2019 at 00:49, on Zulip):

depending on how you override the spans the error can be different

lqd (Jul 20 2019 at 00:51, on Zulip):

if you clobber them all it will have an error like

^ help: use `dyn`: `dyn #`
lqd (Jul 20 2019 at 00:51, on Zulip):

so I set it up like the original code did with quoteand it matches the error you expect

help: use `dyn`: `dyn #[dom_struct]`
davidtwco (Jul 20 2019 at 07:31, on Zulip):

That’s awesome!

davidtwco (Jul 20 2019 at 07:32, on Zulip):

Thanks a bunch.

davidtwco (Jul 20 2019 at 07:36, on Zulip):

Does that still preserve the original warning too on the field?

davidtwco (Jul 20 2019 at 07:36, on Zulip):

(it doesn’t really matter but it’d be interesting)

davidtwco (Jul 20 2019 at 09:11, on Zulip):

I've pushed your changes to the repro repository and modified the attribute crate so that it is also token-for-token identical to what quote/syn was outputting.

lqd (Jul 20 2019 at 09:34, on Zulip):

it's probably not the most minimal repro of the spirit of the bug, as we're making sure the tokens match the original but it's likely small enough to be debugged, and have it as a regression test

davidtwco (Jul 20 2019 at 09:34, on Zulip):

Yeah, I think that's the best that we could hope for.

lqd (Jul 20 2019 at 09:36, on Zulip):

alright cheers David, enjoy the weekend :)

davidtwco (Jul 20 2019 at 09:36, on Zulip):

You too, thanks a ton for the help.

Last update: Nov 11 2019 at 22:20UTC