Stream: t-compiler/rust-analyzer

Topic: support for fn-like proc macros


nitsky (Nov 23 2020 at 16:11, on Zulip):

Hi, I am interested in tackling https://github.com/rust-analyzer/rust-analyzer/issues/5265. Would someone who is familiar with the current state of proc macro support be willing to give some tips on what needs to change and where to get started? Thank you! :)

Jonas Schievink [he/him] (Nov 24 2020 at 15:41, on Zulip):

It seems to me that something like that would only be possible if the proc macro "cooperates" and propagates the input identifiers instead of creating new ones. That way we can at least theoretically map between input and output tokens via mbe::TokenMap.

I don't know if this actually works correctly with proc macros though.

Florian Diebold (Nov 24 2020 at 15:44, on Zulip):

do we expand fn-like proc macros? I'm confused what the actual problem is now, is it expanding them or making IDE features work inside of them? If it's the latter, and we're expanding them already, Jonas is right. But the macros should already be doing that, if they want correct error spans in rustc, right? In that case the question does become whether we're tracking that correctly all the way through

Jonas Schievink [he/him] (Nov 24 2020 at 15:45, on Zulip):

Yeah we should be expanding them now (not correctly, mind you, but simple cases should work)

Jonas Schievink [he/him] (Nov 24 2020 at 15:46, on Zulip):

(renamed the issue to clarify what is being requested)

Florian Diebold (Nov 24 2020 at 15:47, on Zulip):

I have to admit I don't really know in detail how span tracking works in proc macros, don't they nowadays usually stringify and re-parse their input? :thinking:

Florian Diebold (Nov 24 2020 at 15:47, on Zulip):

or is that not the case anymore

Jonas Schievink [he/him] (Nov 24 2020 at 15:48, on Zulip):

I'm not sure either, syn might consume the tokens from the TokenStream directly

nitsky (Nov 24 2020 at 16:10, on Zulip):

@Jonas Schievink @Florian Diebold Using the syn and quote crates it is straightforward to propagate the input identifiers, and blocks wrapped in curly braces are passed through verbatim.

nitsky (Nov 24 2020 at 16:13, on Zulip):

I am already observing some rudimentary support for fn-like proc macros in rust analyzer: the type of the return value is available on hover, even when there is no type annotation nearby that could be hinting it.

Florian Diebold (Nov 24 2020 at 16:14, on Zulip):

yeah, that's what you'd expect when we can expand the macro, but not track the tokens through the expansion

Florian Diebold (Nov 24 2020 at 16:14, on Zulip):

for IDE features to work inside the macro call, we need to know the connection between the tokens written in the source code, and the tokens in the expanded code. And that probably doesn't work right now

nitsky (Nov 24 2020 at 16:16, on Zulip):

that makes sense

nitsky (Nov 24 2020 at 16:16, on Zulip):

I am observing that the rust compiler is keeping track of that correctly. Red squiggles end up in the right places when there are errors.

Florian Diebold (Nov 24 2020 at 16:17, on Zulip):

that's good in the sense that it should at least be possible to do, then ;)

nitsky (Nov 24 2020 at 16:17, on Zulip):

can you provide any guidance on where in the rust analyzer I could start digging to add this support?

Florian Diebold (Nov 24 2020 at 16:18, on Zulip):

we also already do it for macro_rules macros, so the infrastructure exists already. So the missing piece should "just" be to keep track in the proc-macro specific parts

Jonas Schievink [he/him] (Nov 24 2020 at 16:20, on Zulip):

One thing that isn't going to work is IDE support in incomplete proc macro input. The macro has to expand successfully or we don't get any output from it.

Florian Diebold (Nov 24 2020 at 16:20, on Zulip):

ah, that's true :(

nitsky (Nov 24 2020 at 16:21, on Zulip):

ya, but at least getting it working for when the macro expands successfully would be a big step.

nitsky (Nov 24 2020 at 16:22, on Zulip):

what if the macro was designed to be fault-tolerant, i.e. it expanded to produce partial rust code?

Florian Diebold (Nov 24 2020 at 16:24, on Zulip):

then it should work, but it's doubtful that many proc macros are.

Jonas Schievink [he/him] (Nov 24 2020 at 16:25, on Zulip):

syn would have to be able to parse incomplete code too

nitsky (Nov 24 2020 at 16:26, on Zulip):

Thanks, I'm going to start looking into this and will let you know how it goes.

Florian Diebold (Nov 24 2020 at 16:26, on Zulip):

I can't tell you exactly what to do, but I'd start around here. This function gets a Subtree and returns an expanded Subtree. The tokens in this subtree have TokenIds, and those need to be preserved. @Jonas Schievink am I missing something, or do fn-like proc macros also expand through custom_derive?

Jonas Schievink [he/him] (Nov 24 2020 at 16:27, on Zulip):

yes, it's a gross hack :D

Jonas Schievink [he/him] (Nov 24 2020 at 16:27, on Zulip):

you can use custom derives as !-macros and vice-versa

nitsky (Nov 24 2020 at 16:28, on Zulip):

thanks :)

nitsky (Dec 07 2020 at 22:00, on Zulip):

@Jonas Schievink it seems that the changes you made in the last few days to proc-macros have to some extent implemented this feature!

Jonas Schievink [he/him] (Dec 07 2020 at 22:00, on Zulip):

uh, yeah, this seems to work wayyyy better than expected

Jonas Schievink [he/him] (Dec 07 2020 at 22:00, on Zulip):

I wonder if it was the token remapping fix that did this

Jonas Schievink [he/him] (Dec 07 2020 at 22:01, on Zulip):

They now basically work as well as MBE macros, at least in the cases I've tested

nitsky (Dec 07 2020 at 22:02, on Zulip):

thanks! :) I am testing now and so far a lot is working.

nitsky (Dec 07 2020 at 22:05, on Zulip):

I am seeing a lot of "proc macro server crashed" errors, I will try to get something small and reproducible and file an issue.

nitsky (Dec 07 2020 at 22:06, on Zulip):

Is there a way of rebooting the proc macro server without restarting RA?

Jonas Schievink [he/him] (Dec 07 2020 at 22:06, on Zulip):

Not currently, no

Jonas Schievink [he/him] (Dec 07 2020 at 22:06, on Zulip):

But it prints the panic message to stderr

Jonas Schievink [he/him] (Dec 07 2020 at 22:06, on Zulip):

So it should show up in the output panel

Jeremy Kolb (Dec 08 2020 at 13:34, on Zulip):

I think we could send a window/showMessageRequest to the client and prompt a restart if someone wanted to implement that

Jonas Schievink [he/him] (Dec 08 2020 at 13:38, on Zulip):

Yes, that would be good

Last update: Jul 29 2021 at 09:45UTC