Stream: t-compiler/wg-rls-2.0

Topic: Hardcoding Watt Support

Daniel Mcnab (Nov 16 2019 at 10:14, on Zulip):

Would it be a worthwhile goal to hardcode support for Watt procedural macros. Conceivably, we could match on the 'standard' pattern shown in the readme (with different names), although this could be somewhat brittle.
This would have advantages in terms of allowing some basic proc macro support, without needing the full compiler toolchain setup. Additionally, these proc macros are almost certainly guaranteed to be deterministic, which is good for salsa.

matklad (Nov 16 2019 at 10:16, on Zulip):

I definitely think that some form of Watt is a promising way to handle proc macros. Didn't have a change to thing this through yet

Daniel Mcnab (Nov 16 2019 at 10:21, on Zulip):

I'm just trying to work out what the actual API surface for the tokenstream passed into webassembly looks like

matklad (Nov 16 2019 at 10:22, on Zulip):

@Daniel Mcnab isn't it just ?

Daniel Mcnab (Nov 16 2019 at 10:24, on Zulip):

In theory, but I'm just trying to work out how we would actually construct one to pass into wasm

Daniel Mcnab (Nov 16 2019 at 10:34, on Zulip):

I suppose we could actually entirely reuse Watt's implementation if we could work out how to make a proc_macro::TokenStream.
Ahah - impl From<TokenTree> for TokenStream exists.

Daniel Mcnab (Nov 16 2019 at 10:37, on Zulip):

Although I don't think we already have the conversions to proc_macro::TokenTree, so those would need to be written too.

Edwin Cheng (Nov 16 2019 at 10:54, on Zulip):

Maybe for the starter, convert TokenTree to text and parse that text to TokenStream ?

matklad (Nov 16 2019 at 10:56, on Zulip):

Our tokens have ids, so I don't think that building a proper bridge betwen wasm and host would be difficult -- just a question of consrcuting the required side-tables

matklad (Nov 16 2019 at 10:56, on Zulip):

Or, somewhat equivalent, it's trivial to impl Ser/De for TokenStream

Daniel Mcnab (Nov 16 2019 at 15:20, on Zulip):

So just to confirm - this is something which is at least theoretically desirable?
I'm going to start hacking on it, but may need some help understanding how to integrate it and when to expand them. I think the implementation itself should be fairly easy, although I suppose I also need to work out what crate/it should go in or if it should go in a new crate.
I presume the first step is to ensure proc macros are in the macro namespace, based on collecting proc_macro attributes?
Does anyone know if proc macro derives/attribute macros share the same macro namespace as bang macros?

matklad (Nov 16 2019 at 15:29, on Zulip):

Yeah, running proc-macro2 proc-macros via wasm is probably more promising than external process. The hard bit here is integration with cargo

Daniel Mcnab (Nov 16 2019 at 15:33, on Zulip):

Macros which use Watt are already compiled to wasm, which makes our lives easier. What I'm hacking on at the moment is to only add support for watt macros, as a stopgap to experiment. That is - see how a full proc macro expansion would implement with our existing infra and to try and discover if wasm macros could have any unforeseen issues for us (which I doubt)

Daniel Mcnab (Nov 16 2019 at 15:33, on Zulip):

As a bonus, it might be worthwhile trying to remove syn entirely from our direct dependency graph

Daniel Mcnab (Nov 16 2019 at 15:36, on Zulip):

But that is a separate, loosely related experiment.

Daniel Mcnab (Nov 16 2019 at 15:45, on Zulip):

It might also be worth seeing what David Tolnay (silent mention) thinks about this, as the author of watt.

Daniel Mcnab (Nov 17 2019 at 15:16, on Zulip):

OK - I've been hacking on this but have hit a snag at impl HasSource<Ast=ast::MacroCall> for MacroDef, because I've changed MacroDefId to support procedural macros, which only have an ast::FnDef.

matklad (Nov 17 2019 at 15:17, on Zulip):

Hm, yeah, I guess HasSource doesn't make much sense for MacroDef in a proc-macro world

matklad (Nov 17 2019 at 15:18, on Zulip):

Or, rather, we should rename current MacroDef to DeclarativeMacroDef, implment HasSource for that, and introduce a new enum for MacroDef { proc, decl}

Daniel Mcnab (Nov 17 2019 at 15:18, on Zulip):


Daniel Mcnab (Nov 17 2019 at 15:29, on Zulip):

This also allows us to be more flexible if the declaration for compiler builtin macros changes.

Daniel Mcnab (Nov 17 2019 at 18:03, on Zulip):

I presume that Hygiene.def_crate is None for proc macros?

matklad (Nov 17 2019 at 19:29, on Zulip):

I actually don't know for sure, but I think $crate indeed doesn't have any special meaning for proc macros

Daniel Mcnab (Nov 17 2019 at 19:30, on Zulip):

I suppose it can't make sense, given a proc macro crate can't export anything

Daniel Mcnab (Nov 18 2019 at 15:55, on Zulip):

I'm just trying to implement the basic infrastructure for procedural macros, and I'm not sure what makes sense as a layout.
Clearly, in PerNs we need an AnyMacroDefId, but I'm not sure whether it should be split by procedural vs declarative or FnLike / Attribute/ Derive. In terms of data structure sharing, it makes some sense to split on procedural vs declarative (so they can share the AstId<ast::MacroCall>, but in terms of functionality it makes sense to split by how they are used.

Daniel Mcnab (Nov 18 2019 at 15:57, on Zulip):

This is just for name resolution - I'm not implementing expansion yet.
Also, I'm not certain if we should check if a crate has proc-macro=true or just always check for the proc_macro attributes. I think it makes sense to check the crate type, but I'm not sure

Daniel Mcnab (Nov 18 2019 at 17:03, on Zulip):

Ahah - this is a bit like ImplItem

Daniel Mcnab (Nov 19 2019 at 09:39, on Zulip):

Is it possible to upcast an AstId<Something> to an AstId<An enum which contains Something> without an AstDatabase?

Last update: May 27 2020 at 21:30UTC