Stream: t-compiler

Topic: -Z dual-proc-macros


Vadim Petrochenkov (Oct 01 2019 at 23:20, on Zulip):

@Zoxc @Alex Crichton
Could you clarify a couple of details regarding that option?

Vadim Petrochenkov (Oct 01 2019 at 23:22, on Zulip):

Can the compiler rely on metadata in the host and target versions of the proc macro library being the same?
By metadata I mean things like name, edition, etc. (Everything beside the target, basically.)

Vadim Petrochenkov (Oct 01 2019 at 23:25, on Zulip):

In theory, these two crates are loaded using two independent crate searches and you can place crates with entirely different contents to be found by those searches.
On the other hand that's certainly not an intended use.

Vadim Petrochenkov (Oct 01 2019 at 23:29, on Zulip):

Looks like right now this kind of metadata is almost always used from the target version, except when it isn't (?!), e.g. for the crate edition in fn load_proc_macro.

Alex Crichton (Oct 01 2019 at 23:30, on Zulip):

I'm not personally familiar with the rustc implementation of this flag, just the cargo part for myself

Vadim Petrochenkov (Oct 01 2019 at 23:30, on Zulip):

Is this a bug? Should the edition be taken from the target version as well?
Otherwise it looks like the host version is used only for dlsyming.

Alex Crichton (Oct 01 2019 at 23:30, on Zulip):

But in cargo at least and I believe the expected use is that these are compiled exactly the same way except for target

Vadim Petrochenkov (Oct 01 2019 at 23:31, on Zulip):

Yeah, cargo reflects the intended use.

Alex Crichton (Oct 01 2019 at 23:31, on Zulip):

I haven't actually reviewed the rustc implementation at all (or I forget it), so I'll defer to zoxc or when I'm next at a computer tomorrow

Zoxc (Oct 01 2019 at 23:32, on Zulip):

They should be the same except for the target, but I did try to only use stuff from the target libraries for everything but executing code

Vadim Petrochenkov (Oct 01 2019 at 23:34, on Zulip):

only use stuff from the target libraries for everything but executing code

This matches the description in https://github.com/rust-lang/rust/pull/58013#issuecomment-459805962.
Then the edition bit must be a bug.

Vadim Petrochenkov (Oct 01 2019 at 23:35, on Zulip):

Nice, if it's a bug, then it simplifies things.

Zoxc (Oct 01 2019 at 23:41, on Zulip):

Where is the edition related code?

Vadim Petrochenkov (Oct 01 2019 at 23:43, on Zulip):

fn load_proc_macro in decoder.rs, the edition is taken from the host_lib metadata, while everything else is taken from self.root which is the target metadata.

Zoxc (Oct 01 2019 at 23:48, on Zulip):

It kinda make sense since SyntaxExtension is the thing executing the macros. Not sure what it is used for though. Also different editions between the target and host isn't really a supported use case =P

Vadim Petrochenkov (Oct 02 2019 at 08:23, on Zulip):

@Zoxc
One more question: why is -Z dual-proc-macros required only in rustc bootstrap?
The situation "cross compile a library which uses (possibly reexports) a proc macro crate" seems to happen during any cross-compilation?
In this case the host version of the proc macro crate is needed to run code during the build, and the target version is needed to read metadata for reexports etc.

Vadim Petrochenkov (Oct 02 2019 at 08:23, on Zulip):

How is rustc different?

Vadim Petrochenkov (Oct 02 2019 at 08:25, on Zulip):

Also, what happens if we read metadata from the host version? How exactly is it incompatible? (If at least logically the metadata in both host and target versions is supposed to be the same.)

Vadim Petrochenkov (Oct 02 2019 at 08:32, on Zulip):

Also, do the host and target versions have the same CrateDisambiguator or different?
(I'll investigate some of this myself in a couple of days, but leaving questions in the meantime won't hurt, I guess.)

simulacrum (Oct 02 2019 at 11:20, on Zulip):

How is rustc different?

I might be wrong -- and would appreciate @Zoxc confirming -- but I _think_ the difference is that rustc is "unique" in that we have plugins that link against it and want to use the proc macros from its sysroot

simulacrum (Oct 02 2019 at 11:20, on Zulip):

However, if we were to just compile them for the host, then that wouldn't work if the plugin is built for the target

simulacrum (Oct 02 2019 at 11:21, on Zulip):

Though I suspect that might be rather a bug in the compiler (i.e., we should never need to "run" proc macros anywhere but the host) and dual-proc-macros just patch over it

Zoxc (Oct 02 2019 at 17:51, on Zulip):

@Vadim Petrochenkov Normally cargo cross compiles the library only then links that to a proc macro crate for the host. This means that you cannot copy the library over to the target and use it with a Rust compiler running on that target.

simulacrum (Oct 02 2019 at 18:00, on Zulip):

why does the library contain any references to the proc macro?

simulacrum (Oct 02 2019 at 18:01, on Zulip):

I would expect us to compile those out so to speak, i.e., that they only need to be available at build time

Zoxc (Oct 02 2019 at 18:36, on Zulip):

The rlib or dylib metadata will reference the proc macro, and it has to for reexports to work.

Vadim Petrochenkov (Oct 02 2019 at 18:43, on Zulip):

and use it with a Rust compiler running on that target.

Aha, this seems to be the key missing piece - there are actually two compilers running on two different targets!
And they both work with a single library.
I'll try to write some pseudo-code below to understand the exact configuration.

Vadim Petrochenkov (Oct 02 2019 at 18:46, on Zulip):

There are two libraries PM (proc macro) and L (uses the proc macro and reexports it).
There are two targets T1 and T2.

Vadim Petrochenkov (Oct 02 2019 at 18:52, on Zulip):

When cross-compiling L from T1 to T2 we have:
- Compiler built for T1 and running on it.
- Library PM built for T1.
- Library L built for T2, to run proc macros the compiler-T1 dlsyms code from PM-T1, to reexport proc macros the compiler-T1 reads metadata from PM-T1.

Vadim Petrochenkov (Oct 02 2019 at 18:56, on Zulip):

(That's a normal, non-rustc story.)

Vadim Petrochenkov (Oct 02 2019 at 19:02, on Zulip):

Now rustc story when we have a second compiler.
- The second compiler is built for T2 and is running on it.
- We have our previously built library L-T2.
- Using L-T2 - to run proc macros reexported by L-T2 the compiler-T2 dlsymss code from PM-T2, to reexport proc macros reexported by L-T2 further the compiler-T2 reads metadata from PM-T2.

Vadim Petrochenkov (Oct 02 2019 at 19:03, on Zulip):

@Zoxc
Is the description above correct?
If it's correct, then we need to produce PM-T2 at some point, and that explains why we need -Z dual-proc-macros for Cargo.

Vadim Petrochenkov (Oct 02 2019 at 19:05, on Zulip):

It doesn't explain why we need -Z dual-proc-macros for rustc though, in both cases we can dlsym and metadata-read the same library with target matching the compiler's one (aka host).

simulacrum (Oct 02 2019 at 19:35, on Zulip):

So am I right in saying that we wouldn't need this if we never re-exported proc macros? i.e., if there was a direct dep on any proc macro crates. That seems like a reasonable ask (to me); basically saying "we don't ship proc macros in sysroot"

simulacrum (Oct 02 2019 at 19:36, on Zulip):

we'd need some sort of autopublish or w/e business if clippy etc want to directly use rustc_macros for example, but I imagine that would be not too difficult.

Zoxc (Oct 02 2019 at 19:42, on Zulip):

@Vadim Petrochenkov Not quite, the L-T2 library built the "normal" way directly links to the PM-T1 proc macro, so it can't be used with the compiler on T2, since it can't run PM-T1. We need to cross compile L-T2 using -Z dual-proc-macros, where cargo builds both PM-T1 and PM-T2, rustc loads metadata for both; uses PM-T1 to run the macros, but only links the resulting L-T2 to PM-T2. Then we have L-T2 and PM-T2 as if they were built by rustc running on T2.

Vadim Petrochenkov (Oct 02 2019 at 20:00, on Zulip):

the L-T2 library built the "normal" way directly links to the PM-T1 proc macro

Why does it need to link to it?
So far we've been talking only about the compiler dlopening/dlsyming it and reading metadata from it, but not about linking to it.

Zoxc (Oct 02 2019 at 20:01, on Zulip):

When I say link, I mean that the metadata refers to it

Vadim Petrochenkov (Oct 02 2019 at 20:13, on Zulip):

I mean that the metadata refers to it

Hmm, is it somehow hardcoded inside L-T2 through paths or fingerprints?
I thought L-T2 refers to something more flexible like "just something named PM" and while processing L-T2 we can find PM-T1 through crate search if the compiler is T1 and can find PM-T2 thorough crate search if the compiler is T2.

Zoxc (Oct 02 2019 at 20:15, on Zulip):

L-T2 will link to the proc macro with a specific crate disambiguator (and probably target triple too)

Zoxc (Oct 02 2019 at 20:15, on Zulip):

We can have multiple versions of a proc macro in the search path (though -Z dual-proc-macros doesn't support that)

Zoxc (Oct 02 2019 at 20:17, on Zulip):

Since if there's multiple proc macro crates for the host, it won't know which ones corresponds to the proc macro crate for the target

Vadim Petrochenkov (Oct 02 2019 at 20:24, on Zulip):

Ok, I think I understand it now.
What a mess -_-

Vadim Petrochenkov (Oct 02 2019 at 20:28, on Zulip):

I think it would be interesting to find a way to refer to "flexible PM" from L-T2 instead of hardcoding, so it could resolve to different crates (PM-T1 or PM-T2) when L-T2 is processed by different compilers (T1 or T2 respectively).
(In that case we wouldn't need -Z dual-proc-macros at least on the rustc side.)

Vadim Petrochenkov (Oct 02 2019 at 20:43, on Zulip):

@simulacrum

So am I right in saying that we wouldn't need this if we never re-exported proc macros?

Looks like that, except the compiler should known about it somehow (e.g. we can use rustc_macros only through #[no_link] extern crate rustc_macros or something).

simulacrum (Oct 02 2019 at 20:44, on Zulip):

hm, I guess we have e.g. serde = { version = "1", features = ["derive"] } though in deps which might make this not possible

simulacrum (Oct 02 2019 at 20:45, on Zulip):

IIRC we have a special mode in rustc_metadata for "just macro uses" which maybe isn't fully applicable to proc macros, but might be relevant here?

Vadim Petrochenkov (Oct 02 2019 at 20:45, on Zulip):

Ah, of course, rustc_macros is not the only proc macro thing we need.

simulacrum (Oct 02 2019 at 20:46, on Zulip):

though -- I might be wrong here -- but is it true that regular crates build just fine? e.g. if clippy et al used rustc_macros by path and rebuilt it then we'd work just fine

simulacrum (Oct 02 2019 at 20:46, on Zulip):

(and same with serde, etc.)

Vadim Petrochenkov (Oct 02 2019 at 20:46, on Zulip):

IIRC we have a special mode in rustc_metadata for "just macro uses"

Yes, that's exactly what I've been talking about - DepKind::UnexportedMacrosOnly, which is currently expressible only as #[no_link] extern crate in the language.

simulacrum (Oct 02 2019 at 20:46, on Zulip):

ah okay

simulacrum (Oct 02 2019 at 20:46, on Zulip):

that is, the problem only arises if you re-export a proc macro from a precompiled dependency (i.e., in the sysroot)

simulacrum (Oct 02 2019 at 20:47, on Zulip):

if you re-export a proc macro normally then things work fine

Zoxc (Oct 02 2019 at 20:50, on Zulip):

Using webassembly for proc macros would also avoid this =P

Last update: Nov 20 2019 at 02:35UTC