Stream: general

Topic: benchmarking procedural macros


Jake Goulding (Aug 07 2019 at 13:18, on Zulip):

Over in SNAFU, we were blindly micro-optimizing our procedural macro for fun (I think this is the Rust curse). If we really wanted to optimize, i'd want to have some kind of performance test, but I couldn't think how I might do such a thing. Is it possible somehow?

simulacrum (Aug 07 2019 at 13:41, on Zulip):

We don't really provide an easy way to do it, but presumably you could run the compiler in total (cargo check or such) and measure differences in that. It's not great since there will be very high variance but ... well, it's a start

simulacrum (Aug 07 2019 at 13:42, on Zulip):

Maybe can get away with something like -Zexpand-only (I forget the flag) which'll hopefully reduce some of the variance

Jake Goulding (Aug 07 2019 at 13:50, on Zulip):

I guess at some level, it's "just" calling fn my_derive(input: TokenStream) -> TokenStream, so I "just" need to build a TokenStream.

simulacrum (Aug 07 2019 at 14:02, on Zulip):

Right, which is technically possible (rustc has no direct knowledge, in theory, and the interface is ABI stable I think)

simulacrum (Aug 07 2019 at 14:03, on Zulip):

C ABI IIRC

Jake Goulding (Aug 07 2019 at 14:30, on Zulip):

I mean it more basic than that:

extern crate proc_macro;

use proc_macro::TokenStream;

pub fn snafu_derive(input: TokenStream) -> TokenStream {
    unimplemented!("My macro goes here")
}

fn main() {
    let input = r#"struct Foo;"#;
    snafu_derive(input.parse().unwrap());
}
Jake Goulding (Aug 07 2019 at 14:30, on Zulip):

Unfortunately:

thread 'main' panicked at 'procedural macro API is used outside of a procedural macro', src/libproc_macro/bridge/client.rs:315:17
simulacrum (Aug 07 2019 at 14:31, on Zulip):

Right, yeah, you can't create tokenstreams outside of a given proc macro invocation (or indeed pass across proc macro boundaries)

simulacrum (Aug 07 2019 at 14:31, on Zulip):

You could presumably expand recursively and time the innards of the proc_macro function, storing that off into some global though

Jake Goulding (Aug 07 2019 at 14:32, on Zulip):

you can't create tokenstreams outside of a given proc macro invocation

Is there a fundamental reason for this, or is it just "we haven't thought through this yet"?

simulacrum (Aug 07 2019 at 14:35, on Zulip):

Well, it's tied to compiler internals

simulacrum (Aug 07 2019 at 14:35, on Zulip):

And we want to expand in arbitrary order and such

simulacrum (Aug 07 2019 at 14:35, on Zulip):

we basically want the compiler to be in control and not a proc macro crate starting to call the compiler (unexpectedly for the compiler)

simulacrum (Aug 07 2019 at 14:36, on Zulip):

proc macro crates also don't _link_ to the compiler AFAIK, they just know about the C structs that are passed into them

simulacrum (Aug 07 2019 at 14:37, on Zulip):

so they can't actually make a tokenstream if that makes sense (only the compiler knows how)

Jake Goulding (Aug 07 2019 at 14:38, on Zulip):

I guess I could swap to a proc_macro2::TokenStream immediately, and test against that:

extern crate proc_macro;

use proc_macro::TokenStream;

pub fn snafu_derive(input: TokenStream) -> TokenStream {
    snafu_derive_inner(input.into()).into()
}

pub fn snafu_derive_inner(input: proc_macro2::TokenStream) -> proc_macro2::TokenStream {
    unimplemented!("My macro goes here")
}

fn main() {
    let input = r#"struct Foo;"#;
    snafu_derive_inner(input.parse().unwrap());
}
Jake Goulding (Aug 07 2019 at 14:40, on Zulip):

Which might have its own bit of overhead in the conversion, but I think I'm already doing this anyway...

simulacrum (Aug 07 2019 at 14:45, on Zulip):

well, the main function won't work

simulacrum (Aug 07 2019 at 14:46, on Zulip):

you _cannot_ create a tokenstream without the compiler (or another implementor of proc_macro bridge, of which none exist)

simulacrum (Aug 07 2019 at 14:46, on Zulip):

proc_macro2::Tokenstream is just a (somewhat thin) wrapper around proc_macro::Tokenstream

Jake Goulding (Aug 07 2019 at 14:49, on Zulip):

well, the main function won't work

Can you expand on that?

extern crate proc_macro;

use proc_macro::TokenStream;

pub fn snafu_derive(input: TokenStream) -> TokenStream {
    snafu_derive_inner(input.into()).into()
}

pub fn snafu_derive_inner(input: proc_macro2::TokenStream) -> proc_macro2::TokenStream {
    let s = syn::parse2::<syn::ItemStruct>(input).expect("Failed to parse");
    eprintln!("Parsed it: {:?}", s);
    Default::default()
}

fn main() {
    let input = r#"struct Foo;"#;
    snafu_derive_inner(input.parse().unwrap());
}
Parsed it: ItemStruct { attrs: [], vis: Inherited, struct_token: Struct, ident: Ident(Foo), generics: Generics { lt_token: None, params: [], gt_token: None, where_clause: None }, fields: Unit, semi_token: Some(Semi) }

Playground

simulacrum (Aug 07 2019 at 14:50, on Zulip):

Hm, that's surprising to me

simulacrum (Aug 07 2019 at 14:50, on Zulip):

I would expect that to return the 'could not parse' error

simulacrum (Aug 07 2019 at 14:50, on Zulip):

er, could not create w/o proc macro expansion

simulacrum (Aug 07 2019 at 14:51, on Zulip):

Maybe I misunderstood how proc_macro2 works though

Jake Goulding (Aug 07 2019 at 14:51, on Zulip):

I don't believe it's a thin wrapper. I think it basically reparses everything

simulacrum (Aug 07 2019 at 14:51, on Zulip):

Oh interesting

simulacrum (Aug 07 2019 at 14:51, on Zulip):

I did not know that was the case

Jake Goulding (Aug 07 2019 at 14:52, on Zulip):

I haven't looked at the code, so I'm just guessing, mostly.

simulacrum (Aug 07 2019 at 14:52, on Zulip):

Yeah, that's what it looks like based on the API docs

simulacrum (Aug 07 2019 at 14:52, on Zulip):

It makes sense, I guess

Last update: Nov 21 2019 at 23:50UTC