Stream: t-compiler/rust-analyzer

Topic: Generate mbe benchmark tests


Edwin Cheng (Feb 06 2021 at 20:47, on Zulip):

It is quite hard to generate mbe benchmark tests which is self contain: it means we cannot use any name resolution facility. I can find all ast::macro_rules and ast::macro_call inside RA project, but that's not that much (around 2k calls).
The total macro invocations in rust-analyzer analysis-stats . is around 60k which include std and all dependencies, and recursive calls.

If we want to benchmark the real world sitution, maybe only 2k calls are not enough IMO. On the other hand, that 2k call data size is around 100kb, we maybe can just keep it inside git, right?

I feel stuck in both cases, @matklad what do you think about it ?

Laurențiu (Feb 07 2021 at 10:08, on Zulip):

If we want to benchmark the real world sitution, maybe only 2k calls are not enough IMO. On the other hand, that 2k call data size is around 100kb, we maybe can just keep it inside git, right?

How long does it take to run for those?

Laurențiu (Feb 07 2021 at 10:09, on Zulip):

Worst case, we can store the fixture in another repo, ignore the test, and clone it on-demand (on CI?). But I'm not sure it's worth the hassle?

Edwin Cheng (Feb 07 2021 at 11:48, on Zulip):

How long does it take to run for those?

For 60k one, it is around 15s in my low spec machine. I haven't testetd the 2k one yet, but I guess it is proportional to the 60k one.

Edwin Cheng (Feb 07 2021 at 11:56, on Zulip):

Worst case, we can store the fixture in another repo, ignore the test, and clone it on-demand (on CI?). But I'm not sure it's worth the hassle?

Another workaround is adding new argument flags (e.g. dump-macro-rules, dump-macro-call, etc) in rust-analyzer cli and generate these test on CI. But I don't know whether it's worth or not.

Edwin Cheng (Feb 07 2021 at 11:57, on Zulip):

But I can imagine we can reuse these infrastructure to benchmark different part of RA , right ?

Edwin Cheng (Feb 09 2021 at 09:50, on Zulip):

Okay, I think I have an idea : We could just use a pre-generated macro-rules (which is not that big, around 640kb) and generated macro invocations based on these rules. I will try to implement it this week to see how it is.

matklad (Feb 09 2021 at 14:07, on Zulip):

I don't think we necessary should target "real world". I think it's better to have generated synthetic benchmarks. That way, we can easily change the generator to exercise particularly problematic spots

matklad (Feb 09 2021 at 14:08, on Zulip):

Let me maybe do this for accidentally quadratic benchmark, just to get the infra in place

Last update: Jul 26 2021 at 12:00UTC