Stream: t-compiler

Topic: tracking performance regressions


Notification Bot (May 21 2020 at 18:07, on Zulip):

This topic was moved here from #t-compiler/meetings > [weekly meeting] 2020-05-21 #54818 by simulacrum

simulacrum (May 21 2020 at 18:08, on Zulip):

pulled this over here

simulacrum (May 21 2020 at 18:09, on Zulip):

@Josh Triplett One of the problems we have is that "doesn't cause unexpected problems" isn't actually something we can really say too well, unfortunately -- we just don't have a good suite of test cases. For example, #[inline] in panic code can be an optimization if it means it vanishes but definitely hurts compile times for all users of that method (or callers?) as we start to codegen it into downstream crates, which can be quite painful.

Josh Triplett (May 21 2020 at 18:59, on Zulip):

Sure, there are tradeoffs between compilation performance and runtime performance.

Josh Triplett (May 21 2020 at 18:59, on Zulip):

That's the kind of thing I'd hope could be tied to --release.

simulacrum (May 21 2020 at 19:00, on Zulip):

I do think there may be room for --release-but-no-really-i-can-wait

Josh Triplett (May 21 2020 at 19:00, on Zulip):

But in any case, what I'm thinking of by "doesn't cause unexpected problems" would be a rust-timer run.

simulacrum (May 21 2020 at 19:00, on Zulip):

yeah, that's what we'd usually do

Josh Triplett (May 21 2020 at 19:01, on Zulip):

Adding the #[inline] and running rust-timer seems worthwhile.

Josh Triplett (May 21 2020 at 19:01, on Zulip):

We could then evaluate the tradeoff.

simulacrum (May 21 2020 at 19:01, on Zulip):

I do think it makes sense to try a rust-timer run here and see how it goes

simulacrum (May 21 2020 at 19:01, on Zulip):

(presuming there's an inline or whatever that can be added)

Josh Triplett (May 21 2020 at 19:01, on Zulip):

simulacrum said:

I do think there may be room for --release-but-no-really-i-can-wait

Perhaps, yes. Among other things, that could also turn on LTO.

Josh Triplett (May 21 2020 at 19:02, on Zulip):

We'd probably want to start by adding a new optimization level to rustc. But I don't know how much this is on-topic for this particular issue.

simulacrum (May 21 2020 at 19:04, on Zulip):

Yes for this particular case I would agree that trying to fix it seems reasonable

simulacrum (May 21 2020 at 19:04, on Zulip):

but I think if there's not an easy fix (e.g. inline annotations) then there's not much to do

Josh Triplett (May 21 2020 at 19:05, on Zulip):

I do absolutely agree that we can't track every last "why does new rustc generate three more instructions here" regression, and we should focus on the ones that have an impact.

Josh Triplett (May 21 2020 at 19:05, on Zulip):

There's definitely a (fuzzy) line somewhere for what we should track.

Josh Triplett (May 21 2020 at 19:08, on Zulip):

On a different note...I wonder how difficult it would be to have a fully automated regression-bisect bot? For instance, "here's a test case, it's a failure if the compiled output {includes this instruction, doesn't include this instruction, calls this function, has more than X instructions, ...}". @ some github bot, get back a commit hash and a diff of compiled output for the test case.

bjorn3 (May 21 2020 at 19:11, on Zulip):

Once you add support to cargo bisect-rustc, adding support to my bisect bot should just be a matter of parsing the instruction for it and passing it to cargo bisect-rustc.

lqd (May 21 2020 at 19:16, on Zulip):

do we expect a lot of these regressions to be in rustc rather than the one PR updating LLVM to the most recent version ?

simulacrum (May 21 2020 at 19:17, on Zulip):

it's mostly std or llvm, these days, but as we have more mir opts that'll probably change

lqd (May 21 2020 at 19:17, on Zulip):

right

lqd (May 21 2020 at 19:18, on Zulip):

codegen changes in std happen more often than the others, very true

lqd (May 21 2020 at 19:20, on Zulip):

I guess lolbench could also have a role to play in this conversation

lqd (May 21 2020 at 19:25, on Zulip):

(although its regressions had a per-nightly granularity IIRC rather than per-commit; and here the actual code regressing would be known to the bisector, rather than some performance counters stats)

simulacrum (May 21 2020 at 19:31, on Zulip):

lolbench iirc takes forever to run

simulacrum (May 21 2020 at 19:31, on Zulip):

like, days

lqd (May 21 2020 at 19:31, on Zulip):

yeah, part of the reason why it couldn't per-commit iirc

lqd (May 21 2020 at 19:33, on Zulip):

(maybe with enough sponsored hardware + pinning benches to the same host it could be speedier :)

simulacrum (May 21 2020 at 19:36, on Zulip):

perhaps :)

Last update: Jun 04 2020 at 18:20UTC