Stream: t-compiler/wg-prioritization

Topic: perf checks by wg


Santiago Pastorino (May 21 2020 at 14:46, on Zulip):

opening this for discussion

Santiago Pastorino (May 21 2020 at 14:46, on Zulip):

cc @pnkfelix

Santiago Pastorino (May 21 2020 at 14:47, on Zulip):

it would be great to define exactly what should we check and how often

pnkfelix (May 21 2020 at 15:19, on Zulip):

yeah I'm not sure

pnkfelix (May 21 2020 at 15:19, on Zulip):

its important we have something simple to look at

pnkfelix (May 21 2020 at 15:19, on Zulip):

I think the weekly inspection makes sense, if it can be as quick as it as been

pnkfelix (May 21 2020 at 15:20, on Zulip):

the main issue is what metrics to look at.

pnkfelix (May 21 2020 at 15:20, on Zulip):

(another issue is "how to interpret what we look at" -- in other words, what is a red flag indicating we need to discuss at the meeting)

pnkfelix (May 21 2020 at 15:20, on Zulip):

today's meeting led me to wonder if we should be including code size metrics

pnkfelix (May 21 2020 at 15:21, on Zulip):

rather than just instruction count metrics

simulacrum (May 21 2020 at 16:19, on Zulip):

I do think nailing down a set that we care about is good

simulacrum (May 21 2020 at 16:19, on Zulip):

(even if perf doesn't track it)

simulacrum (May 21 2020 at 16:19, on Zulip):

especially as I look to expand what perf does track

Santiago Pastorino (May 21 2020 at 16:45, on Zulip):

@simulacrum one thing I was commenting to @pnkfelix is that I'd like to coordinate the effort of @WG-prioritization with others, my feeling is that there's more people checking perf

Santiago Pastorino (May 21 2020 at 16:45, on Zulip):

I'd like to know how can we help better

Santiago Pastorino (May 21 2020 at 16:45, on Zulip):

what Felix have said is that it may be good that we check perf to raise awareness in weekly meetings about possible regressions

Santiago Pastorino (May 21 2020 at 16:46, on Zulip):

that seems great but again coordinating with whoever does some work here would be nice to avoid duplication of work I guess :)

simulacrum (May 21 2020 at 17:05, on Zulip):

I think it would be good to set an action item for compiler team to indicate what would our goals be

simulacrum (May 21 2020 at 17:05, on Zulip):

@Nicholas Nethercote may have thoughts too

Santiago Pastorino (May 21 2020 at 17:26, on Zulip):

more concretely are you or @Nicholas Nethercote or somebody else periodically checking this? and in that case, what do we think the @WG-prioritization could do, improve and bring into the table?

simulacrum (May 21 2020 at 18:05, on Zulip):

I am, or at least to some extent.

simulacrum (May 21 2020 at 18:05, on Zulip):

Not sure about @Nicholas Nethercote -- I think they are, but I've not talked to them about it. I'm not sure that wg-pri needs to

Santiago Pastorino (May 21 2020 at 18:24, on Zulip):

simulacrum said:

I am, or at least to some extent.

do you think we could help in some way, take this task over or are we fine with you on this?

Santiago Pastorino (May 21 2020 at 18:25, on Zulip):

the main thing the working group should be doing is raising awareness of important issues, like in this case adding performance problems to the agenda of the meeting to ensure that we discuss those

Santiago Pastorino (May 21 2020 at 18:27, on Zulip):

just in case ... what I'm trying to do is avoid duplication of work and have more clear what the responsibilities of the wg should be with this task

simulacrum (May 21 2020 at 18:30, on Zulip):

all I'm doing is glancing at perf.rust-lang.org/index.html somewhat regularly, I don't think it's "hard" and I'm definitely not catching 100% of cases

Santiago Pastorino (May 21 2020 at 18:32, on Zulip):

:+1:

Santiago Pastorino (May 21 2020 at 18:32, on Zulip):

I think I have a better sense of what's needed then :)

Nicholas Nethercote (May 21 2020 at 22:50, on Zulip):

Literally last week I added a calendar item to check for regressions weekly, on my Tuesday morning (Monday afternoon in North America).

Nicholas Nethercote (May 21 2020 at 22:51, on Zulip):

One thing I've always wanted is a way for trusted people to annotate individual commits on the perf.rust-lang.org graphs. E.g. if a regression happens and then is fixed, you could mark the regression commit with an annotation pointing to the fix

simulacrum (May 21 2020 at 22:53, on Zulip):

hm that sounds like it would not be terribly hard to do

Nicholas Nethercote (May 21 2020 at 22:53, on Zulip):

For the regression hunting, this week I looked at instructions and max-rss. Cycles might also be useful, though it correlates very highly with instructions.

simulacrum (May 21 2020 at 22:53, on Zulip):

not sure how to expose it in the UI, maybe on the compare pages

Nicholas Nethercote (May 21 2020 at 22:53, on Zulip):

Binary size would also be interesting, though we don't collect that AFAIK?

simulacrum (May 21 2020 at 22:53, on Zulip):

I plan to soon, though I've yet to investigate which binaries we can look at

Nicholas Nethercote (May 21 2020 at 22:53, on Zulip):

@simulacrum The annotations would be visible on the graphs, ideally

simulacrum (May 21 2020 at 22:54, on Zulip):

@Nicholas Nethercote -- hm, yeah, so we could color the dots? I wouldn't want to stick the text in the tooltip I think

simulacrum (May 21 2020 at 22:54, on Zulip):

or a vertical line maybe, not sure what would be better

Nicholas Nethercote (May 21 2020 at 22:54, on Zulip):

I haven't thought carefully about the UI :)

Nicholas Nethercote (May 21 2020 at 22:55, on Zulip):

if the annotations are per-commit, then vertical lines sound reasonable

simulacrum (May 21 2020 at 22:55, on Zulip):

they can easily be as fine-grained (or not) as we want, though being consistent would be easiest

Nicholas Nethercote (May 21 2020 at 22:56, on Zulip):

For the regression checking, here is a page for Firefox nightly crash checking that may be of interest: https://wiki.mozilla.org/NightlyCrashTriage

Nicholas Nethercote (May 21 2020 at 22:56, on Zulip):

I wrote most of that page, I started up the nightly triage, though I haven't done been involved for a while now

Nicholas Nethercote (May 21 2020 at 22:57, on Zulip):

Key things: a roster, general instructions on how to do it (useful when someone goes on PTO and someone else takes over, or a new person starts doing it), a place for notes

Nicholas Nethercote (May 21 2020 at 22:58, on Zulip):

I'd be happy to write something similar for rustc perf checking, I just need to be told where it would live :)

simulacrum (May 21 2020 at 23:07, on Zulip):

Forge seems good, or rustc-perf maybe (GitHub wiki or just a file in root)

Last update: Jun 05 2020 at 22:50UTC