opening this for discussion
it would be great to define exactly what should we check and how often
yeah I'm not sure
its important we have something simple to look at
I think the weekly inspection makes sense, if it can be as quick as it as been
the main issue is what metrics to look at.
(another issue is "how to interpret what we look at" -- in other words, what is a red flag indicating we need to discuss at the meeting)
today's meeting led me to wonder if we should be including code size metrics
rather than just instruction count metrics
I do think nailing down a set that we care about is good
(even if perf doesn't track it)
especially as I look to expand what perf does track
@simulacrum one thing I was commenting to @pnkfelix is that I'd like to coordinate the effort of @WG-prioritization with others, my feeling is that there's more people checking perf
I'd like to know how can we help better
what Felix have said is that it may be good that we check perf to raise awareness in weekly meetings about possible regressions
that seems great but again coordinating with whoever does some work here would be nice to avoid duplication of work I guess :)
I think it would be good to set an action item for compiler team to indicate what would our goals be
@Nicholas Nethercote may have thoughts too
more concretely are you or @Nicholas Nethercote or somebody else periodically checking this? and in that case, what do we think the @WG-prioritization could do, improve and bring into the table?
I am, or at least to some extent.
Not sure about @Nicholas Nethercote -- I think they are, but I've not talked to them about it. I'm not sure that wg-pri needs to
I am, or at least to some extent.
do you think we could help in some way, take this task over or are we fine with you on this?
the main thing the working group should be doing is raising awareness of important issues, like in this case adding performance problems to the agenda of the meeting to ensure that we discuss those
just in case ... what I'm trying to do is avoid duplication of work and have more clear what the responsibilities of the wg should be with this task
all I'm doing is glancing at perf.rust-lang.org/index.html somewhat regularly, I don't think it's "hard" and I'm definitely not catching 100% of cases
I think I have a better sense of what's needed then :)
Literally last week I added a calendar item to check for regressions weekly, on my Tuesday morning (Monday afternoon in North America).
One thing I've always wanted is a way for trusted people to annotate individual commits on the perf.rust-lang.org graphs. E.g. if a regression happens and then is fixed, you could mark the regression commit with an annotation pointing to the fix
hm that sounds like it would not be terribly hard to do
For the regression hunting, this week I looked at instructions and max-rss. Cycles might also be useful, though it correlates very highly with instructions.
not sure how to expose it in the UI, maybe on the compare pages
Binary size would also be interesting, though we don't collect that AFAIK?
I plan to soon, though I've yet to investigate which binaries we can look at
@simulacrum The annotations would be visible on the graphs, ideally
@Nicholas Nethercote -- hm, yeah, so we could color the dots? I wouldn't want to stick the text in the tooltip I think
or a vertical line maybe, not sure what would be better
I haven't thought carefully about the UI :)
if the annotations are per-commit, then vertical lines sound reasonable
they can easily be as fine-grained (or not) as we want, though being consistent would be easiest
For the regression checking, here is a page for Firefox nightly crash checking that may be of interest: https://wiki.mozilla.org/NightlyCrashTriage
I wrote most of that page, I started up the nightly triage, though I haven't done been involved for a while now
Key things: a roster, general instructions on how to do it (useful when someone goes on PTO and someone else takes over, or a new person starts doing it), a place for notes
I'd be happy to write something similar for rustc perf checking, I just need to be told where it would live :)
Forge seems good, or rustc-perf maybe (GitHub wiki or just a file in root)