@mw could we do a perf run comparing the same commit ? or has this been done somewhere ?
what is the purpose for that?
I don't know what's noise and what's isn't in perf results
is a 10% max-rss difference noise ?
most likely no
looking at perf runs of PRs that don't touch anything memory related, it appears that max-rss differences go up till 40%
do you believe that ? or do you know ?
er, yes is what I meatn
Yeah, max-rss is notoriously unreliable
to my knowledge there's essentially nothing we can do
but I haven't actually investigated myself
I think it would be very useful to run periodically perf runs of master vs master
and hint in other perf runs that anything under the levels reported there is probably noise
we don't really have the capability to do it easily
we can do once per month, every to months, and put the link somewhere
I think what you suggest is probably a good idea, though
look at this PR
we don't really need master v. master either
e.g. just some doc update or such can be enough
it changes the memory allocator used by rustc, from one that's 30kLOC, to one that's 3kLOC
look at all perf numbers of all data
I'd say that this is measuring noise, which means that the allocator wasn't replaced successfully
hm, perhaps, perhaps not; it's hard to say
for allocator benchmarks I wouldn't try to use perf.rlo personally beyond a cursory check
i.e., I'd try to dig down with something like dhat
looking at the PR, they did not override the allocator properly
and they didn't check whether they did
and worse than wasting a couple of days learning nothing, they learned something wrong, that now is believed to be true
I guess.. "okay"? Like, that seems plausibly bad but...
I'm struggling to see what we/I can do :)
my point is that not having error bars there is not harmless
its worse than not having perf
what we can do is manually run a perf run every know and then
once per release or something
of master vs master
the compiler is compiled once, but the run itself is run twice
I can send a PR adding a comma to the readme, and maybe we could schedule a perf run on that
Errors bars would be good; we've wanted them in the past. We technically run many benchmarks 2-3 times so we could try to harvest the error bars from there, or potentially from historical runs
but it would not be a trivial change to the codebase to even add constant error bars
I think we can run perf runs of trivial changes once in a while. And change the message of the bot, when the comparison link is shown, to include a link the last trivial perf run.
Saying something like
Do not forget to compare the results of this perf run, to our latest "noise" run: link .
Everything smaller than the differences reported by the noise run is probably noise.
If you'd like to file a PR against perf that links to e.g. https://perf.rust-lang.org/compare.html?start=9a90d03ad171856dc016c2dcc19292ec49a8a26f&end=fd7f48b3eff67726d848c059574b6aa86675110b&stat=max-rss then that's fine by me.
Will do. Thanks
@simulacrum to perf or to rustc-timing ?
rustc-timing is just data