Stream: t-compiler

Topic: perf of rust vs rust


gnzlbg (Jul 03 2019 at 15:53, on Zulip):

@mw could we do a perf run comparing the same commit ? or has this been done somewhere ?

gnzlbg (Jul 03 2019 at 15:57, on Zulip):

@simulacrum ^^

simulacrum (Jul 03 2019 at 15:58, on Zulip):

what is the purpose for that?

gnzlbg (Jul 03 2019 at 15:58, on Zulip):

I don't know what's noise and what's isn't in perf results

gnzlbg (Jul 03 2019 at 15:58, on Zulip):

is a 10% max-rss difference noise ?

simulacrum (Jul 03 2019 at 15:58, on Zulip):

most likely no

gnzlbg (Jul 03 2019 at 15:59, on Zulip):

looking at perf runs of PRs that don't touch anything memory related, it appears that max-rss differences go up till 40%

gnzlbg (Jul 03 2019 at 15:59, on Zulip):

do you believe that ? or do you know ?

simulacrum (Jul 03 2019 at 15:59, on Zulip):

er, yes is what I meatn

gnzlbg (Jul 03 2019 at 15:59, on Zulip):

ah

simulacrum (Jul 03 2019 at 15:59, on Zulip):

Yeah, max-rss is notoriously unreliable

simulacrum (Jul 03 2019 at 15:59, on Zulip):

to my knowledge there's essentially nothing we can do

simulacrum (Jul 03 2019 at 15:59, on Zulip):

but I haven't actually investigated myself

gnzlbg (Jul 03 2019 at 15:59, on Zulip):

I think it would be very useful to run periodically perf runs of master vs master

gnzlbg (Jul 03 2019 at 16:00, on Zulip):

and hint in other perf runs that anything under the levels reported there is probably noise

simulacrum (Jul 03 2019 at 16:00, on Zulip):

we don't really have the capability to do it easily

gnzlbg (Jul 03 2019 at 16:00, on Zulip):

we can do once per month, every to months, and put the link somewhere

simulacrum (Jul 03 2019 at 16:00, on Zulip):

I think what you suggest is probably a good idea, though

gnzlbg (Jul 03 2019 at 16:00, on Zulip):

look at this PR

simulacrum (Jul 03 2019 at 16:00, on Zulip):

we don't really need master v. master either
e.g. just some doc update or such can be enough

gnzlbg (Jul 03 2019 at 16:00, on Zulip):

https://perf.rust-lang.org/compare.html?start=85ed21e83c9f42203cfe06d41eab3f160bd13518&end=090f4a44679dfd5f618706147f631c44ecebe3f5&stat=max-rss

gnzlbg (Jul 03 2019 at 16:01, on Zulip):

it changes the memory allocator used by rustc, from one that's 30kLOC, to one that's 3kLOC

gnzlbg (Jul 03 2019 at 16:01, on Zulip):

look at all perf numbers of all data

gnzlbg (Jul 03 2019 at 16:01, on Zulip):

I'd say that this is measuring noise, which means that the allocator wasn't replaced successfully

simulacrum (Jul 03 2019 at 16:01, on Zulip):

hm, perhaps, perhaps not; it's hard to say

simulacrum (Jul 03 2019 at 16:02, on Zulip):

for allocator benchmarks I wouldn't try to use perf.rlo personally beyond a cursory check

simulacrum (Jul 03 2019 at 16:02, on Zulip):

i.e., I'd try to dig down with something like dhat

gnzlbg (Jul 03 2019 at 16:02, on Zulip):

looking at the PR, they did not override the allocator properly

gnzlbg (Jul 03 2019 at 16:02, on Zulip):

and they didn't check whether they did

gnzlbg (Jul 03 2019 at 16:03, on Zulip):

and worse than wasting a couple of days learning nothing, they learned something wrong, that now is believed to be true

simulacrum (Jul 03 2019 at 16:03, on Zulip):

I guess.. "okay"? Like, that seems plausibly bad but...

simulacrum (Jul 03 2019 at 16:04, on Zulip):

I'm struggling to see what we/I can do :)

gnzlbg (Jul 03 2019 at 16:04, on Zulip):

my point is that not having error bars there is not harmless

gnzlbg (Jul 03 2019 at 16:04, on Zulip):

its worse than not having perf

gnzlbg (Jul 03 2019 at 16:05, on Zulip):

what we can do is manually run a perf run every know and then

gnzlbg (Jul 03 2019 at 16:05, on Zulip):

once per release or something

gnzlbg (Jul 03 2019 at 16:05, on Zulip):

of master vs master

gnzlbg (Jul 03 2019 at 16:05, on Zulip):

the compiler is compiled once, but the run itself is run twice

gnzlbg (Jul 03 2019 at 16:06, on Zulip):

I can send a PR adding a comma to the readme, and maybe we could schedule a perf run on that

simulacrum (Jul 03 2019 at 16:07, on Zulip):

Errors bars would be good; we've wanted them in the past. We technically run many benchmarks 2-3 times so we could try to harvest the error bars from there, or potentially from historical runs

simulacrum (Jul 03 2019 at 16:07, on Zulip):

but it would not be a trivial change to the codebase to even add constant error bars

gnzlbg (Jul 03 2019 at 16:09, on Zulip):

I think we can run perf runs of trivial changes once in a while. And change the message of the bot, when the comparison link is shown, to include a link the last trivial perf run.

gnzlbg (Jul 03 2019 at 16:10, on Zulip):

Saying something like

Do not forget to compare the results of this perf run, to our latest "noise" run: link .
Everything smaller than the differences reported by the noise run is probably noise.

simulacrum (Jul 03 2019 at 16:11, on Zulip):

If you'd like to file a PR against perf that links to e.g. https://perf.rust-lang.org/compare.html?start=9a90d03ad171856dc016c2dcc19292ec49a8a26f&end=fd7f48b3eff67726d848c059574b6aa86675110b&stat=max-rss then that's fine by me.

gnzlbg (Jul 03 2019 at 16:11, on Zulip):

Will do. Thanks

gnzlbg (Jul 03 2019 at 16:18, on Zulip):

@simulacrum to perf or to rustc-timing ?

simulacrum (Jul 03 2019 at 16:18, on Zulip):

rust-lang-nursery/rustc-perf

simulacrum (Jul 03 2019 at 16:18, on Zulip):

rustc-timing is just data

Last update: Nov 22 2019 at 04:30UTC