Stream: general

Topic: ranting about performance number terminology


Jake Goulding (Oct 14 2018 at 22:23, on Zulip):

pop quiz! What does this mean in "real numbers"?

but a x0.82 regression

simulacrum (Oct 14 2018 at 22:26, on Zulip):

I presume 18% slower (e.g., from 1 second to 1.18 seconds) but not sure

simulacrum (Oct 14 2018 at 22:26, on Zulip):

Or perhaps a 82% slowdown?

Jake Goulding (Oct 14 2018 at 23:22, on Zulip):

Context

Jake Goulding (Oct 14 2018 at 23:22, on Zulip):

It was also phrased as

~ 15% slowdown

simulacrum (Oct 14 2018 at 23:23, on Zulip):

Okay, now that's very interesting. The numbers just don't match up, or at least not that I can tell...

Jake Goulding (Oct 14 2018 at 23:24, on Zulip):

I assume that it comes from https://internals.rust-lang.org/t/help-us-benchmark-saturating-float-casts/6231/14

Jake Goulding (Oct 14 2018 at 23:24, on Zulip):
name        base ns/iter  saturate ns/iter  diff ns/iter  diff %  speedup
bench_rgb   34,507,766    42,325,700           7,817,934  22.66%   x 0.82
Jake Goulding (Oct 14 2018 at 23:25, on Zulip):

And now I'm angry about "speedup" because it seems like any positive "speedup" should increase the speed.

rkruppe (Oct 14 2018 at 23:35, on Zulip):

yeah "speedup" for a measurement where something got slower is ungreat (that's why I wrote "regression" instead) but it's established benchmarking convention to give a ratio where 1.0x is no change and >1.0x is faster. anything with percentages is either misleading or an unconventional presentation of the same ratio

Jake Goulding (Oct 15 2018 at 01:51, on Zulip):

@rkruppe yeah, I'm a big fan of the ratio, but I don't even attempt to use short words. I've never found any that are unambiguous. I always just say something like "is X of the previous time"

Jake Goulding (Oct 15 2018 at 01:52, on Zulip):

"with saturation enabled, the benchmark takes 1.23x of the previous time"

Jake Goulding (Oct 15 2018 at 01:53, on Zulip):

or even more verbose "...the benchmark is slower, taking 1.23..."

RalfJ (Oct 15 2018 at 07:18, on Zulip):

@rkruppe yeah "speedup" didnt work out. I read that and had literally no idea what it meant

Last update: Nov 20 2019 at 13:25UTC