Stream: t-compiler/performance

Topic: coercions-debug noise


simulacrum (May 18 2021 at 15:45, on Zulip):

image.png

simulacrum (May 18 2021 at 15:45, on Zulip):

image.png

simulacrum (May 18 2021 at 15:46, on Zulip):

I'll actually kick some off now

pnkfelix (May 18 2021 at 15:46, on Zulip):

wait what are you kicking off?

simulacrum (May 18 2021 at 15:47, on Zulip):

coercions-debug under cachegrind, I'll run it twice with the same compiler and diff the two; my expectation is I'll see some noise there. If not, a compiler rebuild with slight adjustments and the same thing.

The 8472 (May 18 2021 at 15:52, on Zulip):

well, that's likely hard, or at least, 'bouncing around' is generally an unknown to me

Maybe just measuring the recent variance or some similar statistical measure (nth percentile of step sizes, since it's a discrete series) and showing that in comparisons would help?

simulacrum (May 18 2021 at 15:53, on Zulip):

Maybe. I don't quite know how easy those are to implement pull off, or whether they lie on firm grounding.

simulacrum (May 18 2021 at 15:55, on Zulip):

Definitely not opposed to discussing on an issue or seeing an implementation; I'd love to have some more principled change detection. I've been looking into it in the little free time I have over the past year and hadn't come up with anything that seemed very reliable.

simulacrum (May 18 2021 at 15:59, on Zulip):

Hm, looks like most of it is due to LLVM, but I'll need to dig in more to gain a better understanding of what's going on that causes such large disparities.

lqd (May 18 2021 at 16:15, on Zulip):

I remember some benchmarking work from Laurence Tratt on virtual machines (pypy, hotspot, etc) a few years back where they used change-point analysis with great success for these kinds of issues. Since the softdev team now works on the yorick meta-tracer in rust, we could probably ask them, e.g. Edd Barrett is a regular here :)

(the 2 articles/papers I remember were: https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_1.html and https://tratt.net/laurie/blog/entries/why_arent_more_users_more_happy_with_our_vms_part_2.html)

pnkfelix (May 18 2021 at 16:15, on Zulip):

simulacrum said:

Definitely not opposed to discussing on an issue or seeing an implementation; I'd love to have some more principled change detection. I've been looking into it in the little free time I have over the past year and hadn't come up with anything that seemed very reliable.

The last paper I remember reading on this subject was Rigorous Benchmarking in Reasonable Time by Kalibera and Jones.

Last update: Jun 20 2021 at 01:00UTC