In case you have 40 minutes to spare some time, I really enjoyed watching this talk about performance and profiling: https://youtu.be/r-TLSBdHe1A
The contents somewhat apply to our feature too, especially if we want to do https://github.com/rust-lang/measureme/issues/39.
It also makes me want to improve our perf.rlo infrastructure so we can do more iterations. @nikomatsakis mentioned that the Rust project might get access to a sponsored machine featuring 48 cores or something. We definitely could run benchmarks in parallel but I'm not sure if that would skew results (because of turbo boost behavior of CPU and contention on other resources like the hard disk)
FWIW, I think getting more real dedicated hardware - it need not be all that powerful is already a good start, because we can fairly trivially split across machines.
(running the same benchmark always on one machine)
I don't know how helpful more runs would be, but certainly we could then actually make statistical conclusions about our data.