Stream: t-compiler/wg-polonius

Topic: meeting 2019.11.18


nikomatsakis (Nov 18 2019 at 15:00, on Zulip):

Hey @lqd :wave: -- you around?

lqd (Nov 18 2019 at 15:01, on Zulip):

:wave:

lqd (Nov 18 2019 at 15:01, on Zulip):

how is it going ? :) did you want to do the meeting over zoom or something similar ?

nikomatsakis (Nov 18 2019 at 15:03, on Zulip):

wrapping up something but maybe we start with a zoom call?

lqd (Nov 18 2019 at 15:03, on Zulip):

:thumbs_up:

nikomatsakis (Nov 18 2019 at 15:05, on Zulip):

@lqd (and whomever else is lurking ;) https://mozilla.zoom.us/j/332990221

lqd (Nov 18 2019 at 15:06, on Zulip):

omw

nikomatsakis (Nov 18 2019 at 15:08, on Zulip):

@lqd I heard you

nikomatsakis (Nov 18 2019 at 15:08, on Zulip):

but only a bit :)

nikomatsakis (Nov 18 2019 at 15:09, on Zulip):

Do you hear me? :)

nikomatsakis (Nov 18 2019 at 15:09, on Zulip):

Seems like maybe no

lqd (Nov 18 2019 at 15:09, on Zulip):

not any more, zoom is behaving weirdly rn :/

nikomatsakis (Nov 18 2019 at 15:09, on Zulip):

Maybe we just try zulip then :)

nikomatsakis (Nov 18 2019 at 15:09, on Zulip):

I'm looking at https://github.com/rust-lang/polonius/pull/137

nikomatsakis (Nov 18 2019 at 15:10, on Zulip):

actually why don't we create a better named topic

lqd (Nov 18 2019 at 15:10, on Zulip):

(I think I can try again with the zoom if that's better for you)

Albin Stjerna (Nov 18 2019 at 16:57, on Zulip):

Is there anything really important for me in the gigantonormous placeholder loans thread?

Albin Stjerna (Nov 18 2019 at 16:58, on Zulip):

I've applied for vacation tomorrow by the way :)

nikomatsakis (Nov 18 2019 at 16:58, on Zulip):

@Albin Stjerna not really, but we've created a hackmd that contains the "canonical rules"

nikomatsakis (Nov 18 2019 at 16:58, on Zulip):

it is extended with placeholder loans

Albin Stjerna (Nov 18 2019 at 16:58, on Zulip):

Excellent, thanks!

nikomatsakis (Nov 18 2019 at 16:58, on Zulip):

it is... mostly using the new names :)

nikomatsakis (Nov 18 2019 at 16:58, on Zulip):

I would like to clean it up just a bit more

nikomatsakis (Nov 18 2019 at 16:58, on Zulip):

and I'd like to integrate the move/init stuff into that

Albin Stjerna (Nov 18 2019 at 17:00, on Zulip):

My plan is to spend tomorrow working on move analysis, including doing the same thing for that, and then save my 8 overtime hours for point-wise work during the week

Albin Stjerna (Nov 18 2019 at 17:01, on Zulip):

"the same" means "updating the rules to use the right names and document them in the hackmd"

nikomatsakis (Nov 18 2019 at 17:08, on Zulip):

I've applied for vacation tomorrow by the way :)

geez y'all you humble me btw

lqd (Nov 18 2019 at 17:12, on Zulip):

says the person waking up at 5AM to work on things they otherwise wouldn't have time to do so

nikomatsakis (Nov 18 2019 at 17:12, on Zulip):

I'm super excited to see polonius get unblocked though

nikomatsakis (Nov 18 2019 at 17:12, on Zulip):

every time we take enough time for me to get it back in my head, it's so fun to play with :)

nikomatsakis (Nov 18 2019 at 17:13, on Zulip):

the rest of the time I'm just like "aaaah I can't remember enough"

nikomatsakis (Nov 18 2019 at 17:13, on Zulip):

:P

lqd (Nov 18 2019 at 17:13, on Zulip):

:D

lqd (Nov 18 2019 at 17:14, on Zulip):

(hopefully we can get it to a point where we also have good data about usage, perf, etc about datafrog, for possible usage in rustc's dataflow, or moving some of this from rustc to polonius)

Albin Stjerna (Nov 18 2019 at 17:20, on Zulip):

Right, on the off chance that I have more time to spare tomorrow, is there anywhere I can read about how to generate profiling events using measureme? I've tried googling, but there isn't even a "use this to get started" example anywhere? Can I talk to someone on here?

lqd (Nov 18 2019 at 17:20, on Zulip):

yeah @Wesley Wiser is the person to talk to

Wesley Wiser (Nov 18 2019 at 17:20, on Zulip):

/me :wave:

lqd (Nov 18 2019 at 17:20, on Zulip):

(or mw in an european TZ)

Albin Stjerna (Nov 18 2019 at 17:21, on Zulip):

I'll engage whomever is awake lazily :)

Albin Stjerna (Nov 18 2019 at 17:22, on Zulip):

No need to take up anyone's attention if I don't have time to consume it

lqd (Nov 18 2019 at 17:24, on Zulip):

as rustc is already using measureme, there are a lot of usage examples there

lqd (Nov 18 2019 at 17:25, on Zulip):

and the API seemed easy to use, but I do wonder if you wanted to measure only polonius, or integrate polonius in rustc's measurements

lqd (Nov 18 2019 at 17:26, on Zulip):

both are useful, but I'm not exactly sure how/whether we can do both at the same time ?

nikomatsakis (Nov 18 2019 at 17:29, on Zulip):

ps integrating with measureme seems awesome

nikomatsakis (Nov 18 2019 at 17:29, on Zulip):

and the API seemed easy to use, but I do wonder if you wanted to measure only polonius, or integrate polonius in rustc's measurements

yes, I wonder about this too! it'd be cool if both of them "just worked", seems plausible

lqd (Nov 18 2019 at 17:30, on Zulip):

I'll try to describe the idea here and whenever Wesley has the time to give us a bit of guidance, it'll be much appreciated :)

lqd (Nov 18 2019 at 17:47, on Zulip):

The context is: polonius is in a separate library that rustc calls whenever some flags are passed, it sets up a big database, and calls a polonius function on it. Our goal would be to measure a few things (the first 2 seem straightforward usage of measureme inside rustc):
- the time it takes rustc to set up this data
- the time polonius takes globally to do its job over this data
- inside polonius itself, the computation is made of multiple steps, and we'd like to measure those individually as well.

we wouldn't necessarily need the measurements of polonius' steps inside the -Z self-profile but if that worked it would be lovely :) it might require to pass the sess profiler to polonius I'm not sure ? maybe such a measureme/self-profile integration has already been done in a similar external-library context ?

otherwise, measuring only rustc should IIUC only be a couple of start/end activity calls to the profiler.

if we can't/won't have the polonius' steps in the rustc profile, we'd just need a bit of guidance on how to set-up the profiler and push events to it, but it seems like it'd just be creating a new profiler passing it the folder it should output the 3 files into, and recording events to it

Wesley Wiser (Nov 18 2019 at 17:58, on Zulip):

Yes, the first two things are pretty straightforward. Usually, the easiest thing to do is to use the RAII help methods on SelfProfilerRef. You can probably do something like let _guard = tcx.sess.prof.generic_activity("building polonius db");. That will record the time between when you call generic_activity() and when the guard is dropped.

Wesley Wiser (Nov 18 2019 at 18:00, on Zulip):

For the third thing, we haven't yet tried integrating measurme with a rustc dependency like that. If you just want to be able to have stuff show up in the self profile data dump, then I'd recomend doing what @lqd mentioned and pass in a &measurme::Profiler reference to the entry point for polonius.

lqd (Nov 18 2019 at 18:02, on Zulip):

just to check, we can call generic_activity multiple times with the same id and it'll add up the different times right ? (as the whole "building db" activity is distributed all around the NLL constraint gathering)

Wesley Wiser (Nov 18 2019 at 18:02, on Zulip):

There's some basic docs with info on using measureme directly https://docs.rs/measureme/0.4.0/measureme/#writing-event-trace-files

Wesley Wiser (Nov 18 2019 at 18:04, on Zulip):

just to check, we can call generic_activity multiple times with the same id and it'll add up the different times right ? (as the whole "building db" activity is distributed all around the NLL constraint gathering)

If you call it with the same id, it's considered to be the same thing. Different tools will do different things with that data. For example, the summarization stuff will group it all together so if you want to see individual things from that, I'd recommend formatting some kind of useful id into the id arg like a DefId or something.

Wesley Wiser (Nov 18 2019 at 18:04, on Zulip):

The graphical tools like flamegraph and crox don't collapse events like that so you'll see them individually

lqd (Nov 18 2019 at 18:05, on Zulip):

thanks a lot @Wesley Wiser :)

Wesley Wiser (Nov 18 2019 at 18:05, on Zulip):

Feel free to hop into #t-compiler/wg-self-profile with any questions or ping me directly here!

Wesley Wiser (Nov 18 2019 at 18:06, on Zulip):

I'm also happy to review any PRs or whatever

lqd (Nov 18 2019 at 18:06, on Zulip):

:tada:

Last update: Dec 12 2019 at 00:50UTC