Stream: t-compiler/wg-polonius

Topic: meeting 2019.04.23


lqd (Apr 23 2019 at 19:01, on Zulip):

:wave:

Albin Stjerna (Apr 23 2019 at 19:02, on Zulip):

Hi @WG-polonius!

lokalmatador (Apr 23 2019 at 19:02, on Zulip):

hi

lqd (Apr 23 2019 at 19:03, on Zulip):

PSA: I'll be taking notes for the meeting, if anything seems important, you can mark it with :point_up: and I'll be sure to mention it :)

lqd (Apr 23 2019 at 19:04, on Zulip):

@lokalmatador welcome btw

Albin Stjerna (Apr 23 2019 at 19:04, on Zulip):

Oh, excellent!

lokalmatador (Apr 23 2019 at 19:04, on Zulip):

being the new guy here, I'll briefly introduce myself: 35 year old former postdoc looking for a new challenge in his newly gained spare time ;)

lqd (Apr 23 2019 at 19:04, on Zulip):

as I wasn't there last week I'm not sure about the agenda, I assume these couple things:

nikomatsakis (Apr 23 2019 at 19:05, on Zulip):

well, we have our usual roadmap document in general

lqd (Apr 23 2019 at 19:05, on Zulip):

1) albin's sync, liveness work, and rustc
2) mine
3) saying hi to @lokalmatador
4) the plan till next week ?

nikomatsakis (Apr 23 2019 at 19:05, on Zulip):

sounds about right :)

lqd (Apr 23 2019 at 19:06, on Zulip):

@Albin Stjerna how long is your masters for btw ? is it ending soon ?

Albin Stjerna (Apr 23 2019 at 19:07, on Zulip):

@lqd The crazy answer is "as long as it takes", the long answer "ends in July, roughly"

Albin Stjerna (Apr 23 2019 at 19:07, on Zulip):

That's what I wrote in my project plan anyway

lqd (Apr 23 2019 at 19:07, on Zulip):

do you also feel you need more help with the rustc facts or were niko's notes enough to unblock you ?

Albin Stjerna (Apr 23 2019 at 19:08, on Zulip):

Niko's notes are enough to unblock me! I started working with them today and I already have something that compiles with the new Polonius Output

lqd (Apr 23 2019 at 19:08, on Zulip):

nice !

nikomatsakis (Apr 23 2019 at 19:08, on Zulip):

Cool

lqd (Apr 23 2019 at 19:09, on Zulip):

so the 1st item in the roadmap document is well underway

Albin Stjerna (Apr 23 2019 at 19:09, on Zulip):

I had to implement Atom for mir::Local, which drags Polonius into the mirmodule/crate as well, is that reasonable?

nikomatsakis (Apr 23 2019 at 19:09, on Zulip):

Yep, it seems like it'd be good for @Albin Stjerna and I to lay out a few more steps, but maybe not at this moment in this meeting .

nikomatsakis (Apr 23 2019 at 19:09, on Zulip):

I had to implement Atom for mir::Local

seems expected

Albin Stjerna (Apr 23 2019 at 19:09, on Zulip):

Nah, I have things to do at least for the next couple of days.

lqd (Apr 23 2019 at 19:10, on Zulip):

is the polonius PR ready to go or do we need to wait for something ?

Albin Stjerna (Apr 23 2019 at 19:10, on Zulip):

(that's a good thing with the Easter holiday: I didn't work)

Albin Stjerna (Apr 23 2019 at 19:11, on Zulip):

is the polonius PR ready to go or do we need to wait for something ?

I think it's ok, I addressed @Jake Goulding's comments. It is however a breaking change and doesn't work with rustc as-is, and it's untested for larger inputs

lqd (Apr 23 2019 at 19:12, on Zulip):

I was wondering if liveness was to be incorporated into an/all existing analyses or something a bit apart, so it could maybe be reused in the tests (the programs thingy)

Albin Stjerna (Apr 23 2019 at 19:12, on Zulip):

Or anything non-trivial really, as we don't have fact generation yet

Albin Stjerna (Apr 23 2019 at 19:13, on Zulip):

(deleted)

Albin Stjerna (Apr 23 2019 at 19:13, on Zulip):

I was wondering if liveness was to be incorporated into an/all existing analyses or something a bit apart, so it could maybe be reused in the tests (the programs thingy)

I guess so?

lqd (Apr 23 2019 at 19:13, on Zulip):

the PR is still marked as WIP ?

nikomatsakis (Apr 23 2019 at 19:14, on Zulip):

I think it's ok, I addressed Jake Goulding's comments. It is however a breaking change and doesn't work with rustc as-is, and it's untested for larger inputs

wait, what is a breaking change?

nikomatsakis (Apr 23 2019 at 19:14, on Zulip):

maybe i'm a bit confused, what PR are we talking about exactly?

lqd (Apr 23 2019 at 19:14, on Zulip):

https://github.com/rust-lang/polonius/pull/105 ?

nikomatsakis (Apr 23 2019 at 19:14, on Zulip):

Oh, I see, ok.

Albin Stjerna (Apr 23 2019 at 19:14, on Zulip):

That's what I assumed too

lqd (Apr 23 2019 at 19:14, on Zulip):

what is the breaking change yeah ?

nikomatsakis (Apr 23 2019 at 19:15, on Zulip):

OK, being a breaking change doesn't seem particularly significant :)

nikomatsakis (Apr 23 2019 at 19:15, on Zulip):

I don't think that polonius has much of a stability promise at this point :)

Albin Stjerna (Apr 23 2019 at 19:15, on Zulip):

So it's breaking because it changes Output to take more type parameters

lqd (Apr 23 2019 at 19:15, on Zulip):

it's no big deal to do such breaking changes for polonius ;)

Albin Stjerna (Apr 23 2019 at 19:15, on Zulip):

So literally anything using it has to be updated

lqd (Apr 23 2019 at 19:15, on Zulip):

we're arguably the only ones using it :)

nikomatsakis (Apr 23 2019 at 19:15, on Zulip):

god I hope we are

nikomatsakis (Apr 23 2019 at 19:15, on Zulip):

in any case, that's what semver's for, it's fine

lqd (Apr 23 2019 at 19:15, on Zulip):

agreed :)

Albin Stjerna (Apr 23 2019 at 19:16, on Zulip):

Ok, good, that just means we can keep the old version in rustc until I'm done with my work adding liveness fact generation

nikomatsakis (Apr 23 2019 at 19:16, on Zulip):

right; in any case we don't have to publish right away

Albin Stjerna (Apr 23 2019 at 19:16, on Zulip):

I just wanted to verify this

lqd (Apr 23 2019 at 19:16, on Zulip):

merging the PR and releasing it will be easier to use in rustc than a local build

Albin Stjerna (Apr 23 2019 at 19:16, on Zulip):

Then I...think it's not WIP anymore

nikomatsakis (Apr 23 2019 at 19:17, on Zulip):

merging the PR and releasing it will be easier to use in rustc than a local build

OK, I'm fine with landing it and doing a version bump

Albin Stjerna (Apr 23 2019 at 19:17, on Zulip):

There, I dropped it

nikomatsakis (Apr 23 2019 at 19:17, on Zulip):

So shall we move on in our agenda? :)

lqd (Apr 23 2019 at 19:17, on Zulip):

(we can fiddle with the liveness computation if we need it elsewhere)

Albin Stjerna (Apr 23 2019 at 19:17, on Zulip):

Sure

lqd (Apr 23 2019 at 19:17, on Zulip):

yeah, so

lqd (Apr 23 2019 at 19:18, on Zulip):

apart from the tiny PR I did this week I worked on something that is probably not needed

nikomatsakis (Apr 23 2019 at 19:18, on Zulip):

(we can fiddle with the liveness computation if we need it elsewhere)

(the compiler does a few liveness analyses, but so far they're each a bit different)

lqd (Apr 23 2019 at 19:18, on Zulip):

that is using the location insensitive analyses results in datafrogopt

Albin Stjerna (Apr 23 2019 at 19:19, on Zulip):

that is using the location insensitive analyses results in datafrogopt

Ooo, and why do you think this isn't needed? Also, has this landed? Because I have microbenchmarks I'd love to run on that

lqd (Apr 23 2019 at 19:19, on Zulip):

the couple things I did were easy so I think I have it working, rustc's tests seem to work equally bad

lqd (Apr 23 2019 at 19:19, on Zulip):

it's a bit putting the cart before the horse, until we have everything correct, it maybe unwise to do this kind of work

lqd (Apr 23 2019 at 19:20, on Zulip):

but yeah, running it in your bench suite would be nice

Albin Stjerna (Apr 23 2019 at 19:20, on Zulip):

This is...a different suite :)

Albin Stjerna (Apr 23 2019 at 19:20, on Zulip):

Depending on which one you are referring to.

lqd (Apr 23 2019 at 19:20, on Zulip):

albeit these couple simple things shouldn't matter much I think

nikomatsakis (Apr 23 2019 at 19:20, on Zulip):

the couple things I did were easy so I think I have it working, rustc's tests seem to work equally bad

At some point I had gone through all the regressions and categorized them (and determined they were all acceptable). I wonder if it's worth the trouble to try and make the test run clean (possibly by adding "polonius" versions of the expected output.

lqd (Apr 23 2019 at 19:20, on Zulip):

could be

nikomatsakis (Apr 23 2019 at 19:20, on Zulip):

That said, I think we should spend a bit of time talking to @lokalmatador about where they might fit in :)

nikomatsakis (Apr 23 2019 at 19:21, on Zulip):

(I have to run in a few minutes)

nikomatsakis (Apr 23 2019 at 19:21, on Zulip):

and trying maybe to give them a bit more overall context

lqd (Apr 23 2019 at 19:21, on Zulip):

(I don't remember how to run the tests properly so I used the compare mode polonius, and had 200+ failures)

nikomatsakis (Apr 23 2019 at 19:21, on Zulip):

Hmm, that's what I was doing.

lqd (Apr 23 2019 at 19:22, on Zulip):

I'm not sure we have a lot of work even for us ? that being said maybe the move/overwrite analysis could be interesting for @lokalmatador ?

lokalmatador (Apr 23 2019 at 19:24, on Zulip):

sorry, small network issue

lqd (Apr 23 2019 at 19:24, on Zulip):

@nikomatsakis btw I had a couple questions for the illegal subset relations PR a while back here

lokalmatador (Apr 23 2019 at 19:25, on Zulip):

hm, if someone could enlighten a little more on move/overwrite analysis?

Albin Stjerna (Apr 23 2019 at 19:25, on Zulip):

Anyway, if anyone's interested in my micro-benchmarking setup using Criterion, I have a branch on my fork

lokalmatador (Apr 23 2019 at 19:26, on Zulip):

right now i'm trying to get a better idea and deeper understanding of the whole thing

lqd (Apr 23 2019 at 19:26, on Zulip):

understandable :)

lqd (Apr 23 2019 at 19:26, on Zulip):

so the polonius effort is a bit two fold

lokalmatador (Apr 23 2019 at 19:26, on Zulip):

so i do not know when I can start contributing in reasonable way
but with a little help :)

lqd (Apr 23 2019 at 19:27, on Zulip):

adding new features to NLL, and moving parts of existing computations inside the polonius crate

Albin Stjerna (Apr 23 2019 at 19:27, on Zulip):

right now i'm trying to get a better idea and deeper understanding of the whole thing

Yes, I've been here...a month now, and I'm not sure about this! I spent the weekend (during which I absolutely didn't work) reading the NLL RFC and that helped me a lot

lokalmatador (Apr 23 2019 at 19:27, on Zulip):

or better hints and pointers...

lokalmatador (Apr 23 2019 at 19:27, on Zulip):

I see, let me open that one up

nikomatsakis (Apr 23 2019 at 19:27, on Zulip):

I think that starting by trying to encode the move errors is not a good idea :)

lqd (Apr 23 2019 at 19:28, on Zulip):

dang

lokalmatador (Apr 23 2019 at 19:28, on Zulip):

yeah probably

nikomatsakis (Apr 23 2019 at 19:28, on Zulip):

One thought I had was trying to continue the work that @Albin Stjerna was doing towards an extended perf suite

lqd (Apr 23 2019 at 19:28, on Zulip):

ooooh interesting

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

specifically, @Albin Stjerna gathered various bits of data,

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

I was thinking it would be useful to try and gather some statistics about the various benchmark programs into a spreadsheet

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

to give a bit of context, @lokalmatador -- we had a standard benchmark suite for the rust compiler

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

measuring compilation time

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

we did some measurements of what happens when we enable polonius

lokalmatador (Apr 23 2019 at 19:29, on Zulip):

yep

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

this is expected to be worse. Among other things, polonius is currently running in addition to the standard borrow check

nikomatsakis (Apr 23 2019 at 19:29, on Zulip):

so it could hardly be faster :)

nikomatsakis (Apr 23 2019 at 19:30, on Zulip):

but also it's not been optimized etc etc

nikomatsakis (Apr 23 2019 at 19:30, on Zulip):

anyway, it would be nice to extend the polonius repository with a selected set of tests

nikomatsakis (Apr 23 2019 at 19:30, on Zulip):

that we believe are "representative" somehow

nikomatsakis (Apr 23 2019 at 19:30, on Zulip):

this way we could benchmark polonius just against those tests, which is much, much faster

lokalmatador (Apr 23 2019 at 19:31, on Zulip):

yeah that could probably be something easy to begin with giving me some more time to study; and testing should allow me to understand the whole idea better anyways

Albin Stjerna (Apr 23 2019 at 19:31, on Zulip):

(the micro-benchmarks I toyed with uses the standard input examples in the Polonius crate, for the record)

lokalmatador (Apr 23 2019 at 19:31, on Zulip):

yeah, i think that could be something to start with

Albin Stjerna (Apr 23 2019 at 19:32, on Zulip):

But I also extended the big rustc benchmarks with essentially a Polonius run

nikomatsakis (Apr 23 2019 at 19:32, on Zulip):

@lokalmatador there a few challenges

Albin Stjerna (Apr 23 2019 at 19:32, on Zulip):

For development, I suspect a faster, more micro-oriented benchmark would be a better idea, but I'm also not sure if Criterion is the best way to go, given that it more or less requires benchmarks that run in microseconds

lqd (Apr 23 2019 at 19:32, on Zulip):

it might be interesting to have benchmarks where code fails to compile, since that should be the slowest to analyze

nikomatsakis (Apr 23 2019 at 19:32, on Zulip):

among other things, our benchmark suite consists of entire crates, but polonius operates on individual functions

Albin Stjerna (Apr 23 2019 at 19:33, on Zulip):

Unless you have...a lot of time

lokalmatador (Apr 23 2019 at 19:33, on Zulip):

ok

nikomatsakis (Apr 23 2019 at 19:33, on Zulip):

so what I was hoping to do was to pick some set of crates

nikomatsakis (Apr 23 2019 at 19:33, on Zulip):

well wait sorry

nikomatsakis (Apr 23 2019 at 19:33, on Zulip):

what I was hoping to do was to "Dump the facts" from each crate (That's relatively easy)

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

these are the input tuples to poloonius

lqd (Apr 23 2019 at 19:34, on Zulip):

and locating slow functions ?

lokalmatador (Apr 23 2019 at 19:34, on Zulip):

you're referring to the rustc benchmark and not one for polonius right?

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

put some stats about these facts into a spreadsheet -- perhaps e.g. looking at the maximum size of each kind of tuple from any fn in the crate

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

you're referring to the rustc benchmark and not one for polonius right?

correct

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

basically look for some "outliers" in various dimensions

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

like, ah, here is a benchmark with a lot of subset relations

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

here is another with a lot of xxx relations

nikomatsakis (Apr 23 2019 at 19:34, on Zulip):

whatever it is

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

this might help us narrow down to 2 or 3 "interesting" crates

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

then within those crates to try and find the best functions

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

and extract those

lokalmatador (Apr 23 2019 at 19:35, on Zulip):

sounds reasonable

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

then within those crates to try and find the best functions

the way I did this in the past was to instrument rustc a bit -- it's not too hard -- to profile the time spent per fn

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

as an alternative, we could actually do this up front

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

(I could give you some tips wht to edit)

lqd (Apr 23 2019 at 19:35, on Zulip):

can the self profiling be used for that now ?

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

I don't think it can yet

nikomatsakis (Apr 23 2019 at 19:35, on Zulip):

but it's like 1 line to add a simple measurement

nikomatsakis (Apr 23 2019 at 19:36, on Zulip):

in That case we could make a better spreadsheet

nikomatsakis (Apr 23 2019 at 19:36, on Zulip):

where each row is a fn

nikomatsakis (Apr 23 2019 at 19:36, on Zulip):

and the columns are like "ms" and the number of tuples from each input relation

nikomatsakis (Apr 23 2019 at 19:36, on Zulip):

this might be give us some data

lqd (Apr 23 2019 at 19:36, on Zulip):

datafrog has the ability to dump tuple stats btw

nikomatsakis (Apr 23 2019 at 19:36, on Zulip):

ah yeah, right!

nikomatsakis (Apr 23 2019 at 19:37, on Zulip):

so basically this is the task, then:

nikomatsakis (Apr 23 2019 at 19:37, on Zulip):
nikomatsakis (Apr 23 2019 at 19:37, on Zulip):
lqd (Apr 23 2019 at 19:37, on Zulip):

@lokalmatador are you familar with rustc development by any chance ?

nikomatsakis (Apr 23 2019 at 19:37, on Zulip):

probably me and @lqd can offer some tips here in a separate topic :)

lokalmatador (Apr 23 2019 at 19:38, on Zulip):

lokalmatador are you familar with rustc development by any chance ?

not yet but planning to read the guide this week

lqd (Apr 23 2019 at 19:38, on Zulip):

absolutely

lokalmatador (Apr 23 2019 at 19:38, on Zulip):

oh that would be amazing

nikomatsakis (Apr 23 2019 at 19:38, on Zulip):

one nice thing @lokalmatador is that there is no great rush on this

nikomatsakis (Apr 23 2019 at 19:38, on Zulip):

heh, don't worry, we're not going to ask you to figure everything out on your own ;)

nikomatsakis (Apr 23 2019 at 19:39, on Zulip):

I only wish the compiler were so well documented that were possible :P

lokalmatador (Apr 23 2019 at 19:39, on Zulip):

hehe

lqd (Apr 23 2019 at 19:40, on Zulip):

so Albin is not blocked, I don't think I am either, and @lokalmatador can hit us up for questions and we'll have a dedicated topic for that

lokalmatador (Apr 23 2019 at 19:40, on Zulip):

ok, task is noted down in my todo

Albin Stjerna (Apr 23 2019 at 19:40, on Zulip):

I think you can even try asking me about some lower-hanging things

lqd (Apr 23 2019 at 19:41, on Zulip):

and niko has already too much on their plate already :p

lokalmatador (Apr 23 2019 at 19:41, on Zulip):

great - i thin for now I'll dig into the rustc guide - so for now i won't bug you that much ;)

lokalmatador (Apr 23 2019 at 19:41, on Zulip):

I think you can even try asking me about some lower-hanging things

great, thanks!

lqd (Apr 23 2019 at 19:42, on Zulip):

with that, I think we're good ?

Albin Stjerna (Apr 23 2019 at 19:42, on Zulip):

I'm happy for now!

lokalmatador (Apr 23 2019 at 19:42, on Zulip):

me too

lqd (Apr 23 2019 at 19:42, on Zulip):

if so, good evening everyone, thank you for your time :) :wave:

nikomatsakis (Apr 23 2019 at 19:42, on Zulip):

awesome!

Albin Stjerna (Apr 23 2019 at 19:42, on Zulip):

Thanks! :wave:

nikomatsakis (Apr 23 2019 at 19:42, on Zulip):

let's update the roadmap a bit

nikomatsakis (Apr 23 2019 at 19:42, on Zulip):

to describe the profiling business

nikomatsakis (Apr 23 2019 at 19:43, on Zulip):

it's basically this:

Identify outliers in performance + memory usage

nikomatsakis (Apr 23 2019 at 19:43, on Zulip):

but now we have a bit more detail

lokalmatador (Apr 23 2019 at 19:43, on Zulip):

hopefully soon much more

nikomatsakis (Apr 23 2019 at 19:44, on Zulip):

basic steps outlined here

lqd (Apr 23 2019 at 23:12, on Zulip):

meeting notes PR posted here

Last update: Nov 15 2019 at 21:00UTC