Stream: t-compiler/wg-polonius

Topic: scheduling some sprints


nikomatsakis (Jul 17 2020 at 18:42, on Zulip):

Hey @lqd -- so we failed to schedule any sprints. But I still want to. I was thinking that maybe Aug 2-5 might work? In fact, I am wondering if it'd be possible to schedule some kind of "regular" sprints, or maybe just on my part to kind of dedicate some weeks to helping to investigate and push polonius, doesnt' necessarily have to involve all the rest of y'all

lqd (Jul 17 2020 at 18:43, on Zulip):

I'm game for Aug 2-5

lqd (Jul 17 2020 at 18:45, on Zulip):

(cc @Albin Stjerna)

lqd (Jul 17 2020 at 18:46, on Zulip):

@nikomatsakis by then (hopefully way before that) I'll have written up thoughts and notes about the agenda/topics I mentioned last time, so that you can see if something interests you, to investigate whenever your schedule clears up otherwise

lqd (Jul 17 2020 at 18:47, on Zulip):

(sorry we haven't been able to schedule this earlier)

Albin Stjerna (Jul 18 2020 at 18:50, on Zulip):

Sorry I dropped the ball on the sprint scheduling; I’m also up for Aug 2-5! I, err, may have forgotten what I was doing and/or parts of Rust in general but it will be fun for me!

nikomatsakis (Jul 20 2020 at 17:58, on Zulip):

Hmm I see Aug 2 is a sunday

nikomatsakis (Jul 20 2020 at 17:58, on Zulip):

I'm prob not available then :)

nikomatsakis (Jul 20 2020 at 17:58, on Zulip):

so maybe Aug 3-5 is more accurate

Albin Stjerna (Jul 21 2020 at 09:39, on Zulip):

Off by one, a classic

Albin Stjerna (Jul 29 2020 at 12:57, on Zulip):

Apparently my supervisor thinks Polonius is research, which means I can actually spend some of my working hours working on it!

Albin Stjerna (Jul 29 2020 at 12:57, on Zulip):

Including the sprint

nikomatsakis (Jul 29 2020 at 19:23, on Zulip):

Ooh, nice

nikomatsakis (Jul 29 2020 at 19:23, on Zulip):

I'm looking forward to sprint, I'm presuming it still works for all of you?

nikomatsakis (Jul 29 2020 at 19:23, on Zulip):

I know @lqd you had some specific ideas right about things to pursue?

lqd (Jul 29 2020 at 20:34, on Zulip):

@nikomatsakis yeah I mentioned some here -- plus I think it'll be good for all of us to do a bit of planning, what we can do, what we should do

lqd (Jul 29 2020 at 20:35, on Zulip):

I'm not sure Albin or you saw them yet and if they interest you all though

lqd (Jul 29 2020 at 20:42, on Zulip):

a lot of these items fit into a kind of theme "make sure we can land something that works and scales" -- but I don't think there's a foolproof way of achieving that as of yet (mostly it's the scaling that worries me), so a bit of design/planning to see what I missed in that regard would be greatly appreciated :)

lqd (Jul 29 2020 at 20:50, on Zulip):

(also what changes we will need in order to integrate with chalk eventually)

lqd (Jul 29 2020 at 21:24, on Zulip):

(also a plan for better testing and validating everything)

lqd (Jul 29 2020 at 21:30, on Zulip):

the discussion/design/planning would be very worthwhile to me, as a lot of those coding tasks are easy when we're all on the same page (and I can do a bunch of them on my own) whereas having you all to bounce ideas around is rare

nikomatsakis (Jul 30 2020 at 14:57, on Zulip):

ok, I'm reading over those now...

nikomatsakis (Jul 30 2020 at 14:57, on Zulip):

do you think the best way to start would be just with some dedicated discussion? sounds like yes

lqd (Jul 30 2020 at 15:02, on Zulip):

we could

lqd (Jul 30 2020 at 15:10, on Zulip):

but as long as we are able to do so during the sprint it'll be great :) if there are better tasks to start with, for example related to move errors in rustc or something else, it'd be fine to start with those as well

Albin Stjerna (Aug 03 2020 at 07:18, on Zulip):

My plan is to spend the morning getting back in the saddle, and starting out with getting move errors into rustc

Albin Stjerna (Aug 03 2020 at 07:19, on Zulip):

(there is no chance I will finish that before Niko wakes up)

lqd (Aug 03 2020 at 07:58, on Zulip):

hi :)

lqd (Aug 03 2020 at 08:08, on Zulip):

I wonder if there's an issue with the move errors computation, maybe making it compute too many things: it seems to be taking a lot of time for clap for example. I wonder if it's the rules or the model itself, but maybe this is one those cases where a dataflow context would be more appropriate. maybe the granularity of the model is so fine-grained that a lot of state is computed and cloned (per-point at twice the number of points versus per-block)

lqd (Aug 03 2020 at 08:10, on Zulip):

I'm also not sure one of the fixes I made to remove the false positives (switching the mid/start point where a fact was emitted) was without consequences, even if tests still pass

Albin Stjerna (Aug 03 2020 at 08:31, on Zulip):

I guess those are great questions to ask the rustc test suite once we have the reporting integrated :)

lqd (Aug 03 2020 at 08:32, on Zulip):

hehe yeah the 2nd one for sure

lqd (Aug 03 2020 at 08:35, on Zulip):

I just hope it wouldn't take a lot of effort to discover how to make the computation fast

lqd (Aug 03 2020 at 08:37, on Zulip):

but then again, we could compute the errors differently if datafrog is not helping, or reuse rustc's computation, so it's not a hopeless situation or anything :)

Albin Stjerna (Aug 03 2020 at 08:46, on Zulip):

agreed, I also feel like a lot rests on that

lqd (Aug 03 2020 at 10:16, on Zulip):

ah I forgot and I'll check right now, I remember that when I prototyped fixing the 2 cases of OOM in rustc's fact generation (2 numeric test cases: ui/numbers-arithmetic/saturating-float-casts.rs and ui/wrapping-int-combinations.rs), it seemed to work for the loan and subset errors analysis, but the move errors computation would then OOM instead of rustc

Albin Stjerna (Aug 03 2020 at 11:28, on Zulip):

I'm still at "how do I get stuff to compile" so, err, no rush

Albin Stjerna (Aug 03 2020 at 11:28, on Zulip):

I'm on a new machine and haven't done this for half a year or so, so I guess I'm a little rusty

lqd (Aug 03 2020 at 11:33, on Zulip):

I assume you're following the rustc dev guide ?

Albin Stjerna (Aug 03 2020 at 11:34, on Zulip):

Yes, I am

Albin Stjerna (Aug 03 2020 at 11:35, on Zulip):

I'm so happy it exists

lqd (Aug 03 2020 at 11:35, on Zulip):

yeah it's great, then you won't have issues

lqd (Aug 03 2020 at 11:36, on Zulip):

how's the phd going btw ?! having fun with smt and string operations ? :)

Albin Stjerna (Aug 03 2020 at 11:37, on Zulip):

I think it's too early to tell how it's going, but I'm...doing stuff?

Albin Stjerna (Aug 03 2020 at 11:38, on Zulip):

I do actually have an issue; for whatever reason x.py check doesn't...work

lqd (Aug 03 2020 at 11:39, on Zulip):

how does it fail ?

Albin Stjerna (Aug 03 2020 at 11:39, on Zulip):
./x.py check
Updating only changed submodules
Submodules updated in 0.01 seconds
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
thread 'main' panicked at 'no entry found for key', src/bootstrap/lib.rs:1115:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failed to run: /home/albin/rust/build/bootstrap/debug/bootstrap check
Build completed unsuccessfully in 0:00:00
Albin Stjerna (Aug 03 2020 at 11:40, on Zulip):

Anyway, I'm mainly flailing around trying to figure out what I'm supposed to be doing, which I've heard is normal for a first-year PhD :)

lqd (Aug 03 2020 at 11:40, on Zulip):

good to hear (about the phd, not x.py check failing :)

lqd (Aug 03 2020 at 11:41, on Zulip):

the error is weird and without much info dang

Albin Stjerna (Aug 03 2020 at 11:41, on Zulip):

My hope is to be able to focus more on more rust-adjacent things after finishing my current work on the more "inside the SMT solver" gritty details

Albin Stjerna (Aug 03 2020 at 11:41, on Zulip):

That's exactly what I thought

lqd (Aug 03 2020 at 11:42, on Zulip):

all submodules and directories seem acceptable on git status ?

Albin Stjerna (Aug 03 2020 at 11:42, on Zulip):

the backtrace is also incomprehensible

Albin Stjerna (Aug 03 2020 at 11:42, on Zulip):

It looks like it

Albin Stjerna (Aug 03 2020 at 11:42, on Zulip):

and it should be a clean fork

lqd (Aug 03 2020 at 11:43, on Zulip):

I feel it's not the first time I've seen this panic message mentioned somewhere, at least showing which crate is invalid would be helpful

Albin Stjerna (Aug 03 2020 at 11:43, on Zulip):
git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   Cargo.lock
    modified:   src/librustc_middle/Cargo.toml
    modified:   src/librustc_mir/Cargo.toml
Albin Stjerna (Aug 03 2020 at 11:45, on Zulip):

It's even weirder that _normal compilation_ works fine?!

lqd (Aug 03 2020 at 11:45, on Zulip):

there was a git problem on the rust repo earlier, it may be related to that but I would assume it would have failed earlier or differently if that was actually the case

lqd (Aug 03 2020 at 11:47, on Zulip):

(it was fixed by https://github.com/rust-lang/rust/pull/75092 and some other things)

Albin Stjerna (Aug 03 2020 at 11:53, on Zulip):

Interesting, it works if I don't change my files to use the local polonius

Albin Stjerna (Aug 03 2020 at 11:53, on Zulip):

So something is clearly up with that

Albin Stjerna (Aug 03 2020 at 11:56, on Zulip):

I should point Cargo.toml to the polonius-engine directory, right?

lqd (Aug 03 2020 at 11:56, on Zulip):

yeah

lqd (Aug 03 2020 at 11:57, on Zulip):

I'm still trying to untangle the branch I mentioned earlier but I'll be able to check the same thing on my machine soon as well

lqd (Aug 03 2020 at 12:00, on Zulip):

haha same as you

Albin Stjerna (Aug 03 2020 at 12:02, on Zulip):

Ok so something's happening

Albin Stjerna (Aug 03 2020 at 12:02, on Zulip):

I'm wondering if I even need a newer Polonius though, I think rust is using 0.12.x

Albin Stjerna (Aug 03 2020 at 12:02, on Zulip):

And I think that one has move errors but I'd have to check

lqd (Aug 03 2020 at 12:02, on Zulip):

(there have been some recent changes to x.py, folders moved, etc)

lqd (Aug 03 2020 at 12:02, on Zulip):

it does have it

lqd (Aug 03 2020 at 12:03, on Zulip):

0.12.1

lqd (Aug 03 2020 at 12:03, on Zulip):

so yeah you should be able to use the crates.io version directly

lqd (Aug 03 2020 at 12:04, on Zulip):

while I'll be the one battling thread 'main' panicked at 'can't access crate polonius-engine', src/bootstrap/lib.rs:1115:67

lqd (Aug 03 2020 at 12:10, on Zulip):

hey @simulacrum does :up: ring a bell by any chance ?

lqd (Aug 03 2020 at 12:11, on Zulip):

bootstrap apparently failing while accessing local checkouts of dependencies

lqd (Aug 03 2020 at 12:32, on Zulip):

but it at least seems to work for git checkouts

simulacrum (Aug 03 2020 at 12:57, on Zulip):

it does not, no

simulacrum (Aug 03 2020 at 12:57, on Zulip):

let me take a look though

simulacrum (Aug 03 2020 at 12:59, on Zulip):

@lqd did you make any headway on it?

simulacrum (Aug 03 2020 at 12:59, on Zulip):

it looks like for whatever reason it's not in cargo metadata --no-deps output

simulacrum (Aug 03 2020 at 13:00, on Zulip):

not exactly sure why that'd be though

lqd (Aug 03 2020 at 13:00, on Zulip):

I didn't make much headway on it, just found out it worked with git checkouts so I was pushing my wip work on GH (while LLVM was rebuilding)

simulacrum (Aug 03 2020 at 13:01, on Zulip):

hm okay

simulacrum (Aug 03 2020 at 13:02, on Zulip):

I'd guess that there's some sort of discrepancy in what cargo metadata is spitting out, but I can't think of what it could be right now

lqd (Aug 03 2020 at 13:02, on Zulip):

I tried to clean everything in case it was some stale build artifacts colliding with the recent repository rework, but it was not the case, and now I'm just rebuilding everything :)

simulacrum (Aug 03 2020 at 13:04, on Zulip):

can you gist output of cargo metadata --no-deps --manifest-path Cargo.toml --format-version 1?

lqd (Aug 03 2020 at 13:04, on Zulip):

(it correctly wasn't able to find the crate if I pointed it at a wrong directory)

lqd (Aug 03 2020 at 13:04, on Zulip):

yep I'll try it out now

lqd (Aug 03 2020 at 13:08, on Zulip):

here it is https://gist.githubusercontent.com/lqd/49a6ca9e8ff75ef0b17a9c1b0f48020a/raw/bb223458e6bb05a129a9cdf172f10eab14850e01/metadata.json

lqd (Aug 03 2020 at 13:08, on Zulip):

it doesn't seem to know where the source is

simulacrum (Aug 03 2020 at 13:09, on Zulip):

hm that's actually correct

simulacrum (Aug 03 2020 at 13:09, on Zulip):

I think?

lqd (Aug 03 2020 at 13:09, on Zulip):

but maybe that's ok

lqd (Aug 03 2020 at 13:09, on Zulip):

yeah the other path dependencies don't seem to have it set either

simulacrum (Aug 03 2020 at 13:10, on Zulip):

is this a path dep? or crates.io?

lqd (Aug 03 2020 at 13:10, on Zulip):

path dep

lqd (Aug 03 2020 at 13:11, on Zulip):

hum

simulacrum (Aug 03 2020 at 13:11, on Zulip):

it's not in packages for some reason

lqd (Aug 03 2020 at 13:11, on Zulip):

it's usually a crates io dep, but here I just switched it to using a path dep, is what I meant

simulacrum (Aug 03 2020 at 13:11, on Zulip):

I don't quite understand why

simulacrum (Aug 03 2020 at 13:11, on Zulip):

your lockfile changed, right?

lqd (Aug 03 2020 at 13:12, on Zulip):

yeah

simulacrum (Aug 03 2020 at 13:12, on Zulip):

how are you invoking x.py?

lqd (Aug 03 2020 at 13:12, on Zulip):

./x.py check

lqd (Aug 03 2020 at 13:12, on Zulip):

it doesn't seem to be only on my machine as Albin had the same issue earlier

simulacrum (Aug 03 2020 at 13:13, on Zulip):

do you have a branch I can build?

lqd (Aug 03 2020 at 13:13, on Zulip):

yeah I'll push one

lqd (Aug 03 2020 at 13:17, on Zulip):

so https://github.com/lqd/rust/tree/bootstrap_failure

nikomatsakis (Aug 03 2020 at 13:18, on Zulip):

hi all :)

nikomatsakis (Aug 03 2020 at 13:18, on Zulip):

lots to catch upon here :)

nikomatsakis (Aug 03 2020 at 13:18, on Zulip):

/me starts reading

lqd (Aug 03 2020 at 13:18, on Zulip):

but to see if it reproduces you'd need to change the polonius-engine path dep to a local checkout of https://github.com/rust-lang/polonius

lqd (Aug 03 2020 at 13:18, on Zulip):

hi niko :)

lqd (Aug 03 2020 at 13:19, on Zulip):

not that much to catch up a lot of the backlog is about a weird issue Albin and I are having, which didn't happen a few weeks/months back

nikomatsakis (Aug 03 2020 at 13:19, on Zulip):

I guess first step is that I should try to do the same build...

lqd (Aug 03 2020 at 13:20, on Zulip):

it's not blocking or anything, just weird, as we can still use git dependencies without this issue, only a local checkout doesn't work somehow

nikomatsakis (Aug 03 2020 at 13:21, on Zulip):

I guess what I really want to do is to get "back up to speed" to start -- would you say a good first step is to build polonius git repo, should I also be building a fresh rustc .. I forget how the integration works tbqh

nikomatsakis (Aug 03 2020 at 13:21, on Zulip):

I guess it just needs -Zpolonius?

lqd (Aug 03 2020 at 13:22, on Zulip):

to test using rustc yeah

lqd (Aug 03 2020 at 13:22, on Zulip):

depending on what you want to do, running the cli or unit tests will be faster in polonius directly though

lqd (Aug 03 2020 at 13:23, on Zulip):

but as of right now, there are no real differences between polonius git and the crates.io version

lqd (Aug 03 2020 at 13:23, on Zulip):

(besides some printing in the CLI)

lqd (Aug 03 2020 at 13:24, on Zulip):

I was trying to prepare my WIP work on fixing the fact generation OOMs, and Albin was just building to look at emitting move errors, when we hit this snag

simulacrum (Aug 03 2020 at 13:25, on Zulip):

@lqd yeah so you can almost certainly fix it by using [patch.crates-io] in the root rather than path deps

simulacrum (Aug 03 2020 at 13:25, on Zulip):

I think we don't support path deps going outside the workspace or so

simulacrum (Aug 03 2020 at 13:25, on Zulip):

(maybe a cargo bug? not sure)

lqd (Aug 03 2020 at 13:25, on Zulip):

that would make sense

simulacrum (Aug 03 2020 at 13:26, on Zulip):
diff --git a/Cargo.toml b/Cargo.toml
index 1936e35aa4c..a85567fdce8 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -98,5 +98,7 @@ rustc-std-workspace-std = { path = 'library/rustc-std-workspace-std' }
 # source code for this crate.
 backtrace = { path = "library/backtrace" }

+polonius-engine = { path = "../polonius/polonius-engine" }
+
 [patch."https://github.com/rust-lang/rust-clippy"]
 clippy_lints = { path = "src/tools/clippy/clippy_lints" }
diff --git a/src/librustc_middle/Cargo.toml b/src/librustc_middle/Cargo.toml
index 1257dd351f5..03431cb5a88 100644
--- a/src/librustc_middle/Cargo.toml
+++ b/src/librustc_middle/Cargo.toml
@@ -14,7 +14,7 @@ rustc_arena = { path = "../librustc_arena" }
 bitflags = "1.2.1"
 log = { package = "tracing", version = "0.1" }
 rustc-rayon-core = "0.3.0"
-polonius-engine = { path = "/mnt/d/work/rust/tmp-polonius/lqd-polonius6/polonius-engine" }
+polonius-engine = "0.12.0"
 rustc_apfloat = { path = "../librustc_apfloat" }
 rustc_attr = { path = "../librustc_attr" }
 rustc_feature = { path = "../librustc_feature" }
diff --git a/src/librustc_mir/Cargo.toml b/src/librustc_mir/Cargo.toml
index 8d1f88e01a1..f05c47e0ed6 100644
--- a/src/librustc_mir/Cargo.toml
+++ b/src/librustc_mir/Cargo.toml
@@ -15,7 +15,7 @@ rustc_graphviz = { path = "../librustc_graphviz" }
 itertools = "0.8"
 log = { package = "tracing", version = "0.1" }
 log_settings = "0.1.1"
-polonius-engine = { path = "/mnt/d/work/rust/tmp-polonius/lqd-polonius6/polonius-engine" }
+polonius-engine = "0.12.0"
 rustc_middle = { path = "../librustc_middle" }
 rustc_attr = { path = "../librustc_attr" }
 rustc_data_structures = { path = "../librustc_data_structures" }
lqd (Aug 03 2020 at 13:26, on Zulip):

it definitely used to work so maybe it's more or less recent

simulacrum (Aug 03 2020 at 13:26, on Zulip):

I don't have time myself to file an issue with cargo but seems good to do so

lqd (Aug 03 2020 at 13:27, on Zulip):

yeah I will temporarily use the crates.io patch technique, and look or file an issue on cargo later, thanks for looking at it @simulacrum

nikomatsakis (Aug 03 2020 at 13:28, on Zulip):

ok well I'm building a fresh rustc, and building polonius was successful ;)

nikomatsakis (Aug 03 2020 at 13:28, on Zulip):

kind of sad how long it's been since I had occasion to build rustc :'(

lqd (Aug 03 2020 at 13:29, on Zulip):

thankfully it was only while trying with a local polonius build that it didn't work, so not that big of a deal

Last update: Jun 20 2021 at 01:15UTC