Stream: t-compiler

Topic: rustc-bisect


Santiago Pastorino (Apr 25 2019 at 15:31, on Zulip):

was talking with @nikomatsakis about rustc-bisect after I've heard of it during the weekly meeting

Santiago Pastorino (Apr 25 2019 at 15:32, on Zulip):

would be nice to be able to run cargo bisect-rustc on the project itself

nikomatsakis (Apr 25 2019 at 15:37, on Zulip):

(cc @simulacrum)

nikomatsakis (Apr 25 2019 at 15:37, on Zulip):

Yeah, so, I think cargo-bisect-rustc is awesome but could be made a lot simpler

nikomatsakis (Apr 25 2019 at 15:37, on Zulip):

My "north star" is that when somebody reports a regression

nikomatsakis (Apr 25 2019 at 15:37, on Zulip):

They should be able to go to their project

nikomatsakis (Apr 25 2019 at 15:37, on Zulip):

And instead of doing cargo build do cargo install cargo-bisect-rustc; cargo bisect-rustc

nikomatsakis (Apr 25 2019 at 15:37, on Zulip):

and they should get a "report" they can paste into the bug report

nikomatsakis (Apr 25 2019 at 15:38, on Zulip):

it would say something like:


Regression occurred on nightly-2018-04-28 and specifically PR #12153 (cc @authors of the pr, @rust-lang/compiler)

nikomatsakis (Apr 25 2019 at 15:38, on Zulip):

Then there would be options to do more advanced stuff

nikomatsakis (Apr 25 2019 at 15:38, on Zulip):

The way I imagine this working is a few things:

nikomatsakis (Apr 25 2019 at 15:39, on Zulip):
nikomatsakis (Apr 25 2019 at 15:39, on Zulip):
nikomatsakis (Apr 25 2019 at 15:39, on Zulip):
nikomatsakis (Apr 25 2019 at 15:40, on Zulip):

Ideally, we'd use GH API to include information in that output, like the list of PRs in the nightly etc, but

Santiago Pastorino (Apr 25 2019 at 15:41, on Zulip):

:+1:

nikomatsakis (Apr 25 2019 at 16:34, on Zulip):

and they should get a "report" they can paste into the bug report

though maybe it takes a while to run :)

simulacrum (Apr 25 2019 at 18:25, on Zulip):

I am happy to hand off ownership or review PRs but do not have bandwidth for this sort of overhaul myself

nikomatsakis (Apr 26 2019 at 19:34, on Zulip):

@simulacrum I'm trying to find somebody else to do it, indeed :)

nikomatsakis (Apr 26 2019 at 22:04, on Zulip):

It seems like what needs to happen is for @Santiago Pastorino and I to sit down and kind of draw up a roadmap, eh? I could see this being a (potentially small) working group, though I think there might be enough work that 2 or 3 people could hack on it together

Santiago Pastorino (Apr 27 2019 at 03:00, on Zulip):

@nikomatsakis :+1:

nikomatsakis (Sep 30 2019 at 13:16, on Zulip):

@Santiago Pastorino these are the comments I was referring to

Santiago Pastorino (Oct 03 2019 at 15:44, on Zulip):

@nikomatsakis @simulacrum FYI I'm going to take this task slowly meanwhile I'm doing other things

simulacrum (Oct 03 2019 at 15:45, on Zulip):

:thumbs_up:

simulacrum (Oct 03 2019 at 15:45, on Zulip):

I'm around for questions and such most of the time, though it's been a while since I looked into this

Santiago Pastorino (Oct 03 2019 at 15:46, on Zulip):

:+1:

Santiago Pastorino (Oct 03 2019 at 15:46, on Zulip):

yeah I was going to ask if you had any thought in general

Santiago Pastorino (Oct 03 2019 at 15:46, on Zulip):

first I'm starting to check how the tool works

simulacrum (Oct 03 2019 at 15:47, on Zulip):

Yeah so there's two parts to any bisection tool:

simulacrum (Oct 03 2019 at 15:47, on Zulip):

the actual bisection bit is actually not a trivial algorithm, particularly as we want to do so without actually knowing all of the points

simulacrum (Oct 03 2019 at 15:48, on Zulip):

it's also (AFAICT) a literally not possible problem, we have to use heuristics, for our case

Santiago Pastorino (Oct 03 2019 at 15:49, on Zulip):

nikomatsakis simulacrum FYI I'm going to take this task slowly meanwhile I'm doing other things

I've created this https://github.com/spastorino/cargo-bisect-sample as a sample app so it's easier to play with this tool and in particular it helps to build more automation around cargo-bisect-rustc

Santiago Pastorino (Oct 03 2019 at 15:49, on Zulip):

now yes :)

simulacrum (Oct 03 2019 at 15:58, on Zulip):

it's also (AFAICT) a literally not possible problem, we have to use heuristics, for our case

To elaborate on this a bit...

We don't generally know what a given result means. In particular, say we have compilers A B C D E (ordered by ~time). A given crate may work on A, C, E due to unrelated breakage in B and D. This means that when bisecting, we don't know whether a bad result means we've reached an edge and should go the other way, or if we're just in the middle somewhere.

simulacrum (Oct 03 2019 at 15:58, on Zulip):

This can also be thought of as the problem that bisection only truly works on a monotonic graph -- you need to only go from broken to working or working to broken at most once

simulacrum (Oct 03 2019 at 16:00, on Zulip):

The main pain point with cargo bisect rustc from a usability perspective is that it is rather hard to determine what the user wants, and talking about the range you want to bisect over is also not really easy: what does "start" or "end" mean? If it's seen temporally, e.g. start is 2019-01-01 and end is today, then it's a bit weird that the first thing we'll test is today's nightly

simulacrum (Oct 03 2019 at 16:01, on Zulip):

anyway this is all mostly a bit of a brain dump right now :)

simulacrum (Oct 03 2019 at 16:01, on Zulip):

I can try to answer specific questions -- probably easier to deal with than my saying lots of things

Santiago Pastorino (Oct 03 2019 at 16:13, on Zulip):

yeah, I understand what you meant

Santiago Pastorino (Oct 03 2019 at 16:13, on Zulip):

I guess in a tool like this there should be some premises that may not hold on projects and at that point the tool may not work correctly

Santiago Pastorino (Oct 03 2019 at 16:14, on Zulip):

but I guess it would be sane to assume that you provide a start and an end and there's only one point of failure in between or else the tool may not work

Santiago Pastorino (Oct 03 2019 at 16:15, on Zulip):

we could even start from the last nightly assuming that that's the not working version and going back until there's one that works

Santiago Pastorino (Oct 03 2019 at 16:15, on Zulip):

but again, always assuming that there's just one failure between those two points

Santiago Pastorino (Oct 03 2019 at 16:15, on Zulip):

and it's the one you're looking for

simulacrum (Oct 03 2019 at 16:28, on Zulip):

Right. I personally think we might want to give up on bisection for the common case -- just linear scan

simulacrum (Oct 03 2019 at 16:28, on Zulip):

linear scan is plenty fast if we're doing so recently

pnkfelix (Oct 03 2019 at 18:02, on Zulip):

is part of the plan here to make rustc-bisect able to do its own local builds, to get a finer grain than what the CI nightlies provide?

pnkfelix (Oct 03 2019 at 18:02, on Zulip):

(because there are additional pain points I would anticipate in automating that process, especially if you do not do clean checkouts+builds in between each run)

simulacrum (Oct 03 2019 at 18:09, on Zulip):

hm, I think I would say at this point no

simulacrum (Oct 03 2019 at 18:09, on Zulip):

since 'random person' does not want to compile rustc

simulacrum (Oct 03 2019 at 18:09, on Zulip):

and bisecting between two rustc bors commits is usually not too hard for a compiler dev

lqd (Oct 03 2019 at 18:10, on Zulip):

another pain point is looking for the absence of some output (a warning, or a specific error), grep -v does not help here

simulacrum (Oct 03 2019 at 18:10, on Zulip):

one thought I have had is to run things with rendered-ansi mode and basically UI test it

simulacrum (Oct 03 2019 at 18:11, on Zulip):

it's a bit error prone but maybe having a "collection of allowed outputs" vs. just one is viable there

simulacrum (Oct 03 2019 at 18:11, on Zulip):

I guess those are orthogonal -- I was imagining that we could have a mode for "I want this error code emitted", and another with the UI testing

lqd (Oct 03 2019 at 18:17, on Zulip):

until then I will make a PR to the tutorial to add the git log command to list bors' commits inside a date range, not a big deal but at least can save someone else a couple minutes looking for that

simulacrum (Oct 03 2019 at 18:17, on Zulip):

indeed

Santiago Pastorino (Oct 04 2019 at 00:52, on Zulip):

@nikomatsakis leaving this as a draft we can use to define goals and tasks

Santiago Pastorino (Oct 04 2019 at 00:52, on Zulip):

https://hackmd.io/Ipu9wSL4SCyx9_adR6GKLg

Santiago Pastorino (Oct 04 2019 at 17:24, on Zulip):

@simulacrum about https://hackmd.io/Ipu9wSL4SCyx9_adR6GKLg that we were talking about

Santiago Pastorino (Oct 04 2019 at 17:25, on Zulip):

in particular the tasking

Santiago Pastorino (Oct 04 2019 at 17:25, on Zulip):

first of all I was wondering about the Use currently installed nightly as end point

Santiago Pastorino (Oct 04 2019 at 17:25, on Zulip):

I think that's probably ok if the users has nightly as the default toolchain

Santiago Pastorino (Oct 04 2019 at 17:26, on Zulip):

but what if the default is stable and there's one or two nightlies?

Santiago Pastorino (Oct 04 2019 at 17:26, on Zulip):

should we discover one?

simulacrum (Oct 04 2019 at 17:26, on Zulip):

I think no

Santiago Pastorino (Oct 04 2019 at 17:26, on Zulip):

should we just have a prerequisite of or do have a nightly version as default toolchain or please provide an end point

simulacrum (Oct 04 2019 at 17:26, on Zulip):

that's both hard and not really helpful

Santiago Pastorino (Oct 04 2019 at 17:26, on Zulip):

yeah I think so

simulacrum (Oct 04 2019 at 17:26, on Zulip):

I left some notes in the doc

Santiago Pastorino (Oct 04 2019 at 17:27, on Zulip):

I guess I'd go with if you provide an end, we use that one

Santiago Pastorino (Oct 04 2019 at 17:27, on Zulip):

otherwise the default toolchain is used

Santiago Pastorino (Oct 04 2019 at 17:27, on Zulip):

whatever is that default toolchain

Santiago Pastorino (Oct 04 2019 at 17:27, on Zulip):

but it would abort if the start is newer than that one

simulacrum (Oct 04 2019 at 17:27, on Zulip):

I would personally say actually that we don't accept the end toolchain in anyway except for the default one in the current environment

Santiago Pastorino (Oct 04 2019 at 17:27, on Zulip):

you may want to look for a regression you're hitting on stable I guess

Santiago Pastorino (Oct 04 2019 at 17:28, on Zulip):

I would personally say actually that we don't accept the end toolchain in anyway except for the default one in the current environment

that's exactly what I'm saying, right? :)

Santiago Pastorino (Oct 04 2019 at 17:28, on Zulip):

my english is ungreat and from time to time I need to recheck obvious things :)

simulacrum (Oct 04 2019 at 17:28, on Zulip):

ah, well, "I guess I'd go with if you provide an end, we use that one" I interpreted as being you can provide it in other ways

Santiago Pastorino (Oct 04 2019 at 17:29, on Zulip):

I meant, if you do from the command line end=2019-08-07 that one would be used

simulacrum (Oct 04 2019 at 17:29, on Zulip):

to be clear this means that the expected workflow is strictly that you start by doing cargo +toolchain check, this fails. So before you file a bug report, you run cargo +toolchain bisect-rustc and that'll look for the first successful nightly/SHA going backwards in time.

Santiago Pastorino (Oct 04 2019 at 17:30, on Zulip):

:+1:

simulacrum (Oct 04 2019 at 17:30, on Zulip):

so I'm saying that we would scale back and not provide nearly as many CLI flags -- or maybe "hide" them behind some advanced interface

Santiago Pastorino (Oct 04 2019 at 17:30, on Zulip):

yep agreed

simulacrum (Oct 04 2019 at 17:30, on Zulip):

I think it makes a lot of sense to make the tool be sort of intelligent but overall pretty 'not tunable' at least initially

Santiago Pastorino (Oct 04 2019 at 17:31, on Zulip):

what do you mean by not tunable?

simulacrum (Oct 04 2019 at 17:31, on Zulip):

i.e. not too many options, does basically the same thing everytime

Santiago Pastorino (Oct 04 2019 at 17:32, on Zulip):

ok

simulacrum (Oct 04 2019 at 17:32, on Zulip):

to be clear I think we can issue breaking changes without a problem here at this point

Santiago Pastorino (Oct 04 2019 at 18:17, on Zulip):

hey back, sorry went to have lunch

Santiago Pastorino (Oct 04 2019 at 18:17, on Zulip):

next about Find a starting point going exponentially backwards

Santiago Pastorino (Oct 04 2019 at 18:17, on Zulip):

@simulacrum you added a comment there saying what the tool currently does

Santiago Pastorino (Oct 04 2019 at 18:17, on Zulip):

but I'm a bit confused here

Santiago Pastorino (Oct 04 2019 at 18:18, on Zulip):

the tool works by using a start and end and does what you say

Santiago Pastorino (Oct 04 2019 at 18:18, on Zulip):

in this case we would be looking for a working version

Santiago Pastorino (Oct 04 2019 at 18:19, on Zulip):

unsure if the best way to search would be the same in these cases

simulacrum (Oct 04 2019 at 18:19, on Zulip):

ah yes you are right

Santiago Pastorino (Oct 04 2019 at 18:19, on Zulip):

unsure I got exactly the meaning of your comments

Santiago Pastorino (Oct 04 2019 at 18:19, on Zulip):

ah ok

simulacrum (Oct 04 2019 at 18:19, on Zulip):

I was remembering wrong and the exact handling is wrong

simulacrum (Oct 04 2019 at 18:19, on Zulip):

er, is complicated and probably wrong

Santiago Pastorino (Oct 04 2019 at 18:20, on Zulip):

what is wrong sorry?

simulacrum (Oct 04 2019 at 18:22, on Zulip):

ah my reasoning

Santiago Pastorino (Oct 04 2019 at 18:26, on Zulip):

I still wonder what's the best way to go back

Santiago Pastorino (Oct 04 2019 at 18:27, on Zulip):

isn't maybe nice to try first with the latest available stable?

Santiago Pastorino (Oct 04 2019 at 18:28, on Zulip):

I meant, you want to quickly find a working version in this phase

Santiago Pastorino (Oct 04 2019 at 18:28, on Zulip):

of course you don't want to start from version 1.0 or a previous one :P

Santiago Pastorino (Oct 04 2019 at 18:29, on Zulip):

and if that fails maybe keep jumping through stables?

simulacrum (Oct 04 2019 at 18:35, on Zulip):

I would not do that, it's error prone

simulacrum (Oct 04 2019 at 18:35, on Zulip):

it is fairly fast and effective to do the 2x backwards that I suggested

simulacrum (Oct 04 2019 at 18:36, on Zulip):

this does [today; 2 days ago; 4 days ago; 8 days ago; 16 days ago; 1 month ago; 2 months ago] -- that's 7 hops max for most new things

simulacrum (Oct 04 2019 at 18:36, on Zulip):

and most likely it'll be within the first couple

simulacrum (Oct 04 2019 at 18:36, on Zulip):

you could even be more aggressive and start 1 day ago -> 4 days ago -> 8 days ago -> 16 days ago

nikomatsakis (Oct 04 2019 at 18:37, on Zulip):

(fwiw, everything that's in the hackmd looks great to me, and I agree that avoiding a checkout is good if we can do it)

nikomatsakis (Oct 04 2019 at 18:38, on Zulip):

stepping back 1 day, 2, 4 etc is exactly what I had in mind

Santiago Pastorino (Oct 04 2019 at 18:39, on Zulip):

:+1:

Santiago Pastorino (Oct 04 2019 at 18:42, on Zulip):

(fwiw, everything that's in the hackmd looks great to me, and I agree that avoiding a checkout is good if we can do it)

now about this ^^^

Santiago Pastorino (Oct 04 2019 at 18:42, on Zulip):
    - I would avoid a checkout here, it's pretty expensive disk space/network and quite slow
    - We can either query GitHub API or I would not mind us running a Rust server somewhere on rust infra for this
Santiago Pastorino (Oct 04 2019 at 18:42, on Zulip):

what would be the idea then?

Santiago Pastorino (Oct 04 2019 at 18:43, on Zulip):

you query Github API and get a range of PRs and after that?

Santiago Pastorino (Oct 04 2019 at 18:43, on Zulip):

unsure either what you meant by having a Rust server

Santiago Pastorino (Oct 04 2019 at 18:44, on Zulip):

are you suggesting that in order to use this the user should already have a rustc copy around?

simulacrum (Oct 04 2019 at 18:57, on Zulip):

ah so you should not need a checkout

simulacrum (Oct 04 2019 at 18:58, on Zulip):

a checkout is pretty heavy

simulacrum (Oct 04 2019 at 18:58, on Zulip):

we can query the github API for everything we could get from the checkout

simulacrum (Oct 04 2019 at 18:58, on Zulip):

or, if that's hard or slow, I meant we could have an HTTP server that helps out bisectors by providing helpful and not-really-rate limited endpoints

simulacrum (Oct 04 2019 at 18:59, on Zulip):

internally this could either go to github API or reference local github repo or something

Santiago Pastorino (Oct 04 2019 at 18:59, on Zulip):

but I meant, unsure if we are talking about the same thing

Santiago Pastorino (Oct 04 2019 at 18:59, on Zulip):

once you found that the regression happened on nightly-X-Y-Z

Santiago Pastorino (Oct 04 2019 at 19:00, on Zulip):

you may want to find the rustc commit that've caused the problem

Santiago Pastorino (Oct 04 2019 at 19:00, on Zulip):

with Github API we can find a set of commits where the culprit is included

Santiago Pastorino (Oct 04 2019 at 19:01, on Zulip):

but in order to know which one is it we need to bisect through them

Santiago Pastorino (Oct 04 2019 at 19:01, on Zulip):

still that's what the tool already does

Santiago Pastorino (Oct 04 2019 at 19:01, on Zulip):

maybe that's what you meant

Santiago Pastorino (Oct 04 2019 at 19:02, on Zulip):

all what new code needs to do is find start and end commits and call the existing bisect function

Santiago Pastorino (Oct 04 2019 at 19:02, on Zulip):

so the code would check Github and all that

Santiago Pastorino (Oct 04 2019 at 19:02, on Zulip):

if that's what you meant agree :)

simulacrum (Oct 04 2019 at 19:19, on Zulip):

yes, that's basically it

simulacrum (Oct 04 2019 at 19:19, on Zulip):

we already have full commits from nightlies for start/end so we just need a list of commits

Santiago Pastorino (Oct 04 2019 at 19:20, on Zulip):

also

Santiago Pastorino (Oct 04 2019 at 19:20, on Zulip):

can't we just take sha's from the nightlies?

Santiago Pastorino (Oct 04 2019 at 19:20, on Zulip):

there's no need to go to github api

Santiago Pastorino (Oct 04 2019 at 19:21, on Zulip):

I meant the range in which we can look for a PR is ... the good nigthly and the bad nightly that's next to it

simulacrum (Oct 04 2019 at 19:27, on Zulip):

right

simulacrum (Oct 04 2019 at 19:27, on Zulip):

but to get the per-SHA artifacts you need to enumerate SHAs inbetween those

simulacrum (Oct 04 2019 at 19:27, on Zulip):

that requires API

Santiago Pastorino (Oct 04 2019 at 19:44, on Zulip):

but isn't that we the tool already does?

simulacrum (Oct 04 2019 at 19:47, on Zulip):

right yeah we have that logic already

Santiago Pastorino (Oct 04 2019 at 19:50, on Zulip):

so ... there's nothing we need to add then in that regard :)

Santiago Pastorino (Oct 04 2019 at 19:51, on Zulip):

well the last thing may be bisecting within the PR

Santiago Pastorino (Oct 04 2019 at 19:51, on Zulip):

I guess the tool doesn't do that currently, right?

Santiago Pastorino (Oct 04 2019 at 20:28, on Zulip):
- Bisect on beta/stable
    - go to 'real' nightly by looking at version and looking up appropriate branch point nightly
    - e.g. for 1.39 (currently in beta) branched on September 26th 2019 (date of 1.38 release, ~easily computable with 6 week offsets)
    - so you'd want nightly-2019-09-27 (this is not perfect, but good enough)
Santiago Pastorino (Oct 04 2019 at 20:28, on Zulip):

@simulacrum I'm not sure what that means ^^^

simulacrum (Oct 04 2019 at 20:29, on Zulip):

bisecting within a single PR is not tenable for anyone not a compiler dev, you need to build the compiler, this tool will not support that

simulacrum (Oct 04 2019 at 20:29, on Zulip):

@Santiago Pastorino I was proposing a way to bisect into nightlies when given a beta build as the default

simulacrum (Oct 04 2019 at 20:30, on Zulip):

you can sort of "unwrap" a beta to get the nightly when it branched

simulacrum (Oct 04 2019 at 20:30, on Zulip):

but this is pretty far future, I wouldn't worry about it too much for now

Santiago Pastorino (Oct 04 2019 at 20:30, on Zulip):

:+1:

Santiago Pastorino (Oct 04 2019 at 20:31, on Zulip):

thank you very much for helping me plan this :)

simulacrum (Oct 04 2019 at 20:32, on Zulip):

sure thing

simulacrum (Oct 04 2019 at 20:32, on Zulip):

gets me off of doing it eventually :)

Last update: Nov 22 2019 at 04:45UTC