Stream: t-compiler/cargo-bisect-rustc

Topic: sharing knowledge


Santiago Pastorino (Feb 13 2020 at 16:25, on Zulip):

I was telling @simulacrum that may be a good idea if someone briefly explain the code and decisions to the rest

simulacrum (Feb 13 2020 at 16:25, on Zulip):

You had some list of questions? That's probably a good start

Santiago Pastorino (Feb 13 2020 at 16:25, on Zulip):

my main motivation for this is that I've got some questions about the tool that the only way for me to answer would be to dive into the tool

Santiago Pastorino (Feb 13 2020 at 16:26, on Zulip):

@simulacrum anyway, what I was saying is ... if you don't have time don't worry, the tool is not hard, I can totally invest time and try to explain to the people that needs to know about it

simulacrum (Feb 13 2020 at 16:26, on Zulip):

If folks want to throw questions into this thread I can try and spend some time answering async when I have a few minutes (e.g. waiting for the compiler); I don't know that I have a larger block of time that I can dedicate

Santiago Pastorino (Feb 13 2020 at 16:27, on Zulip):

:+1:

Santiago Pastorino (Feb 13 2020 at 16:27, on Zulip):

my end goal with this is to start building a team of maintainers for the tool with myself included :)

Santiago Pastorino (Feb 13 2020 at 16:28, on Zulip):

but right now I can't answer certain questions like @lqd's one without just diving and investigating the whole thing

Chris Simpkins (Feb 13 2020 at 16:54, on Zulip):

Is it possible to explain the nightly bisect algorithm? It looks like there is an initial stage of testing to identify the ~ first good nightly with jump values that depend upon the duration of the full test range? And then that range is passed to the least_satisfying routine which, to someone without extensive algo background, looks binary search'ish? And this stage 2 search hones in on the commit regression point once the nightly regression is identified?

simulacrum (Feb 14 2020 at 03:13, on Zulip):

So the nightly algorithm, roughly proceeds like this:

First, we select a start and end date. Perhaps confusingly, these are laid out over time -- i.e., end is closest to today. The precise algorithm is a bit pulled together -- mostly "tries to work" no matter what options are given.

Now we try to find the baseline nightly -- this is the "left edge" that we're going to narrow down later. To do so, we start at either --start or --end or today/installed nightly (this seems like a really odd list, to be honest, and I suspect there's bugs here) and then search backwards from that date in increasing jumps depending on how far we've jumped so far.

Overall this just is a complicated looking geometric(ish?) search.

Once we have a baseline, we narrow the other end towards us with the "binary" search -- as with all binary searches, this is not going to work if the underlying function isn't sorted (in our case monotonic from baseline -> regressed).

The slight complication is that it looks like there's some custom code in the least_satisfying function that tries to skip over "unknown" results (e.g., if you land on a nightly that's just broken for other reasons). I don't remember that at all, so can't clarify more here :)

and that's pretty much it for the nightly thing

simulacrum (Feb 14 2020 at 03:14, on Zulip):

@Santiago Pastorino tbh, I think "why do we do X?" are probably better questions at this point than "how does X work?" -- the code itself is pretty far out of cache, but (e.g.) algorithmic choices I probably remember better

Chris Simpkins (Feb 14 2020 at 04:07, on Zulip):

@simulacrum Tyvm for the detailed explanation!

Chris Simpkins (Feb 14 2020 at 04:11, on Zulip):

(this seems like a really odd list, to be honest, and I suspect there's bugs here)

I am seeing cases of duplication at the initial round of checks to confirm that the regression does not occur at the start end of the tests. See https://rust-lang.zulipchat.com/#narrow/stream/217417-t-compiler.2Fcargo-bisect-rustc/topic/Duplicate.20start.20date.20nightly.20pulls.20.2B.20tests. The same nightly is tested twice when there is no regression identified, once in main.rs and once in least_satisfying.rs.

simulacrum (Feb 14 2020 at 12:35, on Zulip):

yes, there's an open issue or PR

Santiago Pastorino (Feb 14 2020 at 14:48, on Zulip):

@simulacrum this is a great start, I'd now encourage everyone to ask questions like why do we do X

Santiago Pastorino (Feb 14 2020 at 14:50, on Zulip):

so one thing that I have seen and I'm not sure what we are doing is ... the way we install/remove installations

Santiago Pastorino (Feb 14 2020 at 14:50, on Zulip):

when would you find something installed in your toolchain?

Santiago Pastorino (Feb 14 2020 at 14:50, on Zulip):

when would you find things locally?

Santiago Pastorino (Feb 14 2020 at 14:50, on Zulip):

why are local things not removed?

Santiago Pastorino (Feb 14 2020 at 14:51, on Zulip):

what's the difference with --preserve given that things end anyway around in some cases

Santiago Pastorino (Feb 14 2020 at 14:51, on Zulip):

I don't know in which cases things are not removed but noticed that something like that happens from time to time

Santiago Pastorino (Feb 14 2020 at 14:52, on Zulip):

and also have seen as someone noticed, cargo-bisect-rustc installing the same nightly twice and things like that

simulacrum (Feb 14 2020 at 15:02, on Zulip):

The design that currently exists is basically let's install everything always - I'm not sure we ever try to not reinstall things etc. By default we try to clean up eagerly to avoid leaving gigabytes of data aroun

Chris Simpkins (Feb 14 2020 at 15:18, on Zulip):

Open to refactoring much of what is in main.rs into a library & adding tests? This would allow us to explore expected behavior in detail. I have time to commit to this.

simulacrum (Feb 14 2020 at 15:29, on Zulip):

I likely don't have time to review, so I'll leave that up to @Santiago Pastorino

Chris Simpkins (Feb 14 2020 at 17:51, on Zulip):

refactoring much of what is in main.rs into a library & adding tests

@Santiago Pastorino interested?

Santiago Pastorino (Feb 15 2020 at 18:02, on Zulip):

Chris Simpkins said:

refactoring much of what is in main.rs into a library & adding tests

Santiago Pastorino interested?

yes, definitely

Chris Simpkins (Feb 16 2020 at 00:26, on Zulip):

Santiago Pastorino said:

Chris Simpkins said:

refactoring much of what is in main.rs into a library & adding tests

Santiago Pastorino interested?

yes, definitely

Sounds good. Let me know how it will work best to collaborate on it. I would be happy to take the first shot at the library refactor or take on some part of it. The part that I understand the best at this point is the nightly testing approach, but I am fine working on any of it. I should have time later this week.

Chris Simpkins (Feb 22 2020 at 23:02, on Zulip):

Santiago Pastorino said:

Chris Simpkins said:

refactoring much of what is in main.rs into a library & adding tests

Santiago Pastorino interested?

yes, definitely

https://github.com/rust-lang/cargo-bisect-rustc/pull/58 refactors all of the custom errors into a new module and broadens error handling across all sources. This eliminates nearly all of the current panic macros and my goal is to eliminate all of them so that users receive non-zero exit status codes with stderr error string reporting rather than a panic. At the moment, this will address two open issues about panics during testing.

I could use input on how to address the error handling through the least_satisfying closure if anyone has thoughts (lines indicated in the OP of the PR). Errors during tests in the least_satisfying function will still panic at this stage.

Last update: Nov 25 2020 at 02:15UTC