Stream: t-compiler

Topic: cargo-bisect as a bot


Félix Fischer (May 02 2020 at 20:17, on Zulip):

How crazy would it be to have cargo-bisect-rustc as a bot we could summon on ICE issues?

The other day I had the pleasure to use it for finding the commit which caused a particular ICE, and it turned out to be pretty fast - it uses the CI artifacts of the rustc repo. It was fast even in my computer, which is by today standards kinda slow :)

I don't know how many ICE do we cover with the following scenario, but I was thinking of this sort of usage:

  1. Authorized user comments like this:
    @bisect_bot run
// Code for minimal lib.rs that causes the ICE
  1. Bot answers
    Running bisection on:
    <prints configuration>
    Use @bisect_bot stop to stop bisection

  2. Bot prints the bisection report after finishing.

Maybe not all scenarios are covered - if to reproduce the ICE one needs more .rs files or a special Cargo.toml, the runs would be kinda hard to configure inside of a github comment. That said, for those cases it could be that you do a command, the bot answers with what it thinks you want to do (like showing you the --dry-run), but without running it yet. After seeing that it's fine, you give it the confirmation and let it do the bisection for you.

Also, scenarios where you need a particular OS, special system dependencies or C libraries - those are definitely out of scope for a bot, I'd think.

simulacrum (May 02 2020 at 20:44, on Zulip):

the main reason we haven't is the (IMO) open question as to a nice UI

simulacrum (May 02 2020 at 20:45, on Zulip):

usually when I bisect I have to do it a couple times

simulacrum (May 02 2020 at 20:45, on Zulip):

but I think doing it as a zulip bot would be more viable

simulacrum (May 02 2020 at 20:45, on Zulip):

bisecting on github would be quite spammy I think

Félix Fischer (May 02 2020 at 21:09, on Zulip):

Wait... ZULIP bots exist??

That's amazing! Oh, Imma look into that. Do we have any Zulip bots in the rust streams, to check how they work UI-wise?

simulacrum (May 02 2020 at 21:09, on Zulip):

triagebot is our only one currently, really

simulacrum (May 02 2020 at 21:09, on Zulip):

that's what I'd add it to if we were to do this

Félix Fischer (May 02 2020 at 21:10, on Zulip):

simulacrum said:

triagebot is our only one currently, really

Ohh, I see. Does it have a github repo or something?

simulacrum (May 02 2020 at 21:10, on Zulip):

https://github.com/rust-lang/triagebot/

bjorn3 (May 03 2020 at 08:33, on Zulip):

I am currently experimenting with this.

bjorn3 (May 03 2020 at 09:53, on Zulip):

It now creates a bisect job when asked in a comment. Next step: report the result back.

Chris Simpkins (May 03 2020 at 14:16, on Zulip):

:+1:

Chris Simpkins (May 03 2020 at 14:41, on Zulip):

@bjorn3 @Félix Fischer Do we need to update the output format to support this? This is a great idea!

bjorn3 (May 03 2020 at 14:42, on Zulip):

My plan currently is to just comment the raw log with just some lines filtered, like the download messages.

Chris Simpkins (May 03 2020 at 14:45, on Zulip):

I just mentioned to @simulacrum that the full report that we generate is not pasted into the repo threads. Most tend to manually add a line indicating that these data were generated by bisect-rustc and this block of text:

searched nightlies: from nightly-2020-04-26 to nightly-2020-04-27
regressed nightly: nightly-2020-04-27
searched commits: from rust-lang/rust@0862458 to rust-lang/rust@e83f756
regressed commit: rust-lang/rust@b592b37
source code: URL OF A REPOSITORY THAT REPRODUCES THE ERROR

or just this:

regressed commit: rust-lang/rust@b592b37

https://github.com/rust-lang/cargo-bisect-rustc/pull/79#issuecomment-623119788

Chris Simpkins (May 03 2020 at 14:46, on Zulip):

It might be worth considering changes on the tool side.

bjorn3 (May 03 2020 at 14:50, on Zulip):

It would be nice if the progress bar was refreshed much less often. Eg 1 time per second. Every time the progress bar gets refreshed, github actions shows another line, which really slows down my browser.

Chris Simpkins (May 03 2020 at 15:07, on Zulip):

How are you filtering the std stream output?

bjorn3 (May 03 2020 at 15:08, on Zulip):

Initially I didn't. Now I do | grep -v "for x86_64-unknown-linux-gnu" to filter the download progress messages.

simulacrum (May 03 2020 at 15:09, on Zulip):

@bjorn3 hm, I would expect that the progress bar should be hidden entirely? we shouldn't show it if there's not a tty attached

bjorn3 (May 03 2020 at 15:09, on Zulip):

On github actions, a tty is attached

simulacrum (May 03 2020 at 15:10, on Zulip):

That is... surprising. Maybe there's some terminfo detection we should be doing

simulacrum (May 03 2020 at 15:10, on Zulip):

or we can env::var_os("GITHUB_CI") or whatever

bjorn3 (May 03 2020 at 15:10, on Zulip):

By the way, does anybody know of a proxy to localhost that doesn't just allow json messages? I currently use smee.io, but it doesn't allow me to upload the log.

Chris Simpkins (May 03 2020 at 15:11, on Zulip):

Or a --verbose vs. --quiet approach that eliminates some elements of the output

simulacrum (May 03 2020 at 15:11, on Zulip):

I use ngrok

simulacrum (May 03 2020 at 15:11, on Zulip):

it doesn't allow uploading logs though either I think

bjorn3 (May 03 2020 at 15:11, on Zulip):

I can try

Chris Simpkins (May 03 2020 at 15:20, on Zulip):

I proposed ANSI escape code output in a new PR on the bisect-rustc repository. This should not affect any lines that you are filtering out. It does add escape codes to the interval reporting on the nightly and commit where a regression is identified. I don't think that this will be an issue unless you are trying to filter on inclusion of that information in an Actions report. The final report is not formatted with escape codes. Let me know if that causes problems for any reason.

bjorn3 (May 03 2020 at 15:33, on Zulip):

When piping the output through grep, it will not use ANSI escape codes, right?

bjorn3 (May 03 2020 at 15:41, on Zulip):

Can cargo bisect-rustc use the github api to get the list of commits, instead of cloning the rust repo? That would save about 3min.

bjorn3 (May 03 2020 at 15:50, on Zulip):

simulacrum said:

it doesn't allow uploading logs though either I think

if the log is short, it allows it, but otherwise it just removes the body.

bjorn3 (May 03 2020 at 15:50, on Zulip):

@simulacrum any ideas how to get the log back to the bot?

simulacrum (May 03 2020 at 15:51, on Zulip):

@bjorn3 it should use https://github.com/rust-lang/cargo-bisect-rustc/issues/77 not github api, but yes, that should be instant

simulacrum (May 03 2020 at 15:51, on Zulip):

@bjorn3 are you planning on running bisection in github actions?

bjorn3 (May 03 2020 at 15:51, on Zulip):

yes

simulacrum (May 03 2020 at 15:52, on Zulip):

I know Pietro had trouble getting access to those logs programmatically

simulacrum (May 03 2020 at 15:52, on Zulip):

maybe we can just link to the run?

simulacrum (May 03 2020 at 15:52, on Zulip):

I would start there

bjorn3 (May 03 2020 at 16:24, on Zulip):

Currently testing the whole pipeline

bjorn3 (May 03 2020 at 16:32, on Zulip):

@simulacrum

image.png

bjorn3 (May 03 2020 at 16:33, on Zulip):

@Félix Fischer Your wish is my command :) ^^^

simulacrum (May 03 2020 at 16:36, on Zulip):

now we just need that integrated into zulip -- I don't want that kind of noise in the less-interactive github

bjorn3 (May 03 2020 at 16:38, on Zulip):

sure, will try that

bjorn3 (May 03 2020 at 16:41, on Zulip):

@simulacrum

Only organization administrators can add bots to this organization

bjorn3 (May 03 2020 at 16:48, on Zulip):

According to the docs, you need to create a bot to access the api. Because I can't create a bot, I can't test the bisect bot once I have programmed something.

bjorn3 (May 03 2020 at 16:59, on Zulip):

Ah, there exists a personal api key

simulacrum (May 03 2020 at 17:01, on Zulip):

oh, well, we'll probably want it as part of triagebot eventually -- just makes sense to avoid having two separate auth implementations and such

simulacrum (May 03 2020 at 17:02, on Zulip):

but yeah if you want to try it out first then that'd be great, I'd love to review a pr

Chris Simpkins (May 03 2020 at 17:41, on Zulip):

Very nice @bjorn3!

Chris Simpkins (May 03 2020 at 17:46, on Zulip):

If the question wasn't answered above you can access Github API with --access=github

Chris Simpkins (May 03 2020 at 17:46, on Zulip):

the default is to clone

Chris Simpkins (May 03 2020 at 17:47, on Zulip):

cc @pnkfelix @Santiago Pastorino re new bot for bisect-rustc

bjorn3 (May 03 2020 at 17:48, on Zulip):

Chris Simpkins said:

If the question wasn't answered above you can access Github API with --access=github

will try

Chris Simpkins (May 03 2020 at 17:49, on Zulip):

it is much faster

bjorn3 (May 03 2020 at 19:04, on Zulip):

Chris Simpkins said:

it is much faster

indeed 1 min vs 4min total bisection time for my test

bjorn3 (May 03 2020 at 19:07, on Zulip):

@simulacrum Here is the zulip bot you wanted:

image.png

simulacrum (May 03 2020 at 19:08, on Zulip):

Looks great! Is that pushed somewhere? I'd be happy to add the bot manually for now too while we figure out triagebot Integration

bjorn3 (May 03 2020 at 19:09, on Zulip):

Currently a private repo, together with the github version. Will make it public in a few minutes.

bjorn3 (May 03 2020 at 19:13, on Zulip):

https://github.com/bjorn3/cargo-bisect-rustc-bot

bjorn3 (May 03 2020 at 19:14, on Zulip):

It works by listening for a comment. It then pushes a commit to https://github.com/bjorn3/cargo-bisect-rustc-bot-jobs. When the check run for that commit finishes, it reads the commit message to determine the correct location to post a message.

bjorn3 (May 03 2020 at 19:14, on Zulip):

It is stateless with the exception of the zulip message listener, which remembers the message queue it registered and the last message id it read.

bjorn3 (May 03 2020 at 19:15, on Zulip):

@simulacrum ^

bjorn3 (May 03 2020 at 19:15, on Zulip):

The code will need cleanup, which I may do tomorrow.

simulacrum (May 03 2020 at 19:16, on Zulip):

Seems reasonable, yeah. I would love a PR to triagebot if you have the time and if not I can add it to my queue :)

bjorn3 (May 03 2020 at 19:17, on Zulip):

I don't know if I have time yet, but I will try.

simulacrum (May 03 2020 at 19:17, on Zulip):

Just file an issue please if not

bjorn3 (May 04 2020 at 11:55, on Zulip):

@simulacrum How do I create and push a commit from triagebot?

simulacrum (May 04 2020 at 12:11, on Zulip):

Hm, probably using the GitHub API

simulacrum (May 04 2020 at 12:12, on Zulip):

(we don't have the functions defined yet, but that's how I'd do it I think)

bjorn3 (May 04 2020 at 12:13, on Zulip):

Does triagebot have permission to write to a repo and permission to change github actions workflows?

bjorn3 (May 04 2020 at 12:13, on Zulip):

The workflow permission is not really necessary, but right now I create a commit from scratch without parent.

simulacrum (May 04 2020 at 13:03, on Zulip):

@bjorn3 We can grant such permissions if necessary to the user account it uses

bjorn3 (May 04 2020 at 13:05, on Zulip):

Ok. Almost got creating a commit using the api working.

bjorn3 (May 04 2020 at 13:14, on Zulip):

Got it working

simulacrum (May 04 2020 at 13:16, on Zulip):

I won't have time today to dig in probably but if you post a PR I'll get to it eventually

bjorn3 (May 04 2020 at 13:17, on Zulip):

I will still need to integrate it with triagebot

pnkfelix (May 04 2020 at 15:26, on Zulip):

bjorn3 said:

Can cargo bisect-rustc use the github api to get the list of commits, instead of cloning the rust repo? That would save about 3min.

Did anyone point you at cargo-bisect-rustc --access ?

pnkfelix (May 04 2020 at 15:27, on Zulip):

oh yes, someone did eventually. sorry for the noise.

pnkfelix (May 04 2020 at 15:29, on Zulip):

(and now I've cross-linked to the #t-compiler/cargo-bisect-rustc stream, here https://rust-lang.zulipchat.com/#narrow/stream/217417-t-compiler.2Fcargo-bisect-rustc/topic/bisect.20via.20bot/near/196196973 )

bjorn3 (May 04 2020 at 15:30, on Zulip):

Should I continue there?

pnkfelix (May 04 2020 at 15:31, on Zulip):

I'm honestly not sure. Is the thing your adding mostly a bot that calls out to cargo-bisect-rustc as if it were just another user?

pnkfelix (May 04 2020 at 15:31, on Zulip):

Or is it making modifications to cargo-bisect-rustc itself (or making direct calls into its internals)?

pnkfelix (May 04 2020 at 15:32, on Zulip):

Or, does it motivate making more changes to cargo-bisect-rustc (in order to enable functionality that so-called "normal users" would not need nor care about) ?

bjorn3 (May 04 2020 at 15:32, on Zulip):

It is a bot that runs cargo-bisect-rustc unmodified.

pnkfelix (May 04 2020 at 15:32, on Zulip):

Okay. Then I suspect we don't have to move the conversation over.

pnkfelix (May 04 2020 at 15:33, on Zulip):

If I were to move it anywhere, it would be somewhere dedicate to bot development

bjorn3 (May 04 2020 at 15:33, on Zulip):

Currently it just accepts jobs and posts a link to the log when the job is finished.

pnkfelix (May 04 2020 at 15:33, on Zulip):

but I don't think we have a dedicated WG for that...

simulacrum (May 04 2020 at 15:52, on Zulip):

#t-release/triagebot is maybe reasonable as a place but :shrug:

Santiago Pastorino (May 04 2020 at 16:04, on Zulip):

just got some time to read this, this is awesome :)

Santiago Pastorino (May 04 2020 at 16:05, on Zulip):

is it integrated with triagebot already?

Santiago Pastorino (May 04 2020 at 16:05, on Zulip):

or how can we use it right now?

simulacrum (May 04 2020 at 16:08, on Zulip):

no not yet integrated

LeSeulArtichaut (May 04 2020 at 16:35, on Zulip):

bjorn3 said:

Ok. Almost got creating a commit using the api working.
Got it working
I will still need to integrate it with triagebot

Santiago Pastorino (May 04 2020 at 17:00, on Zulip):

:+1:

bjorn3 (May 04 2020 at 17:17, on Zulip):

test comment to see which fields a zulip event has when a comment is posted in a channel

bjorn3 (May 04 2020 at 17:42, on Zulip):

is it ok if I test my bot once on this channel? i have already tested for private messages to myself, but public messages use a slightly different api

simulacrum (May 04 2020 at 17:42, on Zulip):

@bjorn3 go for it!

bjorn3 (May 04 2020 at 17:43, on Zulip):

bisect-bot bisect start=2020-04-26 end=2020-04-28

use std::sync::{Arc, RwLock};

#[derive(Copy, Clone)]
pub struct Glfw;

fn fun_call<T>(val: T) -> T { val }

static mut GLFW: Option<Glfw> = None;
pub fn new() -> Result<Arc<RwLock<Glfw>>, Box<dyn std::error::Error>> {
    //TODO: Give the option to choose FAIL / LOG?
    let glfw = unsafe { if let Some(glfw) = GLFW {
        glfw
    } else {
        let glfw = fun_call(Glfw);
        GLFW = Some(glfw);
        glfw
    }};

    Ok(Arc::new(RwLock::new(glfw)))
}
bjorn3 (May 04 2020 at 17:43, on Zulip):

bisection job 643712863 started: https://github.com/bjorn3/cargo-bisect-rustc-bot-jobs/runs/643712863

pnkfelix (May 04 2020 at 17:44, on Zulip):

@bjorn3 I assume the bot will be able to fire off bisections in other streams, right? In particular I imagine we'll see a fair amount from #t-compiler/wg-prioritization

bjorn3 (May 04 2020 at 17:45, on Zulip):

it works in any stream i am subscribed to. i am not subscribed to that channel right now.

pnkfelix (May 04 2020 at 17:45, on Zulip):

gotcha

bjorn3 (May 04 2020 at 17:45, on Zulip):

bisection job 643712863 completed: https://github.com/bjorn3/cargo-bisect-rustc-bot-jobs/runs/643712863

bjorn3 (May 04 2020 at 18:15, on Zulip):

@pnkfelix It should now get notifications from all public streams. Can you try on #t-compiler/wg-prioritization?

bjorn3 (May 04 2020 at 18:15, on Zulip):

Note: the code block must be ```rust . ``` is not enough.

Santiago Pastorino (May 04 2020 at 18:30, on Zulip):

@bjorn3 this is huge :slight_smile:

bjorn3 (May 04 2020 at 18:31, on Zulip):

Did it work? Can't access the logs right now.

Santiago Pastorino (May 04 2020 at 18:32, on Zulip):

I didn't try it, but I meant, the ability to fire this from here is huge :)

LeSeulArtichaut (May 04 2020 at 18:33, on Zulip):

Seems like it worked indeed

Santiago Pastorino (May 04 2020 at 18:34, on Zulip):

I thought @bjorn3 was asking if the thing worked on #t-compiler/wg-prioritization

bjorn3 (May 04 2020 at 18:34, on Zulip):

Indeed

Santiago Pastorino (May 04 2020 at 18:34, on Zulip):

LeSeulArtichaut said:

Seems like it worked indeed

I guess you were saying that it worked here?

Santiago Pastorino (May 04 2020 at 18:35, on Zulip):

because I didn't see any bisection fired on #t-compiler/wg-prioritization

bjorn3 (May 04 2020 at 18:35, on Zulip):

Neither do I see a branch newer than an hour at https://github.com/bjorn3/cargo-bisect-rustc-bot-jobs/branches

LeSeulArtichaut (May 04 2020 at 18:37, on Zulip):

Sorry I thought you were talking about the bisection in this channel

LeSeulArtichaut (May 04 2020 at 18:38, on Zulip):

/me has to stop dropping in conversations

bjorn3 (May 04 2020 at 18:54, on Zulip):

can somebody post a bisect command at #t-compiler/wg-prioritization, so I can see if it works on channels I am not subscribed to?

simulacrum (May 04 2020 at 19:01, on Zulip):

doing

simulacrum (May 04 2020 at 19:03, on Zulip):

@bjorn3 didn't seem to work: https://rust-lang.zulipchat.com/#narrow/stream/227806-t-compiler.2Fwg-prioritization/topic/test.20bisectbot

bjorn3 (May 04 2020 at 19:03, on Zulip):

Yeah, don't see it in the logs either

bjorn3 (May 04 2020 at 19:04, on Zulip):

Can you try again? I changed the code and I think I saw a message from a stream that I am not subscribed to.

simulacrum (May 04 2020 at 19:06, on Zulip):

yeah

bjorn3 (May 04 2020 at 19:06, on Zulip):

It's doing it's job!

simulacrum (May 04 2020 at 19:06, on Zulip):

:tada:

bjorn3 (May 04 2020 at 19:06, on Zulip):

You see the start message, right?

simulacrum (May 04 2020 at 19:08, on Zulip):

yes looks great to me

bjorn3 (May 04 2020 at 19:10, on Zulip):

I don't think I will be able to work on the for the next few days to merge it into triagebot, but I can host it on heroku for now if you want. They have a free teer.

bjorn3 (May 04 2020 at 19:21, on Zulip):

Wait, that won't work for the zulip side, as heroku only runs the program when a http request has been received recently.

bjorn3 (May 04 2020 at 19:50, on Zulip):

I have it running at heroku now. Posting the bisection command at https://github.com/bjorn3/cargo-bisect-rustc-bot/issues/1 works.

bjorn3 (May 04 2020 at 19:51, on Zulip):

If you recently posted a comment at that repo, the zulip side also works.

bjorn3 (May 04 2020 at 19:53, on Zulip):

according to heroku it will keep running for 30min after a request. (github webhook in this case)

Félix Fischer (May 05 2020 at 06:34, on Zulip):

@bjorn3 I'm so happy that you've done this. This is so, so cool :sparkling_heart:

bjorn3 (May 05 2020 at 13:10, on Zulip):

https://github.com/rust-lang/triagebot/issues/506

Last update: May 29 2020 at 17:35UTC