Stream: t-compiler/cargo-bisect-rustc

Topic: Testing


Chris Simpkins (Feb 25 2020 at 18:42, on Zulip):

Moving this to a new thread from Release New Version so that we can continue the conversation

pnkfelix said:

wishes we had a test suite.

pnkfelix said:

actually I guess it wouldn't be that hard to make a test suite, if we're willing to use --end for all the tests, and just cross our finges that the behavior without --end does not break.

Chris Simpkins said:

I've been thinking through how to approach this. The commit level rustc caches expire at the 167 day mark so the tests covering this part of the source would need to be updated periodically.

pnkfelix said:

hmm. that is a good point.
we might at least be able to make the update process relatively painless, by using rustversion and #[rustc_error]...

pnkfelix (Feb 25 2020 at 19:34, on Zulip):

another option would be to just not attempt to test the commit level caches... :(

pnkfelix (Feb 25 2020 at 19:35, on Zulip):

or maybe we could find out if we could ... "bless" certain commits to live beyond the 167 day expiration period, solely for the purpose of supporting the test suite? That might be the simplest option here...

pnkfelix (Feb 25 2020 at 19:35, on Zulip):

@simulacrum any thought on the latter approach?

simulacrum (Feb 25 2020 at 19:42, on Zulip):

I'd just test on nightlies for now, blessing is possible but a bit painful i think

simulacrum (Feb 25 2020 at 19:45, on Zulip):

Or just updated the test once a year ish

simulacrum (Feb 25 2020 at 19:45, on Zulip):

Which might be reasonable

simulacrum (Feb 25 2020 at 19:45, on Zulip):

To test new things

pnkfelix (Feb 25 2020 at 19:49, on Zulip):

wouldn't it would need to be roughly twice a year? (365 / 167 \approx 2?)

pnkfelix (Feb 25 2020 at 19:51, on Zulip):

(and really this is the kind of thing where you need to round up, not down. :slight_smile: so ⌈365/167⌉ = 3 ...

pnkfelix (Feb 25 2020 at 19:51, on Zulip):

nonetheless, updating the tests three times a year probably is better than blessing some random commits

Chris Simpkins (Feb 25 2020 at 20:46, on Zulip):

We just need to make sure that there is at least one good regression every six months... :slight_smile:

simulacrum (Feb 25 2020 at 21:44, on Zulip):

yes, I would personally lean towards updating the test I think for now at least, long-term I think we may want to eventually have commit artifacts stick around even longer -- e.g., we'd preserve every 5th commit after 167 days

simulacrum (Feb 25 2020 at 21:51, on Zulip):

one note: we could "fake" commit bisection in theory for the tests

simulacrum (Feb 25 2020 at 21:51, on Zulip):

i.e., we separately test that we can download commits for example

simulacrum (Feb 25 2020 at 21:51, on Zulip):

but then for the actual test of commit bisection we provide a custom "url generator" or "downloader" impl that actually downloads nightlies behind the back of the commit bisection

simulacrum (Feb 25 2020 at 21:52, on Zulip):

which are, at least in theory, "just" sparse commits

simulacrum (Feb 25 2020 at 21:52, on Zulip):

i.e. everything's a rollup

Chris Simpkins (Feb 25 2020 at 23:09, on Zulip):

Could we mock the regression check part of execution with a handful of strategically designed executables in a repository that is designed to have a series of git commits in a ‘nightly’ and a roll up PR. ‘rustc’ could even be scripts that just spit out the proper exit status codes and stderr messages for the tests.

pnkfelix (Feb 26 2020 at 04:07, on Zulip):

simulacrum said:

yes, I would personally lean towards updating the test I think for now at least, long-term I think we may want to eventually have commit artifacts stick around even longer -- e.g., we'd preserve every 5th commit after 167 days

Or maybe go with a exponentially decaying archive: after 167 days, keep the 2nd, 4th, 8th, 16th, 32nd, ... commits

pnkfelix (Feb 26 2020 at 04:08, on Zulip):

(I am mostly kidding; we're keeping the nightlies anyway)

pnkfelix (Feb 26 2020 at 04:09, on Zulip):

((and hopefully someone will keep paying for that expanding storage in perpetuity ....))

simulacrum (Feb 26 2020 at 14:03, on Zulip):

yeah, I think the main reason we've not done it is it's somewhat non-trivial to do and not really worthwhile

pnkfelix (Mar 02 2020 at 21:11, on Zulip):

okay after poking around a bit, I found a crate rustversion that used a build.rs script to inspect the rustc version. I tried using that crate directly, but it takes too long to repeatedly compile it and its depedencies during bisection. But the strategy itself is easily adapted.

pnkfelix (Mar 02 2020 at 21:11, on Zulip):

I hope to have a PR with some demo tests up in a bit.

Chris Simpkins (Mar 07 2020 at 19:34, on Zulip):

It looks like this is a "stable" ICE for testing? :slight_smile: => https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0ab2bd6a9d722e0f05a95e2a5dcf89cc

Chris Simpkins (Mar 17 2020 at 20:25, on Zulip):

pnkfelix said:

okay after poking around a bit, I found a crate rustversion that used a build.rs script to inspect the rustc version. I tried using that crate directly, but it takes too long to repeatedly compile it and its depedencies during bisection. But the strategy itself is easily adapted.

@pnkfelix Do you have more details about how you intend to approach this?

pnkfelix (Mar 18 2020 at 12:21, on Zulip):

Yeah i have a demo in a branch

Last update: Apr 03 2020 at 18:05UTC