Stream: t-compiler

Topic: half of test suite fails since time immemorial


pnkfelix (Sep 13 2018 at 15:27, on Zulip):

@nagisa wrote this during today's weekly meeting:

for me half of the test suite consistently fails all the time since time immemorial

pnkfelix (Sep 13 2018 at 15:27, on Zulip):

Now, I too often see test failures (especially ones related to LTO)

pnkfelix (Sep 13 2018 at 15:28, on Zulip):

but the scenario alluded to above seems particularly bad

pnkfelix (Sep 13 2018 at 15:28, on Zulip):

and I don't just mean in terms of personal frustration and/or UX

pnkfelix (Sep 13 2018 at 15:28, on Zulip):

I mean in terms of bugs that are not caught early (and are thus left to travis to catch), simply because the tests aren't ... portable?

qmx (Sep 13 2018 at 15:29, on Zulip):

yup, I have a constant grievance with our test suite

pnkfelix (Sep 13 2018 at 15:29, on Zulip):

@nagisa can you provide more info on this? What kinds of tests? And/or hypothetheses about why, e.g. is it some particularlity of your platform?

pnkfelix (Sep 13 2018 at 15:29, on Zulip):

@qmx anyone who's experiencing similar issues should feel free to chime in

qmx (Sep 13 2018 at 15:29, on Zulip):

specially with gdb tests and things that need root permissions and are not obvious at first sight

pnkfelix (Sep 13 2018 at 15:29, on Zulip):

ah okay, that's a good qualifier

qmx (Sep 13 2018 at 15:30, on Zulip):

so run-make-fulldeps is the one that fails more frequently

pnkfelix (Sep 13 2018 at 15:30, on Zulip):

I myself tend to do my regular testing with x.py test src/test/{compile-fail,ui,run-pass,mir-opt}

qmx (Sep 13 2018 at 15:30, on Zulip):

followed by anytime I dare to run the tests in incremental mode

pnkfelix (Sep 13 2018 at 15:30, on Zulip):

so now I can understand why I'm not seeing the failures described by others

davidtwco (Sep 13 2018 at 15:31, on Zulip):

Generally, I only ever run compile-fail, run-pass or ui locally - unless something fails in another suite on Travis. Every few months I get a failure in UI or something like that locally that doesn't happen on Travis and I ignore. Eventually it goes away.

qmx (Sep 13 2018 at 15:31, on Zulip):

then it's random crashes, and I just go and do a full clean build to get the tests to pass

varkor (Sep 13 2018 at 15:45, on Zulip):

the main offenders for me are https://github.com/rust-lang/rust/issues/52528

varkor (Sep 13 2018 at 15:46, on Zulip):

if it gets past those, it's usually fine iirc

nikomatsakis (Sep 13 2018 at 16:11, on Zulip):

specially with gdb tests and things that need root permissions and are not obvious at first sight

I do have problems with gdb tests too, I don't know how to make them work

nikomatsakis (Sep 13 2018 at 16:12, on Zulip):

I use incremental a lot, though; @qmx what problems does incremental cause for you?

qmx (Sep 13 2018 at 16:12, on Zulip):

just that sometimes 50% of the test suite fails at random

qmx (Sep 13 2018 at 16:13, on Zulip):

then I know it's incremental and something got busted

qmx (Sep 13 2018 at 16:13, on Zulip):

never looked into figuring out why

nikomatsakis (Sep 13 2018 at 16:13, on Zulip):

huh, ok

nagisa (Sep 13 2018 at 17:39, on Zulip):

@pnkfelix my failures are usually related to system setup

nagisa (Sep 13 2018 at 17:39, on Zulip):

such as debuginfo tests that always fail due to some gdb incompatibility

nagisa (Sep 13 2018 at 17:40, on Zulip):

there never was a time when I could just run make test and now x.py test and have it just work.

nagisa (Sep 13 2018 at 17:42, on Zulip):

I also have a few run-pass tests that consistently fail, but I do not remember which ones. I suspect them to be related to system setup as well

nagisa (Sep 13 2018 at 17:43, on Zulip):

I don’t bother investigating them much as my work usually does not need the whole test suite to pass for me to be confident in my work (I usually just run x.py test src/test/suite --test-args=certain-test(s).

pnkfelix (Sep 14 2018 at 15:49, on Zulip):

BTW @varkor and @Oli , I don't know how well you two manage your Github notifications (i know I simply cannot keep up with mine), but if you get a chance to respond to https://github.com/rust-lang/rust/issues/52528#issuecomment-421397048 that would be great

nagisa (Sep 17 2018 at 18:07, on Zulip):

never ran tests without optimisation either.

Last update: Nov 20 2019 at 01:30UTC