Stream: general

Topic: CI failure help


Jack Huey (Apr 30 2020 at 16:19, on Zulip):

Can someone help me figure out why CI failed on https://github.com/rust-lang/rust/pull/69406

Charles Lew (Apr 30 2020 at 16:46, on Zulip):

(signal: 9, SIGKILL: kill) , maybe it got OOM-ed?

DPC (Apr 30 2020 at 17:12, on Zulip):

checking it :slight_smile:

Jack Huey (Apr 30 2020 at 17:25, on Zulip):

DPC said:

checking it :)

Thanks :)

RalfJ (Apr 30 2020 at 18:10, on Zulip):

I also saw signal 9 in a rollup

RalfJ (Apr 30 2020 at 18:10, on Zulip):

maybe azure was rebooting some machines...

Jack Huey (Apr 30 2020 at 18:13, on Zulip):

Ok, I have another commit to push, so we'll see if it works this time

marmeladema (Apr 30 2020 at 18:59, on Zulip):

By the way, i've heard than rust-lang might switch entirely from Azure to GHA at some point, how is that going? I've seen more GHA builds popping up recently and they seems so much faster

Jack Huey (Apr 30 2020 at 19:02, on Zulip):

Same thing happened :/

Jack Huey (Apr 30 2020 at 19:08, on Zulip):

It's also sort of weird that it passes on x86_64-gnu-llvm-8

Jack Huey (Apr 30 2020 at 19:09, on Zulip):

But also, with GHA, I can't see the log of mingw check

Jack Huey (Apr 30 2020 at 19:28, on Zulip):

@DPC any idea?

DPC (Apr 30 2020 at 19:28, on Zulip):

uhmm

DPC (Apr 30 2020 at 19:28, on Zulip):

sorry forgot xD checking now

Jack Huey (Apr 30 2020 at 19:28, on Zulip):

no worries

DPC (Apr 30 2020 at 19:28, on Zulip):

let me know if it fails again

Jack Huey (Apr 30 2020 at 19:28, on Zulip):

It did

Jack Huey (Apr 30 2020 at 19:29, on Zulip):

https://github.com/rust-lang/rust/pull/69406/checks?check_run_id=634226168

DPC (Apr 30 2020 at 19:32, on Zulip):

rustc_middle is failing :O

Jack Huey (Apr 30 2020 at 19:33, on Zulip):

Yeah and I don't know why

DPC (Apr 30 2020 at 19:37, on Zulip):

could be spurious not sure

Jack Huey (Apr 30 2020 at 19:38, on Zulip):

twice?

DPC (Apr 30 2020 at 19:45, on Zulip):

third time the charm xD

DPC (Apr 30 2020 at 19:46, on Zulip):

uhm you pushed again?

DPC (Apr 30 2020 at 19:47, on Zulip):

ah that was one hour ago, let's wait for the ci to finish and see what happens

Jack Huey (Apr 30 2020 at 19:51, on Zulip):

I haven't pushed since it failed the second time

Jack Huey (Apr 30 2020 at 20:07, on Zulip):

I also double checked and everything builds/passes locally

Jonas Schievink (Apr 30 2020 at 20:10, on Zulip):

SIGKILL does sound like OOM, have you checked if the max RSS for building rustc_middle perhaps increased by a lot?

Jack Huey (Apr 30 2020 at 20:15, on Zulip):

I haven't. How can I check that

Jack Huey (Apr 30 2020 at 20:15, on Zulip):

(other than just watching memory)

Jonas Schievink (Apr 30 2020 at 20:17, on Zulip):

time on the rustc invocation should be able to tell you

Jonas Schievink (Apr 30 2020 at 20:17, on Zulip):

might depend on the version/variant of time though

Jack Huey (Apr 30 2020 at 20:19, on Zulip):

ah right

Jack Huey (Apr 30 2020 at 20:33, on Zulip):

2870728maxresident

Jack Huey (Apr 30 2020 at 20:33, on Zulip):

just running cargo build in librustc_middle

Jonas Schievink (Apr 30 2020 at 20:40, on Zulip):

so 2.8 GB, how much does it use on master?

Jack Huey (Apr 30 2020 at 20:40, on Zulip):

let me check

Jack Huey (Apr 30 2020 at 20:43, on Zulip):

2931620maxresident

Jack Huey (Apr 30 2020 at 20:43, on Zulip):

So basically the same

simulacrum (Apr 30 2020 at 20:44, on Zulip):

have you tried src/ci/docker/run.sh mingw-check locally?

simulacrum (Apr 30 2020 at 20:44, on Zulip):

that'll be as close as you can get to CI

Jack Huey (Apr 30 2020 at 20:46, on Zulip):

I'm technically on a server I don't have access to docker on

simulacrum (Apr 30 2020 at 20:51, on Zulip):

hm okay

simulacrum (Apr 30 2020 at 20:51, on Zulip):

I'm not sure what else to suggest :)

simulacrum (Apr 30 2020 at 20:51, on Zulip):

I guess I can check it out locally, hold on

Jack Huey (Apr 30 2020 at 20:52, on Zulip):

I'm gonna actually pull it locally and try to do that

Jack Huey (Apr 30 2020 at 20:52, on Zulip):

It'll just take a bit

Jack Huey (Apr 30 2020 at 20:52, on Zulip):

Other than that, I'm not quite sure.

simulacrum (Apr 30 2020 at 20:58, on Zulip):

locally I'm seeing memory growing steadily, at 4.6GB

simulacrum (Apr 30 2020 at 20:59, on Zulip):

5.5GB...

simulacrum (Apr 30 2020 at 21:00, on Zulip):

@Jack Huey yeah I'm guessing something in the bump caused a regression, possibly try enabling parallel-compiler locally

simulacrum (Apr 30 2020 at 21:00, on Zulip):

I'm at 6.6GB memory

Jack Huey (Apr 30 2020 at 21:00, on Zulip):

Oh goodness

simulacrum (Apr 30 2020 at 21:01, on Zulip):

7.6GB...

simulacrum (Apr 30 2020 at 21:01, on Zulip):

it's just slowly growing

simulacrum (Apr 30 2020 at 21:01, on Zulip):

I don't have debug symbols I think

Jack Huey (Apr 30 2020 at 21:02, on Zulip):

sounds like that's the problem

simulacrum (Apr 30 2020 at 21:02, on Zulip):

ah okay gdb worked

simulacrum (Apr 30 2020 at 21:03, on Zulip):
#0  0x00007f3ff38a4c1d in rustc_data_structures::obligation_forest::ObligationForest$LT$O$GT$::register_obligation_at::h11432dda8bed9387 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
(gdb) bt
#0  0x00007f3ff38a4c1d in rustc_data_structures::obligation_forest::ObligationForest$LT$O$GT$::register_obligation_at::h11432dda8bed9387 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#1  0x00007f3ff38a361b in rustc_data_structures::obligation_forest::ObligationForest$LT$O$GT$::process_obligations::hb8ce047e233f85e1 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#2  0x00007f3ff38080b1 in _$LT$rustc_trait_selection..traits..fulfill..FulfillmentContext$u20$as$u20$rustc_infer..traits..engine..TraitEngine$GT$::select_where_possible::h6726f0946fe06511 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#3  0x00007f3ff3807ddd in _$LT$rustc_trait_selection..traits..fulfill..FulfillmentContext$u20$as$u20$rustc_infer..traits..engine..TraitEngine$GT$::select_all_or_error::hc970d0d18b418df5 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#4  0x00007f3ff24164b2 in rustc_typeck::check::FnCtxt::select_all_obligations_or_error::hea0faa68c3ebb13e ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#5  0x00007f3ff24bea2b in rustc_infer::infer::InferCtxtBuilder::enter::h07cc7000c6485bc3 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#6  0x00007f3ff2404927 in rustc_typeck::check::typeck_tables_of::hc11332e26a044801 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#7  0x00007f3ff25ca5f2 in rustc_middle::ty::query::_$LT$impl$u20$rustc_query_system..query..config..QueryAccessors$LT$rustc_middle..ty..context..TyCtxt$GT$$u20$for$u20$rustc_middle..ty..query..queries..typeck_tables_of$GT$::compute::hecf6b31dbb567612 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#8  0x00007f3ff244ee44 in rustc_query_system::dep_graph::graph::DepGraph$LT$K$GT$::with_task_impl::h8aab6675969080c5 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#9  0x00007f3ff25f0097 in rustc_query_system::query::plumbing::get_query::h2ae7de4d18b5bea5 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#10 0x00007f3ff25cc6e0 in rustc_query_system::query::plumbing::ensure_query::h9b5764da1763b6e7 ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
#11 0x00007f3ff240419c in rustc_typeck::check::typeck_item_bodies::haaa829cb10ffc6ea ()
   from /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/../lib/librustc_driver-1fd34dae883ba463.so
simulacrum (Apr 30 2020 at 21:03, on Zulip):

note that this is the beta compiler trying to cargo check rustc_middle on your branch

simulacrum (Apr 30 2020 at 21:04, on Zulip):

8.5gb... anyway, too much :)

Jack Huey (Apr 30 2020 at 21:04, on Zulip):

yeah, definitely a memory issue

Jack Huey (Apr 30 2020 at 21:04, on Zulip):

just not entirely sure what's causing it

simulacrum (Apr 30 2020 at 21:06, on Zulip):

it is a huge pr

simulacrum (Apr 30 2020 at 21:06, on Zulip):

let me check if it is indeed a parallel compiler issue

Jack Huey (Apr 30 2020 at 21:09, on Zulip):

The only thing I can think it might be is the interning: https://github.com/rust-lang/rust/pull/69406/files#diff-1bffadcc6a4ac0564af0f075eb094eb1R99

simulacrum (Apr 30 2020 at 21:09, on Zulip):

seems like it was indeed a parallel compiler thing

simulacrum (Apr 30 2020 at 21:10, on Zulip):

so if you add parallel-compiler=true to your config.toml you can probably reproduce locally

Jack Huey (Apr 30 2020 at 21:10, on Zulip):

okay let me check

Jack Huey (Apr 30 2020 at 21:17, on Zulip):

yeah I'm at 4gb right now

Jack Huey (Apr 30 2020 at 21:21, on Zulip):

Thanks for the help

simulacrum (Apr 30 2020 at 21:22, on Zulip):

sure thing!

Last update: May 29 2020 at 16:45UTC