Stream: t-compiler

Topic: tricks for faster compiler compilation times


Santiago Pastorino (Jan 22 2019 at 16:26, on Zulip):

I wanted to know some trick to make compilation times faster when working on the compiler and what's the explanation for each of those

Santiago Pastorino (Jan 22 2019 at 16:26, on Zulip):

maybe we should document those? I've asked @nikomatsakis several times and keep forgeting

Santiago Pastorino (Jan 22 2019 at 16:26, on Zulip):

so my usual compilation command is ...

Santiago Pastorino (Jan 22 2019 at 16:26, on Zulip):

./x.py build -i --stage 1 src/libstd

Santiago Pastorino (Jan 22 2019 at 16:27, on Zulip):

then there's a little trick which is

Santiago Pastorino (Jan 22 2019 at 16:27, on Zulip):

./x.py build -i --stage 1 --keep-stage 1 src/libstd

nagisa (Jan 22 2019 at 16:27, on Zulip):

/me uses a widely known trick of a-lot-of-cores.

Santiago Pastorino (Jan 22 2019 at 16:27, on Zulip):

:D

Santiago Pastorino (Jan 22 2019 at 16:28, on Zulip):

I've seen Niko use ./x.py check but in cases where compilation fails it's telling me that everything is ok

Santiago Pastorino (Jan 22 2019 at 16:28, on Zulip):

can someone explain those two commands and what check exactly does?

Santiago Pastorino (Jan 22 2019 at 16:28, on Zulip):

I guess I can take the compromise to push something to rustc guides in exchange ;)

nagisa (Jan 22 2019 at 16:29, on Zulip):

well --stage 1 relates to bootstrapping. Since to fully bootstrap a compiler you need to compile it twice over, there’s a lot of unnecessary time spent compiling a kind-of unnecessary compiler as far as local development goes.

Wesley Wiser (Jan 22 2019 at 16:29, on Zulip):

check is the equivalent of cargo check. It just type checks your code but doesn't actually build anything

nagisa (Jan 22 2019 at 16:29, on Zulip):

so --stage=1 makes it compile only once.

davidtwco (Jan 22 2019 at 16:29, on Zulip):

As I understand it, check doesn't run codegen, just type check and borrow check. I'm not sure if #52565 has been fixed though - stopping me from using check.

davidtwco (Jan 22 2019 at 16:30, on Zulip):

I also use @nagisa's lots of cores technique though.

Santiago Pastorino (Jan 22 2019 at 16:30, on Zulip):

well --stage 1 relates to bootstrapping. Since to fully bootstrap a compiler you need to compile it twice over, there’s a lot of unnecessary time spent compiling a kind-of unnecessary compiler as far as local development goes.

yeah, that's the part I know :)

nagisa (Jan 22 2019 at 16:30, on Zulip):

--keep-stage=1 meanwhile avoids invalidating/rebuilding some artifacts that would otherwise be invalidated and rebuilt unnecessarily.

nagisa (Jan 22 2019 at 16:30, on Zulip):

(again, for local development purposes)

Santiago Pastorino (Jan 22 2019 at 16:30, on Zulip):

check is the equivalent of cargo check. It just type checks your code but doesn't actually build anything

for some reason build fails and check pass so my guess is that's not exactly true? in a couple of tries I did

Wesley Wiser (Jan 22 2019 at 16:31, on Zulip):

check is the equivalent of cargo check. It just type checks your code but doesn't actually build anything

for some reason build fails and check pass so my guess is that's not exactly true?

That generally shouldn't happen. What is the build failing with?

nagisa (Jan 22 2019 at 16:31, on Zulip):

-i enables incremental compilation. and src/libstd makes libstd the build target. Normally (with src/libstd being absent) there’s more stuff to be built afterwards, but not that much.

Santiago Pastorino (Jan 22 2019 at 16:32, on Zulip):

check is the equivalent of cargo check. It just type checks your code but doesn't actually build anything

for some reason build fails and check pass so my guess is that's not exactly true?

That generally shouldn't happen. What is the build failing with?

ahh maybe is because of what @davidtwco have said https://github.com/rust-lang/rust/issues/52565

Wesley Wiser (Jan 22 2019 at 16:32, on Zulip):

--keep-stage=1 meanwhile avoids invalidating/rebuilding some artifacts that would otherwise be invalidated and rebuilt unnecessarily.

What confuses me is the combination of --stage 1 and --keep-stage=1. That reads to me like nothing would be compiled.

Wesley Wiser (Jan 22 2019 at 16:32, on Zulip):

Oh, I think cargo has the same issue.

davidtwco (Jan 22 2019 at 16:32, on Zulip):

#52565 wouldn't make a x.py build fail where an x.py check didn't, I don't think.

Wesley Wiser (Jan 22 2019 at 16:33, on Zulip):

Generally, I do ./x.py check until it all compiles and I'm ready to test it and then I do ./x.py build --stage 1

Santiago Pastorino (Jan 22 2019 at 16:33, on Zulip):

--keep-stage=1 meanwhile avoids invalidating/rebuilding some artifacts that would otherwise be invalidated and rebuilt unnecessarily.

What confuses me is the combination of --stage 1 and --keep-stage=1. That reads to me like nothing would be compiled.

yeah, I remember seeing @nikomatsakis and @Alex Crichton discussing about this and believe me that makes sense, it's just that doesn't read nicely :)

lqd (Jan 22 2019 at 16:33, on Zulip):

currently for me, check and incremental collide a bit: after a check everything is rebuilt

nagisa (Jan 22 2019 at 16:34, on Zulip):

I generally develop compiler by running x.py --stage 1 test src/test/whateversuiteisappropriate and going to do my thing meanwhile

Wesley Wiser (Jan 22 2019 at 16:34, on Zulip):

I guess --keep-stage=1 means "keep the dependencies for stage 1"?

Santiago Pastorino (Jan 22 2019 at 16:34, on Zulip):

-i enables incremental compilation. and src/libstd makes libstd the build target. Normally (with src/libstd being absent) there’s more stuff to be built afterwards, but not that much.

yeah, also knew about -i, so passing src/libstd avoids compiling something? what would that be?

Wesley Wiser (Jan 22 2019 at 16:34, on Zulip):

Not "keep the results of stage 1"

nagisa (Jan 22 2019 at 16:37, on Zulip):

At some point --stage=1 meant something else.

nagisa (Jan 22 2019 at 16:37, on Zulip):

and then --keep-stage made more sense.

dlrobertson (Jan 22 2019 at 16:39, on Zulip):

Along the lines of the many cores trick. Has anyone tried compiling rust with sccache?

Santiago Pastorino (Jan 22 2019 at 16:39, on Zulip):

check is the equivalent of cargo check. It just type checks your code but doesn't actually build anything

for some reason build fails and check pass so my guess is that's not exactly true?

That generally shouldn't happen. What is the build failing with?

[santiago@archlinux rust2 (place2b)]$ RUST_BACKTRACE=full ./x.py check -i --stage 1 --keep-stage 1 src/libstd
Updating only changed submodules
Submodules updated in 0.04 seconds
    Finished dev [unoptimized] target(s) in 0.50s
Checking std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.35s
Build completed successfully in 0:00:02
[santiago@archlinux rust2 (place2b)]$ RUST_BACKTRACE=full ./x.py build -i --stage 1 --keep-stage 1 src/libstd
Updating only changed submodules
Submodules updated in 0.03 seconds
    Finished dev [unoptimized] target(s) in 0.28s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.33s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.29s
Copying stage0 test from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling rustc_mir v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_mir)
   Compiling rustc_allocator v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_allocator)
   Compiling rustc_lint v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_lint)
   Compiling rustc_traits v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_traits)
   Compiling rustc_resolve v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_resolve)
   Compiling rustc_plugin v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_plugin)
error[E0614]: type `rustc::mir::Local` cannot be dereferenced=>      ] 115/127: rustc_mir, rustc_traits, rustc_resolve, rustc_plugin
   --> src/librustc_mir/borrow_check/place_ext.rs:100:74
    |
100 |                         let ignore = !has_storage_dead_or_moved.contains(*index) &&
    |                                                                          ^^^^^^

error[E0614]: type `rustc::mir::Local` cannot be dereferenced
   --> src/librustc_mir/borrow_check/place_ext.rs:101:45
    |
101 |                             mir.local_decls[*index].mutability == Mutability::Not;
    |                                             ^^^^^^

   Compiling rustc_privacy v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_privacy)
error: aborting due to 2 previous errors========================>    ] 119/127: rustc_mir

For more information about this error, try `rustc --explain E0614`.
error: Could not compile `rustc_mir`.

To learn more, run the command again with --verbose.
command did not execute successfully: "/home/santiago/src/oss/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--features" "" "--manifest-path" "/home/santiago/src/oss/rust2/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101
failed to run: /home/santiago/src/oss/rust2/build/bootstrap/debug/bootstrap build -i --stage 1 --keep-stage 1 src/libstd
Build completed unsuccessfully in 0:00:14
dlrobertson (Jan 22 2019 at 16:39, on Zulip):

I thought I remember reading that Firefox uses it for compiling the rust crates used there

Wesley Wiser (Jan 22 2019 at 16:40, on Zulip):

I'd love to have a diagram showing the interactions between --stage x and --keep-stage=y. As it is, I never use --keep-stage because I don't really understand what it's keeping and therefore when I can or can't use it.

QuietMisdreavus (Jan 22 2019 at 16:40, on Zulip):

i think that the travis builders use sccache for llvm?

Wesley Wiser (Jan 22 2019 at 16:41, on Zulip):

./x.py check -i --stage 1 --keep-stage 1

That looks kind of suspicious to me. Do you get the same error if you just run ./x.py check -i --stage 1?

Santiago Pastorino (Jan 22 2019 at 16:42, on Zulip):
[santiago@archlinux rust2 (place2b)]$ RUST_BACKTRACE=full ./x.py check -i --stage 1 src/libstd
Updating only changed submodules
Submodules updated in 0.04 seconds
    Finished dev [unoptimized] target(s) in 0.31s
Checking std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.34s
Build completed successfully in 0:00:02
Santiago Pastorino (Jan 22 2019 at 16:42, on Zulip):

it shouldn't build but builds

Santiago Pastorino (Jan 22 2019 at 16:43, on Zulip):

I need to remove -i

Wesley Wiser (Jan 22 2019 at 16:43, on Zulip):

The error is in librustc_mir which I wouldn't expect to be built when you ask for src/libstd

Santiago Pastorino (Jan 22 2019 at 16:43, on Zulip):

well removing -i still builds

Wesley Wiser (Jan 22 2019 at 16:43, on Zulip):

Oh it works when you remove -i?

Wesley Wiser (Jan 22 2019 at 16:43, on Zulip):

Try just ./x.py check --stage 1

Santiago Pastorino (Jan 22 2019 at 16:43, on Zulip):

no no, it still compiles

Zoxc (Jan 22 2019 at 16:44, on Zulip):

What is the error you get?

nagisa (Jan 22 2019 at 16:44, on Zulip):

fwiw, -i is prone to break as the compiler structures get changed

Santiago Pastorino (Jan 22 2019 at 16:44, on Zulip):

@Zoxc running build gives an error and check is fine

Santiago Pastorino (Jan 22 2019 at 16:44, on Zulip):

check is the equivalent of cargo check. It just type checks your code but doesn't actually build anything

for some reason build fails and check pass so my guess is that's not exactly true?

That generally shouldn't happen. What is the build failing with?

[santiago@archlinux rust2 (place2b)]$ RUST_BACKTRACE=full ./x.py check -i --stage 1 --keep-stage 1 src/libstd
Updating only changed submodules
Submodules updated in 0.04 seconds
Finished dev [unoptimized] target(s) in 0.50s
Checking std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.35s
Build completed successfully in 0:00:02
[santiago@archlinux rust2 (place2b)]$ RUST_BACKTRACE=full ./x.py build -i --stage 1 --keep-stage 1 src/libstd
Updating only changed submodules
Submodules updated in 0.03 seconds
Finished dev [unoptimized] target(s) in 0.28s
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.33s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Finished release [optimized] target(s) in 0.29s
Copying stage0 test from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Compiling rustc_mir v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_mir)
Compiling rustc_allocator v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_allocator)
Compiling rustc_lint v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_lint)
Compiling rustc_traits v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_traits)
Compiling rustc_resolve v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_resolve)
Compiling rustc_plugin v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_plugin)
error[E0614]: type rustc::mir::Local cannot be dereferenced=> ] 115/127: rustc_mir, rustc_traits, rustc_resolve, rustc_plugin
--> src/librustc_mir/borrow_check/place_ext.rs:100:74
|
100 | let ignore = !has_storage_dead_or_moved.contains(*index) &&
| ^^^^^^

error[E0614]: type rustc::mir::Local cannot be dereferenced
--> src/librustc_mir/borrow_check/place_ext.rs:101:45
|
101 | mir.local_decls[*index].mutability == Mutability::Not;
| ^^^^^^

Compiling rustc_privacy v0.0.0 (/home/santiago/src/oss/rust2/src/librustc_privacy)
error: aborting due to 2 previous errors========================> ] 119/127: rustc_mir

For more information about this error, try rustc --explain E0614.
error: Could not compile rustc_mir.

To learn more, run the command again with --verbose.
command did not execute successfully: "/home/santiago/src/oss/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--features" "" "--manifest-path" "/home/santiago/src/oss/rust2/src/rustc/Cargo.toml" "--message-format" "json"
expected success, got: exit code: 101
failed to run: /home/santiago/src/oss/rust2/build/bootstrap/debug/bootstrap build -i --stage 1 --keep-stage 1 src/libstd
Build completed unsuccessfully in 0:00:14


@Zoxc ^^^

Zoxc (Jan 22 2019 at 16:45, on Zulip):

That is fine though. If x.py check fails, so should x.py build, not the other way around. x.py check checks fewer things than x.py build

Zoxc (Jan 22 2019 at 16:46, on Zulip):

-i is not prone to break as compiler structures change. It can however break due to hash collisions

nagisa (Jan 22 2019 at 16:47, on Zulip):

-i is not prone to break as compiler structures change. It can however break due to hash collisions

/me shrugs and notes down to try -i again sometime

Zoxc (Jan 22 2019 at 16:48, on Zulip):

@Santiago Pastorino Do x.py clean and try again x.py build and x.py check again without using --keep-stage

Santiago Pastorino (Jan 22 2019 at 16:48, on Zulip):

:+1:

Santiago Pastorino (Jan 22 2019 at 16:49, on Zulip):

takes ages though :)

Santiago Pastorino (Jan 22 2019 at 16:49, on Zulip):

one of the tricks I know I should apply is to buy a new machine ;)

Zoxc (Jan 22 2019 at 16:49, on Zulip):

I want one of AMD's new 12 cores =P

nagisa (Jan 22 2019 at 16:50, on Zulip):

/me wants one of AMD’s new 64-cores...

Santiago Pastorino (Jan 22 2019 at 16:50, on Zulip):

I was looking to something a bit simpler :), going after Systems76 Galago Pro probably with 32GB

nagisa (Jan 22 2019 at 16:51, on Zulip):

By the way there is a 2700 on the GCC Farm, you could play with it and try how it feels before making a decision.

davidtwco (Jan 22 2019 at 16:52, on Zulip):

For a few months I've been using a dedicated server for my Rust development with a "AMD Ryzen 7 1700X (16) @ 3.400GHz", 64GB of RAM and a 512GB SSD - mostly for the convenience of being able to jump between machines and reconnect, attach to tmux and continue where I left off, but the performance is nice too.

Santiago Pastorino (Jan 22 2019 at 16:52, on Zulip):

@davidtwco what dedicated server specifically?

Santiago Pastorino (Jan 22 2019 at 16:53, on Zulip):

I've looked into that too

Zoxc (Jan 22 2019 at 16:53, on Zulip):

The 12 core will hopefully be a drop in replacement for R7 1700, which is what I'm on now

davidtwco (Jan 22 2019 at 16:53, on Zulip):

From Hetzner.

nagisa (Jan 22 2019 at 16:54, on Zulip):

The 12 core will hopefully be a drop in replacement for R7 1700, which is what I'm on now

the socket will be the same, but depending on your board the VRMs may be insufficient

nagisa (Jan 22 2019 at 16:55, on Zulip):

especially if a 16-core is released and you decided to step up to that

Zoxc (Jan 22 2019 at 16:55, on Zulip):

That's why I'm eyeing the 12 core instead =P

nagisa (Jan 22 2019 at 16:56, on Zulip):

That's why I'm eyeing the 12 core instead =P

what’s your board?

Zoxc (Jan 22 2019 at 16:58, on Zulip):

ASUS PRIME X370-PRO

nikomatsakis (Jan 22 2019 at 16:58, on Zulip):

re: --keep-stage, another pattern that works is:

./x.py build -i src/libstd # first time
./x.py build -i src/libstd --keep-stage 0 # second, third, etc time
nikomatsakis (Jan 22 2019 at 16:58, on Zulip):

you have to re-run the first command post rebase and/or if "weird stuff happens"

nikomatsakis (Jan 22 2019 at 16:59, on Zulip):

(one thing I am very interested in is if we can come up with a way to make a safe version of keep-stage...)

nikomatsakis (Jan 22 2019 at 16:59, on Zulip):

the plus-side of this pattern is that it gives you a fully functional compiler (not a stage1 version). The downside is it takes a bit longer.

Vadim Petrochenkov (Jan 22 2019 at 22:10, on Zulip):

nagisa uses a widely known trick of a-lot-of-cores.

This is how I solve the problem as well, more or less.
SSD + CPU with good single-thread performance + many cores make world of difference.

Vadim Petrochenkov (Jan 22 2019 at 22:11, on Zulip):

I never use --keep-stage because it's not guaranteed to be correct.

Vadim Petrochenkov (Jan 22 2019 at 22:12, on Zulip):

I also usually build without debuginfo because "builds for running tests" happen much more often than "builds for getting a backtrace".

Vadim Petrochenkov (Jan 22 2019 at 22:15, on Zulip):

Oh, also -C target-cpu=native, although it made very little difference for rustc the last time I measured it (default -> haswell).

Vadim Petrochenkov (Jan 22 2019 at 22:20, on Zulip):

Also disabling CPU hyper-threading helped in my case, IIRC.

Vadim Petrochenkov (Jan 22 2019 at 22:21, on Zulip):

(Need to remeasure this one though.)

mw (Jan 23 2019 at 10:16, on Zulip):

I remember being a bit disappointed when my shiny new 8C/16T R7 1700X was only slightly faster than 4C/8T Xeon in my laptop. Back then I concluded that single-thread performance was still key for a fast bootstrap. Although that was more than a year ago. Things might have changed a bit since then.

mw (Jan 23 2019 at 10:17, on Zulip):

and I certainly want to change things :) compiling the compiler should scale with the number of cores and not so much with clockspeeds.

pnkfelix (Mar 26 2019 at 10:18, on Zulip):

Along the lines of the many cores trick. Has anyone tried compiling rust with sccache?

I would definitely like to hear more about this. I read that sccache has (experimental) rust support. It would be pretty cool if we could apply it to the rustc builds in some fashion; I've used ccache for LLVM a lot in the past, but I'd like to see the idea generalized.

Last update: Nov 16 2019 at 02:10UTC