Stream: t-compiler/help

Topic: why stage 2?


mark-i-m (May 27 2020 at 22:35, on Zulip):

Honest question: why do we have stage 2 of bootstrapping? Why not just build the compiler once with the beta and be done with it?

davidtwco (May 28 2020 at 09:00, on Zulip):

stage two has been useful for me with the polymorphisation work - it acted like a whole other test suite, catching a ton of cases that running every test with stage one didn't. There are probably better reasons than that, but there's one reason.

lcnr (May 28 2020 at 09:12, on Zulip):

Stage 1 also does not use any compiler optimizations added since the last beta, so stage 2 is faster than stage 1 in some cases.

simulacrum (May 28 2020 at 12:38, on Zulip):

When doing local development, it doesn't matter -- well, beyond what @davidtwco mentions -- but in CI we want to build the compiler twice so that we can ship rustc-dev and let people like clippy or miri developers use a nightly toolchain and link against the compiler libraries

simulacrum (May 28 2020 at 12:39, on Zulip):

w/o stage 2 builds if you wanted to link against the compiler you'd have to build it yourself, as the stage 1 compiler is not ABI compatible with libraries it produces, so you can't link against it

mark-i-m (May 28 2020 at 22:04, on Zulip):

Ah, I see

mark-i-m (May 28 2020 at 22:08, on Zulip):

I've been thinking about this a bit in the context of simplifying contributor setup costs. With infinite time and energy, it seems like it would be possible to eventually get to a place where a new contributor just does rustup install beta then cargo +beta build to build to build the compiler. Likewise, maybe libstd, clippy, miri could be made to use pre-compiled artifacts so that those contributors don't have to build the compiler at all...

mark-i-m (May 28 2020 at 22:08, on Zulip):

Does that seem like a practical future outcome at all? Or are there fundamental blockers to achieving such a thing?

simulacrum (May 28 2020 at 22:21, on Zulip):

the first step to getting that to work is to decouple the compiler from std -- i.e., the compiler stops using std unstable features

simulacrum (May 28 2020 at 22:22, on Zulip):

alternatively, you need built-in-to cargo support for building std (e.g. -Zbuild-std)

mark-i-m (May 28 2020 at 22:23, on Zulip):

hmm... why? doesn't it only mean the compiler has to wait 6 weeks before using unstable features?

mark-i-m (May 28 2020 at 22:23, on Zulip):

I guess we would just need a notion of "unstable on beta"

mark-i-m (May 28 2020 at 22:23, on Zulip):

or am I missing something?

simulacrum (May 28 2020 at 22:24, on Zulip):

yeah, it does

simulacrum (May 28 2020 at 22:24, on Zulip):

so today we build std and then link against it when building the compiler

simulacrum (May 28 2020 at 22:25, on Zulip):

cargo has no fully 100% working support for doing this though, so we run cargo twice

simulacrum (May 28 2020 at 22:25, on Zulip):

I'm hopeful that -Zbuild-std will eventually get to the point that we can use that, though

simulacrum (May 28 2020 at 22:26, on Zulip):

I think a far more likely short-term goal is that we can get something like this to work, which would install the master (or most recent bors commit) for you

./prepare-toolchain.sh
cargo +rustc-master build -p rustc
simulacrum (May 28 2020 at 22:26, on Zulip):

that should basically just work today

mark-i-m (May 28 2020 at 22:26, on Zulip):

Another question: why not just use _yesterday's_ nightly?

mark-i-m (May 28 2020 at 22:27, on Zulip):

though, I guess people would need to keep updating their nightly...

mark-i-m (May 28 2020 at 22:27, on Zulip):

or is that what you just suggested?

simulacrum (May 28 2020 at 22:28, on Zulip):

nightly won't necessarily work

simulacrum (May 28 2020 at 22:28, on Zulip):

but "master" always will

simulacrum (May 28 2020 at 22:28, on Zulip):

(note that every day those two diverge until they resync)

mark-i-m (May 28 2020 at 22:30, on Zulip):

well, it won't always work under the current system, but suppose we said that you need to wait 1 week to use new features on nightly... then we could have a dev channel that you only need to sync every week, and any master commit can be built by that week's dev compiler

mark-i-m (May 28 2020 at 22:30, on Zulip):

or put more generally:

simulacrum (May 28 2020 at 22:31, on Zulip):

note that it's not just about unstable features

simulacrum (May 28 2020 at 22:31, on Zulip):

you also have things like lang items

simulacrum (May 28 2020 at 22:31, on Zulip):

if you don't guarantee master, you need basically cfg(bootstrap) and cfg(some_recent_nightly)

simulacrum (May 28 2020 at 22:31, on Zulip):

which is a bit annoying

simulacrum (May 28 2020 at 22:32, on Zulip):

some_recent_nightly, notably, needs to go into the compiler which we've so far avoided for cfg(bootstrap)

mark-i-m (May 28 2020 at 22:32, on Zulip):

we would make a distribution not ABI compatible with itself

simulacrum (May 28 2020 at 22:32, on Zulip):

I don't see how that helps anything

mark-i-m (May 28 2020 at 22:32, on Zulip):

hmm, i need to chew on that

simulacrum (May 28 2020 at 22:32, on Zulip):

to be clear, I think there are ergonomic wins to be had at low cost today

simulacrum (May 28 2020 at 22:33, on Zulip):

there's just need for some design bandwidth to eke out the best UI and such for it

mark-i-m (May 28 2020 at 22:33, on Zulip):

hmm, yeah, I'm conflating stage-2 with the bootstrap compiler

mark-i-m (May 28 2020 at 22:34, on Zulip):

could you elaborate a bit more regarding lang item? could you give an example?

simulacrum (May 28 2020 at 22:35, on Zulip):

so today when you add a lang item to std, the compiler can always assume it exists

simulacrum (May 28 2020 at 22:35, on Zulip):

but if you're not building with the std in tree, that's no longer true

mark-i-m (May 28 2020 at 22:36, on Zulip):

when you say "the compiler" do you mean the one we are compiling with or the one we are compiling?

simulacrum (May 28 2020 at 22:37, on Zulip):

ah well, okay, so actually I guess you're going to end up building std anyway to use it

simulacrum (May 28 2020 at 22:37, on Zulip):

so that's not a concern too much

simulacrum (May 28 2020 at 22:38, on Zulip):

(i.e., you'd need to run something like cargo build -p rustc && cargo +stage1 build -p std)

mark-i-m (May 28 2020 at 22:39, on Zulip):

ah, i see... that seems unfortunate

mark-i-m (May 28 2020 at 22:40, on Zulip):

it would be nice to reach an end state where you only need to build what you're working on

mark-i-m (May 28 2020 at 22:40, on Zulip):

i.e. if you are working on the compiler, you don't need to build std; if you are working on std, you don't need to build the compiler

simulacrum (May 28 2020 at 22:40, on Zulip):

well I mean you don't need to build std

simulacrum (May 28 2020 at 22:41, on Zulip):

but then you can't compile anything with that compiler

mark-i-m (May 28 2020 at 22:41, on Zulip):

right

mark-i-m (May 28 2020 at 22:41, on Zulip):

including running most of the tests

mark-i-m (May 28 2020 at 22:41, on Zulip):

for the new compiler

simulacrum (May 28 2020 at 22:41, on Zulip):

sure, yes, 99.9% of the tests are not no_core

simulacrum (May 28 2020 at 22:41, on Zulip):

to bypass this you basically need at least some level of ABI stability

mark-i-m (May 28 2020 at 22:41, on Zulip):

though, it might not be that bad... build std only takes ~30 seconds on my laptop

simulacrum (May 28 2020 at 22:42, on Zulip):

right yes the ergonomic hit is way worse than the actual build time

simulacrum (May 28 2020 at 22:42, on Zulip):

I will note though that technically there's nothing stopping us from pseudo-building std for you

simulacrum (May 28 2020 at 22:42, on Zulip):

e.g. you run something like cargo run -p rustc-std

simulacrum (May 28 2020 at 22:42, on Zulip):

and that's the equivalent of x.py build --stage 1 src/libtest today, basically

simulacrum (May 28 2020 at 22:43, on Zulip):

obviously it's not as nice because run != build and you need to emulate check and such atop that somehow

mark-i-m (May 28 2020 at 22:43, on Zulip):

Ideally in the long term, the tests would just list std as a Cargo dep and it would just get built

simulacrum (May 28 2020 at 22:43, on Zulip):

Ah... well, maybe

simulacrum (May 28 2020 at 22:43, on Zulip):

I mean you could basically do that today

simulacrum (May 28 2020 at 22:44, on Zulip):

just add a build.rs to compiletest that runs cargo +stage1 build -p std

simulacrum (May 28 2020 at 22:44, on Zulip):

(and in theory it could even do rerun-if-changed on all compiler sources and build the compiler first)

mark-i-m (May 28 2020 at 22:45, on Zulip):

hmm... interesting

simulacrum (May 28 2020 at 22:45, on Zulip):

or, actually, it need not be a build.rs I think

simulacrum (May 28 2020 at 22:45, on Zulip):

hm now I kinda want to try this out

mark-i-m (May 28 2020 at 22:48, on Zulip):

so it seems like building the compiler without std is hard, but what about building std with, say, a beta compiler instead? that would just mean std can't use features until they hit beta, right?

simulacrum (May 28 2020 at 22:48, on Zulip):

well std today already must support building with beta

simulacrum (May 28 2020 at 22:49, on Zulip):

I think the first thing I'd do here is just prototype a no-dep empty crate with build.rs that invokes x.py appropriately

simulacrum (May 28 2020 at 22:49, on Zulip):

and then the thing to do is cargo build -p that-crate

simulacrum (May 28 2020 at 22:50, on Zulip):

(and maybe you have, say, 2-3 of these crates for the common build configs)

mark-i-m (May 28 2020 at 22:50, on Zulip):

do you mean cargo +beta build -p that-crate?

simulacrum (May 28 2020 at 22:50, on Zulip):

sure

simulacrum (May 28 2020 at 22:50, on Zulip):

(this crate, being stable-compatible and nearly empty, can use any toolchain though)

mark-i-m (May 28 2020 at 22:51, on Zulip):

oh yeah

simulacrum (May 28 2020 at 22:51, on Zulip):

it would to start just run x.py as everyone does today, so it wouldn't get special treatment in terms of downloading compilers and such initially

mark-i-m (May 28 2020 at 22:51, on Zulip):

but what I really meant is more like ./x.py +beta... I think you're getting to what I was asking

simulacrum (May 28 2020 at 22:52, on Zulip):

well, yes, the future is that you do cargo +beta build -p that-crate and that compiler is what we use instead of the current approach of downloading one

mark-i-m (May 28 2020 at 22:52, on Zulip):

right

simulacrum (May 28 2020 at 22:52, on Zulip):

but realistically the one-off download is a pretty small hit I think

mark-i-m (May 28 2020 at 22:53, on Zulip):

at least for me, it's a common annoyance because it takes a long time on my internet

mark-i-m (May 28 2020 at 22:53, on Zulip):

also, it's weird for newcommers

simulacrum (May 28 2020 at 22:54, on Zulip):

oh sure

mark-i-m (May 28 2020 at 22:54, on Zulip):

it would reduce the perceived complexity a lot (I think) if people could just rustup install beta and start building std without anything else and without building a compiler first

simulacrum (May 28 2020 at 22:54, on Zulip):

well, today that's already true?

simulacrum (May 28 2020 at 22:54, on Zulip):

well modulo rustup

simulacrum (May 28 2020 at 22:54, on Zulip):

like you can totally just x.py build --stage 0 src/libstd

mark-i-m (May 28 2020 at 22:55, on Zulip):

hmm... that's true

simulacrum (May 28 2020 at 22:55, on Zulip):

and really even cargo +beta build -p std works IIRC

simulacrum (May 28 2020 at 22:55, on Zulip):

the hard bit is that using that std is pretty annoying today, but we could make that better without too much trouble

mark-i-m (May 28 2020 at 22:55, on Zulip):

that seems like it could be a good step

mark-i-m (May 28 2020 at 22:56, on Zulip):

also one other thing that occurs to me is that the default flags don't do what most std devs would want (--stage 0)

mark-i-m (May 28 2020 at 22:57, on Zulip):

but really using cargo would be even better

simulacrum (May 28 2020 at 22:57, on Zulip):

yeah I think we could get cargo working

simulacrum (May 28 2020 at 22:57, on Zulip):

maybe I can spend some cycles on this soon

mark-i-m (May 28 2020 at 22:58, on Zulip):

one other thing: you mentioned that moving std out-of-tree would prevent the compiler from using unstable features

simulacrum (May 28 2020 at 22:58, on Zulip):

what would be great is if someone typed up a doc basically saying "here is what I want to do as a std dev"

mark-i-m (May 28 2020 at 22:59, on Zulip):

but what if we instead made it a subtree and moved std to its own repo?

simulacrum (May 28 2020 at 22:59, on Zulip):

where it's not so much x.py commands as things like "I want to build std and use it for some crate outside of the compiler tree to test my changes"

mark-i-m (May 28 2020 at 22:59, on Zulip):

simulacrum said:

what would be great is if someone typed up a doc basically saying "here is what I want to do as a std dev"

that's why we're doing the contributor survey :)

simulacrum (May 28 2020 at 22:59, on Zulip):

I mean you could but I don't think that buys you much if you can get std buildable via cargo direct w/o x.py

simulacrum (May 28 2020 at 23:00, on Zulip):

I would definitely want more experience with subtrees before doing that

mark-i-m (May 28 2020 at 23:01, on Zulip):

I mean you could but I don't think that buys you much if you can get std buildable via cargo direct w/o x.py

Perhaps not _functionaly_ but it buys a lot of simplicity. That is, a random person could just clone the repo and use cargo like they would for any other crate.

mark-i-m (May 28 2020 at 23:01, on Zulip):

(well, mostly)

simulacrum (May 28 2020 at 23:01, on Zulip):

yes that's somewhat true

simulacrum (May 28 2020 at 23:01, on Zulip):

I would want to see how far we can get in the compiler tree with that

mark-i-m (May 28 2020 at 23:02, on Zulip):

but the point is, it would be a lot less intimidating and would require a somewhat smaller repo clone

mark-i-m (May 28 2020 at 23:02, on Zulip):

simulacrum said:

I would want to see how far we can get in the compiler tree with that

with what? subtree?

simulacrum (May 28 2020 at 23:02, on Zulip):

no, with "just build with cargo"

simulacrum (May 28 2020 at 23:03, on Zulip):

if we don't require submodules etc then I think there's not that much overhead from having the compiler next to you

simulacrum (May 28 2020 at 23:03, on Zulip):

sure, the clone is a bit bigger

simulacrum (May 28 2020 at 23:03, on Zulip):

but Rust developers need large disks anyway because rustc is disk-hungry :)

mark-i-m (May 28 2020 at 23:04, on Zulip):

yeah, just that from skimming the survey comments a lot of people seem to feel intimidated by massive monorepo, and there seems to be a lot of interest to contributing to libstd, so making that feel more "normal" might be a good direction

mark-i-m (May 28 2020 at 23:05, on Zulip):

but I get where you're coming from, from a practicality standpoint

mark-i-m (May 28 2020 at 23:05, on Zulip):

simulacrum said:

but Rust developers need large disks anyway because rustc is disk-hungry :)

lol, this is a bit of an understatement :P
(also something that comes up a lot in the survey, btw)

simulacrum (May 28 2020 at 23:05, on Zulip):

I think people do get intimidated but I don't know how much of that is "oh no big repo" rather than our current strategies for building etc

mark-i-m (May 28 2020 at 23:06, on Zulip):

hmm, I guess that's a fair point

simulacrum (May 28 2020 at 23:06, on Zulip):

like, it might even be a big win to move std and its deps into say std/ and have the top-level be std/ compiler/ tools/ or so

simulacrum (May 28 2020 at 23:06, on Zulip):

and that while a huge diff

simulacrum (May 28 2020 at 23:06, on Zulip):

is trivial

mark-i-m (May 28 2020 at 23:06, on Zulip):

:+1: I do like that idea

simulacrum (May 28 2020 at 23:06, on Zulip):

(someone proposed this somewhere)

mark-i-m (May 28 2020 at 23:07, on Zulip):

do you remember what the reaction to it was?

simulacrum (May 28 2020 at 23:07, on Zulip):

fizzled out more than anything I think

mark-i-m (May 28 2020 at 23:07, on Zulip):

I see

simulacrum (May 28 2020 at 23:07, on Zulip):

I imagine a MCP or w/e today would go through pretty quick though

simulacrum (May 28 2020 at 23:07, on Zulip):

especially if we start by just moving std rather than everything

simulacrum (May 28 2020 at 23:08, on Zulip):

e.g. we'd have src/* and std/

mark-i-m (May 28 2020 at 23:08, on Zulip):

:+1: I was just going to ask

mark-i-m (May 28 2020 at 23:08, on Zulip):

perhaps I will open such an MCP a bit later (have to go now)

mark-i-m (May 28 2020 at 23:09, on Zulip):

thanks for the discussion @simulacrum ! This has been very enlightening

Joshua Nelson (May 30 2020 at 05:28, on Zulip):

On a related point - why is stage2 the default? I can see how it would sometimes be useful, but it seems to me that 99% of the time stage1 is what you actually want and you should have to pass --stage 2 explicitly if you need it for some reason

mark-i-m (May 31 2020 at 04:23, on Zulip):

@Joshua Nelson yeah, I expect that after the contributor survey we might want to re-look at some of the defaults of x.py to make common things a bit more straightforward

Last update: Sep 28 2020 at 15:30UTC