Stream: t-compiler/major changes

Topic: Improve defaults in x.py compiler-team#326


Joshua Nelson (Jul 07 2020 at 21:40, on Zulip):

Will do

Josh Triplett (Jul 07 2020 at 21:40, on Zulip):

Thank you, I appreciate it. :)

Josh Triplett (Jul 07 2020 at 21:40, on Zulip):

And thanks for working on this!

Joshua Nelson (Jul 07 2020 at 21:41, on Zulip):

Oh hmm editing zulip streams does not work the way I thought it did haha

Josh Triplett (Jul 07 2020 at 21:41, on Zulip):

(As an aside, I'm so thankful for the Rust community providing the comfort and safety to be able to raise things like that without a high probability of backlash.)

Josh Triplett (Jul 07 2020 at 21:42, on Zulip):

@Joshua Nelson I can edit the rustbot message to link to the right stream.

Joshua Nelson (Jul 07 2020 at 21:42, on Zulip):

No, I mean it split this channel into two streams

Josh Triplett (Jul 07 2020 at 21:43, on Zulip):

Oh, that's strange.

Joshua Nelson (Jul 07 2020 at 21:43, on Zulip):

the other one I can't edit, I think because I hadn't commented yet?

nikomatsakis (Jul 07 2020 at 22:10, on Zulip):

fwiw I don't think we need an fcp merge here to get total team consensus

nikomatsakis (Jul 07 2020 at 22:10, on Zulip):

I think a rustbot second suffices

nikomatsakis (Jul 07 2020 at 22:10, on Zulip):

also, I'm glad you raised the "sane" question, I wanted to bring it up ...

triagebot (Jul 07 2020 at 22:11, on Zulip):

@T-compiler: Proposal #326 has been seconded, and will be approved in 10 days if no objections are raised.

Josh Triplett (Jul 07 2020 at 22:20, on Zulip):

@nikomatsakis Where is the repo for rustbot and the MCP automation? I'd like to file an issue regarding title changes.

Joshua Nelson (Jul 07 2020 at 22:37, on Zulip):

unfortunately github doesn't let me change the name of the branch without making an entirely new PR :/

lqd (Jul 07 2020 at 22:37, on Zulip):

the repo is https://github.com/rust-lang/triagebot/ I think

Joshua Nelson (Jul 13 2020 at 21:51, on Zulip):

so @Pietro Albini @simulacrum how much is this going to break CI? I can probably figure out most of the straight-up failures if I debug enough, but I'm worried I'm going to break stable releases or something

Joshua Nelson (Jul 13 2020 at 21:51, on Zulip):

should I just sed -i s/x.py/x.py --stage 2/g src/ci/*?

simulacrum (Jul 13 2020 at 21:53, on Zulip):

That sounds fine to me as a start. We should note any problems fairly early on -- try builds and the like -- and we're coming up on the end of a cycle so we'll have a full 12 weeks to get it all nailed down if we land it fairly soon.

Joshua Nelson (Jul 13 2020 at 21:53, on Zulip):
$ rg x.py src/ci --no-filename --no-heading | wc -l
90

:whistles:

Joshua Nelson (Jul 13 2020 at 21:54, on Zulip):

oh nice, I forgot about try builds

Joshua Nelson (Jul 13 2020 at 21:57, on Zulip):

going to let https://github.com/rust-lang/rust/pull/73964/checks?check_run_id=866988967 run to make sure all the defaults for --stage 1 work then I'll push a commit changing it all to use --stage 2

Joshua Nelson (Jul 13 2020 at 22:54, on Zulip):

@simulacrum bootstrap tests are failing because they expect --stage 2 to be the default. Do you want me to update them to match --stage 1 or to pass --stage 2 explicitly?

Joshua Nelson (Jul 13 2020 at 23:12, on Zulip):

(going to pass --stage 2 unless I hear otherwise, you'll see it again in the PR)

Joshua Nelson (Jul 13 2020 at 23:39, on Zulip):

I guess this means x.py now needs a way to specify 'build rustc but not anything else', right now there's no way to build stage2 rustc without passing test --stage 2.

Joshua Nelson (Jul 14 2020 at 00:44, on Zulip):

I just realized I didn't address rustdoc at all in the MCP - are people ok with not building rustdoc by default? Or should I leave that as is, so that x.py build will build both src/libstd and src/tools/rustdoc?

Joshua Nelson (Jul 14 2020 at 00:45, on Zulip):

personally I think it makes sense not to build it unless it was explicitly requested but I don't feel strongly about it

simulacrum (Jul 14 2020 at 12:32, on Zulip):

yeah I'd expect to pass stage 2 there, though we may want some stage 1 tests as well

Santiago Pastorino (Jul 14 2020 at 21:14, on Zulip):

hey @Joshua Nelson I was wondering the following from your proposal ...

Santiago Pastorino (Jul 14 2020 at 21:14, on Zulip):
The proposed new behavior is as follows:

    x.py build --stage 0:
        stage0 libstd

This is suitable for contributors only working on the standard library,
as it means rustc never has to be compiled.

    x.py build --stage 1:
        stage0 libstd
        stage1 rustc
        stage1 libstd

This is suitable for contributors working on the compiler. It ensures
that you have a working rustc and libstd without having to pass
src/libstd in addition.

    x.py build --stage 2:
        stage0 libstd
        stage1 rustc
        stage1 libstd
        stage2 rustc
        stage2 libstd
        stage2 rustdoc, tools, etc.
Santiago Pastorino (Jul 14 2020 at 21:14, on Zulip):

this seems more clear in my opinion to what we have

Santiago Pastorino (Jul 14 2020 at 21:14, on Zulip):

but I wonder something else here ...

Santiago Pastorino (Jul 14 2020 at 21:15, on Zulip):

I guess I may not be understanding well what we mean by some stage N or I may have a different idea

Santiago Pastorino (Jul 14 2020 at 21:16, on Zulip):

but couldn't this be more like ...

Santiago Pastorino (Jul 14 2020 at 21:17, on Zulip):
The proposed new behavior is as follows:

    x.py build --stage 0 (doesn't make sense, stage 0 is beta compiler and std)

    x.py build --stage 1:
        stage1 libstd
        stage1 rustc

    x.py build --stage 2:
        stage1 libstd
        stage1 rustc
        stage2 libstd
        stage2 rustc
Santiago Pastorino (Jul 14 2020 at 21:17, on Zulip):

I know that if you do this the typical workflow would change a bit, but on the other hand things are more clear

Santiago Pastorino (Jul 14 2020 at 21:17, on Zulip):

and you would need stage 3

Santiago Pastorino (Jul 14 2020 at 21:18, on Zulip):

just in case, to be clear, more than suggesting an alternative or discussing it, I want to understand why do we have what we have :)

Joshua Nelson (Jul 14 2020 at 21:18, on Zulip):

stage 0 is beta compiler and std

This depends how you number it. Currently numbering is split between 'what it is' and 'what compiler it was built by'

Joshua Nelson (Jul 14 2020 at 21:18, on Zulip):

if you say build --stage 0 src/libstd, that means build libstd with the beta compiler

Joshua Nelson (Jul 14 2020 at 21:18, on Zulip):

it does not mean use the distributed beta libstd

Joshua Nelson (Jul 14 2020 at 21:19, on Zulip):

doesn't make sense

I don't see why not? You can build libstd without building the compiler first, that's what I expect people contributing to libstd do

Santiago Pastorino (Jul 14 2020 at 21:20, on Zulip):

I meant, if you give --stage N the meaning of build the stage N compiler, stage 0 doesn't make sense because it already exists

Joshua Nelson (Jul 14 2020 at 21:20, on Zulip):

and you would need stage 3

Yeah I missed that in the MCP, it's in the PR. I'll update the description.

Santiago Pastorino (Jul 14 2020 at 21:21, on Zulip):

Joshua Nelson said:

and you would need stage 3

Yeah I missed that in the MCP, it's in the PR. I'll update the description.

what I meant here is that in my schema stage 3 is really needed

Joshua Nelson (Jul 14 2020 at 21:21, on Zulip):

that's not the current meaning of --stage N though

Santiago Pastorino (Jul 14 2020 at 21:21, on Zulip):

yes I know

Santiago Pastorino (Jul 14 2020 at 21:21, on Zulip):

I was just trying to understand

Santiago Pastorino (Jul 14 2020 at 21:21, on Zulip):

the problem I have is ...

Santiago Pastorino (Jul 14 2020 at 21:22, on Zulip):

if you give --stage 1 the meaning of use stage 1 compiler to produce artifacts, shouldn't those artifacts be stage 2 artifacts and be named like so?

Joshua Nelson (Jul 14 2020 at 21:22, on Zulip):

I don't propose in this MCP to change the meaning of --stage N (even though I think it's a good change)

Santiago Pastorino (Jul 14 2020 at 21:23, on Zulip):

I think one of the most confusing things is giving --stage 1 the meaning of use stage 1 compiler to produce artifacts and then name those artifacts with the number 1

Joshua Nelson (Jul 14 2020 at 21:23, on Zulip):

I'm only saying "don't build rustc by default for x.py build"

Santiago Pastorino (Jul 14 2020 at 21:23, on Zulip):

Joshua Nelson said:

I don't propose in this MCP to change the meaning of --stage N (even though I think it's a good change)

:+1:, again, I'm not trying to discuss your proposal :)

Santiago Pastorino (Jul 14 2020 at 21:23, on Zulip):

was just interested in understanding a bit better this

Santiago Pastorino (Jul 14 2020 at 21:23, on Zulip):

Joshua Nelson said:

I'm only saying "don't build rustc by default for x.py build"

:+1:

Joshua Nelson (Jul 14 2020 at 21:23, on Zulip):

oh I see what you mean

Joshua Nelson (Jul 14 2020 at 21:24, on Zulip):

yeah I named them confusingly

Joshua Nelson (Jul 14 2020 at 21:24, on Zulip):

should I say 'stage 1 artifacts' instead?

Joshua Nelson (Jul 14 2020 at 21:24, on Zulip):

currently I use 'stage1 rustc' to mean the thing in stage1/bin/rustc but 'stage1 libstd' to mean the thing built by stage1 rustc

Santiago Pastorino (Jul 14 2020 at 21:25, on Zulip):

yeah to be honest I'm not really sure what currently happens :)

Santiago Pastorino (Jul 14 2020 at 21:25, on Zulip):

it's a bit confusing to me, didn't spend time trying to understand properly

Santiago Pastorino (Jul 14 2020 at 21:26, on Zulip):

but what I think should look better is ...

Joshua Nelson (Jul 14 2020 at 21:26, on Zulip):

Yeah it is definitely confusing, I had a discussion about this yesterday in https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/freeze.20stage0

Santiago Pastorino (Jul 14 2020 at 21:26, on Zulip):

if you want to give --stage 0 the meaning of use stage 0 to produce artifacts, maybe a more explicit name for that should be better, like --use-stage 0

Santiago Pastorino (Jul 14 2020 at 21:26, on Zulip):

and then your proposal be something like ...

Joshua Nelson (Jul 14 2020 at 21:27, on Zulip):

Ooh I definitely like the name --use-stage better

Santiago Pastorino (Jul 14 2020 at 21:27, on Zulip):
The proposed new behavior is as follows:

    x.py build --use-stage 0:
        stage1 libstd

This is suitable for contributors only working on the standard library,
as it means rustc never has to be compiled.

    x.py build --use-stage 1:
        stage1 libstd
        stage1 rustc
        stage2 libstd

This is suitable for contributors working on the compiler. It ensures
that you have a working rustc and libstd without having to pass
src/libstd in addition.

    x.py build --use-stage 2:
        stage1 libstd
        stage1 rustc
        stage2 libstd
        stage2 rustc
        stage3 libstd
        stage3 rustdoc, tools, etc.
Joshua Nelson (Jul 14 2020 at 21:28, on Zulip):

as opposed to currently I think people assume it means --build-stage, which is not correct

Santiago Pastorino (Jul 14 2020 at 21:28, on Zulip):

right

Santiago Pastorino (Jul 14 2020 at 21:28, on Zulip):

it can also have a shorter way like -s 1

Santiago Pastorino (Jul 14 2020 at 21:28, on Zulip):

unsure if there's something like that currently

Joshua Nelson (Jul 14 2020 at 21:29, on Zulip):

I'll make a PR adding that alias separately I think, and then we can use --use-stage in documentation

Joshua Nelson (Jul 14 2020 at 21:29, on Zulip):

there's no -s shorthand I know of currently

Santiago Pastorino (Jul 14 2020 at 21:31, on Zulip):

Santiago Pastorino said:

The proposed new behavior is as follows:

    x.py build --use-stage 0:
        stage1 libstd

This is suitable for contributors only working on the standard library,
as it means rustc never has to be compiled.

    x.py build --use-stage 1:
        stage1 libstd
        stage1 rustc
        stage2 libstd

This is suitable for contributors working on the compiler. It ensures
that you have a working rustc and libstd without having to pass
src/libstd in addition.

    x.py build --use-stage 2:
        stage1 libstd
        stage1 rustc
        stage2 libstd
        stage2 rustc
        stage3 libstd
        stage3 rustdoc, tools, etc.

@Joshua Nelson what do you think about :point_up: ?, do you think it make sense?

Joshua Nelson (Jul 14 2020 at 21:31, on Zulip):

I don't want to change the terminology though

Santiago Pastorino (Jul 14 2020 at 21:31, on Zulip):

yeah, I'm not talking about your PR

Santiago Pastorino (Jul 14 2020 at 21:31, on Zulip):

maybe like something for the future

Santiago Pastorino (Jul 14 2020 at 21:31, on Zulip):

but I want to know if what I'm saying even make sense to others

Joshua Nelson (Jul 14 2020 at 21:32, on Zulip):

cc @eddyb he feels strongly about stage naming

Santiago Pastorino (Jul 14 2020 at 21:32, on Zulip):

and if so, why don't we have an schema like that

Joshua Nelson (Jul 14 2020 at 21:32, on Zulip):

@eddyb what do you think about keeping the behavior as discussed but we change the names we use for things?

Joshua Nelson (Jul 14 2020 at 21:32, on Zulip):

^ see the chart Santiago Pastorino wrote above

eddyb (Jul 14 2020 at 21:32, on Zulip):

@Santiago Pastorino I think you're falling into the same trap.... stageN directories are not the stages

Santiago Pastorino (Jul 14 2020 at 21:33, on Zulip):

right :)

eddyb (Jul 14 2020 at 21:33, on Zulip):

if we changed something, the first thing is to make those as clear as possible

Santiago Pastorino (Jul 14 2020 at 21:33, on Zulip):

eddyb said:

Santiago Pastorino I think you're falling into the same trap.... stageN directories are not the stages

yeah I know, but isn't that confusing?

eddyb (Jul 14 2020 at 21:33, on Zulip):

so --stage 1 currently really is "build stage1" where "stage1" is stage1-* and stage1 just happens to be where we put the sysroot

Santiago Pastorino (Jul 14 2020 at 21:34, on Zulip):

ok, so, it's build stage1? or is it build with stage1?

eddyb (Jul 14 2020 at 21:34, on Zulip):

it's the former, and stage1 is a badly named directory

eddyb (Jul 14 2020 at 21:34, on Zulip):

stage1/bin/rustc is an input to stage1

Santiago Pastorino (Jul 14 2020 at 21:34, on Zulip):

isn't that different to what @Joshua Nelson was saying then?

eddyb (Jul 14 2020 at 21:34, on Zulip):

stage1 is the stage1 sysroot, but improperly labelled

Joshua Nelson (Jul 14 2020 at 21:35, on Zulip):

wait now I'm confused :(

eddyb (Jul 14 2020 at 21:35, on Zulip):

I would call that directory stage1-sysroot or stage0to1

eddyb (Jul 14 2020 at 21:35, on Zulip):

it's not stage1 as much as it's used by/to build stage1

eddyb (Jul 14 2020 at 21:35, on Zulip):

it's the least stage1 directory out of all the ones with that prefix :P

Joshua Nelson (Jul 14 2020 at 21:35, on Zulip):

so --use-stage is not correct? I thought --stage N src/libstd means 'build libstd with stage N'

eddyb (Jul 14 2020 at 21:35, on Zulip):

it's build at stage N

eddyb (Jul 14 2020 at 21:35, on Zulip):

so it builds stageN-std

eddyb (Jul 14 2020 at 21:36, on Zulip):

the newer directories with dashes in them fit much better overall than the old nonsense

eddyb (Jul 14 2020 at 21:36, on Zulip):

and by newer I mean post-makefiles :P

Joshua Nelson (Jul 14 2020 at 21:36, on Zulip):

so then ... stage0 libstd and stage0 beta compiler are part of the same stage ???

eddyb (Jul 14 2020 at 21:37, on Zulip):

"stage0 beta compiler" isn't a thing

eddyb (Jul 14 2020 at 21:37, on Zulip):

stage0, just like stage1, is the inputs to that stage

Joshua Nelson (Jul 14 2020 at 21:37, on Zulip):

ok, so then stage0 libstd and the first rustc built by beta are the same stage

eddyb (Jul 14 2020 at 21:37, on Zulip):

correct

eddyb (Jul 14 2020 at 21:38, on Zulip):

stage0/bin/rustc is the compiler that will be used to build stage0, and unlike renaming stage1 tostage0to1, there isn't something to rename stage0 to, with the same scheme, lol

Santiago Pastorino (Jul 14 2020 at 21:38, on Zulip):

I guess what I feel is that there are kind of different meanings of what really stage means flying around

Joshua Nelson (Jul 14 2020 at 21:38, on Zulip):

I understand those are the current directory names (stage0-std, stage0-rustc and all that), but can you explain again why it makes sense to consider those both part of the same stage?

eddyb (Jul 14 2020 at 21:38, on Zulip):

short of stage0-sysroot (which is already taken up because we need two sysroots lol)

eddyb (Jul 14 2020 at 21:39, on Zulip):

@Joshua Nelson because the only difference is which crate is being compiled

eddyb (Jul 14 2020 at 21:39, on Zulip):

you could imagine src/librustc_* just explicitly depending on src/libstd

eddyb (Jul 14 2020 at 21:39, on Zulip):

it's all one big compilation

eddyb (Jul 14 2020 at 21:40, on Zulip):

you don't switch from one stage to the next without changing the compiler... that's "staging"

Joshua Nelson (Jul 14 2020 at 21:40, on Zulip):

ok, they belong to each other because they're linked together?

eddyb (Jul 14 2020 at 21:40, on Zulip):

think of, idk, a rocket

eddyb (Jul 14 2020 at 21:40, on Zulip):

Joshua Nelson said:

ok, they belong to each other because they're linked together?

yes! and that has a lot of implications

eddyb (Jul 14 2020 at 21:41, on Zulip):

the sections of the rocket are the "stages", whereas the engine would be the compiler

eddyb (Jul 14 2020 at 21:41, on Zulip):

the engines are more or less between the stages

Joshua Nelson (Jul 14 2020 at 21:41, on Zulip):

so then you would say there is no stage3

eddyb (Jul 14 2020 at 21:41, on Zulip):

so you have stage0/bin/rustc as the bottom-most engine, followed by an entire stage0, then this repeats for each stage

eddyb (Jul 14 2020 at 21:42, on Zulip):

no, not unless you build stage3-std and maybe stage3-rustc (which you could trivially use to link stage4/bin/rustc)

Joshua Nelson (Jul 14 2020 at 21:42, on Zulip):

right

Joshua Nelson (Jul 14 2020 at 21:42, on Zulip):

ok I will update the numbering in the MCP to match that

eddyb (Jul 14 2020 at 21:42, on Zulip):

we could have it so instead of stage1/bin/rustc you use stage0-output/bin/rustc

eddyb (Jul 14 2020 at 21:42, on Zulip):

maybe that would make it more obvious

eddyb (Jul 14 2020 at 21:42, on Zulip):

and then hide the sysroot better or something

Joshua Nelson (Jul 14 2020 at 21:43, on Zulip):

personally I think stage1 is silly and we should use stage0-rustc directly

eddyb (Jul 14 2020 at 21:43, on Zulip):

it's only confusing because there's "segments" and "points" between them (that are the output of one segment and the input of the next)

Joshua Nelson (Jul 14 2020 at 21:43, on Zulip):

then you don't have people getting confused like this

eddyb (Jul 14 2020 at 21:43, on Zulip):

oh yeah I definitely agree

eddyb (Jul 14 2020 at 21:44, on Zulip):

I was saying elsewhere that I'd prefer if we symlink'd stage0-rustc/$HOST/release/rustc directly into the rustup toolchain dir or w/e

eddyb (Jul 14 2020 at 21:44, on Zulip):

I much prefer the dashed directories

eddyb (Jul 14 2020 at 21:44, on Zulip):

partly because of what I remember from the makefile days, cursed be their memory

Joshua Nelson (Jul 14 2020 at 21:46, on Zulip):

and remind me why rustdoc is off-by-one? it links against the previous compiler or something?

eddyb (Jul 14 2020 at 21:49, on Zulip):

yeah, IMO running the tests should build the previous stage of tools, that way building rustdoc itself shouldn't need an off by one

eddyb (Jul 14 2020 at 21:50, on Zulip):

basically today x.py test --stage N src/test/ui/rustdoc depends on x.py build --stage N src/tools/rustdoc and the latter uses x.py build --stage N-1 compiler libs, not x.py build --stage N compiler libs, so it feels wrong

eddyb (Jul 14 2020 at 21:50, on Zulip):

instead we could have x.py test --stage N src/test/ui/rustdoc depend on x.py build --stage N-1 src/tools/rustdoc

Joshua Nelson (Jul 14 2020 at 21:51, on Zulip):

ok, so how does this new numbering look

Joshua Nelson (Jul 14 2020 at 21:51, on Zulip):

https://gist.github.com/jyn514/e3da90ee203d5eb9194255afef428d2c

Joshua Nelson (Jul 14 2020 at 21:51, on Zulip):

rustdoc is the one I'm not sure about

Joshua Nelson (Jul 14 2020 at 21:57, on Zulip):

err actually that was off a bit, I updated it

Joshua Nelson (Jul 14 2020 at 22:06, on Zulip):

Updated the MCP to use the revised numbering

eddyb (Jul 14 2020 at 22:17, on Zulip):

I would use the same numbering for rustc and rustdoc

eddyb (Jul 14 2020 at 22:17, on Zulip):

so that only libstd is on the last stage

Joshua Nelson (Jul 14 2020 at 22:19, on Zulip):

even though what you're calling stage1 rustdoc is built with build --stage 2 src/tools/rustdoc?

eddyb (Jul 14 2020 at 22:19, on Zulip):

I think the current build behavior for rustdoc is wrong

eddyb (Jul 14 2020 at 22:20, on Zulip):

either that or all the other tools are wrong and we have to find a good way to handle all of them

eddyb (Jul 14 2020 at 22:20, on Zulip):

but technically rustdoc is in the minority with its behavior

Joshua Nelson (Jul 14 2020 at 22:20, on Zulip):

Sure, it's definitely it's the odd man out

Joshua Nelson (Jul 14 2020 at 22:20, on Zulip):

but I think we should keep consistency with x.py

eddyb (Jul 14 2020 at 22:20, on Zulip):

and it would make your gist read much better

eddyb (Jul 14 2020 at 22:21, on Zulip):

to what end? it just introduces confusion

eddyb (Jul 14 2020 at 22:21, on Zulip):

also note that x.py build will build other tools as well, if they're enabled in config.toml

Joshua Nelson (Jul 14 2020 at 22:21, on Zulip):

ok, I'll say stageN tools then

eddyb (Jul 14 2020 at 22:21, on Zulip):

and in your gist, they're like rustc not like rustdoc

eddyb (Jul 14 2020 at 22:21, on Zulip):

thanks ^_^

Joshua Nelson (Jul 14 2020 at 22:21, on Zulip):

where N is the stage of the compiler they're linked to

Joshua Nelson (Jul 14 2020 at 22:25, on Zulip):

wait no now I'm confused again

Joshua Nelson (Jul 14 2020 at 22:25, on Zulip):

If I say x.py build --stage 1 src/tools/cargo, does that build rustc once or twice?

eddyb (Jul 14 2020 at 22:25, on Zulip):

good question

eddyb (Jul 14 2020 at 22:26, on Zulip):

Cargo doesn't depend on rustc libraries, so probably only once

eddyb (Jul 14 2020 at 22:26, on Zulip):

(i.e. x.py build --stage 1 src/libstd + Cargo build)

eddyb (Jul 14 2020 at 22:26, on Zulip):

cc @simulacrum

Joshua Nelson (Jul 14 2020 at 22:26, on Zulip):

so in the case of tools, it really is --use-stage

Joshua Nelson (Jul 14 2020 at 22:26, on Zulip):

It's only for things that link to rustc that it's --build-stage

eddyb (Jul 14 2020 at 22:27, on Zulip):

I mean, you might be conflating a few things? the various steps still have dependencies

simulacrum (Jul 14 2020 at 22:27, on Zulip):

Should be correct - not sure if entirely the case today (cargo tests do have rustc sysroot deps, currently)

eddyb (Jul 14 2020 at 22:28, on Zulip):

so the difference is Cargo depends on std but clippy depends on rustc (and transitively on std)

eddyb (Jul 14 2020 at 22:28, on Zulip):

all of these dependencies are satisfied within/"at" the stage provided in --stage

Joshua Nelson (Jul 14 2020 at 22:28, on Zulip):

maybe I should just omit the tools altogether lol

Joshua Nelson (Jul 14 2020 at 22:28, on Zulip):

I'm not changing how they behave

eddyb (Jul 14 2020 at 22:29, on Zulip):

you'd only need to change rustdoc for them to appear sensible shrug

eddyb (Jul 14 2020 at 22:29, on Zulip):

@simulacrum wait what why would Cargo tests want to do that

simulacrum (Jul 14 2020 at 22:29, on Zulip):

The difference between std-dep tools and rustc-dep tools, including rustc itself in some sense is one of the core confusions I think with stage-naming

eddyb (Jul 14 2020 at 22:29, on Zulip):

surely it's not for proc macros?

simulacrum (Jul 14 2020 at 22:29, on Zulip):

@eddyb I don't think it's "want" more so just no one's purged all of that

simulacrum (Jul 14 2020 at 22:29, on Zulip):

No it's like plugin testing or something

eddyb (Jul 14 2020 at 22:30, on Zulip):

I wonder why Cargo wanted to ever test that :/

eddyb (Jul 14 2020 at 22:30, on Zulip):

maybe it has something to do with Servo shrug

simulacrum (Jul 14 2020 at 22:30, on Zulip):

It can probably be removed/replaced today

Joshua Nelson (Jul 14 2020 at 22:32, on Zulip):

/me is more confused than when he started

eddyb (Jul 14 2020 at 22:33, on Zulip):

this is what you get from touching staging,

Joshua Nelson (Jul 14 2020 at 22:33, on Zulip):

I'm going to leave the MCP the same pending me figuring out what's going on

Joshua Nelson (Jul 14 2020 at 22:33, on Zulip):

I don't think the numbering of the tools affects it much anyway

eddyb (Jul 14 2020 at 22:34, on Zulip):

I kept wanting to make things clearer for years but dreaded what you're going through :P

Joshua Nelson (Jul 14 2020 at 22:34, on Zulip):

I'm not even trying to make them clearer, just less painful lol

eddyb (Jul 14 2020 at 22:34, on Zulip):

or what I'm going through trying to explain why some things make sense to me, and failing most of the time

Joshua Nelson (Jul 14 2020 at 22:35, on Zulip):

the end goal is like @LeSeulArtichaut said, to make it so that x.py build 'just works' and people don't have to think about it

eddyb (Jul 14 2020 at 22:36, on Zulip):

the simplest way IMO is to just say that x.py by default does minimal, instead of maximal, work. and only ./configure && make retains the maximal behavior

Joshua Nelson (Jul 14 2020 at 22:36, on Zulip):

/me forgot to update configure and make in the PR

eddyb (Jul 14 2020 at 22:37, on Zulip):

so x.py build and x.py build --stage 1 and x.py build src/libstd and x.py build --stage 1 src/libstd would all do the same thing

eddyb (Jul 14 2020 at 22:37, on Zulip):

i.e. if you don't pass a stage, it's 1, if you don't pass a filter, it's src/libstd

eddyb (Jul 14 2020 at 22:37, on Zulip):

anything more explicit doesn't change at all

Joshua Nelson (Jul 14 2020 at 22:37, on Zulip):

That's not actually the current behavior, it will still build rustdoc by default

Joshua Nelson (Jul 14 2020 at 22:38, on Zulip):

Because I forgot to mention it in the original MCP

eddyb (Jul 14 2020 at 22:38, on Zulip):

lol

eddyb (Jul 14 2020 at 22:38, on Zulip):

maybe that should be treated just like all the other tools and you have to put it into config.toml that you want rustdoc

eddyb (Jul 14 2020 at 22:38, on Zulip):

or maybe it can be default shrug

Joshua Nelson (Jul 14 2020 at 22:39, on Zulip):

I would be happy with making rustdoc more like other tools

Joshua Nelson (Jul 14 2020 at 22:39, on Zulip):

but I think that's a different MCP

eddyb (Jul 14 2020 at 22:39, on Zulip):

but ideally x.py build --stage 1 src/libstd would also build the same tools (at stage 0)

eddyb (Jul 14 2020 at 22:39, on Zulip):

at least that's the simple approach for this in my view

Joshua Nelson (Jul 14 2020 at 22:40, on Zulip):

at this point I think the code speaks more clearly than I can https://github.com/rust-lang/rust/pull/73964/commits/41a339d1c8753a655a691f605d29c20fba856e46

Joshua Nelson (Jul 14 2020 at 22:40, on Zulip):

(not sure about the all_crates change but I figure someone will notice if it breaks things)

tmandry (Jul 17 2020 at 18:42, on Zulip):

Since I didn't see it mentioned in the MCP: will the default stage for the dist and install commands also change?

tmandry (Jul 17 2020 at 18:43, on Zulip):

this would break people who are using x.py to build and distribute rust themselves (like we do on fuchsia)

tmandry (Jul 17 2020 at 18:43, on Zulip):

which I'm fine with, but --stage 1 doesn't seem like the right default for those commands to me

tmandry (Jul 17 2020 at 18:46, on Zulip):

Also FYI: rustbot's link to this stream on the MCP issue was broken

Camelid (Jul 17 2020 at 19:13, on Zulip):

@tmandry I filed an issue: rust-lang/triagebot#701

Joshua Nelson (Jul 17 2020 at 21:58, on Zulip):

Since I didn't see it mentioned in the MCP: will the default stage for the dist and install commands also change?

Hmm that's a good point. I don't see any reason they should change, since it seems very rare to want to install the stage 1 compiler. I'll clarify that in the MCP.

Joshua Nelson (Jul 17 2020 at 22:05, on Zulip):

ok here are the subcommands that look relevant:

I have no idea what run does except for the suggestion of expand-yaml-anchors (cc @Pietro Albini who wrote that command - -h -v doesn't show the available paths). Anyone have suggestions?
The default for dist and install seems like it should stay at 2.
The default for build and test should change to 1 (that's kind of the point of the MCP :P).
I think doc should actually be 0, since normally when you document things you're just fixing a typo. For this release cycle only that will have very different results between beta and nightly (due to https://github.com/rust-lang/rust/pull/73101) but in general I think stage 0 is a good default.
Presumably bench should use stage 2 since it's measuring performance? I've never used it before.

eddyb (Jul 18 2020 at 02:18, on Zulip):

that sounds about right

eddyb (Jul 18 2020 at 02:18, on Zulip):

@tmandry FWIW you should probably use ./configure and make to get the "distro-facing" stable interface

eddyb (Jul 18 2020 at 02:19, on Zulip):

worst case, if you need some special config.toml settings (not exposed otherwise), you can still pass them to ./configure with --set I believe

tmandry (Jul 18 2020 at 02:36, on Zulip):

@eddyb thanks, good to know.

Joshua Nelson (Jul 18 2020 at 03:34, on Zulip):

updated the MCP with the stage changes

Joshua Nelson (Jul 18 2020 at 03:47, on Zulip):

btw @tmandry thanks for bringing it up, I would never have thought of x.py dist :sweat_smile:

comex (Jul 18 2020 at 05:39, on Zulip):

The confusing naming is something that's really confused me when I've dabbled with rustc, and this seems like a significant improvement!

comex (Jul 18 2020 at 05:47, on Zulip):

FWIW, another thing I'd love to see (not sure if it exists already) is a way to rebuild rustc but use it with an existing libstd/libcore - which isn't possible for all changes, but I think would be good enough to test small changes.

comex (Jul 18 2020 at 05:49, on Zulip):

I'm interested in both "build libstd/libcore, but only once, not every time I change anything in rustc" and "don't build libstd/libcore at all, just use the one from the rustup nightly".

comex (Jul 18 2020 at 05:50, on Zulip):

Again, I have limited experience with building rustc, so I don't know if this is possible already somehow.

eddyb (Jul 18 2020 at 06:15, on Zulip):

@comex I believe the path forward for stuff like that is your second point, more than the first. @simulacrum and I have discussed it before, but idk if anyone has made any progress in allowing the use of rustup nightly (or more realistically, last PR build)

eddyb (Jul 18 2020 at 06:16, on Zulip):

if you change something in rustc, it'd be unsound in general to use a stale core/std build

eddyb (Jul 18 2020 at 06:16, on Zulip):

I've recently observed some strange behavior arising from the fact that stage1-* doesn't get entirely nuked whenever I rebuild rustc

eddyb (Jul 18 2020 at 06:16, on Zulip):

I guess I should've filed an issue

eddyb (Jul 18 2020 at 06:17, on Zulip):

at best, it ICEs because of some incoherence. at worst, it generates broken code that behaves incorrectly at runtime

comex (Jul 18 2020 at 06:19, on Zulip):

Still, when the goal is to make the edit-compile-run cycle as fast as possible, and the change might be something as simple as "add a println somewhere"...

eddyb (Jul 18 2020 at 06:20, on Zulip):

but it's like 40 seconds

eddyb (Jul 18 2020 at 06:20, on Zulip):

maybe I'm still getting used to how fast building rustc is nowadays (with incremental), heh

comex (Jul 18 2020 at 06:21, on Zulip):

Don't know. I haven't built rustc very often :)

comex (Jul 18 2020 at 06:21, on Zulip):

But I do work on Clang from time to time, and I'm used to Clang incremental builds being like 10-20 seconds total, and I think that's slow.

eddyb (Jul 18 2020 at 06:21, on Zulip):

40 seconds is about the time it takes to build core/std from scratch, that is. maybe 2 minutes on less capable machines? not sure

eddyb (Jul 18 2020 at 06:22, on Zulip):

and I tend to run everything through the entire test suite anyway, so it's not like the risks are worth the fraction that is taken up by core/std

comex (Jul 18 2020 at 06:22, on Zulip):

Another thing...

comex (Jul 18 2020 at 06:23, on Zulip):

I also want to be able to build rustc in debug mode, without the cost of building libcore/libstd with rustc in debug mode.

eddyb (Jul 18 2020 at 06:23, on Zulip):

why? I don't know anyone who actually uses that except to debug maybe some "rustc calling into LLVM" failures

comex (Jul 18 2020 at 06:24, on Zulip):

It's faster, isn't it? (If the actual debug info level is held constant. I just mean the optimization level.)

eddyb (Jul 18 2020 at 06:24, on Zulip):

I mean using debug rustc. it's only there in case something really bad happens and there's no normal way to handle it

eddyb (Jul 18 2020 at 06:24, on Zulip):

once we move to tracing, most of my gripes with the debug logging should go away, due to the hierarchical nesting information

eddyb (Jul 18 2020 at 06:25, on Zulip):

there's just no way to get that kind of detail from using an actual debugger

eddyb (Jul 18 2020 at 06:25, on Zulip):

not without us putting a lot of effort into integrating the rustc codebase with debuggers

comex (Jul 18 2020 at 06:26, on Zulip):

Well. My primary interest is not actually a debugger but just building in unoptimized mode.

eddyb (Jul 18 2020 at 06:26, on Zulip):

it's not like LLVM where most stuff is fields and printable through a ->dump() call or whatever

comex (Jul 18 2020 at 06:27, on Zulip):

I'd love if Rust had a better debugger experience but that's another matter :)

eddyb (Jul 18 2020 at 06:27, on Zulip):

it's not really about Rust, we're kind of limited by archaic tooling and don't have the bandwidth to develop our own

eddyb (Jul 18 2020 at 06:28, on Zulip):

@comex oh, sorry, that's optimize = false not debug = true and even less requested

comex (Jul 18 2020 at 06:28, on Zulip):

(in my opinion, traditional debuggers are fundamentally broken even for C++, since they can't instantiate functions at runtime. Swift does this right.)

eddyb (Jul 18 2020 at 06:28, on Zulip):

or talked about at all

eddyb (Jul 18 2020 at 06:28, on Zulip):

does Swift have its own debugger?

eddyb (Jul 18 2020 at 06:28, on Zulip):

or does it do something fancy by integrating lldb and swiftc?

comex (Jul 18 2020 at 06:28, on Zulip):

The latter.

eddyb (Jul 18 2020 at 06:28, on Zulip):

I am jealous now

eddyb (Jul 18 2020 at 06:29, on Zulip):

but I guess I have previously suggested gdb/lldb could be connected up to miri somehow

eddyb (Jul 18 2020 at 06:29, on Zulip):

so that no JITing would be necessary

eddyb (Jul 18 2020 at 06:30, on Zulip):

@comex anyway in practice, it's not worth it. even if you bypass building core/std, anything you build will still be really slow

eddyb (Jul 18 2020 at 06:30, on Zulip):

maybe @bjorn3 has good news regarding the cranelift backend, i.e. maybe it's less hopeless than LLVM

eddyb (Jul 18 2020 at 06:31, on Zulip):

(there's also MIR optimizations we haven't finished/turned on by default, or even started working on, which could take up the slack)

comex (Jul 18 2020 at 06:31, on Zulip):

Well, I don't know if the time spent just loading core makes this not worth it. I did actually try this once, but I think I used no_core for the test case.

eddyb (Jul 18 2020 at 06:31, on Zulip):

it'd be useful for like maybe one test case

comex (Jul 18 2020 at 06:31, on Zulip):

But the thing I want to build in that case was a 10 line test case.

eddyb (Jul 18 2020 at 06:32, on Zulip):

in general it takes more time to write a #![no_core] testcase than the extra time it takes to build the compiler with optimizations :P

comex (Jul 18 2020 at 06:32, on Zulip):

Then it's still a win, because I only have to write the testcase once, but rebuild the compiler many times ;)

eddyb (Jul 18 2020 at 06:33, on Zulip):

oh wait

eddyb (Jul 18 2020 at 06:33, on Zulip):

if the test is #![no_core] you can just set optimize = false and just don't build core/std

eddyb (Jul 18 2020 at 06:33, on Zulip):

./x.py build --stage 0 src/rustc or something should be enough (the explicit src/rustc is to account for this MCP)

comex (Jul 18 2020 at 06:34, on Zulip):

Yeah, I actually did do something like that. I forget which command I used. But I was hoping I could get a similar effect while still being able to use core, by reusing an existing built one.

comex (Jul 18 2020 at 06:34, on Zulip):

But as I said, maybe it takes a long time just to load the serialized data or something; I haven't tried.

eddyb (Jul 18 2020 at 06:34, on Zulip):

it's risky but I guess --keep-std could be offered as a non-broken version of --keep-stage (the latter which likes to break your build directory until you nuke most of it...)

eddyb (Jul 18 2020 at 06:35, on Zulip):

it's mostly loaded on-demand, which actually matters, e.g. I remember cutting 4 seconds off the hello world compile time on raspberry pi lol

comex (Jul 18 2020 at 06:36, on Zulip):

I see. well, maybe "less broken" :)

eddyb (Jul 18 2020 at 06:37, on Zulip):

one of the reasons I'm opposed is you could gain minutes but lose hours if you touch one of many things that would actually result in a different core/std. I wish there was a way to tell if the only change between two builds is one has more debug logging sigh

comex (Jul 18 2020 at 06:38, on Zulip):

Fair enough...

eddyb (Jul 18 2020 at 06:39, on Zulip):

but I guess if it warns that it's not rebuilding core/std and explains what could go wrong, every single time... it shouldn't be that bad

Jake Goulding (Jul 18 2020 at 11:18, on Zulip):

since it seems very rare to want to install the stage 1 compiler

@Joshua Nelson that's actually all I've been doing recently... https://github.com/shepmaster/rust/blob/silicon/silicon/README.md

Is there a "translation guide" from the old style to the new style commands?

eddyb (Jul 18 2020 at 11:19, on Zulip):

ideally we can just keep them (the dist/install commands) behaving the same

eddyb (Jul 18 2020 at 11:20, on Zulip):

as for x.py build commands, depends how explicit they were - if you pass both --stage and e.g. src/libstd (or some other path), I'm against that changing behavior

eddyb (Jul 18 2020 at 11:22, on Zulip):

@Jake Goulding all the x.py commands I can see in that README look explicit enough to not need any changes

eddyb (Jul 18 2020 at 11:23, on Zulip):

like, I think you will be able to shorten:
../../x.py build -i --stage 1 src/libstd
to:
../../x.py build -i
but you won't have to

eddyb (Jul 18 2020 at 11:28, on Zulip):

also that's an interesting problem, I didn't notice the two-step process before. I guess there's not much to do there until the changes are upstreamed, presumably there's no way to get the new target to be accepted if using a single build dir (and a stage2 build)? but also the split build dir approach allows rebuilding only silicon/cross, kind of like --keep-stage (but not unsound/broken)

Joshua Nelson (Jul 18 2020 at 12:41, on Zulip):

Currently the MCP says rustdoc will also be built by default. I don't feel strongly about that, though.

Joshua Nelson (Jul 18 2020 at 12:41, on Zulip):

So you would still need src/libstd to avoid building rustdoc

Joshua Nelson (Jul 18 2020 at 12:43, on Zulip):

The confusing naming is something that's really confused me when I've dabbled with rustc, and this seems like a significant improvement!

Thank you! This doesn't actually change the numbering, it just makes the defaults somewhat better. Changing the numbering is another whole can of worms :laughing:

Joshua Nelson (Jul 20 2020 at 01:50, on Zulip):

It's been 12 days since nikomatsakis seconded the MCP - the description says it should have been accepted after 10 days. Is there a bug in the automation maybe?

Yuki Okushi (Jul 20 2020 at 08:30, on Zulip):

Joshua Nelson said:

It's been 12 days since nikomatsakis seconded the MCP - the description says it should have been accepted after 10 days. Is there a bug in the automation maybe?

it isn't a _bug_ but missing functionality. currently someone has to add major-change-accepted manually iirc

Santiago Pastorino (Jul 20 2020 at 20:30, on Zulip):

I usually manually accept MCPs just before our thursday's meetings

Joshua Nelson (Jul 20 2020 at 20:31, on Zulip):

got it, thanks :) I'll try to contain my impatience :laughing:

Joshua Nelson (Jul 21 2020 at 21:12, on Zulip):

@eddyb I figured out why so many people think stage0 means beta:

https://rustc-dev-guide.rust-lang.org/building/how-to-build-and-run.html?highlight=config.toml#bootstrapping

Stage 0: the stage0 compiler is usually (you can configure x.py to use something else) the current beta rustc compiler

Joshua Nelson (Jul 21 2020 at 21:13, on Zulip):

I'll make a PR fixing it

eddyb (Jul 21 2020 at 21:13, on Zulip):

I mean our terminology has been loose for ages

eddyb (Jul 21 2020 at 21:14, on Zulip):

"the stage0 compiler" is technically ambiguous and could be used to mean "the compiler that compiles all of stage0"

eddyb (Jul 21 2020 at 21:14, on Zulip):

rather than anything built at stage0

eddyb (Jul 21 2020 at 21:14, on Zulip):

so it's not "simply wrong", it's misleadingly ambiguous :P

Joshua Nelson (Jul 21 2020 at 21:15, on Zulip):

I mean, I would prefer it mean 'the thing I get when I run x.py build --stage 0 src/rustc

eddyb (Jul 21 2020 at 21:15, on Zulip):

if you go looking you'll probably find stuff like that everywhere

eddyb (Jul 21 2020 at 21:15, on Zulip):

sure but that's not inherent in "stage N compiler"

eddyb (Jul 21 2020 at 21:15, on Zulip):

you have to be more specific either way IMO

Joshua Nelson (Jul 21 2020 at 21:15, on Zulip):

that's how people think of it though

eddyb (Jul 21 2020 at 21:16, on Zulip):

"the compiler produced at/by X" vs "the compiler that compiles X"

eddyb (Jul 21 2020 at 21:16, on Zulip):

@Joshua Nelson it's not guaranteed which means we shouldn't use that phrasing IMO

eddyb (Jul 21 2020 at 21:16, on Zulip):

at least not in documentation

eddyb (Jul 21 2020 at 21:17, on Zulip):

some formulations just don't distinguish input from output clearly. and that's a "feature" of natural languages

Joshua Nelson (Jul 21 2020 at 21:18, on Zulip):

I think we should clarify we mean 'the compiler used to build stage0' then

eddyb (Jul 21 2020 at 21:18, on Zulip):

although I believe it's rather common for "the X Yer" to mean "the Y who Ys X" so the situation may be more ambiguous than you think

Joshua Nelson (Jul 21 2020 at 21:18, on Zulip):

leaving it ambiguous almost seems malicious :P

eddyb (Jul 21 2020 at 21:19, on Zulip):

I'd argue against even keeping the ambiguous phrasing unless it comes as a note that this may be how it's sometimes referred and a warning that it's ambiguous

eddyb (Jul 21 2020 at 21:19, on Zulip):

an example for what I was saying above: "the truck driver" means "the driver who drives the truck"

eddyb (Jul 21 2020 at 21:20, on Zulip):

@Joshua Nelson specifically what I think would be ideal is that the main language we use is unambiguous and anything else is an "aside" rather than go "ambiguous thing (i.e. unambiguous explanation)"

Joshua Nelson (Jul 21 2020 at 21:21, on Zulip):

well the docs certainly aren't using it in that sense

Joshua Nelson (Jul 21 2020 at 21:21, on Zulip):

because they mention 'stage3'

Joshua Nelson (Jul 21 2020 at 21:21, on Zulip):

(Optional) Stage 3: to sanity check our new compiler, we can build the libraries with the stage2 compiler. The result ought to be identical to before, unless something has broken.

eddyb (Jul 21 2020 at 21:21, on Zulip):

so they're using it in both senses :)

eddyb (Jul 21 2020 at 21:21, on Zulip):

or are confused. or both!

triagebot (Jul 22 2020 at 15:39, on Zulip):

This proposal has been accepted: #326.

Joshua Nelson (Jul 25 2020 at 10:35, on Zulip):

@simulacrum this is just waiting for your review of the implementation I think :) https://github.com/rust-lang/rust/pull/73964

Joshua Nelson (Aug 14 2020 at 12:39, on Zulip):

eddyb said:

so x.py build and x.py build --stage 1 and x.py build src/libstd and x.py build --stage 1 src/libstd would all do the same thing

eddyb said:

like, I think you will be able to shorten:
../../x.py build -i --stage 1 src/libstd
to:
../../x.py build -i
but you won't have to

so ... I think this is a great idea (and @RalfJ agreed on https://github.com/rust-lang/blog.rust-lang.org/pull/666#discussion_r470428415) but it didn't make it into the MCP. Is it ok if I make a follow-up MR with the implementation? Or should I open a new MCP for changing that default?

Joshua Nelson (Aug 14 2020 at 12:40, on Zulip):

to be clear, the current default is that x.py build and x.py build --stage 1 library/std src/tools/* are the same

Joshua Nelson (Sep 03 2020 at 13:22, on Zulip):

heh, this makes me really glad I made the change: https://github.com/rust-lang/compiler-team/issues/326#issuecomment-686464359

Last update: May 07 2021 at 07:00UTC