Stream: t-compiler

Topic: more principled `--keep-stage 1`


nikomatsakis (Oct 10 2018 at 13:11, on Zulip):

So @mw I was thinking... --keep-stage 1 is an amazing time saver, but if you change the definition of things that wind up in metadata (or the metadata code itself) then things go haywire...

nikomatsakis (Oct 10 2018 at 13:11, on Zulip):

but it seems plausible to me that we could hash the definitions of all types that may wind up in metadata

nikomatsakis (Oct 10 2018 at 13:12, on Zulip):

I guess I'm thinking: it's plausible that we could make a variant of --keep-stage 1 that's "always on" and approximates when it can be safely used. Well, always-on isn't right of course, but you could make it a lot safer

nikomatsakis (Oct 10 2018 at 13:13, on Zulip):

I guess if you modified (e.g.) the type checker, you would always be foregoing the check to see that libstd and friends still type-check with your new code

nikomatsakis (Oct 10 2018 at 13:13, on Zulip):

(which in turn might affect — if not the structure of metadata — then its contents)

RalfJ (Oct 17 2018 at 09:52, on Zulip):

@nikomatsakis you mean --keep-stage 0 right? --keep-stage 1 will NOT keep stage 0. I have not yet found a case where keeping stage 1 is useful, but your workflow may differ?

pnkfelix (Oct 17 2018 at 10:30, on Zulip):

We have got to fix the rustc guide

pnkfelix (Oct 17 2018 at 10:31, on Zulip):

I stopped trying to use --keep-stage some time ago because I couldn't understand/predict when it would go wrong

pnkfelix (Oct 17 2018 at 10:31, on Zulip):

and then I recently tried to use --keep-stage 1, based on the rustc guide. And I again cannot understand/predict when it will go wrong. Except perhaps the prediction is "all the time" is about right.

pnkfelix (Oct 17 2018 at 10:44, on Zulip):

In any case, any idea that @nikomatsakis and/or @mw have for making --keep-stage N be more defensive about its behavior gets a big :thumbs_up: from me

oli (Oct 17 2018 at 10:46, on Zulip):

I use --keep-stage 0 --keep-stage 1 when I'm modifying unit tests in e.g. librustc_datastructures

pnkfelix (Oct 17 2018 at 11:18, on Zulip):

So, here's a simple question: Is --keep-stage 0 expected to fail in weird ways if you use the flag on a clean checkout?

pnkfelix (Oct 17 2018 at 11:20, on Zulip):

In particular, I keep a shell script that I use to drive my builds. Currently it doesn't attempt to keep track whether this is a clean build or if we're building atop a previous build attempt. I had assumed that --keep-stage would fall back on building stuff if the things it needs are not present.

pnkfelix (Oct 17 2018 at 11:24, on Zulip):

I should probably go look at how its actually implemented rather than try to guess the semantics here.

RalfJ (Oct 17 2018 at 11:36, on Zulip):

@pnkfelix yeah I'd assume --keep-stage to just skip even trying to do any stage 0 build job

pnkfelix (Oct 17 2018 at 11:37, on Zulip):

I guess I would prefer if it fell back on building the stuff it needs if the needed files aren't even there in the first place

pnkfelix (Oct 17 2018 at 11:38, on Zulip):

potentially with some printouts saying that its doing so

RalfJ (Oct 17 2018 at 11:38, on Zulip):

so what I usually do when I work on librustc_mir and need to build miri (which needs a stage 2 build): I first do ./x.py build src/rustc, once, to get a clean everything; then then subsequent builds are ./x.py --keep-stage 0 build src/rustc. That takes around 5min then (I feel it was faster some weeks ago...)

pnkfelix (Oct 17 2018 at 11:38, on Zulip):

you do src/rustc instead of src/libstd ?

RalfJ (Oct 17 2018 at 11:38, on Zulip):

yeah I need a fully working stage 2

RalfJ (Oct 17 2018 at 11:38, on Zulip):

IIRC src/libstd doesnt give me that

RalfJ (Oct 17 2018 at 11:39, on Zulip):

the difference between the two is negligible though with stage 2

RalfJ (Oct 17 2018 at 11:39, on Zulip):

it just copies some more stuff around

pnkfelix (Oct 17 2018 at 11:39, on Zulip):

oh sorry I overlooked that part of your message

RalfJ (Oct 17 2018 at 11:39, on Zulip):

("fully working" = including proc macros)

RalfJ (Oct 17 2018 at 11:39, on Zulip):

if it wasn't for that I'd just always do ./x.py --stage 1 build src/libstd and would never even need --keep-stage

nikomatsakis (Oct 17 2018 at 13:06, on Zulip):

I never use --keep-stage 0

nikomatsakis (Oct 17 2018 at 13:07, on Zulip):

I have no idae what it does

nikomatsakis (Oct 17 2018 at 13:07, on Zulip):

I use only --keep-stage 1, which I understand :)

nikomatsakis (Oct 17 2018 at 13:07, on Zulip):

so, no, I did not mean --keep-stage 0

nikomatsakis (Oct 17 2018 at 13:07, on Zulip):

The instructions in the rustc-guide match precisely what I do, but I'm open to there being a better set of things to do :)

nikomatsakis (Oct 17 2018 at 13:08, on Zulip):

in particular, I build with --keep-stage 1 but only after I've already got a succesful build lying around ...

nikomatsakis (Oct 17 2018 at 13:09, on Zulip):

and then I recently tried to use --keep-stage 1, based on the rustc guide. And I again cannot understand/predict when it will go wrong. Except perhaps the prediction is "all the time" is about right.

interesting @pnkfelix . I do not find this at all. It basically goes wrong if I have rebased or something, but if I've just made a small edit to some fn, it is generally fine. (And if things "seem fishy", then I just try removing it.)

Interestingly, this aligns basically perfectly with my experiences with makefiles in the past. That is, makefiles never truly capture all the dependencies, so it's usually "hmm something is weird, better make clean". :P

nikomatsakis (Oct 17 2018 at 13:10, on Zulip):

but I guess my basic attitude is to "add" --keep-stage 1 when iterating, not to use it by default

pnkfelix (Oct 17 2018 at 13:10, on Zulip):

I think the core of my problem was my assumption was that i didn't actually need to start with the base run

nikomatsakis (Oct 17 2018 at 13:10, on Zulip):

ah. yes, that would go wrong

pnkfelix (Oct 17 2018 at 13:10, on Zulip):

i.e. that --keep-stage 1 would be "smart" and build things that are truly not present

pnkfelix (Oct 17 2018 at 13:11, on Zulip):

which is .... well, I won't claim its what make always does. But its what make is supposed to do, given a "correct" Makefile

nikomatsakis (Oct 17 2018 at 13:31, on Zulip):

which is .... well, I won't claim its what make always does. But its what make is supposed to do, given a "correct" Makefile

yes just ime never actually the case :P

nikomatsakis (Oct 17 2018 at 13:32, on Zulip):

or, much like with x.py, if the makefile is correct -- then it is overconservative, and it offers some escape valves, which is exactly what --keep-stage 1 is...

RalfJ (Oct 17 2018 at 13:41, on Zulip):

I use only --keep-stage 1, which I understand :)

funny. I am dead scared of it.^^

RalfJ (Oct 17 2018 at 13:41, on Zulip):

it recompiles stage0 but then does NOT recompile stage1.

RalfJ (Oct 17 2018 at 13:43, on Zulip):

I have no idea what your workflow is like, to make that useful^^

nikomatsakis (Oct 17 2018 at 13:44, on Zulip):

1. build compiler
2. add a debug! log
3. rebuild compiler, but do not rebuild libraries <-- this is where --keep-stage 1 makes sense

nikomatsakis (Oct 17 2018 at 13:44, on Zulip):

a variation on (2) is "modify some part of the compiler that the libraries do not use"

RalfJ (Oct 17 2018 at 13:48, on Zulip):

so what is your full invocation then?

nikomatsakis (Oct 17 2018 at 13:49, on Zulip):

./x.py build -i --stage 1 src/libstd is my "base invocation" 99% of the time

nikomatsakis (Oct 17 2018 at 13:49, on Zulip):

I tack on --keep-stage 1 if I just made a small alteration

nikomatsakis (Oct 17 2018 at 13:50, on Zulip):

which of course is actually the vast majority of the time

nikomatsakis (Oct 17 2018 at 13:50, on Zulip):

so in terms of # of builds I suppose that's the most common thing

RalfJ (Oct 17 2018 at 13:52, on Zulip):

oh so you have --stage 1 --keep-stage 1. that makes some more sense, I get it.

RalfJ (Oct 17 2018 at 13:52, on Zulip):

libstd builds in <60s without -i so I dont usually bother optimizing that

RalfJ (Oct 17 2018 at 13:53, on Zulip):

and for proc macros I need the starts to align or so, haven't found a way to cut that shorter. I guess I could try --keep-stage 0 --keep-stage 2 --stage 2...

Last update: Nov 20 2019 at 01:45UTC