Stream: t-compiler

Topic: do we really need stage2..?


nikomatsakis (Mar 24 2020 at 19:12, on Zulip):

Hey @eddyb -- I was idly thinking to myself about how to make rustc development more amenable to incremental compilation. One of the challenges of course is the need to build a full stage2 build to get things working. If I recall, the main reason we need to do this is for symbol names and dynamic linking, right? (I mean, it's a good idea for when doing a full bootstrap anyway, but in terms of day-to-day development, it'd be nice to avoid.) Does the new work you did on a more a specified dynamic linking algorithm help at all here?

eddyb (Mar 24 2020 at 19:12, on Zulip):

I... I'll need to break that down

eddyb (Mar 24 2020 at 19:13, on Zulip):

Does the new work you did on a more a specified dynamic linking algorithm help at all here?

doesn't sound like anything I worked on

nikomatsakis (Mar 24 2020 at 19:13, on Zulip):

sorry I was just typing nonsense

nikomatsakis (Mar 24 2020 at 19:13, on Zulip):

doing too many things at once

nikomatsakis (Mar 24 2020 at 19:13, on Zulip):

I meant symbol naming algorithm

eddyb (Mar 24 2020 at 19:13, on Zulip):

building the compiler locally is never really needed for local work (unless you want to build clippy or miri which need custom drivers)

eddyb (Mar 24 2020 at 19:14, on Zulip):

I don't think symbol names factor into this at all

nikomatsakis (Mar 24 2020 at 19:14, on Zulip):

I could be just totally off base :)

nikomatsakis (Mar 24 2020 at 19:14, on Zulip):

but I definitely remember that at some point stage1 builds would fail with stuff like custom derive, and I believe this was related to symbol naming

centril (Mar 24 2020 at 19:14, on Zulip):

@nikomatsakis do people actually use stage2 for day to day dev?

eddyb (Mar 24 2020 at 19:15, on Zulip):

I've tested on stage1 for years now, we just don't have a way to tell x.py aka bootstrap to only build the compiler once but otherwise run as many tests as possible (instead of whitelisting a dozen testsuites like I do)

eddyb (Mar 24 2020 at 19:15, on Zulip):

@nikomatsakis aaaah I see. no, symbol names were just what you were seeing in errors

eddyb (Mar 24 2020 at 19:15, on Zulip):

that's incompatible proc_macro ABI to be exact

eddyb (Mar 24 2020 at 19:15, on Zulip):

I fixed it... some time last year, I think?

nikomatsakis (Mar 24 2020 at 19:15, on Zulip):

I used to use stage1 exclusively but at some point I changed to using stage2 because I was encountering too many annoying problems with stage2

nikomatsakis (Mar 24 2020 at 19:16, on Zulip):

but I'd be happy to know that's all outdated

centril (Mar 24 2020 at 19:16, on Zulip):

I use ./x.py --keep-stage 1 test --stage 1 --pass check --bless src/test/ui/ myself

nikomatsakis (Mar 24 2020 at 19:16, on Zulip):

(in which case I wonder if we should consider changing the default, or making some easy way so that when compiler devs setup their environment for home builds, stage1 is the default?)

nikomatsakis (Mar 24 2020 at 19:16, on Zulip):

at minimum we should adjust our rustc-dev-guide instructions here, although I confess I've not checked what they currently say

centril (Mar 24 2020 at 19:17, on Zulip):

(well, --keep-stage 1 after the first build on a new branch)

bjorn3 (Mar 24 2020 at 19:17, on Zulip):

eddyb said:

I fixed it... some time last year, I think?

You mean #49219?

eddyb (Mar 24 2020 at 19:17, on Zulip):

so, here's the thing: ./x.py test --stage 1 doesn't build the compiler only once

eddyb (Mar 24 2020 at 19:17, on Zulip):

@bjorn3 yeah I was looking for the link, but you're faster :P

eddyb (Mar 24 2020 at 19:17, on Zulip):

@nikomatsakis anyway you want @simulacrum. I've discussed this sort of thing with them before

Jonas Schievink (Mar 24 2020 at 19:18, on Zulip):

The only test that always fails for me when testing stage1 is the hotplug-codegen-backend test, so that might indeed have to do with dynamic linking

bjorn3 (Mar 24 2020 at 19:18, on Zulip):

Yeah, hotplug-codegen-backend needs to be compiled for the same rustc libraries as the rustc it will be linked to.

eddyb (Mar 24 2020 at 19:19, on Zulip):

it's so much worse

eddyb (Mar 24 2020 at 19:19, on Zulip):

so first of all that test clearly needs // ignore-stage1 or w/e

eddyb (Mar 24 2020 at 19:20, on Zulip):

but also, it's fulldeps. which means that even at --stage 1 it builds stage2!

eddyb (Mar 24 2020 at 19:20, on Zulip):

fulldeps tests are the bane of my existence and we should (IMO) eradicate them

eddyb (Mar 24 2020 at 19:20, on Zulip):

or at least change the way we run them? I don't even know anymore

Jonas Schievink (Mar 24 2020 at 19:21, on Zulip):

Why are they needed instead of being stage2-only tests?

simulacrum (Mar 24 2020 at 19:21, on Zulip):

imo, fulldeps tests should be eradicated, yes, and if really needed can be replaced with custom drivers I think (e.g., we build rustc-test-xx and the only thing it does is run a modified rustc that acts like the current fulldeps test did), which should be possible to do at any stage

eddyb (Mar 24 2020 at 19:22, on Zulip):

for the record, this is (still) my VSCode tasks.json:

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "remote",
            "type": "shell",
            "command": "yes '' | head -n $(tput lines) && clear -x && git push build -qf $((git stash create; echo HEAD) | head -n1):build-1 && ssh -t build.lyken.rs \"env SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt nix-shell -E 'with import<nixpkgs>{};clangStdenv.mkDerivation{name=\\\"name\\\";hardeningDisable=[\\\"all\\\"];buildInputs=[ccache cmake fish file gdb libedit libgit2 libxml2 ncurses ninja openssl pkgconfig python37 ripgrep swig which zlib];}' --run 'cd /home/eddy/Projects/rust-1 && git checkout -q --detach build-1 && ./x.py test --stage 0 src/tools/tidy && rm -f src/test/debuginfo/issue-22656.rs && ./x.py test --no-fail-fast --stage 1 src/test/{rustdoc,rustdoc-ui,mir-opt,codegen,codegen-units,incremental,pretty,debuginfo,compile-fail,run-make,run-fail,ui}'\"",
            "group": {
                "kind": "build",
                "isDefault": true
            },
            "options": {
                "shell": {
                    "executable": "bash",
                    "args": [
                        "-c"
                    ]
                }
            },
            "presentation": {
                "echo": false
            }
        }
    ]
}
eddyb (Mar 24 2020 at 19:22, on Zulip):

@Jonas Schievink the ignore-stage1 thing came later

bjorn3 (Mar 24 2020 at 19:23, on Zulip):

eddyb said:

so first of all that test clearly needs // ignore-stage1 or w/e

#70368

eddyb (Mar 24 2020 at 19:24, on Zulip):

@Jonas Schievink it's... weird. some *-fulldeps tests just use a tiny compiler dependency, like graphviz (I am not kidding, run-make-fulldeps/save-analysis is literally just that)

bjorn3 (Mar 24 2020 at 19:26, on Zulip):

^ For reference: https://github.com/rust-lang/rust/blob/2dcf54f564c6d8bbf48960fb9aaec88a0e2e062a/src/test/run-make-fulldeps/save-analysis/foo.rs

eddyb (Mar 24 2020 at 19:26, on Zulip):

lmao

eddyb (Mar 24 2020 at 19:27, on Zulip):

I love how some shotgun/exhaustive tests can read like a shitpost (the classic one is weird-syntax or w/e it was called)

bjorn3 (Mar 24 2020 at 19:28, on Zulip):

You mean weird-exprs.rs?

bjorn3 (Mar 24 2020 at 19:28, on Zulip):

// Just a grab bag of stuff that you wouldn't want to actually write.

bjorn3 (Mar 24 2020 at 19:28, on Zulip):

s/wouldn't/shouldn't :)

davidtwco (Mar 24 2020 at 19:42, on Zulip):

Stage two was useful for finding bugs in my polymorphisation work that compiling std or the tests didn't catch, there should still be a easy way to build stage two locally and it should be tested in CI. Prior to the polymorphisation, very rarely did I ever need stage two when hacking on rustc.

nikomatsakis (Mar 24 2020 at 19:44, on Zulip):

Yes to both those points

eddyb (Mar 24 2020 at 19:51, on Zulip):

IMO we should treat it like "full bootstrap" today (which today means stage2-std is actually built and not a copy of stage1-std)

eddyb (Mar 24 2020 at 19:51, on Zulip):

as in, you might want it in some cases, and it's on for CI

eddyb (Mar 24 2020 at 19:53, on Zulip):

eddyb said:

for the record, this is (still) my VSCode tasks.json:

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "2.0.0",
    "tasks": [
        {
            "label": "remote",
            "type": "shell",
            "command": "yes '' | head -n $(tput lines) && clear -x && git push build -qf $((git stash create; echo HEAD) | head -n1):build-1 && ssh -t build.lyken.rs \"env SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt nix-shell -E 'with import<nixpkgs>{};clangStdenv.mkDerivation{name=\\\"name\\\";hardeningDisable=[\\\"all\\\"];buildInputs=[ccache cmake fish file gdb libedit libgit2 libxml2 ncurses ninja openssl pkgconfig python37 ripgrep swig which zlib];}' --run 'cd /home/eddy/Projects/rust-1 && git checkout -q --detach build-1 && ./x.py test --stage 0 src/tools/tidy && rm -f src/test/debuginfo/issue-22656.rs && ./x.py test --no-fail-fast --stage 1 src/test/{rustdoc,rustdoc-ui,mir-opt,codegen,codegen-units,incremental,pretty,debuginfo,compile-fail,run-make,run-fail,ui}'\"",
            "group": {
                "kind": "build",
                "isDefault": true
            },
            "options": {
                "shell": {
                    "executable": "bash",
                    "args": [
                        "-c"
                    ]
                }
            },
            "presentation": {
                "echo": false
            }
        }
    ]
}

oops I forgot about this. what I meant to say is that this is the test command:
./x.py test --no-fail-fast --stage 1 src/test/{rustdoc,rustdoc-ui,mir-opt,codegen,codegen-units,incremental,pretty,debuginfo,compile-fail,run-make,run-fail,ui}

eddyb (Mar 24 2020 at 19:53, on Zulip):

oh wow literally a dozen test suites

centril (Mar 24 2020 at 19:59, on Zulip):

(run-fail & compile-fail are vestigial)

eddyb (Mar 24 2020 at 20:06, on Zulip):

but they still exist, right? ideally we'd get rid of them

eddyb (Mar 24 2020 at 20:06, on Zulip):

I wish refactoring compiletest would be easy

centril (Mar 24 2020 at 20:12, on Zulip):

@eddyb yea we want to move them to UI tests

mark-i-m (Mar 24 2020 at 23:53, on Zulip):

For whatever reason, I often end up with weird errors when I try incremental stage 1 builds, and now I just don't run anything and let the CI warn me if I can

mark-i-m (Mar 24 2020 at 23:54, on Zulip):

Also related: would it be possible to do a check-only compiler? that is, a compiler that is incapable of doing anything except a cargo check. Would that allow us to just run UI tests by building the check-only compiler with beta + incremental and then running ui tests with the check-only compiler?

eddyb (Mar 25 2020 at 05:57, on Zulip):

the only thing you could skip is LLVM I guess?

eddyb (Mar 25 2020 at 05:57, on Zulip):

@mark-i-m at a minimum you can just run ./x.py test --stage 1 src/ui

eddyb (Mar 25 2020 at 05:58, on Zulip):

would be useful to know what commands you got errors for and how you're enabling incremental, too

eddyb (Mar 25 2020 at 05:58, on Zulip):

since I set incremental = true in config.toml and never pass -i

eddyb (Mar 25 2020 at 05:58, on Zulip):

I think the -i flag is weird and easy to miss if you're typing commands by hand and I frankly don't know what will happen then, probably nothing good

mark-i-m (Mar 26 2020 at 04:50, on Zulip):

oh, hmm... maybe that's it... I didn't know there was a config.toml setting for that. I always use ./x.py test --stage 1 -i except that often results in weird compile errors or strange test failures, so then I resort to ./x.py test --stage 1 which often works but sometimes it doesn't. I haven't been keeping track of the exact tests that fail though. I can do that in the future

Santiago Pastorino (Mar 27 2020 at 22:07, on Zulip):

eddyb said:

since I set incremental = true in config.toml and never pass -i

I'm not doing this in general because incremental gives me troubles from time to time

Santiago Pastorino (Mar 27 2020 at 22:07, on Zulip):

right now ...

Santiago Pastorino (Mar 27 2020 at 22:07, on Zulip):
error: unused attribute
    --> src/libcore/num/mod.rs:4176:13
     |
2446 | / macro_rules! uint_impl {
2447 | |     ($SelfT:ty, $ActualT:ty, $BITS:expr, $MaxV:expr, $Feature:expr, $EndFeature:expr,                                                                                                                                                2448 | |         $rot:expr, $rot_op:expr, $rot_result:expr, $swap_op:expr, $swapped:expr,
2449 | |         $reversed:expr, $le_bytes:expr, $be_bytes:expr,
...    |
4176 | |             #[allow_internal_unstable(const_fn_union)]
     | |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...    |
4334 | |     }
4335 | | }
     | |_- in this expansion of `uint_impl!`
...
4850 | /     uint_impl! { u64, u64, 64, 18446744073709551615, "", "", 12, "0xaa00000000006e1", "0x6e10aa",
4851 | |     "0x1234567890123456", "0x5634129078563412", "0x6a2c48091e6a2c48",
4852 | |     "[0x56, 0x34, 0x12, 0x90, 0x78, 0x56, 0x34, 0x12]",
4853 | |     "[0x12, 0x34, 0x56, 0x78, 0x90, 0x12, 0x34, 0x56]",
4854 | |     "", ""}
     | |___________- in this macro invocation
Santiago Pastorino (Mar 27 2020 at 22:07, on Zulip):

getting a bunch of errors like this ^^^

Santiago Pastorino (Mar 27 2020 at 22:08, on Zulip):

so I prefer to pass -i unless it doesn't work like right now :)

eddyb (Mar 27 2020 at 22:12, on Zulip):

@Santiago Pastorino yeah that's a known bug, it just doesn't constantly trigger. I tend to nuke stage0-std when that happens

eddyb (Mar 27 2020 at 22:12, on Zulip):

still totally worth it

eddyb (Mar 27 2020 at 22:12, on Zulip):

(but yeah we should fix that bug already lol)

Santiago Pastorino (Mar 27 2020 at 22:43, on Zulip):

eddyb said:

Santiago Pastorino yeah that's a known bug, it just doesn't constantly trigger. I tend to nuke stage0-std when that happens

what do you mean exactly?

eddyb (Mar 27 2020 at 22:45, on Zulip):

exactly by what?

eddyb (Mar 27 2020 at 22:45, on Zulip):

it's an incremental bug that triggers from time to time and I rm -rf build/*/stage0-std when it happens

Santiago Pastorino (Mar 27 2020 at 22:46, on Zulip):

and after that it goes away?, it was consistently happening to me

eddyb (Mar 27 2020 at 22:46, on Zulip):

well, yes, unless you change libcore again

eddyb (Mar 27 2020 at 22:47, on Zulip):

I don't know the exact condition, I think enough code in libcore has to change, so that attributes go out of sync

Santiago Pastorino (Mar 27 2020 at 22:47, on Zulip):

I was doing some librustc_mir changes

eddyb (Mar 27 2020 at 22:47, on Zulip):

how could you see that failure? libcore can't change if you just edit librustc_mir

eddyb (Mar 27 2020 at 22:47, on Zulip):

were you switching between branches or rebasing?

Santiago Pastorino (Mar 27 2020 at 22:49, on Zulip):

so I've compiled last master

Santiago Pastorino (Mar 27 2020 at 22:49, on Zulip):

then switched to a branch that has one extra commit

Santiago Pastorino (Mar 27 2020 at 22:49, on Zulip):

rebased to put that one on top of master

Santiago Pastorino (Mar 27 2020 at 22:49, on Zulip):

and tried to compile

eddyb (Mar 27 2020 at 22:49, on Zulip):

in the process you've touched something in libcore

Santiago Pastorino (Mar 27 2020 at 22:49, on Zulip):

nope

eddyb (Mar 27 2020 at 22:49, on Zulip):

you can avoid that by rebasing directly, without switching to the branch you want to rebase first

Santiago Pastorino (Mar 27 2020 at 22:50, on Zulip):

but I see what you meant

eddyb (Mar 27 2020 at 22:50, on Zulip):

e.g. git rebase master my-branch

Santiago Pastorino (Mar 27 2020 at 22:50, on Zulip):

yeah, makes sense :+1:

Last update: Jun 04 2020 at 18:40UTC