Stream: t-compiler/major changes

Topic: two-step bootstrap process for aarch64-apple-darwin


Jake Goulding (Jul 18 2020 at 11:40, on Zulip):

I landed Teach bootstrap about target files vs target triples, which did allow for me to cross-compile rustc in a single step, but then the created rustc failed when it tried to generate code as it couldn't find the host information.

eddyb (Jul 18 2020 at 11:44, on Zulip):

uh oh

Jake Goulding (Jul 18 2020 at 11:44, on Zulip):

This is the error we got

❯ ./stage1/bin/rustc hello.rs
error: Error loading target specification: Could not find specification for target "/Users/shep/Projects/rust/silicon/aarch64-apple-darwin.json". Use `--print target-list` for a list of built-in targets
eddyb (Jul 18 2020 at 11:44, on Zulip):

is that with a JSON target spec file? or by changing rustc_target::spec?

eddyb (Jul 18 2020 at 11:45, on Zulip):

I don't think we support anything other than the latter for the host, but it might be possible to make it work (I just haven't heard of this being attempted before)

Jake Goulding (Jul 18 2020 at 11:46, on Zulip):

I was attempting to use the JSON spec to avoid requiring the double-compiler-cross-compile-bootstrap.

Jake Goulding (Jul 18 2020 at 11:49, on Zulip):

I expect that the next change would be to modify code around here to support some new -Z --host-target-override myfile.json

eddyb (Jul 18 2020 at 11:49, on Zulip):

hmm. if we had landed it as a built-in target, it would be in beta by now (but I don't know what the requirements and stability implications are around that)

eddyb (Jul 18 2020 at 11:50, on Zulip):

we could still backport a new target to the beta, to make this easier (assuming that's all that's needed)

Jake Goulding (Jul 18 2020 at 11:51, on Zulip):

I haven't wanted to submit the target as a PR because it requires forcibly linking to zlib, as mentioned in another thread.

eddyb (Jul 18 2020 at 11:51, on Zulip):

ahhhh fun

eddyb (Jul 18 2020 at 11:51, on Zulip):

I would've thought that's also doable through RUSTFLAGS but maybe not that specific "post link flags" aspect

eddyb (Jul 18 2020 at 11:52, on Zulip):

also wow GitHub likes to do something very weird for that kind of link

eddyb (Jul 18 2020 at 11:52, on Zulip):

(scroll down a bit after refreshing to see it)

eddyb (Jul 18 2020 at 11:54, on Zulip):

@Jake Goulding anyway if the static zlib stuff gets fixed on beta, maybe you can coordinate with @simulacrum to get this target in as well into that beta build

eddyb (Jul 18 2020 at 11:55, on Zulip):

(but again I have no idea what kind of approval would be needed for a new builtin target, or if we have a way to make it unstable for now - technically possible, but idk if the plumbing has been implemented)

Jake Goulding (Jul 18 2020 at 11:55, on Zulip):

I think that @simulacrum has mentioned that they'd be open to beta backport of the target due to the relative interest for the target in general.

eddyb (Jul 18 2020 at 11:55, on Zulip):

nice :D

Jake Goulding (Jul 18 2020 at 11:56, on Zulip):

I'm also hoping for a new release of libc so the two dependent crates can be updated to stable releases.

eddyb (Jul 18 2020 at 11:57, on Zulip):

as for making things work with a JSON file, I don't see how we could do that without serializing the actual contents and ignoring the file path. maybe we do that already to some extent? unsure exactly how all that works atm

simulacrum (Jul 18 2020 at 11:57, on Zulip):

Yeah, if we're not testing it on ci IMO it's fine to backport to beta, doesn't really matter. We can remove it before it hits stable if there's any concerns there too.

Jake Goulding (Jul 18 2020 at 11:58, on Zulip):

I wonder if (lack of) stability of targets and their names should be discussed on the tiers RFC

Jake Goulding (Jul 18 2020 at 12:00, on Zulip):

eddyb said:

as for making things work with a JSON file, I don't see how we could do that without serializing the actual contents and ignoring the file path. maybe we do that already to some extent? unsure exactly how all that works atm

Why wouldn't a permanently unstable command line option, like I suggested earlier, be a viable option?

eddyb (Jul 18 2020 at 12:01, on Zulip):

seems like it would require the file to always exist, and that flag to be always passed?

eddyb (Jul 18 2020 at 12:01, on Zulip):

like there's no doubt it could work, I just don't see it being ergonomic

Jake Goulding (Jul 18 2020 at 12:02, on Zulip):

Yes, and I think that's appropriate. AFAICT, this is only useful during initial exploratory testing of cross-compiling rustc to a new platform.

eddyb (Jul 18 2020 at 12:02, on Zulip):

oh wait I was thinking about rlibs but rustc itself would have to know about it heh

eddyb (Jul 18 2020 at 12:02, on Zulip):

like, when it starts up. I wonder why this doesn't "just work" with rustc_target::spec being modified

eddyb (Jul 18 2020 at 12:03, on Zulip):

I guess it remembers the file path?

eddyb (Jul 18 2020 at 12:03, on Zulip):

instead of the triple

Jake Goulding (Jul 18 2020 at 12:03, on Zulip):

eddyb said:

I guess it remembers the file path?

Yes, the error message seems to suggest that.

eddyb (Jul 18 2020 at 12:04, on Zulip):

maybe the simplest workaround is similar to what you did to src/bootstrap, but for the compiler's hardcoded host target

eddyb (Jul 18 2020 at 12:05, on Zulip):

@Jake Goulding like, here: https://github.com/rust-lang/rust/blob/master/src/bootstrap/builder.rs#L1250

eddyb (Jul 18 2020 at 12:06, on Zulip):

although that looks like it already passes the triple and not the file? confusing

eddyb (Jul 18 2020 at 12:06, on Zulip):

yeah you literally changed that lol

eddyb (Jul 18 2020 at 12:07, on Zulip):

@Jake Goulding was the error before or after the src/bootstrap changes?

eddyb (Jul 18 2020 at 12:10, on Zulip):

to be clear: this is the only place I could find where the information is passed in https://github.com/rust-lang/rust/blob/d3df8512d2c2afc6d2e7d8b5b951dd7f2ad77b02/src/librustc_session/config.rs#L568-L578

eddyb (Jul 18 2020 at 12:10, on Zulip):

@simulacrum btw can we move this topic somewhere else? maybe #t-compiler/arm? I just noticed we're in the MCP stream oops

Jake Goulding (Jul 18 2020 at 12:11, on Zulip):

I haven't tested cross-compiling with only the JSON spec file with the most recent set of bootstrap changes. I'd expect that the error will have changed from cant find target /path/to/my-target.json to cant find target my-target. That is, the triple is passed, but rustc still wouldn't know about it.

eddyb (Jul 18 2020 at 12:11, on Zulip):

if you change rustc_target::spec and use a JSON file, both beta and the built compiler should work, I think?

eddyb (Jul 18 2020 at 12:12, on Zulip):

since the compiler you just built will look in rustc_target::spec for the triple

eddyb (Jul 18 2020 at 12:12, on Zulip):

but idk if anything else will break

Jake Goulding (Jul 18 2020 at 12:13, on Zulip):

Ah, making both changes at the same time; I didn't get that.

eddyb (Jul 18 2020 at 12:13, on Zulip):

sorry, I assumed (without checking) that the fork includes this commit

eddyb (Jul 18 2020 at 12:14, on Zulip):

(and that the bootstrap changes were only to get beta to use the target JSON file)

Jake Goulding (Jul 18 2020 at 12:17, on Zulip):

Well, that's certainly worth trying, as it would reduce the total cross-compile time from 1.5 hours to 1 hour (at least until the target is available in beta). It might also allow me to cheat and have the -lz hack in the JSON file but not in rustc_target, which would make me feel better about merging it sooner.

eddyb (Jul 18 2020 at 12:17, on Zulip):

heh, good idea

eddyb (Jul 18 2020 at 12:18, on Zulip):

@simulacrum @nagisa okay this might be silly, but there's nothing (at the OS level, I guess) stopping --target or an env var from taking in an entire JSON object, is there :P

eddyb (Jul 18 2020 at 12:19, on Zulip):

could be really compact if it refers to some other triple as the base, and just changes a field or two

simulacrum (Jul 18 2020 at 12:19, on Zulip):

Hm on Windows there's a really rather low command line length limit, but you could always use the file-with-arguments syntax to work around that

eddyb (Jul 18 2020 at 12:20, on Zulip):

/me grumbles

eddyb (Jul 18 2020 at 12:20, on Zulip):

@simulacrum anyway what I was thinking was more on the lines of src/bootstrap passing an entire JSON string in an env var to the rustc build (i.e. for an env!("..."))

simulacrum (Jul 18 2020 at 12:21, on Zulip):

Yeah I'd personally be fine with that

eddyb (Jul 18 2020 at 12:21, on Zulip):

we can avoid it this time, I think, but it could be a fun way to make ad-hoc rustc's that only know of their host target spec that way

eddyb (Jul 18 2020 at 12:22, on Zulip):

like a "go fast" rustc specialized to the latest Intel or AMD CPUs etc.

Jake Goulding (Jul 18 2020 at 12:23, on Zulip):

Or ARM CPUs... :upside_down:

eddyb (Jul 18 2020 at 12:23, on Zulip):

@Jake Goulding oh also it's possible rustc didn't pass host_triple() through the "load this is it's a file path" codepath

eddyb (Jul 18 2020 at 12:24, on Zulip):

ahahaha yes https://github.com/rust-lang/rust/blob/d3df8512d2c2afc6d2e7d8b5b951dd7f2ad77b02/src/librustc_session/config.rs#L1383-L1394

eddyb (Jul 18 2020 at 12:24, on Zulip):

it doesn't treat host_triple() as a default for --target, but rather only as a non-file triple

eddyb (Jul 18 2020 at 12:25, on Zulip):

match Some(matches.opt_str("target").unwrap_or(host_triple())) might be enough for a quick test

eddyb (Jul 18 2020 at 12:26, on Zulip):

(with src/bootstrap changed to pass the path instead of the triple, to CFG_COMPILER_HOST_TRIPLE)

Jake Goulding (Jul 18 2020 at 12:26, on Zulip):

We hacked it to forcibly use it as a path. It didn't work, but I don't recall why.

eddyb (Jul 18 2020 at 12:27, on Zulip):

awww, alright

Notification Bot (Jul 18 2020 at 13:34, on Zulip):

This topic was moved by simulacrum to #t-compiler/arm > two-step bootstrap process for aarch64-apple-darwin

Last update: May 07 2021 at 06:15UTC