When cross-compiling from x86_64 to aarch64, the native rustlib directory is empty in the target compiler. However, my x86_64 compiler has the rustlibs for the _target_:

% ls build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib | cat
libLLVM.dylib

% ls build/x86_64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib | cat
liballoc-bc1bed71af59030d.rlib
libbacktrace-908ce5b600cbb5e4.rlib
libbacktrace_sys-73640f972bd0168e.rlib
libcfg_if-ed32d5eef52668e4.rlib
libcompiler_builtins-48b6ebeb945f7f23.rlib
libcore-eae2616245cf1435.rlib
libgetopts-4868046cc68f6a07.rlib
libhashbrown-4019ac1df1726b61.rlib
liblibc-147e2bbc2c8920b5.rlib
libpanic_abort-2d6d15cfb868ff37.rlib
libpanic_unwind-5fba9238dd90a238.rlib
libproc_macro-7b75bc3478afb07d.rlib
librustc_demangle-022c7d6aca3d2ccc.rlib
librustc_std_workspace_alloc-322b374d69dcba49.rlib
librustc_std_workspace_core-3322a8e8a27ca844.rlib
librustc_std_workspace_std-12218c207bcf61af.rlib
libstd-6247a4aa0eaf3fee.dylib
libstd-6247a4aa0eaf3fee.rlib
libterm-fe16d30007d35333.rlib
libtest-23728a41e0e6f04a.dylib
libtest-23728a41e0e6f04a.rlib
libunicode_width-e2337f69b5becf1b.rlib
libunwind-65eb1f72892d4a26.rlib
self-contained

Is this "normal" for a cross-compile? Did I do something wrong?

probably. there are 3 triples: "target", "host" and "build"

I can never remember what the "host" vs "build" distinction is

but AFAIK you use --target to get cross-compilation stdlib

@Jake Goulding actually, wait, I misread, why would build/aarch64-apple-darwin exist/work?

it was in the host array, I think

unless you can execute aarch64-apple-darwin binaries, that directory shouldn't make sense I don't think?

we can

is this using the OS-level emulation layer thing?

no, I have physical arm64 macOS hardware

and that build above produces a rustc that works, but complains about a missing libstd

I am talking about a system that can execute both aarch64-apple-darwin and x86_64-apple-darwin

oh I see, yes, the physical hardware can execute both

AFAIK that's the only reason to have both under ./build/

so what's the right way to cross-compile from x86_64 to aarch64 then?

@André Arko the hardware you're building Rust on?

which one is the native and which one is it emulating?

okay let me back up :)

I have 1 intel x86_64 macOS machine, and 1 apple arm64 macOS machine

to be clear: I'm asking about the machine that has both directories under ./build/

no other machine

that directory was produced by @Jake Goulding's x86_64 machine, where he was trying to cross-compile a rustc that would work an arm64

he succeeded, the output rustc works when copied to the arm64 hardware

can the x86_64 machine run aarch64 executables?

no

okay then AFAIK build/aarch64-apple-darwin shouldn't exist on that machine

the one inside stage1, or the one containing stage1?

oh you mean the top level one?

yes, that's the one that depends on the hardware you're running on. or at least I haven't seen it any other way

...your message is very confusing because the entire point of running the build was to create that directory

like, cross compiling an arm64 rustc from x86_64

hmmmmmm

and it worked, build/aarch64-apple-darwin contained a working rustc at the end

oh I see, our buildsystem is even more confusing than I expected

but, confusingly, it was missing libstd

unlike the x86_64 directory, which was not missing libstd

(it can build anything cross-compiled into build/x86_64-apple-darwin, it's just the sysroot that wouldn't make sense)

(literally copying the libs from x86_64 to aarch64 made it a full, working rustc, which I have then used to cargo install ripgrep and some other things successfully)

huh okay

in that case, what would be the "right" way to do that?

not sure. to expand a bit, we have build/$tripleX/stage0-rustc/release/ and build/$tripleX/stage0-rustc/$tripleY/release/

the former will contain things like build scripts and proc macros, that have to run on the same hardware that's doing the build

while the latter will contain cross-compilation artifacts

and I'm not even sure $tripleX matters for those

ah ok, interesting

(meaning we might be able to simplify our directory structure. not sure though)

so (iirc) the configuration was build="x86_64-apple-darwin" and host=["aarch64-apple-darwin"]

however, what you are talking about is the sysroots, e.g. build/$tripleX/stage1 (which has bin/rustc and lib/rustlib/$tripleZ etc.)

that's what I was missing at first, those don't have more than $tripleX for the binaries

ahh, yes that seems right

@André Arko @Jake Goulding okay now that I'm not (as) confused... did you copy directories from build/ or try to use installing into a temporary directory?

the former is not really supported

except for rustup toolchain link and that's for non-cross-compiled rustc anyway

I think you need something like DESTDIR=some/empty/dir ./x.py install

and then that directory should contain bin, lib, etc, etc.

ahh, okay. I didn't see anything in the rustc book that explained that, but maybe I missed it

I was confused about why everything seemed so scattered in build/ :)

yeah you're not supposed to poke around in there

I guess this is just ./configure && make install tbh

that's where the DESTDIR env var convention comes from

this is how distros (are supposed to) build rustc

got it, I'm not familiar with DESTDIR (only ./configure --prefix) so that didn't occur to me

we also had the small intermediate problem of needing to edit rustc, build a new one, and use that one to run the cross-compile build

--prefix is where you'll install stuff (so /usr typically), DESTDIR is whatever dir to dump the actual files in

that the packager will put into a rpm, deb, etc.

got it

thanks, this is helpful

I only learned about it a few months ago, from someone trying to package Rust without using Python (that was... a lot of fun :P)

you should also be able to create whatever rustup downloads and install, using... ./x.py dist I think?, but presumably that's harder to use than installing into a directory that you then copy over

@André Arko this seems useful https://www.gnu.org/prep/standards/html_node/DESTDIR.html

this also mentions DESTDIR https://www.debian.org/doc/manuals/maint-guide/modify.en.html#destdir

so yeah it's a common convention

yeah, makes sense! I clearly have never built linux distro packages before :)

@simulacrum what do you think about replacing build/$triple/stage$n-$foo with just build/stage$n-$foo? given that inside each of those, there's $triple/release already for the target, as per Cargo. although maybe this is a half-baked idea

maybe build/$triple is for the "build" triple which if you changed, a different beta would be downloaded, right? so it'd make sense to keep the current stratification

but then why would this:

so (iirc) the configuration was build="x86_64-apple-darwin" and host=["aarch64-apple-darwin"]

result in build/aarch64-apple-darwin existing?

Otherwise you'd clobber the stage2/bin/rustc (and everything else in there)

@simulacrum maybe the question i should be asking is "what is the right invocation of x.py?"

You're trying to produce a full toolchain, cross-compiling?

Yeah. I have one, but I'm doing copy-pasting of files myself.

https://github.com/shepmaster/rust/blob/silicon-bootstrap/silicon/README.md#copy-the-cross-compiler-to-the-dtk

I would look at one of the dist builders, say for ppc - IIRC, those cross compile currently so you can likely make use of the same invocation

(swapping dist for build perhaps)

That seems overly simple

Hm that looks about right. I'd try it

Target for std, host for rustc

So x.py dist --host aarch64-apple-darwin --target aarch64-apple-darwin and that will produce a single directory I could copy around?

Can I mix in a stage 1 in there too?

Should, I think, yeah

I think so

s/dist/build/

Though it won't speed things up I imagine

Since you're building the compiler again anyway

I could be wrong though

It is the nature of the beast

@Jake Goulding in case you didn't see it, I suggested using DESTDIR=some/empty/dir x.py install instead of copying things from the build directory

wasn't sure what dist does but it sounds similar

Dist produces the .gz and .xz files. Install is probably more want I want for “copy things around”

simulacrum said:

Otherwise you'd clobber the stage2/bin/rustc (and everything else in there)

if you're not going to execute it, why should it be in stageN/bin/rustc?

@Jake Goulding note that the intended "automated distro build" workflow is ./configure ... && make && make DESTDIR=... install

so you don't even need to touch x.py or config.toml if you want to automate this

"automate" is not quite a goal, beyond allowing rustup install. Mostly just documenting what we need to do as we get to there.

fair enough. I wonder if you can make x.py dist work with rustup install without having the components published on the official server, I sadly doubt it because I've seen a separate tool being needed even just to install PR artifacts (which are similar to nightlies but on a different S3 bucket or something)

Yeah. I think I asked about something similar before — would have loved to publish AVR-enabled rustc builds that were usable by rustup somehow.

@simulacrum basically I'm suggesting that the triple in build/$triple always be the build triple (and all of the sysroots would be of that triple), and host triples would only exist under e.g. stageN-rustc/$triple/release

@simulacrum and only x.py install / x.py dist would know how to create a sysroot for a host triple different than the build triple

is that too convoluted? idk

oh it's probably possible, just not what we do today and may break some expectations

seems like a "good" change though generally speaking

also if this works the next question is "do we need the stratification of build/$triple or can we just expect the build triple to just never change on a given system"

IIRC someone complained when I said it probably never changes, iirc talking about running i686 and x86_64 compilers on the same platform or something

(i.e., build compilers)

Ditto x86_64 and aarch64 in my case :-)

no, build triples you have to be able to run on the machine you're building on

Yes, and the new apple machines can run both :-)

you have a x86_64 build triple and a aarch64 host triple

wrt to making rustup work, you can do it if you don't mind being "nightly" or "beta" or "stable"

@Jake Goulding sure, that's why I kept asking confusing questions last night :P

(we do so for say dev-static)

RUSTUP_DIST_SERVER=https://dev-static.rust-lang.org rustup -v update stable with a URL that's to, say, jakes-apple-builds.com would in theory work, you'd just need the same directory structure and manifests

(because it wasn't clear what machine these dirs were on. IMO the only reasons to have more than one dir under build/ is 32-bit vs 64-bit or musl vs glibc, that sort of t hing)

even then we can probably say "edge case" and tell people to use different build/ directories

@simulacrum huuuuh do PR builds not work like that because rustup doesn't understand their versioning?

well rustup has 3 channels

it wasn't clear what machine these dirs were on

Yep, we've been bouncing around a bit — as you do when getting a new platform going :-)

maybe you could finagle the PR builds as "future nightlies" but there's not really support in rustup for hashes I think

it's long been on my todo list to try and get that support integrated

NixOS patches rustup to auto-patch toolchains to work directly, so anything that uses rustup as a dependency, as opposed to executing the one on the system, will be missing the requisite patchelf invocations :(

I have a bunch of patchelf commands in my fish history so this isn't too bad but I don't like it very much

@simulacrum anyway, thanks for the explanation! I was mostly trying to clean up my own understanding. and hoping we can make this less confusing in the future

@Jake Goulding sure, but based on having both dirs under build/ I thought this was a machine that could run both, but that was wrong (as made clear by @André Arko)

Well, it depends who and when you ask :wink:. I currently am building the rustc compiler on the aarch64 machine. that machine can run both aarch64 and x86-64 binaries natively.

uhhhhhhh @André Arko claimed the situation you were in was on a x64 machine

"natively" as in ./some-x86-binary and ./some-aarch64-binary; there's still some emulation happening somewhere

That's what I mean by "when". At one point we only were able to cross-compile. Now we are able to natively compile.

But I'm still doing steps on both machines, in order to be able to reproduce this elsewhere.

@simulacrum the thing that is fascinating to me and which wasn't really there ages ago in this form, is the existence of stageN-X/$triple/release, which comes from how Cargo target dir works I guess, and which can service both host and target triples equally, leaving stageN-X/release to be the build triple, implicitly

hm yeah that's been a thing in cargo for as long as I remember but maybe we weren't always passing --target to cargo? not sure

we weren't using Cargo pre-"rustbuild" :P

oh right

and I'm not sure we've fully looked into the consequences of that - another example is the effort being put into making "rustbuild" less necessary for doing builds, and relying on Cargo more

@Jake Goulding oh btw, speaking of this whole thing, are you aware of the person trying to get aarch64 builds going, on Discord?

they had a lot of trouble and I wasn't sure who to point them to

which channel?

#compiler, scroll up and you'll find a bunch of stuff

I expected other people would also try to do this sort of thing, and that someone would know more about LLVM and buildsystems, enough to deal with everything quicker, I just didn't know who to point them to

they eventually got rustc running on iOS

yeah, the ios person

I think this is them https://github.com/rust-lang/rust/issues/73628

I just feel bad if they wasted their time

due to lack of coordination

jokes on you, I know nothing, other than to ask smarter people :wink:

I mean idk who the macOS ppl are

other than acrichto who is probably too busy

that issue sounds relevant but I wonder if there's two people

oh, @aspen, they're on here as well

The main place for ARM stuff is #t-compiler/arm. I've just been popping out into other streams for help from a broader perspective

  DESTDIR=/tmp/crossed \
  ../../x.py install --stage 1 --host aarch64-apple-darwin --target aarch64-apple-darwin,x86_64-apple-darwin --warnings warn

Does seem to do what I'd want though!

nice :D

isn't there a cargo-lipo as well :P?

I know of it, but I've never used it. Figured I'd do the simpler thing for now. I'm sure that project will be picking up steam soon.