Stream: t-compiler/help

Topic: cargo for rustc


Jason Hu (Jul 22 2020 at 14:52, on Zulip):

It's probably a random question: why are we using a separate x.py for build of rustc, instead of using cargo? I can see many disadvantages of x.py liking rebuilding after certain actions. I am just wondering what's the motivation behind.

simulacrum (Jul 22 2020 at 14:54, on Zulip):

cargo is insufficiently powerful currently for staged builds and such

simulacrum (Jul 22 2020 at 14:54, on Zulip):

we're slowly moving towards more cargo and less custom tooling atop it, though

lzutao (Jul 22 2020 at 14:54, on Zulip):

It is documented somewhat at https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html#bootstrapping-the-compiler

Jake Goulding (Jul 22 2020 at 14:58, on Zulip):

liking rebuilding after certain actions

Give rebuilding LLVM 2-3 times a shot to compare the alternative ;-)

Goat (Jul 22 2020 at 15:03, on Zulip):

we're slowly moving towards more cargo and less custom tooling atop it, though

Isn'tcargo make powerful enough for this already? I know it's not part of official Cargo, but it's not out of the question that someday we might integrate cargo make functionality into cargo build or something like that, to make things vastly simpler for projects with FFI or which otherwise depend on cargo make running to get a full functional build.

simulacrum (Jul 22 2020 at 15:11, on Zulip):

I think we could plausibly switch to cargo subcommands but it'd be a big migration and not really worth it IMO

Jake Goulding (Jul 22 2020 at 15:12, on Zulip):
alias cargo-bootstrap x.py
Joshua Nelson (Jul 22 2020 at 15:14, on Zulip):

Jake Goulding said:

alias cargo-bootstrap x.py

in case anyone does want to use that - alias cargo-bootstrap='cd $(git rev-parse --show-toplevel) && ./x.py' should work even if you're not in the top-level directory

Joshua Nelson (Jul 22 2020 at 15:15, on Zulip):

it won't work with cargo bootstrap though because it's an alias, you'll have to put it in ~/bin/cargo-bootstrap or something for that to work

Josh Triplett (Jul 23 2020 at 18:32, on Zulip):

Is the staged bootstrapping that can't be done with Cargo still complex enough that it needs Python, or could it theoretically be done with a shell script?

Josh Triplett (Jul 23 2020 at 18:32, on Zulip):

It would be nice to reduce the set of tools required to bootstrap Rust.

Josh Triplett (Jul 23 2020 at 18:33, on Zulip):

Especially since Rust will become part of more and more bootstrap cycles in other parts of a normal system.

Joshua Nelson (Jul 23 2020 at 19:15, on Zulip):

@Josh Triplett bootstrap is written in both python and rust. I'm not sure how complicated the pure python bit is, @simulacrum would know better, but here's how big it is:

(-bash@build-server) ~/.../src/bootstrap $ wc -l *.py
 1026 bootstrap.py
  107 bootstrap_test.py
  474 configure.py
 1607 total
Joshua Nelson (Jul 23 2020 at 19:16, on Zulip):

but it also requires a bunch of host tools as well: https://rustc-dev-guide.rust-lang.org/building/prerequisites.html?highlight=curl#dependencies

Joshua Nelson (Jul 23 2020 at 19:17, on Zulip):

and it also has a bunch of undocumented dependencies, like uname

Joshua Nelson (Jul 23 2020 at 19:19, on Zulip):

I think it parses config.toml, which I would hate to have to do in shell

simulacrum (Jul 23 2020 at 19:31, on Zulip):

python is basically "I don't have rust installed and need something to download it for me

Josh Triplett (Jul 23 2020 at 22:25, on Zulip):

In many cases, I wouldn't want anything downloading binaries by default anyway. :)

Josh Triplett (Jul 23 2020 at 22:26, on Zulip):

Are there any bits of configuration that need to be parsed by something that runs before the rust bootstrap gets compiled?

simulacrum (Jul 23 2020 at 22:29, on Zulip):

Hm, so I want to say mostly no -- or at least it wouldn't be mandatory to do so

simulacrum (Jul 23 2020 at 22:29, on Zulip):

you could definitely run the bootstrap manually if you had a rust compiler around to build it with

simulacrum (Jul 23 2020 at 22:30, on Zulip):

there's a bunch of bits it expects the python script to extract for it and pass in, but there's no reason that python couldn't be bypassed

simulacrum (Jul 23 2020 at 22:30, on Zulip):

That said, the situation in which you do have a Rust compiler but don't have python seems unlikely?

simulacrum (Jul 23 2020 at 22:31, on Zulip):

Bootstrapping Rust without the rust build system is not possible, though

simulacrum (Jul 23 2020 at 22:32, on Zulip):

To be clear, our current setup is:

  1. Run x.py / bootstrap.py -- this detects the current architecture, (optionally) downloads a toolchain, and builds the rust portion of the build system with that toolchain
  2. Run the rust build system, this is pretty complex today, I wouldn't want to replace it with anything else
  3. Rust build system (rustbuild) invokes cargo a bunch of times
simulacrum (Jul 23 2020 at 22:33, on Zulip):

I guess gcc and clang either use cmake or makefiles directly instead of having something like rustbuild

simulacrum (Jul 23 2020 at 22:33, on Zulip):

but given that they need a c compiler anyway I'm not sure that's very helpful

Joshua Nelson (Jul 23 2020 at 22:40, on Zulip):

simulacrum said:

you could definitely run the bootstrap manually if you had a rust compiler around to build it with

I mean, maybe in the abstract sense it's possible, but I don't think you could do it and still have all the options in config.toml work: https://github.com/rust-lang/rust/blob/master/src/bootstrap/bootstrap.py#L707

simulacrum (Jul 23 2020 at 22:53, on Zulip):

I guess that's sort of true? But seems not too interesting

simulacrum (Jul 23 2020 at 22:54, on Zulip):

like, if you're running cargo build yourself "of course" you'll need to specify options to it yourself

simulacrum (Jul 23 2020 at 22:55, on Zulip):

that said, I would love to get cargo to natively support a bunch of the options we currently need to feed to it from config.toml

simulacrum (Jul 23 2020 at 22:55, on Zulip):

things like switching debuginfo on/off and such for subsets of packages via configuration that's not Cargo.toml would be great

simulacrum (Jul 23 2020 at 22:55, on Zulip):

(Probably solveable today with "just" support for something like include = "./config.toml" which acts optionally)

simulacrum (Jul 23 2020 at 22:56, on Zulip):

I've meant to talk to Cargo team about this for a while now but haven't found the time

simulacrum (Jul 23 2020 at 22:57, on Zulip):

I don't think that really moves the needle in terms of whether you can build the compiler with just cargo or not, because all of those flags are optional

simulacrum (Jul 23 2020 at 22:59, on Zulip):

@Josh Triplett I would generally be interested in hearing more about the concrete constraints we're dealing with -- or documentation on those -- and would be happy to chat about how we can move Rust's build system closer to those.

Josh Triplett (Jul 23 2020 at 23:00, on Zulip):

@simulacrum In general, distributions want to build each version of Rust packages with the previous packaged version of Rust, and not touch the network.

simulacrum (Jul 23 2020 at 23:01, on Zulip):

If that's the only requirement I think we meet support for that fully today.

Josh Triplett (Jul 23 2020 at 23:01, on Zulip):

The Python requirement isn't a huge problem today, but as more people use Rust in more places, it might be nice to reduce the bootstrapping requirements.

Josh Triplett (Jul 23 2020 at 23:02, on Zulip):

And yeah, I'd be happy to see Cargo provide more of the needed functionality as well.

simulacrum (Jul 23 2020 at 23:03, on Zulip):

Hm, interesting. I think removing Python is mostly a problem of "defaults" so to speak -- as I mentioned, you can basically do everything it does yourself without all that much trouble

simulacrum (Jul 23 2020 at 23:03, on Zulip):

in some sense cargo run --manifest-path src/bootstrap/Cargo.toml --bin bootstrap -- build --stage 1 src/libstd would mostly work today, with some env variables set and so forth

simulacrum (Jul 23 2020 at 23:04, on Zulip):

and we could likely get that to "just work"

Jake Goulding (Jul 23 2020 at 23:09, on Zulip):

Run strace on x.py, capture all the command invocations and environment variables it does, then move that to whatever build environment :slight_smile:

Jubilee (Sep 06 2020 at 23:48, on Zulip):

Python is actually a bit more of an obstacle for building than I think is being stated, lots of people have trouble when interacting with their Python configurations, even on Unix systems which usually don't give Python that much grief, and it introduces a knowledge dependency for contributors.

Last update: Sep 28 2020 at 15:30UTC