I'm trying to build the compiler, and I'm repeatedly finding that I run out of memory compiling
I initially thought it might be because my
parallel-compiler = true and
codegen-units = 0, but I've since reverted both of those settings, and I still haven't managed to successfully build
rustc_middle without having
rustc killed by the OOM killer.
(I have 8GB of RAM.)
I would appreciate some help trying to figure out if I've done something wrong here.
I recall having had trouble building rustc with 8G of memory at about the time 2018 edition was being cooked. Given that the compiler only ever gets more complex (and LLVM ever so slightly more quadratic), I can see it just genuinely not fitting.
Though there are a couple of things you can try: a clean checkout and just
x.py build without even creating a config.toml (so the defaults are used) is one.
I'll give that a try, just to make sure.
I've been using
./x.py build -i --stage 1 src/libstd. Is that likely to be a problem?
I would try without
-i, given that it is a scratch build anyway.
Already kicked off a build with
-i and no
config.toml, but if that fails too then I'll try without.
Related question: is there any option in the compiler to report a "high water mark" of total memory usage, from the allocator?
Would be useful for memory usage profiling and optimization.
Well, it's making it further, but still coming very close to OOM.
...and there it went. Killed after 13 minutes of CPU time.
Trying again with
./x.py clean and then just
Too early to tell, but compiling
rustc_middle seems to be using drastically less memory without
I don’t believe rustc or x.py has an option for high watermark, but running code under a memory profiler such as massif would give that information.
(genuinely not sure if its even feasible to get x.py use massif, @Nicholas Nethercote would know better)
Looks like that did it. Incremental compilation massively increases memory usage.
I just verified and locally building rustc can take up to 10GB
Even more in fact
I can say it works on my laptop with 32 GB :-)
I've been able to successfully build it with 8GB, but it takes some care, and that's unfortunate.
Once I finish the project I'm working on, I may try doing some memory profiling and see if I can reduce peak memory consumption enough to make it fit more comfortably, and support incremental builds.
(Also, I managed to get incremental builds working by freeing up a little more memory, doing the build once, and now as long as I don't touch anything that goes into
rustc_middle I can keep doing incremental builds. But that's not ideal either.)
Maybe splitting up
rustc_middle into separate crates would solve the problem? Obviously a huge undertaking though
What's the difference between
@Camelid As I understand it, that's been a desirable goal for a long time, but that's separate from incremental compilation taking up an excessive amount of memory.
I guess it would just fix the symptom of the problem rather than the problem itself
I usually add some swap space so I don't oom, but yeah this regularly happens to me... it's one of the reasons I try to use CI to test
The other thing is that if there are multiple jobs running it's worse
I've never done it myself. I'm sure it's possible in some fashion, but it might require modifying
@Camelid the reason
rustc_hir even exists is part of the start of that process
rustc_middle is still where everything came from but I'm not sure who is still working on trying to break it up
some of the approaches are less great than others, there are nice ways and there are hacks that involve redirecting stuff through e.g.
fn pointers in global variables etc.
Yeah, breaking this stuff up looks really hard!
Thanks for the background information by the way :)