Stream: t-compiler

Topic: cost of linking / dsymutil


nikomatsakis (Nov 08 2019 at 20:57, on Zulip):

So I was visiting a company today (same place that @Esteban Küber works, as it happens...) and they were showing me some of their woes with regard to incremental compilation costs. We whipped out self-profile (yay @WG-self-profile, this worked great) and we found that 97% of the time was in link_crate -- and it looks like that might have been excluding dsymutil costs, which were large (like, we had top running and dsymutil was sitting there for a while). Or at least cargo said 1min of time and the self-profile included only ~35s.

Which brings me to my question. How much have we thought about ways to avoid running the linker at all in incremental scenarios? I feel like I've heard some ideas for epic hacks (maybe from @eddyb? at least that's where my mind turns when I think of epic hacks...). Maybe from @Alex Crichton?

Also dear god what can we do about dsymutil!!

eddyb (Nov 08 2019 at 20:59, on Zulip):

there are certainly hacks we can do, but what exactly is being linked?

simulacrum (Nov 08 2019 at 21:00, on Zulip):

if it's dsymutil, I feel like that's debuginfo? in which case the problem might be that we generate a ton of data -- e.g., multiple gigabytes of debuginfo if set to "full" vs. just line numbers

eddyb (Nov 08 2019 at 21:01, on Zulip):

oh yeah unless you're using a debugger, don't set it to full

eddyb (Nov 08 2019 at 21:01, on Zulip):

maybe we're relying on a bad default for debug mode

nikomatsakis (Nov 08 2019 at 21:24, on Zulip):

I'm not sure if we set it to full, but I think they are trying to use debuggers

nikomatsakis (Nov 08 2019 at 21:24, on Zulip):

That doesn't seem like such an unreasonable thing to want to do :)

nikomatsakis (Nov 08 2019 at 21:24, on Zulip):

dsymutil was I think like 50% of the linking cost

nikomatsakis (Nov 08 2019 at 21:24, on Zulip):

there are certainly hacks we can do, but what exactly is being linked?

I'm not sure how to answer this -- rust code? :P

Alex Crichton (Nov 08 2019 at 21:36, on Zulip):

There's some background for dsymutil at https://github.com/rust-lang/rust/issues/47240

Alex Crichton (Nov 08 2019 at 21:36, on Zulip):

and https://github.com/rust-lang/rust/pull/47784 initially started by trying to remove it altogether but we weren't able to do that

Alex Crichton (Nov 08 2019 at 21:37, on Zulip):

(but it ended up in a -Z flag to avoid it)

Alex Crichton (Nov 08 2019 at 21:37, on Zulip):

avoiding the linker will likely be pretty hard, and maybe not that useful? That's only really possible if zero object files changed, but it seems like at least one object file is likely to always change

Alex Crichton (Nov 08 2019 at 21:39, on Zulip):

we also have a very old issue - https://github.com/rust-lang/rust/issues/34651 - to use split-dwarf on all platforms to reduce the cost of linking with debuginfo

Alex Crichton (Nov 08 2019 at 21:42, on Zulip):

Another theory is that LLD may be faster than the system linker on OSX

Alex Crichton (Nov 08 2019 at 21:42, on Zulip):

you should be able to switch with rustc foo.rs -C linker-flavor=ld64.lld -Clinker=rust-lld

Alex Crichton (Nov 08 2019 at 21:42, on Zulip):

although I just did that and it produced a segfaulting binary

Alex Crichton (Nov 08 2019 at 21:42, on Zulip):

so it is likely not production quality yet

Alex Crichton (Nov 08 2019 at 21:43, on Zulip):

the tl;dr; of dsymutil is that if we don't run it RUST_BACKTRACE doesn't work

Alex Crichton (Nov 08 2019 at 21:43, on Zulip):

and historically that's been more important than saving compile time

Alex Crichton (Nov 08 2019 at 21:44, on Zulip):

@nikomatsakis is this project available though to poke around in?

nikomatsakis (Nov 08 2019 at 21:44, on Zulip):

It's closed source

nikomatsakis (Nov 08 2019 at 21:44, on Zulip):

I was thinking about looking at some other repos that may have similar problems

nikomatsakis (Nov 08 2019 at 21:44, on Zulip):

it was 'plain old cargo' though

nagisa (Nov 08 2019 at 21:44, on Zulip):

linking debug info will take a fair share of time on all platforms.

nikomatsakis (Nov 08 2019 at 21:44, on Zulip):

the pattern was a lot of crates, and then a very shallow "tip" crate

nikomatsakis (Nov 08 2019 at 21:45, on Zulip):

(much like rustc_driver etc)

nagisa (Nov 08 2019 at 21:45, on Zulip):

At work we have a semi-trivial 500-crate graph that takes 100% longer to link with debug info than when building with --release.

nikomatsakis (Nov 08 2019 at 21:45, on Zulip):

avoiding the linker will likely be pretty hard, and maybe not that useful? That's only really possible if zero object files changed, but it seems like at least one object file is likely to always change

yeah I mean the goal would be to somehow do some surgery such that we can run the code without actually linking or something. I have no idea how to achieve this. =)

nagisa (Nov 08 2019 at 21:46, on Zulip):

lld has incremental linking support

nikomatsakis (Nov 08 2019 at 21:46, on Zulip):

This is why I mentioned epic hacks ;)

nikomatsakis (Nov 08 2019 at 21:46, on Zulip):

Yes, that's another possibility

nagisa (Nov 08 2019 at 21:46, on Zulip):

and Microsoft’s linker does too AFAIK

nikomatsakis (Nov 08 2019 at 21:46, on Zulip):

Sounds promising

Alex Crichton (Nov 08 2019 at 21:46, on Zulip):

some recommendations we can make for now possibly are:

Alex Crichton (Nov 08 2019 at 21:47, on Zulip):

I've never tested out incremental linker support myself, but we probably want a working LLD linker before we switch to an incremental version

Alex Crichton (Nov 08 2019 at 21:48, on Zulip):

IIRC we don't activate MSVC's incremental linker support

nikomatsakis (Nov 08 2019 at 21:49, on Zulip):

OK. Definitely exploring incremental linking is the obvious route.

nagisa (Nov 08 2019 at 21:49, on Zulip):

I said lld has incremental linking support, but now that I spent 5 minutes not finding anything on ddg, I’m not so sure anymore

nikomatsakis (Nov 08 2019 at 21:50, on Zulip):

I too have heard this before

nikomatsakis (Nov 08 2019 at 21:50, on Zulip):

I've never tested out incremental linker support myself, but we probably want a working LLD linker before we switch to an incremental version

But I mean this is also true :)

Alex Crichton (Nov 08 2019 at 21:51, on Zulip):

at least the -h page for rust-lld doesn't show anything about incremental

Alex Crichton (Nov 08 2019 at 21:51, on Zulip):

that being said when it works LLD is typically much faster than the system linker

Alex Crichton (Nov 08 2019 at 21:51, on Zulip):

I don't think the macos linker is winning any awards for its speed

nagisa (Nov 08 2019 at 21:51, on Zulip):

The references I can find are all about thinlto incremental stuff (https://clang.llvm.org/docs/ThinLTO.html#incremental)

nagisa (Nov 08 2019 at 21:52, on Zulip):

but yeah, if linking time is a problem, the only thing I know of to resolve it right now is to reduce the amount of work for linker to do

nagisa (Nov 08 2019 at 21:53, on Zulip):

which means not giving it gigabytes of debuginfo to link for the first obvious step.

Alex Crichton (Nov 08 2019 at 21:53, on Zulip):

@nikomatsakis in general our performance monitoring is pretty bad for non-x86_64-unknown-linux-gnu platforms

Alex Crichton (Nov 08 2019 at 21:54, on Zulip):

it's largely a technical problem where we just can't profile everything everywhere all the time

nagisa (Nov 08 2019 at 21:54, on Zulip):

This problem is as relevant on x86_64 linux gnu tho

Alex Crichton (Nov 08 2019 at 21:54, on Zulip):

but I ran across https://github.com/rust-lang/rust/issues/66192 yesterday which was very surprising

Alex Crichton (Nov 08 2019 at 21:54, on Zulip):

and yes this is a problem without split dwarf on linux but ld + dsymutil is for me I feel so much slower than just ld on linux

Alex Crichton (Nov 08 2019 at 21:54, on Zulip):

like the linux linkers are generally pretty fast

Alex Crichton (Nov 08 2019 at 21:55, on Zulip):

but not profiling these sorts of things on OSX means that this just comes up less often

Alex Crichton (Nov 08 2019 at 21:55, on Zulip):

it's all stuff we should fix but it's not something we're prioritizing

nikomatsakis (Nov 08 2019 at 21:56, on Zulip):

yeah, sounds correct

nikomatsakis (Nov 08 2019 at 21:56, on Zulip):

I know that basically every time I talked to Anthony Jones at Mozilla he brings up dsymutil :)

nikomatsakis (Nov 08 2019 at 21:56, on Zulip):

He is...probably not wrong to do so

eddyb (Nov 08 2019 at 22:44, on Zulip):

@nikomatsakis when I said "what is being linked" I meant what kind of artifact

eddyb (Nov 08 2019 at 22:44, on Zulip):

is all that time spent linking an executable?

Ivan Dubrov (Nov 09 2019 at 00:25, on Zulip):

Yes, one huge executable.

nagisa (Nov 09 2019 at 00:42, on Zulip):

@eddyb if you want I can concoct an example

eddyb (Nov 09 2019 at 00:43, on Zulip):

okay an executable makes sense

nagisa (Nov 09 2019 at 00:43, on Zulip):

but I think you can easily do that yourself by just adding deps on a couple of crates that have large dep trees.

eddyb (Nov 09 2019 at 00:43, on Zulip):

that's trickier to deal with than "we're wasting time making pointless rlib's"

David G. Horsman (Nov 09 2019 at 08:55, on Zulip):

Hello excuse the interruption. I will be monitoring (lurking). My goal is audited source to executable dumps at each step which is very compatible with Rust. As you all know optimization makes this less deterministic but that will not be a problem for audit.

This is for open source software and hardware. Thanks so very much for all your hard work. When the word "awesome" really says something. Go Rust.

David G. Horsman (Nov 09 2019 at 08:58, on Zulip):

Additionally I will looking at where NN will benefit Rust within Build, dependencies and deployment.

David G. Horsman (Nov 09 2019 at 09:00, on Zulip):

@nikomatsakis I am unfamiliar with your work culture here. Delete anything off topic please.

mw (Nov 11 2019 at 09:20, on Zulip):

Here is an old issue about incremental linking: https://github.com/rust-lang/rust/issues/37543

mw (Nov 11 2019 at 09:21, on Zulip):

Note that incremental ThinLTO and incremental linking are two different things.

mw (Nov 11 2019 at 09:22, on Zulip):

the former will only avoid redoing the ThinLTO optimization passes. We already support that but it won't change the time spent in the actual linking step.

Last update: Nov 16 2019 at 01:05UTC