Is there anyone informed about rustc's debug info generation who could chime in on an LLVM diff? https://reviews.llvm.org/D74169#1990625
The tl;dr is that Rust currently generates a massive amount of unnecessary debug info (https://github.com/rust-lang/rust/issues/56068) because its codegen units are _huge_. This isn't a problem for the actual text segment because the linker garbage collects unused sections, but no linker presently garbage collects the DWARF associated with those unused sections. The diff linked above adds a
--gc-debuginfo flag to lld that seems exactly like what we need, but there is presently a massive link-time penalty. One of the LLVM devs upstream is asking whether there is existing work to optimize this on Rust's end, and I don't know the answer!
No active work from what I know, but maybe @eddyb has something in the pipeline. Seems like that issue went under the radar.
nope but I would love to be able to dedup debuginfo between CGUs
lld growing functionality like this is awesome!
Agreed! Unfortunately it is sloooooooow. The lld patch is still pretty WIP, so unclear if there's just missing optimizations or if there's something fundamentally polynomial about the approach.
--gc-sections increased link time from 10s to 240s (!) on the binary I was testing.
@eddyb would you expect to see a decrease in debug info size with
codegen-units = 1? That's interesting.
well, it can't be less than that
but I was also thinking cross-crate
which I guess we could measure
you compile all the dependency crates with that flag to only serialize MIR and do no codegen, then compile the final crate with 1 CGU
that should be the minimal size both in terms of LLVM IR and debuginfo (but LLVM passes may increase the total machine code)
Is that the
Hmm, that flag doesn't seem to exist. Anyway, just compiling everything with
codegen-units = 1 cut the size of
.debug_info by 40% (!). Unfortunately it makes compile times on this project untenable, but still intriguing.
right, it removes parallelism
@Nikhil Benesch hmm,
-Z always-encode-mir but what you mention makes more sense
I wonder when that went away
Yeah, sorry, compile time increase with 1 CGU is totally expected. I'm just musing that even in our CI release pipeline we are unlikely to want to take the 4x increase in compile time for a 40% reduction in debug info.
I would, however, be interested in poking at doing some DWARF deduplication in rustc! But I'd probably need a pointer or two.
you could try to compare it to full LTO
@Nikhil Benesch we can't do it in rustc. or rather, this is a linker thing
you'd have to parse DWARF, deduplicate, then re-encode it
(full LTO should have a similar effect to doing codegen only in the final crate)
I'm wondering if rustc could be smarter about creating codegen units, though?
Smarter specifically wrt trying to group bunches of symbols together that can all be dropped on the floor when linking debug info. Maybe that's an impossible ask though.
doesn't sound easy, while what I was thinking is stuff like type descriptions that are replicated across CGUs/libraries
Gotcha. I'll try to get some full LTO stats.
Btw, while I have you, https://github.com/rust-lang/rust/issues/46034 looks pretty easy to address. The tl;dr is that the
debug_pubtypes sections are pretty useless. Removed in DWARF 5, and no longer generated by default by gcc/clang. There's even a working patch for Rust that unconditionally omits them.
I think the only hold up is deciding if it's acceptable to change rustc to unconditionally omit those on Linux, or if it needs to be behind some codegen flag that gets stabilized.
Happy to pick that patch up and submit for review if you (or anyone) can advise. I'm not familiar with what sort of stability promises rustc makes on the debug info front.
me neither tbh
Hah, ok! Do you know who would know?
mw is away from Rust for a while
How does it change user-visible behavior?
@bjorn3 did a bunch of work on debuginfo for the cranelift backend
and @Hanna Kruppe, @nagisa and @Nikita Popov are the go-to people for LLVM things, that I know of
@Jonas Schievink AIUI you can tell the linker to build
debug_pubtypes into a GDB index that speeds up the initial boot of gdb/lldb. But doesn't seem like anyone regularly uses them.
And thanks, eddyb, that's super helpful!
So users would have to customize the linker invocation to benefit from this? Seems fine to remove this then
Here's some additional context on debug_pubnames and debug_pubtypes from Fedora: https://fedoraproject.org/wiki/Features/GdbIndex#Detailed_Description
To the best of my knowledge no program in the distro (and probably no program anywhere) uses these sections. They are just wasted space.
Yeah exactly. In order to make use of it you're already going to need to edit the linker args. Unfortunately there is no LLVM arg that controls their generation, so right now you have no choice but to accept them, and with the existing patch you have no choice but to not have them.
Our collector is already fairly good about grouping related stuff together into codegen units, to facilitate incremental compilation. If there’s a possibility to improve it, it would be primarily driven by incremental compilation needs I feel.
I’d be cool with disabling pubnames/types by default, especially because, I believe, we generate fairly recent version (4) of dwarf in the first place