@Alex Crichton A while back I think you said you could be ok with bytecode not being compressed? I've been experimenting with that, it reduces compile time by up to 5% on debug builds.
I need to measure the impact on the .rlib file sizes.
I'd personally be totally fine decompressing the bytecode
and/or investigating faster compression algorithms
something like a lower compression level or maybe even zstd might get 90% of the wins
I'd be fine removing the compression entirely myself
it is worth noting that a common complaint is that Rust produces enormous target directories
Though I personally think the solution there is not compression but rather storing different data (and/or "intelligent" compression) -- I am also unopposed to not compressing
it may also be plausible that we can easily compress on a separate thread?
We already use "Fast" compression (as opposed to "Best", "Default", or "None")
@Alex Crichton there are mentions of "bitcode" and "bytecode". Are these the same thing?
Hmm, a rough comparison on one project suggests that disabling compression increases .rlib file size by typically 10-25%
Given that they are regular 1MiB+, I'm reluctant to cause that much disk usage increase
@nnethercote I think bitcode and bytecode are the same
@nnethercote honestly actually I think there's a better solution here, albeit more involved
the only purpose of bytecode in rlibs is for LTO
but most projects don't use LTO
actually, so here's a real kicker
ok so ignore libstd
if you execute LTO
then you do a huge amount of object file codegen in the middle, none of which is used, all of which is wasted time
(For this whole project (https://github.com/mozilla/fix-stacks) the rlibs total size goes from 276,561,064 to 347,907,278, a 1.26x increase)
if you dont' do LTO then you're creating a bunch of bitcode that's never used
so the real solution here is to probably add two flags to the compiler which Cargo automatically passes:
That's both compile time wins and disk space wins all across the board
and everything will be turned on by default as soon as we land it in cargo
does that make sense?
So when would bitcode be omitted, and when would the object be omitted?
if you compile with LTO, there's no need to put object files in rlibs
(nor is there any reason to compress the bitcode found in the rlib)
we have a flag like that with
-C for cross-lang LTO, but rust projects should be using that as well
ok, so the rlib should contain: object XOR bitcode
except for libstd, but ignore that
for everything cargo produces locally it should be xor
so we either (a) don't produce any bitcode, nor compress it, or (b) don't codegen something that's not needed
both of which can be pretty significant savings
It seems that uncompressed bitcode/bytecode is also a thing: https://github.com/rust-lang/rust/blob/6576f4be5af31a5e61dfc0cf50b7130e6c6dfb35/src/librustc/dep_graph/graph.rs#L910-L914
yeah so @mw might be able to help out there to explain more
but there's a
-C flag which makes our rlibs "cross lang lto compatible"
which means that all the
*.o files are actually uncompress bitcode
note though that this is a pretty simple concept
but there's a fair amont of legwork to get it all hooked up in the compiler
for example the LTO passes in rustc need to get updated of how they look for bitcode, it's either a "libstd rlib" where it's located adjacent to the object or it's a "cargo rlib" where it's the
@Alex Crichton You've lost me... is this "legwork" about the original idea (object XOR bitcode), or about the uncompressed bitcode stuff?
@nnethercote the xor idea
I suspect uncompressed bitcode is dead in the water if it inflates sizes that much
I agree my original idea of not compressing bitcode is not good. But 'object XOR bitcode' would make it much less interesting anyway, since only LTO builds would end up with bitcode.
@nnethercote so actually
To start I think we just need a flag to skip bitcode
And that's it
that sounds easy :)
The hard part later is making lto faster but that's less interesting
If you add that flag I can whip up a patch for cargo
cool, I'll poke around, try to work it out
thanks for the help!
So I'm.basically countering your uncompressed bitcode
With let's delete bitcode lol
"make it faster" is good, but "don't do it at all" is better :)
https://github.com/rust-lang/rust/pull/66598 is the PR, for anyone following along. Big wins!
@mw Thanks for writing up #66961! I'm considering adding this to my list of tasks for Q1 2020, because I suspect it won't get done otherwise. Does that sound right to you?
@nnethercote Yes, that sounds good. I can do the reviewing then.