Stream: general

Topic: `rust-std` component became so big


Charles Lew (Jul 08 2019 at 18:19, on Zulip):

@Zoxc Forgive my ignorance, but why does rust-std component include librustc-<hash>.rlib? I mean, when will this library be linked?

Zoxc (Jul 08 2019 at 18:41, on Zulip):

Never on stable/beta, for custom compiler frontends on nightly

Charles Lew (Jul 08 2019 at 18:46, on Zulip):

Thanks for the reply! so maybe it makes sense to package those into a separate component, and only those using custom compiler frontends needs to download them :) that will save more than 100MB space, i think~

Zoxc (Jul 08 2019 at 18:53, on Zulip):

Yes. I've suggested doing that before.

Charles Lew (Jul 08 2019 at 18:54, on Zulip):

I have another question: the rustc_driver-<hash>.dll.lib is almost 6MB, and it's only a import library... Using tool i can see that rustc_driver-<hash>.dll is exporting all kinds of functions...almost 9k functions. Is that expected?

Charles Lew (Jul 08 2019 at 18:56, on Zulip):

there are functions defined within serde_json, termcolor, parking_lot, syntax, ...

Zoxc (Jul 08 2019 at 18:58, on Zulip):

That's expected. It basically has all the compiler code and the code of its dependencies.

rkruppe (Jul 08 2019 at 18:59, on Zulip):

Yes. Though I wouldn't stake my life on all of these being strictly necessary, by nature it will export a ton of symbols. It's a dylib, not a C dylib, so it exposes a Rust ABI interface: anything that was already compile to machine code in any of the upstream crates is put there and exposed for downstream crates to reference so they don't have to duplicate that machine code. It's literally the dynamically linked equivalent of an rlib (well, all of the rlibs that make up rustc_driver aggregated into one library).

Charles Lew (Jul 08 2019 at 19:02, on Zulip):

i see. thanks~

Zoxc (Jul 08 2019 at 19:03, on Zulip):

I'd like us to ship a single binary containing rustc, clippy and rls, and only use static linking / rlibs. In which case there would be no lib files and less duplication of code.

Zoxc (Jul 08 2019 at 19:04, on Zulip):

We could then use LTO and export no symbols for faster load times (possibly)

Charles Lew (Jul 08 2019 at 19:12, on Zulip):

i'm also thinking about something else. In the wild quite a few people want to build dylib-based plugin systems. However from what i learnt just now, the common dependency part, as a dylib will export everything in every crate, and dead code elimination is impossible. This is a little... suboptimal?

rkruppe (Jul 08 2019 at 19:15, on Zulip):

I can't recommend strongly enough against building anything, plugin system or otherwise, on top of dylibs. Everything that needs dynamic linking should use cdylibs.

Charles Lew (Jul 08 2019 at 19:16, on Zulip):

But in cdylibs, passing rust values through the dll boundaries are so inconvenient...

Charles Lew (Jul 08 2019 at 19:17, on Zulip):

If it's convenient enough, rustc_driver.dll can also be a cdylib, right?

Charles Lew (Jul 08 2019 at 19:21, on Zulip):

maybe we're just one very-smart-proc-macro away from doing the necessary data marshaling, i don't know, but we're just not there.

rkruppe (Jul 08 2019 at 19:23, on Zulip):

I know the downsides and pains of Rust-Rust interfaces channeled through the C ABI very well, but dylibs are just so much worse in every other respect.

Charles Lew (Jul 08 2019 at 19:24, on Zulip):

yeah, i agree

RalfJ (Jul 11 2019 at 15:25, on Zulip):

oh so I didnt just imagine that rust-std grew in size enormously? given it is a mandatory component, seems like an odd choice to put things in there that only experimental alternative backends need

simulacrum (Jul 11 2019 at 15:26, on Zulip):

there's definitely some truth to us probably not wanting to do that

simulacrum (Jul 11 2019 at 15:26, on Zulip):

otoh, we didn't actually change what's in the component -- just the encoding of said things

RalfJ (Jul 11 2019 at 15:27, on Zulip):

but it did regress a lot recently? I've seen https://github.com/rust-lang/rust/pull/59800 being mentioned

RalfJ (Jul 11 2019 at 15:28, on Zulip):

is there a bug for that particular regression? I just found a very general "things are too big" at https://github.com/rust-lang/rust/issues/61978

simulacrum (Jul 11 2019 at 15:28, on Zulip):

yes, it did by around ~50 MB I think? It's really noticeable primarily because we used to be around ~80 MB download and are now above 100

simulacrum (Jul 11 2019 at 15:29, on Zulip):

https://github.com/rust-lang/rust/issues/62486

RalfJ (Jul 11 2019 at 15:29, on Zulip):

I remember rustc and rust-std being about the same size usually, now rust-std is three times the size

RalfJ (Jul 11 2019 at 15:29, on Zulip):

looks more like a 100MB regression to me, rust-std has 170MB

simulacrum (Jul 11 2019 at 15:29, on Zulip):

ah, I think that's installed size

RalfJ (Jul 11 2019 at 15:30, on Zulip):

the "downloading" status shows installed size...?

RalfJ (Jul 11 2019 at 15:30, on Zulip):

(of rustup)

simulacrum (Jul 11 2019 at 15:30, on Zulip):

I think no? but then you get another progress meter thingy for the installation

RalfJ (Jul 11 2019 at 15:30, on Zulip):

they show the same size

RalfJ (Jul 11 2019 at 15:31, on Zulip):
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2019-07-11, rust version 1.38.0-nightly (cd2cd4c96 2019-07-10)
info: downloading component 'rustc'
 65.6 MiB /  65.6 MiB (100 %)  21.0 MiB/s in  3s ETA:  0s
info: downloading component 'rust-std'
171.1 MiB / 171.1 MiB (100 %)  22.2 MiB/s in  8s ETA:  0s
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: downloading component 'rust-src'
info: downloading component 'rust-analysis'
info: removing component 'rustc'
info: removing component 'rust-std'
info: removing component 'cargo'
info: removing component 'rust-docs'
info: removing component 'rust-src'
info: removing component 'rust-analysis'
info: installing component 'rustc'
 65.6 MiB /  65.6 MiB (100 %)  11.6 MiB/s in  5s ETA:  0s
info: installing component 'rust-std'
171.1 MiB / 171.1 MiB (100 %)  27.4 MiB/s in  6s ETA:  0s
info: installing component 'cargo'
info: installing component 'rust-docs'
 11.6 MiB /  11.6 MiB (100 %)   8.4 MiB/s in  1s ETA:  0s
info: installing component 'rust-src'
info: installing component 'rust-analysis'
info: checking for self-updates
simulacrum (Jul 11 2019 at 15:31, on Zulip):

yeah that's what I see locally

simulacrum (Jul 11 2019 at 15:31, on Zulip):

I think I misread -- I saw rust-src and was really concerned when you said that was 170MB now

simulacrum (Jul 11 2019 at 15:32, on Zulip):

but you typo'd it looks like from std

simulacrum (Jul 11 2019 at 15:32, on Zulip):

installed size on disk appears to have gone up by ~200 MB

simulacrum (Jul 11 2019 at 15:33, on Zulip):

I think it's because we're now including some things twice, but not sure

simulacrum (Jul 11 2019 at 15:34, on Zulip):

i.e., we before had a lot of common code in librustc...so but now that's duplicated across rlibs

simulacrum (Jul 11 2019 at 15:34, on Zulip):

but I could be wrong, I'm not terribly familiar with how we link things

RalfJ (Jul 11 2019 at 15:34, on Zulip):

yeah sorry I wrote rust-src accidentally. we dont have 170MB of source code yet. :P

simulacrum (Jul 11 2019 at 15:35, on Zulip):

nightly:

   83.7 MiB [ 15.9%]  librustc-0a6217d7d74c7abb.rlib
   76.8 MiB [ 14.6%]  libLLVM-8-rust-1.38.0-nightly.so
   56.2 MiB [ 10.7%]  librustc_driver-28b978599ed51f4d.so
   41.0 MiB [  7.8%]  librustc_mir-7a92cf707c2a230b.rlib
   24.6 MiB [  4.7%]  libcore-c4f58d8b9ed1ef3b.rlib
   20.5 MiB [  3.9%]  librustc_typeck-904f9abd82bc8c39.rlib
   19.5 MiB [  3.7%]  libsyntax-c413d5cb9dc5c44f.rlib
   14.8 MiB [  2.8%]  librustc_metadata-227115a80841d294.rlib
   10.6 MiB [  2.0%]  librustc_interface-890a457e8a5c59b9.rlib

stable:

   76.8 MiB [ 27.1%]  libLLVM-8-rust-1.36.0-stable.so
   25.7 MiB [  9.1%]  libcore-5594cb4f559bc761.rlib
   25.1 MiB [  8.8%]  librustc-d6e5ab7261915b59.so
   11.8 MiB [  4.2%]  librustc_mir-5c472af620e2968c.so
    9.4 MiB [  3.3%]  libstd-9895e8982b0a79e7.rlib
    8.1 MiB [  2.8%]  librustc_macros-e6df7be6834ec89b.so
    6.3 MiB [  2.2%]  librustc_typeck-575efaf425292d9e.so
    6.2 MiB [  2.2%]  libsyntax-aa56a60c3365997c.so
    6.0 MiB [  2.1%]  libjemalloc_sys-4dfe88c31e2f050c.rlib
gnzlbg (Jul 17 2019 at 21:35, on Zulip):

there are functions defined within serde_json, termcolor, parking_lot, syntax, ...

Is libtest in there or how did termcolor Land there?

Charles Lew (Jul 18 2019 at 17:37, on Zulip):

1. no, libtest is not in there 2. i did a cargo tree and it seems there's rustc_driver => env_logger => termcolor & rustc_driver => rustc_interface => rustc_errors => termcolor

Last update: Nov 20 2019 at 11:30UTC