Stream: t-compiler

Topic: dynamic linkage with cdylibs

Matthew Maurer (Nov 21 2019 at 20:41, on Zulip): #4 seems to imply that dynamically linked rust crates would be available in the cdylib case, though it just says dynamic library, so perhaps it refers only to rust dylibs. indicates otherwise, stating that both staticlibs and cdylibs must have all static rust crates.

In my group's use case, it is important that memory usage be kept to a minimum. One way we intended to tackle this was for the standard library (and other components, but the standard library is an obvious large piece) to have their code size costs amortized across all the places it is linked in by using dynamic linking. Unfortunately, we also intended to use cdylibs to gradually introduce Rust into a mostly C++ codebase, and these two concerns are now at odds.

Is there a rationale for this decision, or is support simply unimplemented? The potential issues I can see are:

If these are non-issues, we could consider allowing dynamic linking in general for staticlibs and cdylibs, but still defaulting to static linking for ease of use.
If some of these are potential issues, we could possibly mask this off at the Cargo layer (because separately combining cdylibs with dynamic linkage from two cargo invocations could lead to some of the issues above), but allow it at the rustc layer so that monolithic build systems could use this functionality.

I'd also love to hear about any other gotchas you think we might run into if we tried to do dynamic linkage of rust crates into a cdylib.

pnkfelix (Nov 21 2019 at 21:06, on Zulip):

I am so close to diving in and trying to force myself to explore all the points in this design space. (And how would I force myself to do so? In my usual fashion: By composing a blog post that takes me a month to write...!)

Matthew Maurer (Jan 19 2020 at 20:11, on Zulip):

If I wanted to try to work on this, would the correct first step be to file an issue or a RFC?

Matthew Maurer (Jan 19 2020 at 21:02, on Zulip):

cc @Alex Crichton since he added the test checking that cdylib deps are forced static when he added cdylib support

Alex Crichton (Jan 21 2020 at 19:11, on Zulip):

@Matthew Maurer as you can imagine there's a lot of tricky issues with dynamic libraries, and what we have now is basically "this works" rather than "this was meticulously designed and is opinionated as to what we should do"

Alex Crichton (Jan 21 2020 at 19:11, on Zulip):

in that sense I think tweaking things is fine, but it's notoriuosly hard to do so because there's so many subtelties involved here

Alex Crichton (Jan 21 2020 at 19:12, on Zulip):

a lot of the subtelties aren't obvious and makes some decisions seem wonky, so I can try help clarify anything existing if it seems weird

Matthew Maurer (Jan 21 2020 at 20:30, on Zulip):

@Alex Crichton
Can you clarify why we cannot link rust dylibs into a cdylib? If not, should I just upload a PR to remove this restriction, or is there a process I should go through to test or gain consensus?

(non-tl;dr below)

The specific decision that seems wonky is that if I build a cdylib and try to -C prefer-dynamic or even use --extern to point it at a crate which only has a dylib form, current rustc will reject this linkage pattern. I tried simply removing this restriction, and it Seems To Work (TM) (Patch is at home, I'll upload it after work for reference purposes.)

The justification for wanting this is to allow amortized disk and RAM cost of large rust components like the standard library in an environment where Rust and C++ are coexisting.

I enumerated the things I thought might go wrong above in my intro post for the thread, but:

Alex Crichton (Jan 21 2020 at 22:12, on Zulip):

@Matthew Maurer I believe we can just remove that as you've seen

Alex Crichton (Jan 21 2020 at 22:12, on Zulip):

I always that that was the case!

Alex Crichton (Jan 21 2020 at 22:12, on Zulip):

I didn't realize we had a hard check rejecting dylibs linked into cdylibs

Matthew Maurer (Jan 21 2020 at 22:53, on Zulip):

OK, thanks. I thought there might be something subtle and wanted to check first. I'll send a PR when I get home.

Last update: May 27 2020 at 20:50UTC