Stream: t-compiler

Topic: cross-lang-lto


gnzlbg (Nov 04 2018 at 11:33, on Zulip):

@mw i'm trying to set up jemalloc with -Zcross-lang-lto on macosx with xcode10 (should have a recent enough clang) so that jemalloc is inlined into Rust:

CFLAGS="-flto=thin" RUSTFLAGS="-Zcross-lang-lto -Ccodegen-units=1" cargo test --all --release

fails with: ld: unknown option: -plugin-opt=O3

I guess I'm missing something?

mw (Nov 05 2018 at 11:04, on Zulip):

I never tested the OSX linker with this. LLD works best in my experience (at least on Windows and Linux). Also, since nightly Rust is based on the yet unreleased LLVM 8.0, you might have to compile your own Clang because the LLVM IR format has changed.

mw (Nov 13 2018 at 10:00, on Zulip):

@gnzlbg Do you have a specific project you want to try this with?

mw (Nov 13 2018 at 10:01, on Zulip):

I'm thinking about creating a sample project with some benchmarks

mw (Nov 13 2018 at 10:01, on Zulip):

if you already had something suitable, we could use that

gnzlbg (Nov 13 2018 at 10:49, on Zulip):

jemalloc-sys would be nice

gnzlbg (Nov 13 2018 at 10:49, on Zulip):

jemalloc-sys would be nice

gnzlbg (Nov 13 2018 at 10:49, on Zulip):

It already has benchmarks that might benefit from this.

mw (Nov 13 2018 at 14:29, on Zulip):

ok, great! existing benchmarks are a big plus.

mw (Nov 13 2018 at 14:30, on Zulip):

I'll look into it over the next few days

gnzlbg (Nov 13 2018 at 16:26, on Zulip):

i'm in a conference till thursday, but have the zulip app on the phone and check it now and then

I don't know if you want to try this out spontaneously once you get some time for it, or if you want to allocate a slot of time for trying that out in advance. If you do, let me know when, I can try to be online in case anything doesn't work. Compiling jemalloc-sys should "just work" but it builds jemalloc which is a C library with a non-trivial makefile system, so the experience isn't always smooth. If you get stuck I might be able to help though.

mw (Nov 13 2018 at 16:28, on Zulip):

thanks!

mw (Nov 13 2018 at 16:28, on Zulip):

I don't have a fixed time slot allocated for it.

gnzlbg (Nov 13 2018 at 16:29, on Zulip):

ok, then ping me if you run into any issues, maybe i'm online

mw (Nov 14 2018 at 13:51, on Zulip):

OK, I had success with the following command:
EXTRA_CFLAGS=-flto=thin CC="/home/mw/llvm-build/bin/clang" RUSTFLAGS="-Clinker=/home/mw/llvm-build/bin/clang -Zcross-lang-lto -Zpre-link-arg=-fuse-ld=lld" cargo test --release --all

mw (Nov 14 2018 at 13:52, on Zulip):

but I had to modify a file in jemalloc because an extra LD_FLAG is needed

mw (Nov 14 2018 at 13:53, on Zulip):

LD_FLAGS=-fuse-ld=lld and EXTRA_LD_FLAGS=-fuse-ld=lld didn't work unfortunately

mw (Nov 14 2018 at 13:53, on Zulip):

another problem is that the commandline options for LLD seem to have changed in a non-backwards compatible way

mw (Nov 14 2018 at 13:54, on Zulip):

I had to modify rustc to not pass a flag anymore that was still accepted a while ago

mw (Nov 14 2018 at 13:54, on Zulip):

see https://github.com/rust-lang/rust/pull/55947

mw (Nov 14 2018 at 13:55, on Zulip):

I used the rustc version from the PR (basically nightly)

mw (Nov 14 2018 at 13:55, on Zulip):

and the latest clang master

gnzlbg (Nov 15 2018 at 09:30, on Zulip):

cool! I was trying to use apple clang and was failing, going to give the latest clang master a try instead. Shouldn't we ship this too with the llvm-tools-preview ?

mw (Nov 15 2018 at 10:29, on Zulip):

Shouldn't we ship this too with the llvm-tools-preview

I was thinking that too. The only problem I see is that we would be shipping an unstable/not officially released version of Clang, which doesn't sound ideal to me. On the other hand, stable Clang is unlikely to ever catch up with rustc's LLVM version...

gnzlbg (Nov 15 2018 at 11:41, on Zulip):

I think it is fair to assume that if you want to use cross-lang-lto you need to use a clang version that's in sync with rust's llvm version (bundled or not), and that's what llvm-tools is there for if a bundled LLVM is shipped. I also wanted to add llc to llvm-tools, so a PR to update that could probably just do both.

gnzlbg (Nov 15 2018 at 11:46, on Zulip):

Do you have a branch of jemallocator that builds with cross-lang-lto locally ?

mw (Nov 15 2018 at 13:06, on Zulip):

let me create one...

mw (Nov 15 2018 at 13:08, on Zulip):

hm, it's not so easy since the change is in the jemalloc sources...

mw (Nov 15 2018 at 13:39, on Zulip):

this should work:
git clone https://github.com/michaelwoerister/jemallocator.git --branch clang-thinlto-hack --recursive

mw (Nov 15 2018 at 13:40, on Zulip):

you still need the right version of clang and rustc though

gnzlbg (Nov 15 2018 at 13:43, on Zulip):

ok, we should send a patch to jemalloc with a configuration option that forces using lld for linking

gnzlbg (Nov 15 2018 at 13:48, on Zulip):

@mw I wonder, isn't it enough during configure (e.g. in jemalloc-sys/build.rs) to call configure with the environment variable ("LDFLAGS", &format("{} -fuse-ld=lld")) ?

gnzlbg (Nov 15 2018 at 13:49, on Zulip):

might be less intrusive than modifying jemalloc at this point

mw (Nov 15 2018 at 13:51, on Zulip):

I tried LD_FLAGS="-fuse-ld=lld" cargo build and EXTRA_LD_FLAGS="-fuse-ld=lld" cargo build

mw (Nov 15 2018 at 13:51, on Zulip):

but the configure script would ignore them

mw (Nov 15 2018 at 13:52, on Zulip):

using CFLAGS this way worked just fine

mw (Nov 15 2018 at 13:52, on Zulip):

maybe cargo isn't passing the flags on? or the configure script doesn't support LD_FLAGS being set from the outside

mw (Nov 15 2018 at 13:53, on Zulip):

also, in theory it should work with other linkers too, if the right --plugin option is passed

mw (Nov 15 2018 at 13:54, on Zulip):

in practice things didn't work so well with ld the last time I tried

mw (Nov 15 2018 at 13:54, on Zulip):

gold worked at some point and later didn't

mw (Nov 15 2018 at 13:55, on Zulip):

so I'm telling people to use LLD, but for jemalloc a more general solution would be needed, I think

gnzlbg (Nov 15 2018 at 14:19, on Zulip):

https://github.com/gnzlbg/jemallocator/commit/6eb2b293f46e85b5c2d9190658507a76293dd031

gnzlbg (Nov 15 2018 at 14:20, on Zulip):

so independently of how to pass this (cflags, ldflags, ...) something like this should work

gnzlbg (Nov 15 2018 at 14:20, on Zulip):

once the new nightly is produced, we should try to at least upstream this to jemallocator, and get the macosx build bots to use it

gnzlbg (Nov 15 2018 at 14:21, on Zulip):

we could add an x86_64-unknown-linux-gnu build bot that has clang trunk installed in travis (llvm apt repos provide this)

gnzlbg (Nov 15 2018 at 14:21, on Zulip):

and get it going on linux as well

gnzlbg (Nov 15 2018 at 14:21, on Zulip):

ideally behind an opt-in option at the beginning

gnzlbg (Nov 15 2018 at 14:21, on Zulip):

so that people can enable it if it works for them

gnzlbg (Nov 15 2018 at 14:22, on Zulip):

we can then try to use this in rustc, and once it works, maybe turn that to opt-out so that cross-lang lto is used by default

mw (Nov 15 2018 at 14:22, on Zulip):

oh by the way, I only tested this on linux so far

gnzlbg (Nov 15 2018 at 14:23, on Zulip):

i'll test macos once the next nightly is released, it is not working there for me yet

mw (Nov 15 2018 at 14:24, on Zulip):

did you verify that the patch above works?

mw (Nov 15 2018 at 14:24, on Zulip):

i.e. is it cargo that's not passing on the LD_FLAGS variable?

gnzlbg (Nov 15 2018 at 14:26, on Zulip):

nope, I haven't yet, last time i tried I was still passing it more flags (use lto thin, etc.)

mw (Nov 15 2018 at 14:28, on Zulip):

but jemalloc-sys built? because that's where it failed for me

mw (Nov 15 2018 at 14:28, on Zulip):

while linking some shared library

mw (Nov 15 2018 at 14:29, on Zulip):

btw, do you know why cargo bench fails?

mw (Nov 15 2018 at 14:29, on Zulip):

it gives a lot of errors while building the benchmark

gnzlbg (Nov 15 2018 at 14:31, on Zulip):

with the LTO options it did not build, I just assumed that it was because I was passing it more options than you, and that because I am using a nightly without your latest PR

gnzlbg (Nov 15 2018 at 14:31, on Zulip):

it might be that the benchmarks are not tested on CI ?

gnzlbg (Nov 15 2018 at 14:32, on Zulip):

at some point it switched to use cargo test --all but maybe that does not build the benchmarks and they got out of sync

mw (Nov 15 2018 at 14:33, on Zulip):

I get this:

mw (Nov 15 2018 at 14:33, on Zulip):
error[E0599]: no method named `dealloc` found for type `jemallocator::Jemalloc` in the current scope
    --> benches/roundtrip.rs:249:26
     |
249  |                   Jemalloc.dealloc(ptr, new_layout);
     |                            ^^^^^^^
...
4873 | / rt_realloc_excess_used!(
4874 | |     rt_primes_4194301bytes_32align_realloc_excess_used,
4875 | |     4194301,
4876 | |     32
4877 | | );
     | |__- in this macro invocation
     |
     = help: items from traits can only be used if the trait is in scope
help: the following trait is implemented but not in scope, perhaps add a `use` for it:
     |
9    | use std::alloc::GlobalAlloc;
     |

error: aborting due to 6301 previous errors
mw (Nov 15 2018 at 14:34, on Zulip):

might be just a small fix that's needed

gnzlbg (Nov 15 2018 at 14:34, on Zulip):

@mw i'll send a PR fixing those, the benchmarks are not built on CI anymore, and they don't have any of the GlobalAlloc / Alloc changes of the last... year at least ... so nothing builds

gnzlbg (Nov 15 2018 at 14:35, on Zulip):

working on a PR now

mw (Nov 15 2018 at 14:35, on Zulip):

thanks!

gnzlbg (Nov 15 2018 at 15:22, on Zulip):

PR is open (https://github.com/alexcrichton/jemallocator/pull/98), once CI is green I'll merge.

gnzlbg (Nov 15 2018 at 15:23, on Zulip):

rust chokes a bit compiling the benchmarks though

gnzlbg (Nov 16 2018 at 12:58, on Zulip):

@mw the benchmark changes have been merged and there is a nightly containing the linker changes already

gnzlbg (Nov 16 2018 at 12:59, on Zulip):

until clang is shipped with the llvm-tools we can just query the clang version at >= 8

mw (Nov 16 2018 at 15:02, on Zulip):

rustc -vV prints the LLVM version. we could use that

gnzlbg (Nov 16 2018 at 15:46, on Zulip):

sounds good

mw (Nov 16 2018 at 17:15, on Zulip):

I have the benchmarks running but they are taking so long, I think I won't wait for them

mw (Nov 16 2018 at 17:16, on Zulip):

fyi: I'll be on vacation until wednesday, so I probably won't be online

mw (Nov 16 2018 at 17:18, on Zulip):

hm, you can do a cargo benchcmp on the results printed so far...

mw (Nov 16 2018 at 17:19, on Zulip):

so these are not all benchmarks, but quite a few. they look pretty good mostly:

mw (Nov 16 2018 at 17:19, on Zulip):

https://gist.githubusercontent.com/michaelwoerister/2c9e2b03f0343d70a8d57ceb637b9895/raw/c98db6398d4553e5fa3f0da573bf7bf883932142/partial-benchcmp-jemallocator-xtlo.txt

Last update: Nov 22 2019 at 04:35UTC