Stream: t-compiler/wg-llvm

Topic: LLVM upgrade effort


J. Ryan Stinnett (Nov 19 2019 at 01:07, on Zulip):

I am trying to get a sense of how much pain / time / effort is required each time Rust goes through an upgrade (not just a cherry picking a bug fix) of LLVM. Along with that, I am curious about the kinds of changes typically required to adapt to the new version of LLVM.

Looking over the docs in the rustc guide, it feels like quite an involved process that someone might be focused on full time for a while and seems to occur a few times per year for specific features and also to stay current with each LLVM release. Is most of the time spent adapting the Rust <-> LLVM interface for API changes, or is it more dominated by fighting surprise behavior changes in LLVM and finding workarounds?

I have several ideas for tooling to help with these LLVM upgrade tasks, but I want to check whether my assumptions match the experience of those who've worked through an upgrade in past for Rust.

simulacrum (Nov 19 2019 at 02:13, on Zulip):

I think we've not historically experienced much pain from LLVM upgrades, beyond breakage -- but that's generally on tier 2 or tier 3 platforms -- @cuviper and @Alex Crichton may be able to say more, I think they've been involved with the last couple upgrades

Alex Crichton (Nov 19 2019 at 02:22, on Zulip):

From what I remember 99% of the pain of an upgrade is fixing regressions in the ci builds we see through bors

Alex Crichton (Nov 19 2019 at 02:22, on Zulip):

The API changes are mostly quite smooth

Alex Crichton (Nov 19 2019 at 02:22, on Zulip):

Sometimes it's taken forever to narrow down issues

Alex Crichton (Nov 19 2019 at 02:22, on Zulip):

Other times it's been super smooth

Alex Crichton (Nov 19 2019 at 02:22, on Zulip):

I haven't been too too involved in the last one or two upgrades though, and I think they were relatively painless

centril (Nov 19 2019 at 08:17, on Zulip):

(Don't know how relevant it is, but generally speaking, having frequent LLVM submodule bumps is undesirable because all the rust-lang/rust devs have to rebuild LLVM, which is painful)

J. Ryan Stinnett (Nov 19 2019 at 11:57, on Zulip):

It sounds like it's actually easier and less painful than I assumed then. :happy: (The text "Unlike bugfixes, updating to pick up a new feature of LLVM typically requires a lot more work." in the rustc guide led me to assume it was usually a long / arduous adventure.)

Good to know about the additional complexity of LLVM changes forcing rebuilds for rustc devs... I hadn't realised that particular impact of more frequent LLVM upgrades.

simulacrum (Nov 19 2019 at 12:44, on Zulip):

It's not really an impact that I think is too important, or at least I barely notice it. I do have decent hardware though :)

It's also something that we can (and should) fix, it's just that it requires some design work and the exact solution isn't completely clear.

mati865 (Nov 19 2019 at 13:08, on Zulip):

I think we've not historically experienced much pain from LLVM upgrades, beyond breakage -- but that's generally on tier 2 or tier 3 platforms

FYI LLVM 9 upgrade broke tier1 targets when using non default options: #64609 #65618

J. Ryan Stinnett (Nov 19 2019 at 13:39, on Zulip):

For hunting down LLVM-related regressions, does that typically mean compiling many random revisions of LLVM locally to try to bisect the issue? (If so, it sounds slow since I imagine it takes a while to build at each bisection step.)

Alex Crichton (Nov 19 2019 at 14:14, on Zulip):

For me I typically extract ir first to get a standalone test I can reproduce with just llvm tools, then I reduce (sometimes also with llvm tools)

Alex Crichton (Nov 19 2019 at 14:14, on Zulip):

And then yeah bisection is typically next

J. Ryan Stinnett (Nov 19 2019 at 14:23, on Zulip):

Ah right, makes sense to reproduce in LLVM standalone where possible to simplify. LLVM has also added there own llvm-reduce tool for IR reduction recently, maybe that will be of some help here.

J. Ryan Stinnett (Nov 19 2019 at 14:30, on Zulip):

For LLVM standalone, there's also llvmlab bisect for accelerated bisection via a shared cache.

RalfJ (Nov 19 2019 at 14:40, on Zulip):

It's not really an impact that I think is too important, or at least I barely notice it. I do have decent hardware though :)

It's also something that we can (and should) fix, it's just that it requires some design work and the exact solution isn't completely clear.

LLVM builds take 30-60min on my hardware, which is not totally shabby either. Multiply this by 3 for 3 rustc checkouts (so that I can work on more than one PR at a time). It's a big pain.

RalfJ (Nov 19 2019 at 14:41, on Zulip):

in comparison, building one stage (so that I can run stage 1 tests) without having to build LLVM usually takes <20min.

simulacrum (Nov 19 2019 at 17:30, on Zulip):

in theory e.g. ccache (not sccache) would solve the multiply by 3

Last update: Dec 12 2019 at 00:50UTC