Stream: t-compiler

Topic: Miscompilation with target-cpu=znver1 (AMD Ryzen 1000/2000


Siavosh Zarrasvand (Nov 15 2019 at 04:24, on Zulip):

Hey, this was pinged to llvm-icebreakers https://github.com/rust-lang/rust/issues/63959

I'm happy to take it on, but see my comment. I lack the target CPU on both machines, would that be an issue?

pnkfelix (Nov 15 2019 at 09:40, on Zulip):

I suspect it will be difficult to do much without the hardware. @eddyb did volunteer to run testcases or dump data on their own hardware ... but still, its probably better for us to focus effort on finding someone with the hardware ...

mati865 (Nov 15 2019 at 11:16, on Zulip):

@eddyb was able to reproduce the crash on Intel CPU by cross compiling to Windows and using Wine to run the binary. So it shouldn't be an issue.

Siavosh Zarrasvand (Nov 15 2019 at 14:56, on Zulip):

@pnkfelix Fine. If I could suggest some improvements, I would suggest clarifying these things when marking issues for llvm-icebreakers. If people volunteer too many times and are told they cannot due to <random issue> they lose interest. I'm not there yet but there's only so many times you volunteer with non-constructive outcome.

pnkfelix (Nov 15 2019 at 15:00, on Zulip):

@Siavosh Zarrasvand noted

nagisa (Nov 15 2019 at 15:24, on Zulip):

There is a ryzen box on https://cfarm.tetaneutral.net/news/

nagisa (Nov 15 2019 at 15:24, on Zulip):

if it reproduces with wine, should be good enough

Siavosh Zarrasvand (Nov 15 2019 at 15:43, on Zulip):

@nagisa @mati865 Seems plausible. I'm happy to take it on but @pnkfelix might know more. I can wait and if you cannot find someone with the hardware, feel free to let me know...

pnkfelix (Nov 15 2019 at 15:44, on Zulip):

at this point I trust @mati865 and @nagisa 's interpretation of things more than my own

pnkfelix (Nov 15 2019 at 15:45, on Zulip):

(I wasn't am not sure if the reproduction with Wine required running Wine atop a Ryzen box, though...)

Siavosh Zarrasvand (Nov 15 2019 at 15:45, on Zulip):

at this point I trust mati865 and nagisa 's interpretation of things more than my own

Oki, cool. First step for me is to reproduce the issue on my hardware then. I will do that during the weekend and let you guys know how I progress :slight_smile:

pnkfelix (Nov 15 2019 at 15:46, on Zulip):

best thing might be to ping @eddyb and just ask for clarification, if their notes on the bug do not suffice.

Siavosh Zarrasvand (Nov 15 2019 at 15:46, on Zulip):

(I wasn't am not sure if the reproduction with Wine required running Wine atop a Ryzen box, though...)

Yeah, fair point. Hence why I listen, no point wasting effort, as that is even more discouraging. Let me see if I can reproduce it, I will report back here during the weekend

Siavosh Zarrasvand (Nov 15 2019 at 15:47, on Zulip):

best thing might be to ping eddyb and just ask for clarification, if their notes on the bug do not suffice.

Also valid, but he would only know if he tried wine on both AMD and intel. If he tried on AMD only he might not know for Intel. I will ask.

mati865 (Nov 15 2019 at 16:02, on Zulip):

I reproduced crash on Ryzen box on both Windows and Wine, eddyb reproduced it using Wine on Intel CPU.
https://github.com/rust-lang/rust/issues/63959#issuecomment-552219951 isn't totally clear but I thought people would get the point.

mati865 (Nov 15 2019 at 16:02, on Zulip):

Always targetting znver1 ofc.

pnkfelix (Nov 15 2019 at 16:06, on Zulip):

Okay, so: sorry for the confusion on my part @Siavosh Zarrasvand ; it sounds like you should be able to have success on other hardware as long as you have access to Wine.

Siavosh Zarrasvand (Nov 15 2019 at 16:09, on Zulip):

Okay, so: sorry for the confusion on my part @Siavosh Zarrasvand ; it sounds like you should be able to have success on other hardware as long as you have access to Wine.

No worries, thank you for doublechecking

mati865 (Nov 15 2019 at 16:12, on Zulip):

I believe it can be reproduced natively on Windows on non Ryzen box as well, at least with windows-gnu target I haven't found specific instructions that would not run on any other "modern" CPU.
So you should be able to use whatever OS and CPU you want but binaries have to be compiled for znver1 CPU and Windows target.

eddyb (Nov 15 2019 at 21:15, on Zulip):

whoops didn't see this thread. I left a comment on the issue

Siavosh Zarrasvand (Nov 17 2019 at 00:19, on Zulip):

@eddyb All good, I'm installing all targets now, will attempt to reproduce after. Do I understand it correctly that znver1 is not really required, but compiling for Windows is, as the miscompilation happens for the windows targets?

eddyb (Nov 17 2019 at 00:19, on Zulip):

"all targets"?

eddyb (Nov 17 2019 at 00:20, on Zulip):

@Siavosh Zarrasvand you might not need a "1st generation AMD Ryzen" (i.e. znver1) CPU but you do need to run rustc with -C target-cpu=znver1

Siavosh Zarrasvand (Nov 17 2019 at 00:22, on Zulip):

"all targets"?

Hahaha, I figured I might as well... probably will need them at some point if I hang around in this channel. Is in rustup target add all

eddyb (Nov 17 2019 at 00:22, on Zulip):

the assembly looks very different from other target CPUs, likely due to the different cost tables (which are so different because AMD made Ryzen unlike existing CPUs)

eddyb (Nov 17 2019 at 00:22, on Zulip):

@Siavosh Zarrasvand that seems excessive and a waste of your time, you should only add each target you need at a time

Siavosh Zarrasvand (Nov 17 2019 at 00:22, on Zulip):

What is the full compiler command?

Siavosh Zarrasvand (Nov 17 2019 at 00:23, on Zulip):

Siavosh Zarrasvand that seems excessive and a waste of your time, you should only add each target you need at a time

Are they that many? I thought I'd give it 20 minutes...

eddyb (Nov 17 2019 at 00:23, on Zulip):

every time you update nightly you will have to download probably entire GBs

eddyb (Nov 17 2019 at 00:23, on Zulip):

for no real good reason

Siavosh Zarrasvand (Nov 17 2019 at 00:24, on Zulip):

Omg...

Siavosh Zarrasvand (Nov 17 2019 at 00:24, on Zulip):

oki, I will have to remove them. Can I abort, clean and add targets again?

eddyb (Nov 17 2019 at 00:24, on Zulip):

I think you can just Ctrl+C it yeah

eddyb (Nov 17 2019 at 00:24, on Zulip):

run rustup target list

eddyb (Nov 17 2019 at 00:25, on Zulip):

it will tell you which ones are installed

eddyb (Nov 17 2019 at 00:25, on Zulip):

there are 89 targets in total (listed for me)

eddyb (Nov 17 2019 at 00:26, on Zulip):

I don't actually know how to cross-compile for windows, you might need help from @mati865 on that

Siavosh Zarrasvand (Nov 17 2019 at 00:26, on Zulip):

Yeah, removing now.

Which triple do I need to reproduce the error?

eddyb (Nov 17 2019 at 00:26, on Zulip):

(I've never tried it, only copied windows binaries from a windows machine to linux and ran wine on it)

eddyb (Nov 17 2019 at 00:26, on Zulip):

x86_64-pc-windows-gnu or x86_64-pc-windows-msvc

eddyb (Nov 17 2019 at 00:27, on Zulip):

I think both reproduce but I'm not sure which one is easier to cross-compile for

Siavosh Zarrasvand (Nov 17 2019 at 00:27, on Zulip):

Oki, cool, I will try those two

Siavosh Zarrasvand (Nov 17 2019 at 01:02, on Zulip):

This compiles fine cargo rustc --release --target=x86_64-pc-windows-gnu -- -C target-cpu=znver1 -C linker=x86_64-w64-mingw32-gcc

Siavosh Zarrasvand (Nov 17 2019 at 01:02, on Zulip):

Am I supposed to get the issue while I compile, or when I run the binary in wine?

Siavosh Zarrasvand (Nov 17 2019 at 01:19, on Zulip):

@eddyb How did you manage to get it to crash under wine? Does it mean that you installed Windows binaries for rustup on wine, and compiled using your wine installation of Rust?

Siavosh Zarrasvand (Nov 17 2019 at 01:21, on Zulip):

My cross compilation works, and it runs through wine
It works for the first code example: https://github.com/rust-lang/rust/issues/63959#issuecomment-525482889

For the later examples in the thread on Github, I can compile but not run the binaries, the error I get is related to my nvidia drivers.

eddyb (Nov 17 2019 at 01:38, on Zulip):

when you run the binary

eddyb (Nov 17 2019 at 01:39, on Zulip):

@Siavosh Zarrasvand you shouldn't use Cargo, it's just more complicated

eddyb (Nov 17 2019 at 01:40, on Zulip):

adding your flags to my command:

rustc main.rs -C opt-level=3 -C codegen-units=1 -C target-cpu=znver1 --edition=2018 --target=x86_64-pc-windows-gnu -C linker=x86_64-w64-mingw32-gc
eddyb (Nov 17 2019 at 01:40, on Zulip):

@Siavosh Zarrasvand oh you should also only use my latest testcase

eddyb (Nov 17 2019 at 01:41, on Zulip):

before I started reducing all of the testcases crash while rustc is running code from a proc macro, so those cannot reproduce except on windows

Siavosh Zarrasvand (Nov 17 2019 at 01:55, on Zulip):

before I started reducing all of the testcases crash while rustc is running code from a proc macro, so those cannot reproduce except on windows

I think this is it, right:

thread 'main' panicked at 'assertion failed: (left == right) left: ([1, 1, 1, 1, 1, 1, 1, 1], [2, 2, 2, 2, 2, 2, 2, 2], [3, 3, 3, 3, 3, 3, 3, 3]), right: ([16, 250, 50, 0, 0, 0, 0, 0], [32, 32, 32, 32, 32, 32, 32, 32], [3, 3, 3, 3, 3, 3, 3, 3])', ./src/main.rs:37:13 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.

Siavosh Zarrasvand (Nov 17 2019 at 01:59, on Zulip):

Alright, it is 2 AM here in London. If this is correct, then I continue tomorrow to see if I can debug it :slight_smile:

eddyb (Nov 17 2019 at 02:01, on Zulip):

@Siavosh Zarrasvand yupp, nice!

eddyb (Nov 17 2019 at 02:03, on Zulip):

I think if you replace -C target-cpu=znver1 with -C target-cpu=native, the assertion failure will go away

eddyb (Nov 17 2019 at 02:05, on Zulip):

@Siavosh Zarrasvand btw, on github, you need to put triple backticks (```) on their own lines

Siavosh Zarrasvand (Nov 17 2019 at 02:09, on Zulip):

I think if you replace -C target-cpu=znver1 with -C target-cpu=native, the assertion failure will go away

That is correct, cpu-target=native will not produce the assertion error. Would be interested to understand how you debugged it to this degree so far :slight_smile:

eddyb (Nov 17 2019 at 02:10, on Zulip):

I didn't debug it, I just took the code and reduced it while keeping it crashing (and then later switched to an assertion failure)

eddyb (Nov 17 2019 at 02:11, on Zulip):

oh and if I haven't mentioned this already, the optimized LLVM IR doesn't seem to differ much between windows and linux, but the assembly does, by a lot, because of windows calling conventions, I think

Siavosh Zarrasvand (Nov 17 2019 at 02:15, on Zulip):

oh and if I haven't mentioned this already, the optimized LLVM IR doesn't seem to differ much between windows and linux, but the assembly does, by a lot, because of windows calling conventions, I think

Would that not be the case in general, that the assembly is very different? They do compile for different ABIs after all and specially the optimised versions optimize for different OS-level schedulers, maybe? My ASM is a bit rusty (pun intended) so I do not remember how different the results should be.

eddyb (Nov 17 2019 at 02:17, on Zulip):

OS schedulers can't have anything to do with userspace code

eddyb (Nov 17 2019 at 02:18, on Zulip):

and the ABIs typically only affect function prologue/epilogue

eddyb (Nov 17 2019 at 02:18, on Zulip):

the only reason the difference is so big here is the heavy use of SIMD, I think

eddyb (Nov 17 2019 at 02:18, on Zulip):

(that's also why those types have to be so big, so that LLVM uses a lot of SIMD instructions to move data around)

Siavosh Zarrasvand (Nov 17 2019 at 02:19, on Zulip):

OS schedulers can't have anything to do with userspace code

What I meant was that at compile time, the strategies of how to lay out the ASM could be different based on which scheduler the compiler is tasked to optimize for.

eddyb (Nov 17 2019 at 02:20, on Zulip):

nope

eddyb (Nov 17 2019 at 02:21, on Zulip):

what is being optimized for is CPU microarchitectural details, like how many instructions of a certain category can be executed in parallel, how many cycles an instruction is expected to take, etc.

eddyb (Nov 17 2019 at 02:22, on Zulip):

the OS primarily only affects ABI (including exception handling, which can be pretty different between linux and windows)

eddyb (Nov 17 2019 at 02:23, on Zulip):

I was hoping I'd find a simple explanation so I went and searched for all mentions of windows in LLVM's X86 target

eddyb (Nov 17 2019 at 02:24, on Zulip):

and there isn't anything noticeable. looking at the assembly, the SIMD usage differs and that can totally be just from the ABI register assignment

Siavosh Zarrasvand (Nov 18 2019 at 19:57, on Zulip):

Quick update on this, I've been preoccupied since where we left it during the weekend. I intend to pick it up again before Friday, ideally by compiling a couple of similar binaries for different cpu-targets, and then use gdb or IDA Pro to debug my way through the binaries and understand the ASM-level difference between them. Then post a gist about that and wait for feedback.

Does this sounds like a way forward?

Last update: Dec 12 2019 at 00:55UTC