Stream: wg-secure-code

Topic: Reproducible builds


Joshua Liebow-Feeser (Dec 14 2018 at 19:24, on Zulip):

Somebody managed to bootstrap rustc from mrustc. It's not fully reproducible yet, but definitely a step in the right direction. https://www.gnu.org/software/guix/blog/2018/bootstrapping-rust/

Tony Arcieri (Dec 14 2018 at 19:37, on Zulip):

neat

Tony Arcieri (Dec 14 2018 at 19:38, on Zulip):

also re: the overall topic, if anyone knows the state of rustc in that regard, I'm curious

Joshua Liebow-Feeser (Dec 14 2018 at 19:38, on Zulip):

They discuss it briefly in the blog post

Joshua Liebow-Feeser (Dec 14 2018 at 19:39, on Zulip):

"Rust takes reproducible builds seriously, but there are some reproducibility problems left in earlier compilers that pop up very sporadically (mostly because of LLVM, and some because of Rust hashtable poisoning). Help wanted, especially from LLVM people!"

Tony Arcieri (Dec 14 2018 at 19:39, on Zulip):

more specifically I'd be curious about a QEMU (or otherwise) recipe which, given a cargo project, always produces the same binary, byte-for-byte

Tony Arcieri (Jan 06 2019 at 17:37, on Zulip):

Anyone familiar with reprotest?

Tony Arcieri (Jan 06 2019 at 17:37, on Zulip):

it looks very interesting but I can't find any information about it

Tony Arcieri (Jan 06 2019 at 17:38, on Zulip):

here's something using it https://github.com/kpcyrd/sniffglue/blob/master/ci/reprotest.sh

Tony Arcieri (Jan 06 2019 at 17:38, on Zulip):

I guess it's this? https://pypi.org/project/reprotest/

Shnatsel (Jan 06 2019 at 22:51, on Zulip):

Question: if I make a binary with a Cargo.lock in the repo, and then publish that binary to crates.io, does that Cargo.lock propagate to users, or whenever someone types cargo install my-program they actually get latest semver-compatible versions?

Alex Gaynor (Jan 06 2019 at 22:54, on Zulip):

You can't publish binaries to crates.io at all, can you? I'd assume any Cargo.lock you have around is ignored.

Shnatsel (Jan 06 2019 at 23:06, on Zulip):

Right. I meant executables that are built from source on the user's machine. So Cargo.lock is not included in the upload to crates.io?

Tony Arcieri (Jan 07 2019 at 15:38, on Zulip):

no, it can't be because Cargo.lock's resolution depends on the constraints of all of the other dependencies

briansmith (Jan 18 2019 at 01:32, on Zulip):

You can definitely publish binary components in crates, without source code. It is done by some crates. The only requirement is that your Cargo.toml must generate something; you can include whatever else you want.

Shnatsel (Jan 18 2019 at 01:37, on Zulip):

so like, I can have a binary embedded, a hello world source code, a build.rs that copies my binary to target/release, and everything is fine and dandy and I can push that to crates.io?

briansmith (Jan 18 2019 at 01:38, on Zulip):

I don't know about that. Probably your build.rs has to build an executable. But you can have another executable included that it copies to the output and executes, I think.

Shnatsel (Jan 18 2019 at 01:38, on Zulip):

ewww, gross

briansmith (Jan 18 2019 at 01:39, on Zulip):

I know you can have binary libraries (*.a) that you ship in your crate, for example.

Shnatsel (Jan 18 2019 at 01:39, on Zulip):

on the other hand, I don't know what I expected

briansmith (Jan 18 2019 at 01:40, on Zulip):

My understanding is that the new golang build/module system will require that what is in github matches what you have.

briansmith (Jan 18 2019 at 01:40, on Zulip):

But, even then, I don't know if it lets you put your *.a and *.exe files in github and then says "OK". I'm guessing not.

Shnatsel (Jan 18 2019 at 01:42, on Zulip):

I will bypass any heuristics they make up

Shnatsel (Jan 18 2019 at 01:42, on Zulip):

the only thing that could sorta-reliably stop it is manual review

Tony Arcieri (Jan 18 2019 at 06:14, on Zulip):

there are like

Tony Arcieri (Jan 18 2019 at 06:14, on Zulip):

two concurrent threads I was trying to get to cross streams, heh

Tony Arcieri (Jan 18 2019 at 06:14, on Zulip):

@Shnatsel's embedding Cargo.lock and such in binaries

Tony Arcieri (Jan 18 2019 at 06:15, on Zulip):

and this one: https://github.com/rust-lang/cargo/issues/5654

Tony Arcieri (Jan 18 2019 at 06:15, on Zulip):

because I think you can take @Shnatsel's thing and put it in a crate and solve ^^^ problem

Tony Arcieri (Jan 22 2019 at 20:41, on Zulip):

anyone familiar with guix challenge? https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-challenge.html

Tony Arcieri (May 18 2019 at 16:19, on Zulip):

Well, I made a repo/crate for this: https://github.com/rust-secure-code/cargo-repro

Tony Arcieri (May 18 2019 at 16:19, on Zulip):

it doesn't do anything yet :wink:

Tony Arcieri (May 18 2019 at 16:19, on Zulip):

would love to gather input here: https://github.com/rust-secure-code/cargo-repro/issues/3

DevQps (May 20 2019 at 11:46, on Zulip):

Nice! I just came back from vacation. Just some questions to get a good feeling for the goal of this tool (I might have missed some prior conversations here and there :))

Is it used to check if packages are the same (as in: project code and .zip archives downloaded from crates.io) or to check if binaries are the same? Or even mixes between them? (Like to check if an archive outputs the same binary?)

Of course I guess checking binaries really depends on the compiler that is used right?

Tony Arcieri (May 20 2019 at 13:58, on Zulip):

to me it's two parts:

Tony Arcieri (May 20 2019 at 14:00, on Zulip):

1) building binaries which are reproducible, and gathering the necessary environmental information to reproduce them. the former part is largely just cargo build --locked but I think such a tool should try to collect any potential sources of nondeterminism and potentially encode them into the resulting binary

Tony Arcieri (May 20 2019 at 14:01, on Zulip):

2) verifying a build is actually reproducible, given the source tree it was built from as a cargo project. there's reprotest for this, but it's a bit cumbersome to use and also trying to be a general purpose reproducible build utility

Tony Arcieri (May 20 2019 at 14:02, on Zulip):

mostly the problem to me is reproducible builds in Rust right now seem pretty cargo cult and I haven't even gotten one to work myself yet, so I'd like a tool that at least encodes the "recipe" for doing so somewhere other than a bunch of forum posts and GitHub issues

Tony Arcieri (May 20 2019 at 14:02, on Zulip):

"cargo cult" yeesh unintentional pun :sweat_smile:

Tony Arcieri (Aug 12 2019 at 17:02, on Zulip):

some very, very preliminary functionality in this PR: https://github.com/rust-secure-code/cargo-repro/pull/5

Last update: Nov 11 2019 at 22:40UTC