Stream: t-compiler

Topic: leveraging oss-fuzz #39916


pnkfelix (Nov 28 2019 at 10:11, on Zulip):

hi @Siavosh Zarrasvand , lets discuss fuzzing ideas here

pnkfelix (Nov 28 2019 at 10:12, on Zulip):

(for now I'm using issue #39916 as an anchor, just because I like to put some issue into each topic when I can. But its possible we will eventually want to spawn off a separate issue dedicated solely to OSS-Fuzz itself. Issue #39916 is a bit broader, dare I say "fuzzier" in scope.)

pnkfelix (Nov 28 2019 at 10:13, on Zulip):

I suspect first steps would be to do a survey of what we already have implemented in https://github.com/rust-fuzz

pnkfelix (Nov 28 2019 at 10:14, on Zulip):

also, it seems this group has a dedicated irc channel (#rust-fuzz on irc.mozilla.org). I wonder if they can be convinced to come to zulip as well.

pnkfelix (Nov 28 2019 at 10:18, on Zulip):

now, to be fair, I think the stuff in https://github.com/rust-fuzz is dedicated to fuzzing projects implemented in Rust. While my own personal interest is in specifically fuzzing the rustc compiler

Siavosh Zarrasvand (Nov 28 2019 at 13:49, on Zulip):

@pnkfelix Initially, it seems to me we need to identify two things.

  1. What tools we are going to use
  2. What we're going to fuzz

OSS-Fuzz supports libfuzzer and American Fuzzy Lop (AFL), the tests would need to be written in those frameworks. If they are not, there is a chance Google would not accept the submission to be run on their servers, but of course, we can try to speak to them if we choose not to use them.

libfuzzer: http://llvm.org/docs/LibFuzzer.html
AFL: http://lcamtuf.coredump.cx/afl/

There is one QuickCheck like crate for Rust: https://docs.rs/quickcheck/0.9.0/quickcheck/
QuickCheck: http://hackage.haskell.org/package/QuickCheck

Two articles on Fuzzing vs QuickCheck/Property testing:

https://danluu.com/testing/
https://medium.com/@oherrala/quickcheck-or-fuzzing-which-one-to-use-98d200e9609b

Siavosh Zarrasvand (Nov 28 2019 at 13:50, on Zulip):

now, to be fair, I think the stuff in https://github.com/rust-fuzz is dedicated to fuzzing projects implemented in Rust. While my own personal interest is in specifically fuzzing the rustc compiler

Same here. The rust compiler and some runtime. Tokio is very high on my priorities...

Siavosh Zarrasvand (Nov 28 2019 at 13:53, on Zulip):

My personal preference:

I would go for fuzzing as it gives higher coverage. Well setup, we should be able perform property testing through fuzzing as well, I don't really see why we wouldn't be able to do that.

With regards to targets, I would say

rustc
std, for example async-std
runtimes like Tokio

pnkfelix (Nov 28 2019 at 13:53, on Zulip):

(I probably won't be able to chat live at this point because I have to prep for the compiler meeting and then I have to go home and take care of my kids for the evening)

pnkfelix (Nov 28 2019 at 13:54, on Zulip):

I'm pretty ignorant of APIs for libfuzzer and AFL, but I can review them.

pnkfelix (Nov 28 2019 at 13:55, on Zulip):

there is an AFL crate in rust-fuzz: https://github.com/rust-fuzz/afl.rs

pnkfelix (Nov 28 2019 at 13:55, on Zulip):

but I don't know if that's a wrapper around http://lcamtuf.coredump.cx/afl/, or if it is something else

Siavosh Zarrasvand (Nov 28 2019 at 13:56, on Zulip):

I'm pretty ignorant of APIs for libfuzzer and AFL, but I can review them.

Happy to answer questions as I can learn that way. My background on the lower layers is reverse engineering, exploit development, some embedded programming. So I have no experience in compiler development but really eager to understand your thought process.

If it saves you time, write the questions and I'll provide answers :slight_smile:

pnkfelix (Nov 28 2019 at 13:57, on Zulip):

I understand what it means to fuzz rustc, in terms of observing the compiler's behavior on variations of legal source code.

pnkfelix (Nov 28 2019 at 13:57, on Zulip):

I'm not sure I understand what you mean by fuzzing std, though.

pnkfelix (Nov 28 2019 at 13:58, on Zulip):

is that just a matter of trying to identify new test inputs from existing tests for std ?

Siavosh Zarrasvand (Nov 28 2019 at 13:59, on Zulip):

is that just a matter of trying to identify new test inputs from existing tests for std ?

Using a fuzzing testing approach to test some std-libraries, ideally the IO-related ones. For example tokio, fuzzing http-request/response methods to identify edge cases.

pachi (Nov 28 2019 at 14:22, on Zulip):

I suppose you're already aware of it, but @Shnatsel did explore fuzzying in rust, and has written some docs about the experience. I'm pinging them just in case, and this is an interesting link with some of their findings https://medium.com/@shnatsel/how-ive-found-vulnerability-in-a-popular-rust-crate-and-you-can-too-3db081a67fb

Siavosh Zarrasvand (Nov 28 2019 at 15:28, on Zulip):

I suppose you're already aware of it, but Shnatsel did explore fuzzying in rust, and has written some docs about the experience. I'm pinging them just in case, and this is an interesting link with some of their findings https://medium.com/@shnatsel/how-ive-found-vulnerability-in-a-popular-rust-crate-and-you-can-too-3db081a67fb

I did read that post a while ago I believe. Also read a couple of other fuzzing attempts, one in particular at the compiler that found 8 issues. I posted it somewhere here before.

Perhaps useful to make a list of past approaches for reference and put it on gist. I'm out now but will sort when back :slight_smile:

nagisa (Nov 28 2019 at 18:55, on Zulip):

also, it seems this group has a dedicated irc channel (#rust-fuzz on irc.mozilla.org). I wonder if they can be convinced to come to zulip as well.

Shouldn’t be too hard, you’ve already convinced at least 1 person there. @Manish Goregaokar is also here.

Manish Goregaokar (Nov 28 2019 at 18:55, on Zulip):

I don't really actively monitor zulip though

nagisa (Nov 28 2019 at 18:55, on Zulip):

now, to be fair, I think the stuff in https://github.com/rust-fuzz is dedicated to fuzzing projects implemented in Rust. While my own personal interest is in specifically fuzzing the rustc compiler

My undergrad was about that specifically. Fuzzing a compiler with traditional tools is… not that feasible.

Siavosh Zarrasvand (Nov 28 2019 at 22:49, on Zulip):

now, to be fair, I think the stuff in https://github.com/rust-fuzz is dedicated to fuzzing projects implemented in Rust. While my own personal interest is in specifically fuzzing the rustc compiler

My undergrad was about that specifically. Fuzzing a compiler with traditional tools is… not that feasible.

Please elaborate :slight_smile:

nagisa (Nov 28 2019 at 23:13, on Zulip):

The state becomes very large very quickly. You want to fuzz separate components, but in compiler code later phases/stages etc often have many presumptions based on what the previous stages do and filter out.

nagisa (Nov 28 2019 at 23:14, on Zulip):

Those are two major problems. If you had infinite memory, fuzzing whole compiler with traditional branch-tracking tools wouldn’t be a problem. Alas.

Siavosh Zarrasvand (Nov 29 2019 at 00:46, on Zulip):

Those are two major problems. If you had infinite memory, fuzzing whole compiler with traditional branch-tracking tools wouldn’t be a problem. Alas.

I do not disagree with this. I do have some ideas for mitigation, will write after I trial and error.

Siavosh Zarrasvand (Nov 29 2019 at 00:47, on Zulip):

Ah, I had a look, and Google's OSS-Fuzz docs even mention what I had in my head:
https://github.com/google/oss-fuzz/blob/master/docs/faq.md#can-i-launch-an-additional-process-eg-a-daemon-from-my-fuzz-target

i.e, dump fuzzing is efficient, on very limited scopes. Have many fuzz targets instead.

nagisa (Nov 29 2019 at 01:36, on Zulip):

Yeah hence: fuzz separate components. but that is a non trivial problem too. Could adapt the compiler code, of course, but that’s also non-trivial problem.

Siavosh Zarrasvand (Nov 29 2019 at 18:13, on Zulip):

Leaving this here for now for reference. Please ignore it. https://github.com/SiavoshZarrasvand/Awesome-Fuzzing/blob/master/awesome-fuzzing.md

Siavosh Zarrasvand (Dec 02 2019 at 18:26, on Zulip):

When building rustc from source, is it good to set

# Build a multi-threaded rustc parallel-compiler = true

?

Siavosh Zarrasvand (Dec 02 2019 at 18:29, on Zulip):

Also, is there a way to alternate between different settings when compiling and fuzzing the targets? For example to have one rustc compiled with jemalloc as the target, and one without.

simulacrum (Dec 02 2019 at 18:56, on Zulip):

generally speaking, for now, probably no -- we don't ship a parallel compiler by default for now

simulacrum (Dec 02 2019 at 18:57, on Zulip):

You can alternate between them probably most easily by just having separate checkouts, though that's pretty expensive on disk space

Last update: Dec 12 2019 at 00:50UTC