Stream: wg-secure-code

Topic: integration-test-only-apis


briansmith (Nov 26 2018 at 01:32, on Zulip):

We need a mechanism to expose APIs to integration tests (not within-module unit tests, but the tests under tests/ parallel to src/) that aren't publicly exposed elsewhere. @Tony Arcieri gave one use case in integer-overflow (mocking APIs) and also kindly demonstrated the flaw in the most common workaround for this missing feature: making such APIs available only in debug mode, and running tests only in debug mode, not test --release.

briansmith (Nov 26 2018 at 01:33, on Zulip):

I also have a need for such test-only APIs, not only exposed to my own integration tests, but also the integration tests to the users of my crates.

briansmith (Nov 26 2018 at 01:33, on Zulip):

In particular, it is very convenient to be able to swap out a a good PRNG for a deterministic one for fuzzing, known-answer tests, and other tests, but we don't generally want the PRNG to be swapped out in non-test situations.

brycx (Nov 26 2018 at 11:31, on Zulip):

In particular, it is very convenient to be able to swap out a a good PRNG for a deterministic one for fuzzing, known-answer tests, and other tests, but we don't generally want the PRNG to be swapped out in non-test situations.

In case you didn't know about honggfuzz, it has an attribute for exactly that: https://github.com/rust-fuzz/honggfuzz-rs#conditional-compilation

RalfJ (Nov 26 2018 at 11:42, on Zulip):

@briansmith you probably thought of feature flags, but I'm still curious why they are not good enough?

Tony Arcieri (Nov 26 2018 at 14:46, on Zulip):

in my case, I'm using feature flags, but there's a feature I want to make sure is never used in a release build

briansmith (Nov 26 2018 at 18:32, on Zulip):

Just like Tony, I want to make sure the feature is never used in non-test situations. The whole point of the API is to be secure and this extension point totally breaks the guarantee of security.

Zach Reizner (Nov 26 2018 at 18:48, on Zulip):

@briansmith if the integration test feature is enabled, you could print out warnings (so a human can see), kill the current process after a short time (so a headless daemon can't last very long), or refuse to run if the current environment isn't a dev environment (e.g. current directory tree has a Cargo.toml, current process tree has cargo).

Shnatsel (Nov 26 2018 at 22:27, on Zulip):

In case you didn't know about honggfuzz, it has an attribute for exactly that: https://github.com/rust-fuzz/honggfuzz-rs#conditional-compilation

Actually that's supported by all three Rust fuzzers: cargo-fuzz, afl.rs, honggfuzz-rs.

Last update: Nov 11 2019 at 23:05UTC