Currently, we have only one way to build fuzz targets (e.g. clang-fuzzer) in LLVM: specify -DLLVM_USE_SANITIZE_COVERAGE=On and -DLLVM_USE_SANITIZER=Address I would like to have this be more flexible. 1. Fuzz targets should be built in the default configuration (no asan, no coverage). They won't be suitable for fuzzing this way, but this will ensure that the code still builds (fuzz targets need to be build during check-all) 2. [stretch] each fuzz target foo-fuzz needs to have a build rule check-foo-fuzz that will execute the fuzz target on a fixed set of inputs (from the same repo) as a way of regression testing. This will ensure that the fuzz target not only builds, but runs. This is also a great regression test. 3. Fuzz targets should work with any of the sanitizers, or with none at all (-DLLVM_USE_SANITIZER=Address should not be mandatory) 4. There should be a way to specify the compile-time coverage instrumentation flags and link-time fuzzing engine flag. In particular, I'd like to support the env. vars. defined by OSS-Fuzz: CC, CXX, CFLAGS, CXXFLAGS, LIB_FUZZING_ENGINE (by passing these flags to cmake variables). This way we'll be able to build the fuzz targets with e.g. AFL, hoggfuzz, and with Clang Coverage (for coverage visualization)
For 1: Should clang-fuzzer build by default then rather than having to specify "ninja clang-fuzzer"? For 2: By fixed set of inputs, do you mean a corpus? How large should it be? And does this mean the default build should link with StandaloneFuzzTargetMain rather than doing -fsanitize=fuzzer? For 4: So we want to build all of LLVM with the CC, CXX, CFLAGS, CXXFLAGS from OSS-Fuzz, and then link just the fuzz target with LIB_FUZZING_ENGINE, right?
> For 1: Should clang-fuzzer build by default then rather than having to specify "ninja clang-fuzzer"? If there's a `check-clang-fuzzer` target then clang-fuzzer will be built as part of `ninja check`, which is good enough. I don't see much point in building clang-fuzzer as part of plain `ninja` when fuzzing is disabled in that case. > For 2: By fixed set of inputs, do you mean a corpus? How large should it be? And does this mean the default build should link with StandaloneFuzzTargetMain rather than doing -fsanitize=fuzzer? I'd go with just some very basic inputs for the fuzzer - some of these could basically be empty, unless there are particularly meaningful inputs for the fuzzer. For example, some valid bitcode would make sense for isel-fuzzer.
I made some progress on (1) and (2) with the following changes: r312200 cmake: Invent add_llvm_fuzzer to set up fuzzer targets r312338 llvm-isel-fuzzer: Make buildable and testable without libFuzzer r312425 Move some CLI utils out of llvm-isel-fuzzer and into the library r312427 llvm-isel-fuzzer: Add some basic tests Notably, all of the fuzzers in LLVM are now set up with a new cmake function ( add_llvm_fuzzer) and can optionally provide a main to be used without fuzzers. There's also a helper in FuzzMutate/FuzzerCLI.h to make writing that main function quite trivial.
[somehow my response was lost, repeating] (In reply to Matt Morehouse from comment #1) > For 1: Should clang-fuzzer build by default then rather than having to > specify "ninja clang-fuzzer"? Agree with Justin. Having check-foo-fuzzer as part of "check" would be enough. check-foo-fuzzer would build foo-fuzzer and run it against a fixed set of inputs. > > For 2: By fixed set of inputs, do you mean a corpus? How large should it > be? E.g. for llvm-dwarfdump-fuzzer those would be files in test/Object/Inputs > And does this mean the default build should link with > StandaloneFuzzTargetMain rather than doing -fsanitize=fuzzer? I would try to make it work with -fsanitize=fuzzer (don't instrument by default, but link with libFuzzer). Compared StandaloneFuzzTargetMain, libFuzzer can read dirs recursively. > > For 4: So we want to build all of LLVM with the CC, CXX, CFLAGS, CXXFLAGS > from OSS-Fuzz, and then link just the fuzz target with LIB_FUZZING_ENGINE, > right? Yes. Nore that you may still need to use -DLLVM_USE_SANITIZER=Address (Memory, Undefined) since those may pass extra flags to the build.
#4 is done in LLVM (r315629) and Clang (r315486, r315603, r315630). For #3, ASan is no longer required (I believe this changed when LLVM_USE_SANITIZER switched to -fsanitize=fuzzer-no-link instrumentation). But some fuzzers may still not work with other sanitizers. I know that clang-proto-fuzzer doesn't work with MSan yet.