Clarify libFuzzer and Code Coverage docs plus other minor fixes.

Bug: 539572
Change-Id: I83ae6f7e74b29be928bd81d50704e59ec864394c
Reviewed-on: https://chromium-review.googlesource.com/c/1340600
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Abhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609045}

testing/libfuzzer/README.md

@@ -4,8 +4,8 @@
 
 *** aside
 [Getting Started](getting_started.md)
-| [Buildbot](https://build.chromium.org/p/chromium.fyi/buildslaves/slave43-c1)
-| [ClusterFuzz Status](https://clusterfuzz.com/fuzzer-stats)
+| [Buildbot]
+| [ClusterFuzz Stats]
 | [Cover Bug]
 ***
 
@@ -55,8 +55,7 @@ libFuzzer.
 ## Project Links
 * [libFuzzer Infrastructure Bugs]
 
-[libFuzzer]: http://llvm.org/docs/LibFuzzer.html
-[crbug.com/539572]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572
+[Buildbot]: https://ci.chromium.org/p/chromium/g/chromium.fuzz/builders
 [Cover Bug]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572
 [Getting Started Guide]: getting_started.md
 [Efficient Fuzzer Guide]: efficient_fuzzer.md
@@ -66,8 +65,11 @@ libFuzzer.
 [Reproducing on Linux, Mac, and Android]: https://github.com/google/clusterfuzz-tools
 [Reference]: reference.md
 [ClusterFuzz Bugs]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label:Stability-LibFuzzer%20label:ClusterFuzz&sort=-modified&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified
+[ClusterFuzz Stats]: https://clusterfuzz.com/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan
 [Pdfium Bugs]: https://bugs.chromium.org/p/pdfium/issues/list?can=1&q=libfuzzer&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles
 [Manual Bugs]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3AStability-LibFuzzer+-label%3AClusterFuzz&sort=-modified&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids
 [OSS Trophies]: http://llvm.org/docs/LibFuzzer.html#trophies
 [Guided in-process fuzzing of Chrome components]: https://security.googleblog.com/2016/08/guided-in-process-fuzzing-of-chrome.html
+[crbug.com/539572]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572
+[libFuzzer]: http://llvm.org/docs/LibFuzzer.html
 [libFuzzer Infrastructure Bugs]: https://bugs.chromium.org/p/chromium/issues/list?q=label:LibFuzzer-Infra

testing/libfuzzer/clusterfuzz.md

@@ -32,23 +32,19 @@ provides links to crashes and coverage reports.
 
 
 ## Corpus
-ClusterFuzz uses two corpus types with libFuzzer:
 
-* **Seed** (or **static**) corpus: files manually uploaded by developers.
-ClusterFuzz uses these files for fuzzing but doesn't delete/overwrite them.
+Chromium developers can access the corpus stored in the [Corpus GCS Bucket] via
+web interface or by using `gsutil` tool (the latter is easier for downloading):
 
-* **General** (or **working**) corpus: files generated by fuzzers themselves.
-These corpus files are frequently modified during fuzzing sessions and can be
-deleted during corpus minimization.
+```bash
+mkdir local_corpus_dir
+gsutil -m cp -r gs://clusterfuzz-corpus/libfuzzer/<fuzz_target> local_corpus_dir
+```
 
-A fuzz target has two input corpus directories, seed and general, but its output
-goes into general corpus directory. Seed corpus is read-only.
-
-
-[Buildbot]: https://build.chromium.org/p/chromium.fyi/buildslaves/slave43-c1
+[Buildbot]: https://ci.chromium.org/p/chromium/g/chromium.fuzz/builders
 [Code Coverage]: https://chromium-coverage.appspot.com/reports/latest_fuzzers_only/linux/index.html
 [chromium_libfuzzer.py]: https://code.google.com/p/chromium/codesearch#chromium/build/scripts/slave/recipes/chromium_libfuzzer.py
-[ClusterFuzz Fuzzer Stats]: https://clusterfuzz.com/v2/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan
+[ClusterFuzz Fuzzer Stats]: https://clusterfuzz.com/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan
 [ClusterFuzz libFuzzer Logs]: https://console.cloud.google.com/storage/browser/clusterfuzz-libfuzzer-logs
 [Corpus GCS Bucket]: https://console.cloud.google.com/storage/clusterfuzz-corpus/libfuzzer
 [fuzzer_test.gni]: https://code.google.com/p/chromium/codesearch#chromium/src/testing/libfuzzer/fuzzer_test.gni

testing/libfuzzer/efficient_fuzzer.md

@@ -83,7 +83,8 @@ script provides detailed instructions as well as a usage example.
 
 Note that code coverage of a fuzz target **depends heavily** on the corpus
 provided when running the target, i.e. code coverage report generated by a fuzz
-target launched without any corpus would not make much sense.
+target launched without any corpus would not make much sense. To download the
+corpus from ClusterFuzz, see [ClusterFuzz Corpus].
 
 ## Corpus Size
 
@@ -291,6 +292,7 @@ there is no intended API to disable checksum verification, or when target code
 uses random generator that affects reproducibility of crashes.
 
 [AFL]: http://lcamtuf.coredump.cx/afl/
+[ClusterFuzz Corpus]: clusterfuzz.md#Corpus
 [ClusterFuzz status]: clusterfuzz.md#Status-Links
 [Corpus GCS Bucket]: https://console.cloud.google.com/storage/clusterfuzz-corpus/libfuzzer
 [issue 638836]: https://bugs.chromium.org/p/chromium/issues/detail?id=638836

testing/libfuzzer/getting_started.md

@@ -21,8 +21,13 @@ doesn't require any special configuration and gives meaningful output quickly
 for speed, coverage and other parameters.
 
 ```bash
-# With address sanitizer
-gn gen out/libfuzzer '--args=use_libfuzzer=true is_asan=true is_debug=false is_component_build=true' --check
+# AddressSanitizer is the default config we recommend testing with.
+# Linux:
+tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux ASan' out/libfuzzer
+# Mac:
+tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Mac ASan' out/libfuzzer
+# Windows:
+python tools\mb\mb.py gen -m chromium.fuzz -b "Libfuzzer Upload Windows ASan" out\libfuzzer
 ```
 
 Supported sanitizer configurations are:
@@ -32,14 +37,11 @@ Supported sanitizer configurations are:
 | `is_asan=true` | Enables [Address Sanitizer] to catch problems like buffer overruns. (only supported sanitizer on Windows and Mac)|
 | `is_msan=true` | Enables [Memory Sanitizer] to catch problems like uninitialized reads<sup>\[[*](reference.md#MSan)\]</sup>. |
 | `is_ubsan_security=true` | Enables [Undefined Behavior Sanitizer] to catch<sup>\[[*](reference.md#UBSan)\]</sup> undefined behavior like integer overflow. |
-| | It is possible to run libfuzzer without any sanitizers; *probably not what you want*.|
+| | It is possible to run fuzz targets without any sanitizers; *probably not what you want*.|
 
 Fuzz targets are built with minimal symbols by default. The symbol level
 can be adjusted in the usual way by setting `symbol_level`.
 
-To get the exact GN configuration that are used on our builders, see
-[Build Config].
-
 ## Write Fuzz Target
 
 Create a new `<my_fuzzer>.cc` file and define a `LLVMFuzzerTestOneInput`
@@ -61,7 +63,10 @@ do not use `testing/libfuzzer/fuzzers` directory, this was a directory used for
 initial sample fuzz targets and is no longer recommended for landing new fuzz
 targets.
 
-[quic_stream_factory_fuzzer.cc] is a good example of real-world fuzz target.
+Most of the fuzz targets are very small. They may perform one or a few API calls
+using the data provided by fuzzing engine as an argument. Other fuzz targets may
+be more complex if a certain initialization procedure needs to be performed.
+[quic_stream_factory_fuzzer.cc] is a good example of a complex fuzz target.
 
 ## Define GN Target
 
@@ -80,15 +85,21 @@ fuzzer_test("my_fuzzer") {
 Build with ninja as usual and run:
 
 ```bash
+# Build the fuzz target.
 ninja -C out/libfuzzer url_parse_fuzzer
-./out/libfuzzer/url_parse_fuzzer
+# Create an empty corpus directory.
+mkdir corpus
+# Run the fuzz target.
+./out/libfuzzer/url_parse_fuzzer corpus
+# If have other corpus directories, pass their paths as well:
+./out/libfuzzer/url_parse_fuzzer corpus seed_corpus_dir_1 seed_corpus_dir_N
 ```
 
 Your fuzz target should produce output like this:
 
 ```
 INFO: Seed: 1511722356
-INFO: Loaded 2 modules   (115485 guards): 22572 [0x7fe8acddf560, 0x7fe8acdf5610), 92913 [0xaa05d0, 0xafb194), 
+INFO: Loaded 2 modules   (115485 guards): 22572 [0x7fe8acddf560, 0x7fe8acdf5610), 92913 [0xaa05d0, 0xafb194),
 INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
 INFO: A corpus is not provided, starting from an empty corpus
 #2  INITED cov: 961 ft: 48 corp: 1/1b exec/s: 0 rss: 48Mb
@@ -147,6 +158,13 @@ if (size < kMinInputLength || size > kMaxInputLength)
   return 0;
 ```
 
+* Generate a [code coverage report]. Note that code coverage of a fuzz target
+depends heavily on the corpus provided when running the target. To generate an
+actionable code coverage report for a new fuzz target, it is recommended to run
+fuzz target built with ASan locally for a little while (several minutes / hours)
+in order to produce some corpus, which then should be used for generating code
+coverage report.
+
 ### Disable noisy error message logging
 
 If the code that you are fuzzing generates lot of error messages when
@@ -165,7 +183,7 @@ struct Environment {
 
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
   static Environment env;
-  
+
   // Put your fuzzing code here and use data+size as input.
   return 0;
 }
@@ -316,3 +334,4 @@ performance and for optimization hints.
 [its own documentation]: http://llvm.org/docs/LibFuzzer.html#output
 [Getting Started with libprotobuf-mutator in Chromium]: libprotobuf-mutator.md
 [base::FuzzedDataProvider]: https://cs.chromium.org/chromium/src/base/test/fuzzed_data_provider.h
+[code coverage report]: efficient_fuzzer.md#Code-Coverage

testing/libfuzzer/reference.md

@@ -31,12 +31,12 @@ running:
 
 | Builder | Description |
 |---------|-------------|
-|Linux ASan | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux ASan' out/Directory` |
-|Linux ASan Debug | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux ASan Debug' out/Directory` |
-|Linux MSan \[[*](#MSan)\] | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux MSan' out/Directory` |
-|Linux UBSan \[[*](#UBSan)\]| `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux UBSan' out/Directory` |
-|Mac ASan | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Mac ASan' out/Directory` |
-|Windows ASan | `tools/mb/mb.py gen -m chromium.fyi -b "Libfuzzer Upload Windows ASan" out/Directory` |
+|Linux ASan | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux ASan' out/Directory` |
+|Linux ASan Debug | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux ASan Debug' out/Directory` |
+|Linux MSan \[[*](#MSan)\] | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux MSan' out/Directory` |
+|Linux UBSan \[[*](#UBSan)\]| `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux UBSan' out/Directory` |
+|Mac ASan | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Mac ASan' out/Directory` |
+|Windows ASan | `python tools\mb\mb.py gen -m chromium.fuzz -b "Libfuzzer Upload Windows ASan" out\Directory` |
 
 
 ### Linux

testing/libfuzzer/reproducing_on_windows.md

@@ -18,7 +18,7 @@ powershell.
 2. Generate gn build configuration for libFuzzer:
 
 ```
-python tools\mb\mb.py gen -m chromium.fyi -b "Libfuzzer Upload Windows ASan" out\libFuzzer
+python tools\mb\mb.py gen -m chromium.fuzz -b "Libfuzzer Upload Windows ASan" out\libFuzzer
 ```
 
 3. Build the fuzzer:

tools/code_coverage/coverage.py

@@ -42,13 +42,14 @@
 
   python tools/code_coverage/coverage.py pdfium_fuzzer \\
       -b out/coverage -o out/report \\
-      -c 'out/coverage/pdfium_fuzzer -runs=<runs> <corpus_dir>' \\
+      -c 'out/coverage/pdfium_fuzzer -runs=0 <corpus_dir>' \\
       -f third_party/pdfium
 
   where:
     <corpus_dir> - directory containing samples files for this format.
-    <runs> - number of times to fuzz target function. Should be 0 when you just
-             want to see the coverage on corpus and don't want to fuzz at all.
+
+  To learn more about generating code coverage reports for fuzz targets, see
+  https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/efficient_fuzzer.md#Code-Coverage
 
   * Sample workflow for running Blink web tests: