Skip to content

GitLab

T
tangled
Commit 13c2318c authored by Max Moroz's avatar Max Moroz Committed by Commit Bot
Browse files
Options

Clarify libFuzzer and Code Coverage docs plus other minor fixes. 

Bug: 539572
Change-Id: I83ae6f7e74b29be928bd81d50704e59ec864394c
Reviewed-on: https://chromium-review.googlesource.com/c/1340600
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Reviewed-by: default avatarAbhishek Arya <inferno@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609045}
parent e337fdb8
Show whitespace changes
Inline Side-by-side
Showing with 57 additions and 37 deletions
testing/libfuzzer/README.md
View file @ 13c2318c
...@@ -4,8 +4,8 @@ ...@@ -4,8 +4,8 @@
*** aside *** aside
[Getting Started](getting_started.md) [Getting Started](getting_started.md)
| [Buildbot](https://build.chromium.org/p/chromium.fyi/buildslaves/slave43-c1) | [Buildbot]
| [ClusterFuzz Status](https://clusterfuzz.com/fuzzer-stats) | [ClusterFuzz Stats]
| [Cover Bug] | [Cover Bug]
*** ***
...@@ -55,8 +55,7 @@ libFuzzer. ...@@ -55,8 +55,7 @@ libFuzzer.
## Project Links ## Project Links
* [libFuzzer Infrastructure Bugs] * [libFuzzer Infrastructure Bugs]
[libFuzzer]: http://llvm.org/docs/LibFuzzer.html [Buildbot]: https://ci.chromium.org/p/chromium/g/chromium.fuzz/builders
[crbug.com/539572]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572
[Cover Bug]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572 [Cover Bug]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572
[Getting Started Guide]: getting_started.md [Getting Started Guide]: getting_started.md
[Efficient Fuzzer Guide]: efficient_fuzzer.md [Efficient Fuzzer Guide]: efficient_fuzzer.md
...@@ -66,8 +65,11 @@ libFuzzer. ...@@ -66,8 +65,11 @@ libFuzzer.
[Reproducing on Linux, Mac, and Android]: https://github.com/google/clusterfuzz-tools [Reproducing on Linux, Mac, and Android]: https://github.com/google/clusterfuzz-tools
[Reference]: reference.md [Reference]: reference.md
[ClusterFuzz Bugs]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label:Stability-LibFuzzer%20label:ClusterFuzz&sort=-modified&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified [ClusterFuzz Bugs]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label:Stability-LibFuzzer%20label:ClusterFuzz&sort=-modified&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified
[ClusterFuzz Stats]: https://clusterfuzz.com/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan
[Pdfium Bugs]: https://bugs.chromium.org/p/pdfium/issues/list?can=1&q=libfuzzer&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles [Pdfium Bugs]: https://bugs.chromium.org/p/pdfium/issues/list?can=1&q=libfuzzer&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles
[Manual Bugs]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3AStability-LibFuzzer+-label%3AClusterFuzz&sort=-modified&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids [Manual Bugs]: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=label%3AStability-LibFuzzer+-label%3AClusterFuzz&sort=-modified&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids
[OSS Trophies]: http://llvm.org/docs/LibFuzzer.html#trophies [OSS Trophies]: http://llvm.org/docs/LibFuzzer.html#trophies
[Guided in-process fuzzing of Chrome components]: https://security.googleblog.com/2016/08/guided-in-process-fuzzing-of-chrome.html [Guided in-process fuzzing of Chrome components]: https://security.googleblog.com/2016/08/guided-in-process-fuzzing-of-chrome.html
[crbug.com/539572]: https://bugs.chromium.org/p/chromium/issues/detail?id=539572
[libFuzzer]: http://llvm.org/docs/LibFuzzer.html
[libFuzzer Infrastructure Bugs]: https://bugs.chromium.org/p/chromium/issues/list?q=label:LibFuzzer-Infra [libFuzzer Infrastructure Bugs]: https://bugs.chromium.org/p/chromium/issues/list?q=label:LibFuzzer-Infra
testing/libfuzzer/clusterfuzz.md
View file @ 13c2318c
...@@ -32,23 +32,19 @@ provides links to crashes and coverage reports. ...@@ -32,23 +32,19 @@ provides links to crashes and coverage reports.
## Corpus ## Corpus
ClusterFuzz uses two corpus types with libFuzzer:
* **Seed** (or **static**) corpus: files manually uploaded by developers. Chromium developers can access the corpus stored in the [Corpus GCS Bucket] via
ClusterFuzz uses these files for fuzzing but doesn't delete/overwrite them. web interface or by using `gsutil` tool (the latter is easier for downloading):
* **General** (or **working**) corpus: files generated by fuzzers themselves. ```bash
These corpus files are frequently modified during fuzzing sessions and can be mkdir local_corpus_dir
deleted during corpus minimization. gsutil -m cp -r gs://clusterfuzz-corpus/libfuzzer/<fuzz_target> local_corpus_dir
```
A fuzz target has two input corpus directories, seed and general, but its output [Buildbot]: https://ci.chromium.org/p/chromium/g/chromium.fuzz/builders
goes into general corpus directory. Seed corpus is read-only.
[Buildbot]: https://build.chromium.org/p/chromium.fyi/buildslaves/slave43-c1
[Code Coverage]: https://chromium-coverage.appspot.com/reports/latest_fuzzers_only/linux/index.html [Code Coverage]: https://chromium-coverage.appspot.com/reports/latest_fuzzers_only/linux/index.html
[chromium_libfuzzer.py]: https://code.google.com/p/chromium/codesearch#chromium/build/scripts/slave/recipes/chromium_libfuzzer.py [chromium_libfuzzer.py]: https://code.google.com/p/chromium/codesearch#chromium/build/scripts/slave/recipes/chromium_libfuzzer.py
[ClusterFuzz Fuzzer Stats]: https://clusterfuzz.com/v2/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan [ClusterFuzz Fuzzer Stats]: https://clusterfuzz.com/fuzzer-stats/by-fuzzer/fuzzer/libFuzzer/job/libfuzzer_chrome_asan
[ClusterFuzz libFuzzer Logs]: https://console.cloud.google.com/storage/browser/clusterfuzz-libfuzzer-logs [ClusterFuzz libFuzzer Logs]: https://console.cloud.google.com/storage/browser/clusterfuzz-libfuzzer-logs
[Corpus GCS Bucket]: https://console.cloud.google.com/storage/clusterfuzz-corpus/libfuzzer [Corpus GCS Bucket]: https://console.cloud.google.com/storage/clusterfuzz-corpus/libfuzzer
[fuzzer_test.gni]: https://code.google.com/p/chromium/codesearch#chromium/src/testing/libfuzzer/fuzzer_test.gni [fuzzer_test.gni]: https://code.google.com/p/chromium/codesearch#chromium/src/testing/libfuzzer/fuzzer_test.gni
testing/libfuzzer/efficient_fuzzer.md
View file @ 13c2318c
...@@ -83,7 +83,8 @@ script provides detailed instructions as well as a usage example. ...@@ -83,7 +83,8 @@ script provides detailed instructions as well as a usage example.
Note that code coverage of a fuzz target **depends heavily** on the corpus Note that code coverage of a fuzz target **depends heavily** on the corpus
provided when running the target, i.e. code coverage report generated by a fuzz provided when running the target, i.e. code coverage report generated by a fuzz
target launched without any corpus would not make much sense. target launched without any corpus would not make much sense. To download the
corpus from ClusterFuzz, see [ClusterFuzz Corpus].
## Corpus Size ## Corpus Size
...@@ -291,6 +292,7 @@ there is no intended API to disable checksum verification, or when target code ...@@ -291,6 +292,7 @@ there is no intended API to disable checksum verification, or when target code
uses random generator that affects reproducibility of crashes. uses random generator that affects reproducibility of crashes.
[AFL]: http://lcamtuf.coredump.cx/afl/ [AFL]: http://lcamtuf.coredump.cx/afl/
[ClusterFuzz Corpus]: clusterfuzz.md#Corpus
[ClusterFuzz status]: clusterfuzz.md#Status-Links [ClusterFuzz status]: clusterfuzz.md#Status-Links
[Corpus GCS Bucket]: https://console.cloud.google.com/storage/clusterfuzz-corpus/libfuzzer [Corpus GCS Bucket]: https://console.cloud.google.com/storage/clusterfuzz-corpus/libfuzzer
[issue 638836]: https://bugs.chromium.org/p/chromium/issues/detail?id=638836 [issue 638836]: https://bugs.chromium.org/p/chromium/issues/detail?id=638836
......
testing/libfuzzer/getting_started.md
View file @ 13c2318c
...@@ -21,8 +21,13 @@ doesn't require any special configuration and gives meaningful output quickly ...@@ -21,8 +21,13 @@ doesn't require any special configuration and gives meaningful output quickly
for speed, coverage and other parameters. for speed, coverage and other parameters.
```bash ```bash
# With address sanitizer # AddressSanitizer is the default config we recommend testing with.
gn gen out/libfuzzer '--args=use_libfuzzer=true is_asan=true is_debug=false is_component_build=true' --check # Linux:
tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux ASan' out/libfuzzer
# Mac:
tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Mac ASan' out/libfuzzer
# Windows:
python tools\mb\mb.py gen -m chromium.fuzz -b "Libfuzzer Upload Windows ASan" out\libfuzzer
``` ```
Supported sanitizer configurations are: Supported sanitizer configurations are:
...@@ -32,14 +37,11 @@ Supported sanitizer configurations are: ...@@ -32,14 +37,11 @@ Supported sanitizer configurations are:
| `is_asan=true` | Enables [Address Sanitizer] to catch problems like buffer overruns. (only supported sanitizer on Windows and Mac)| | `is_asan=true` | Enables [Address Sanitizer] to catch problems like buffer overruns. (only supported sanitizer on Windows and Mac)|
| `is_msan=true` | Enables [Memory Sanitizer] to catch problems like uninitialized reads<sup>\[[*](reference.md#MSan)\]</sup>. | | `is_msan=true` | Enables [Memory Sanitizer] to catch problems like uninitialized reads<sup>\[[*](reference.md#MSan)\]</sup>. |
| `is_ubsan_security=true` | Enables [Undefined Behavior Sanitizer] to catch<sup>\[[*](reference.md#UBSan)\]</sup> undefined behavior like integer overflow. | | `is_ubsan_security=true` | Enables [Undefined Behavior Sanitizer] to catch<sup>\[[*](reference.md#UBSan)\]</sup> undefined behavior like integer overflow. |
| | It is possible to run libfuzzer without any sanitizers; *probably not what you want*.| | | It is possible to run fuzz targets without any sanitizers; *probably not what you want*.|
Fuzz targets are built with minimal symbols by default. The symbol level Fuzz targets are built with minimal symbols by default. The symbol level
can be adjusted in the usual way by setting `symbol_level`. can be adjusted in the usual way by setting `symbol_level`.
To get the exact GN configuration that are used on our builders, see
[Build Config].
## Write Fuzz Target ## Write Fuzz Target
Create a new `<my_fuzzer>.cc` file and define a `LLVMFuzzerTestOneInput` Create a new `<my_fuzzer>.cc` file and define a `LLVMFuzzerTestOneInput`
...@@ -61,7 +63,10 @@ do not use `testing/libfuzzer/fuzzers` directory, this was a directory used for ...@@ -61,7 +63,10 @@ do not use `testing/libfuzzer/fuzzers` directory, this was a directory used for
initial sample fuzz targets and is no longer recommended for landing new fuzz initial sample fuzz targets and is no longer recommended for landing new fuzz
targets. targets.
[quic_stream_factory_fuzzer.cc] is a good example of real-world fuzz target. Most of the fuzz targets are very small. They may perform one or a few API calls
using the data provided by fuzzing engine as an argument. Other fuzz targets may
be more complex if a certain initialization procedure needs to be performed.
[quic_stream_factory_fuzzer.cc] is a good example of a complex fuzz target.
## Define GN Target ## Define GN Target
...@@ -80,8 +85,14 @@ fuzzer_test("my_fuzzer") { ...@@ -80,8 +85,14 @@ fuzzer_test("my_fuzzer") {
Build with ninja as usual and run: Build with ninja as usual and run:
```bash ```bash
# Build the fuzz target.
ninja -C out/libfuzzer url_parse_fuzzer ninja -C out/libfuzzer url_parse_fuzzer
./out/libfuzzer/url_parse_fuzzer # Create an empty corpus directory.
mkdir corpus
# Run the fuzz target.
./out/libfuzzer/url_parse_fuzzer corpus
# If have other corpus directories, pass their paths as well:
./out/libfuzzer/url_parse_fuzzer corpus seed_corpus_dir_1 seed_corpus_dir_N
``` ```
Your fuzz target should produce output like this: Your fuzz target should produce output like this:
...@@ -147,6 +158,13 @@ if (size < kMinInputLength || size > kMaxInputLength) ...@@ -147,6 +158,13 @@ if (size < kMinInputLength || size > kMaxInputLength)
return 0; return 0;
``` ```
* Generate a [code coverage report]. Note that code coverage of a fuzz target
depends heavily on the corpus provided when running the target. To generate an
actionable code coverage report for a new fuzz target, it is recommended to run
fuzz target built with ASan locally for a little while (several minutes / hours)
in order to produce some corpus, which then should be used for generating code
coverage report.
### Disable noisy error message logging ### Disable noisy error message logging
If the code that you are fuzzing generates lot of error messages when If the code that you are fuzzing generates lot of error messages when
...@@ -316,3 +334,4 @@ performance and for optimization hints. ...@@ -316,3 +334,4 @@ performance and for optimization hints.
[its own documentation]: http://llvm.org/docs/LibFuzzer.html#output [its own documentation]: http://llvm.org/docs/LibFuzzer.html#output
[Getting Started with libprotobuf-mutator in Chromium]: libprotobuf-mutator.md [Getting Started with libprotobuf-mutator in Chromium]: libprotobuf-mutator.md
[base::FuzzedDataProvider]: https://cs.chromium.org/chromium/src/base/test/fuzzed_data_provider.h [base::FuzzedDataProvider]: https://cs.chromium.org/chromium/src/base/test/fuzzed_data_provider.h
[code coverage report]: efficient_fuzzer.md#Code-Coverage
testing/libfuzzer/reference.md
View file @ 13c2318c
...@@ -31,12 +31,12 @@ running: ...@@ -31,12 +31,12 @@ running:
| Builder | Description | | Builder | Description |
|---------|-------------| |---------|-------------|
|Linux ASan | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux ASan' out/Directory` | |Linux ASan | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux ASan' out/Directory` |
|Linux ASan Debug | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux ASan Debug' out/Directory` | |Linux ASan Debug | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux ASan Debug' out/Directory` |
|Linux MSan \[[*](#MSan)\] | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux MSan' out/Directory` | |Linux MSan \[[*](#MSan)\] | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux MSan' out/Directory` |
|Linux UBSan \[[*](#UBSan)\]| `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Linux UBSan' out/Directory` | |Linux UBSan \[[*](#UBSan)\]| `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Linux UBSan' out/Directory` |
|Mac ASan | `tools/mb/mb.py gen -m chromium.fyi -b 'Libfuzzer Upload Mac ASan' out/Directory` | |Mac ASan | `tools/mb/mb.py gen -m chromium.fuzz -b 'Libfuzzer Upload Mac ASan' out/Directory` |
|Windows ASan | `tools/mb/mb.py gen -m chromium.fyi -b "Libfuzzer Upload Windows ASan" out/Directory` | |Windows ASan | `python tools\mb\mb.py gen -m chromium.fuzz -b "Libfuzzer Upload Windows ASan" out\Directory` |
### Linux ### Linux
......
testing/libfuzzer/reproducing_on_windows.md
View file @ 13c2318c
...@@ -18,7 +18,7 @@ powershell. ...@@ -18,7 +18,7 @@ powershell.
2. Generate gn build configuration for libFuzzer: 2. Generate gn build configuration for libFuzzer:
``` ```
python tools\mb\mb.py gen -m chromium.fyi -b "Libfuzzer Upload Windows ASan" out\libFuzzer python tools\mb\mb.py gen -m chromium.fuzz -b "Libfuzzer Upload Windows ASan" out\libFuzzer
``` ```
3. Build the fuzzer: 3. Build the fuzzer:
......
tools/code_coverage/coverage.py
View file @ 13c2318c
...@@ -42,13 +42,14 @@ ...@@ -42,13 +42,14 @@
python tools/code_coverage/coverage.py pdfium_fuzzer \\ python tools/code_coverage/coverage.py pdfium_fuzzer \\
-b out/coverage -o out/report \\ -b out/coverage -o out/report \\
-c 'out/coverage/pdfium_fuzzer -runs=<runs> <corpus_dir>' \\ -c 'out/coverage/pdfium_fuzzer -runs=0 <corpus_dir>' \\
-f third_party/pdfium -f third_party/pdfium
where: where:
<corpus_dir> - directory containing samples files for this format. <corpus_dir> - directory containing samples files for this format.
<runs> - number of times to fuzz target function. Should be 0 when you just
want to see the coverage on corpus and don't want to fuzz at all. To learn more about generating code coverage reports for fuzz targets, see
https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/efficient_fuzzer.md#Code-Coverage
* Sample workflow for running Blink web tests: * Sample workflow for running Blink web tests:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment