Commit 4c3160e0 authored by Ryan Heise's avatar Ryan Heise Committed by Commit Bot

Improve documentation in tools/perf/README.md

* Restructure document to be a simple overview + functional information (i.e. how do I X).
* Remove discussion of implementation details (WPR, TBM, etc.)
  that are covered in child documents.
* Add discussion on how to create a test.

Change-Id: Id60882846b15aa557620d23516a1584f2ff62cea
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2450715
Commit-Queue: Ryan Heise <heiserya@google.com>
Auto-Submit: Ryan Heise <heiserya@google.com>
Reviewed-by: default avatarJohn Chen <johnchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#814019}
parent 86d2bfa6
<!-- Copyright 2019 The Chromium Authors. All rights reserved.
<!-- Copyright 2020 The Chromium Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.
-->
# Chrome Benchmarking System
# Overview
This directory contains benchmarks and infrastructure to test Chrome and
Chromium and output performance measurements. The benchmarks are run on the
[perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console). Most
of the benchmarks are run using
[Telemetry](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/README.md),
which is a framework for driving Chrome through test cases ("stories").
Telemetry often makes use of
[Web Page Replay (WPR)](https://chromium.googlesource.com/catapult.git/+/HEAD/web_page_replay_go/README.md)
to save a version of a website so that we can measure performance without fear
of the website changing underneath us.
For doing measurements, Telemetry usually makes use of an event tracing system
built into Chromium. Event data is tracked as Chromium runs. After Chromium
finishes running that event data is processed by either
[TBMv2](https://chromium.googlesource.com/catapult.git/+/HEAD/tracing/tracing/metrics)
(which is the current system) or
[TBMv3](https://chromium.googlesource.com/chromium/src/+/HEAD/tools/perf/core/tbmv3)
(which is an experimental new system) to create measurements.
Those measurements are uploaded to the
[ChromePerf Dashboard](https://chromeperf.appspot.com/) which charts
measurements over time and alerts on regressions. Regressions can be bisected
using [Pinpoint](https://pinpoint-dot-chromeperf.appspot.com/) to figure out
which Chromium change caused them.
Pinpoint can also be used to run try jobs against machines in our data centers.
Information about available platforms for testing is available to Googlers at
[Chrome Benchmarking Sheet](https://goto.google.com/chrome-benchmarking-sheet).
Please also read the [Chrome Speed][speed] documentation to learn more about the
team organization and, in particular, the top level view of
[How Chrome Measures Performance][chrome_perf_how].
# Performance tools
Chromium and output performance measurements. These benchmarks are continuously
run on the [perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console).
For more information on how Chrome measures performance, see
[here](/docs/speed/how_does_chrome_measure_performance.md).
# Using The Chrome Benchmarking System
## Analyzing Results From The Perf Waterfall
The [ChromePerf Dashboard](https://chromeperf.appspot.com/) is the destination
for all metrics generated by the perf waterfall. It provides tools to set up a
dashboard for performance of a set of tests + metrics over time. In addition, it
provides the ability to launch a bisection by selecting a point on the
dashboard.
## Running A Single Test
The Chrome Benchmarking System has two methods for manually running performance tests:
run_benchmark and Pinpoint.
run_benchmark is useful for creating and debugging benchmarks using local
devices. Run from the command line, it has a number of flags useful for
determining the internal state of the benchmark. For more information, see
[here](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry/docs/run_benchmarks_locally.md).
[Pinpoint](https://pinpoint-dot-chromeperf.appspot.com/) wraps run_benchmark and
provides the ability to remotely run A/B benchmarks using any platform available
in our lab. It will run a benchmark for as many iterations as needed to get a
statistically significant result, then visualize it.
## Creating New Tests (stories)
[This document](https://chromium.googlesource.com/catapult.git/+/HEAD/telemetry)
provides an oveview of how tests are structured and some of the underlying
technologies. After reading that doc, figure out if your story fits into an
existing benchmark by checking
[here](https://goto.google.com/chrome-benchmarking-sheet) (or
[here](https://bit.ly/chrome-benchmarks) for non-Googlers).
* If it does, follow the instructions next to it. If there are no instructions,
find the test type in src/tools/perf/page_sets.
* Otherwise, read [this](https://docs.google.com/document/d/1ni2MIeVnlH4bTj4yvEDMVNxgL73PqK_O9_NUm3NW3BA/edit).
After figuring out where your story fits, create a new one. There is a
considerable amount of variation between different benchmarks, so use a nearby
story as a model. You may also need to introduce custom JavaScript to drive
interactions on the page or to deal with nondeterminsim. For an example, search
[this file](https://source.chromium.org/chromium/chromium/src/+/master:tools/perf/page_sets/system_health/browsing_stories.py?q=browsing_stories.py&ss=chromium)
for browse:tools:sheets:2019.
Next, we need to use WPR (WebPageReplay) to record all of the content requested by the test. By default,
tests spin up a local webserver using these recordings, removing one source of
nondeterminism. To do that, run:
```./tools/perf/record_wpr --browser=system --story-filter=STORY_NAME BENCHMARK_NAME```
Next, we need to verify the recording works. To do so, run the test:
```./tools/perf/run_benchmark run BENCHMARK_NAME --browser=system --story-filter=STORY_NAME ```
After running this, you will need to verify the following:
* Does the browser behave the same as it did when creating the recording? If not, is the difference in behavior acceptable?
* Are there any concerning errors generated by Chrome when running run_benchmark? These will appear in the output of run_benchmark.
* Check the benchmarks in the link generated by run_benchmark. Does everything look reasonable?
If any problems were encountered, review or add custom JavaScript as described in the previous section. Alternatively, ask for help.
If everything looks good, upload your WPR archive by following the instructions
in [Upload the recording to Cloud Storage](https://sites.google.com/a/chromium.org/dev/developers/telemetry/record_a_page_set)
and create a CL.
# Tools In This Directory
This directory contains a variety of tools that can be used to run benchmarks,
interact with speed services, and manage performance waterfall configurations.
It also has commands for running functional unittests.
[speed]: /docs/speed/README.md
[chrome_perf_how]: /docs/speed/how_does_chrome_measure_performance.md
## run_tests
This command allows you to run functional tests against the python code in this
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment