Commit cf8139e2 authored by behdad's avatar behdad Committed by Commit Bot

Added docs for representative perf tests

. README added for representative perf tests.
. Update pixel-wrangling instructions to include looking at
  repr_perf_test results.
. Linked the pixel-wrangling doc to the representative perf test's for
  more detail.

Bug: chromium:1000837
Change-Id: Iefe03f7c722ccc9c8d72cf3e5ada7816e240d1cc
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1937848Reviewed-by: default avatarSadrul Chowdhury <sadrul@chromium.org>
Reviewed-by: default avatarYuly Novikov <ynovikov@chromium.org>
Commit-Queue: Behdad Bakhshinategh <behdadb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#720547}
parent beabaf9d
...@@ -79,6 +79,8 @@ test the code that is actually shipped. As of this writing, the tests included: ...@@ -79,6 +79,8 @@ test the code that is actually shipped. As of this writing, the tests included:
`src/gpu/gles2_conform_support/BUILD.gn` `src/gpu/gles2_conform_support/BUILD.gn`
* `gl_tests`: see `src/gpu/BUILD.gn` * `gl_tests`: see `src/gpu/BUILD.gn`
* `gl_unittests`: see `src/ui/gl/BUILD.gn` * `gl_unittests`: see `src/ui/gl/BUILD.gn`
* `rendering_representative_perf_tests` (on the chromium.gpu.fyi waterfall):
see `src/chrome/test/BUILD.gn`
And more. See And more. See
[`src/testing/buildbot/README.md`](../../testing/buildbot/README.md) [`src/testing/buildbot/README.md`](../../testing/buildbot/README.md)
...@@ -236,8 +238,9 @@ shift, and a calendar appointment. ...@@ -236,8 +238,9 @@ shift, and a calendar appointment.
by Telemetry, rather than a Gtest harness. The tests and their by Telemetry, rather than a Gtest harness. The tests and their
expectations are contained in [src/content/test/gpu/gpu_tests/test_expectations] . See expectations are contained in [src/content/test/gpu/gpu_tests/test_expectations] . See
for example <code>[webgl_conformance_expectations.txt]</code>, for example <code>[webgl_conformance_expectations.txt]</code>,
<code>[gpu_process_expectations.txt]</code> and <code>[gpu_process_expectations.txt]</code>,
<code>[pixel_expectations.txt]</code>. <code>[pixel_expectations.txt]</code> and
[rendering_representative_perf_tests].
1. See the header of the file a list of modifiers to specify a bot 1. See the header of the file a list of modifiers to specify a bot
configuration. It is possible to specify OS (down to a specific configuration. It is possible to specify OS (down to a specific
version, say, Windows 7 or Mountain Lion), GPU vendor version, say, Windows 7 or Mountain Lion), GPU vendor
...@@ -276,6 +279,7 @@ https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7-rel ...@@ -276,6 +279,7 @@ https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7-rel
[pixel_expectations.txt]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations/pixel_expectations.txt [pixel_expectations.txt]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations/pixel_expectations.txt
[stamping out flakiness]: gpu_testing.md#Stamping-out-Flakiness [stamping out flakiness]: gpu_testing.md#Stamping-out-Flakiness
[gtest-DISABLED]: https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#temporarily-disabling-tests [gtest-DISABLED]: https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#temporarily-disabling-tests
[rendering_representative_perf_tests]: ../testing/rendering_representative_perf_tests.md#Updating-Expectations
### When Bots Misbehave (SSHing into a bot) ### When Bots Misbehave (SSHing into a bot)
......
# Representative Performance Tests for Rendering Benchmark
`rendering_representative_perf_tests` runs a sub set of stories from rendering
benchmark on CQ, to prevent performance regressions. For each platform there is
a `story_tag` which describes the representative stories used in this test.
These stories will be tested using the [`run_benchmark`](../../tools/perf/run_benchmark) script. Then the recorded values for `frame_times` will be
compared with the historical upper limit described in [`src/testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json`](../../testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json).
[TOC]
## Clustering the Benchmark and Choosing Representatives
The clustering of the benchmark is based on the historical values recorded for
`frame_times`. For steps on clustering the benchmark check [Clustering benchmark stories](../../tools/perf/experimental/story_clustering/README.md).
Currently there are three sets of representatives described by story tags below:
* `representative_mac_desktop`
* `representative_mobile`
* `representative_win_desktop`
Adding more stories to representatives or removing stories from the set is
managed by adding and removing story tags above to stories in [rendering benchmark](../../tools/perf/page_sets/rendering).
## Updating the Upper Limits
The upper limits for averages and confidence interval (CI) ranges of
`frame_times` described in [`src/testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json`](../../testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json)
are used to passing or failing a test. These values are the 95 percentile of
the past 30 runs of the test on each platform (for both average and CI).
This helps with catching sudden regressions which results in a value higher
than the upper limits. But in case of gradual regressions, the upper limits
may not be useful in not updated frequently. Updating these upper limits also
helps with adopting to improvements.
Updating these values can be done by running [`src/tools/perf/experimental/representative_perf_test_limit_adjuster/adjust_upper_limits.py`](../../tools/perf/experimental/representative_perf_test_limit_adjuster/adjust_upper_limits.py)and committing the changes.
The script will create a new JSON file using the values of recent runs in place
of [`src/testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json`](../../testing/scripts/representative_perf_test_data/representatives_frame_times_upper_limit.json).
## Updating Expectations
To skip any of the tests, update
[`src/tools/perf/expectations.config`](../../tools/perf/expectations.config) and
add the story under rendering benchmark.
\ No newline at end of file
The code is this directory provides support for clustering and choosing # Clustering Benchmark Stories
The code in this directory provides support for clustering and choosing
representatives for benchmarks. representatives for benchmarks.
Input needed for the clustering methods are: Input needed for the clustering methods are:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment