Skip to content

GitLab

T
tangled
Commit c4650ad0 authored by Brian Sheedy's avatar Brian Sheedy Committed by Commit Bot
Browse files
Options

Merge GPU pixel documentation 

Merges various parts of the GPU Gold documentation from
gpu_pixel_testing_with_gold.md into the other parts of the GPU
documentation. Also removes references to the legacy pixel tests and
their triage process since the code for them has been removed.

Bug: 985007
Change-Id: I19d5480de77f1066046ef8fe0f076264cff01ee7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1721523
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Auto-Submit: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: default avatarKenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#681868}
parent 0cc71129
Show whitespace changes
Inline Side-by-side
Showing with 105 additions and 207 deletions
docs/gpu/gpu_pixel_testing_with_gold.md
View file @ c4650ad0
# GPU Pixel Testing With Gold
This page describes in detail the GPU pixel tests that utilize Skia Gold for
storing and comparing images (the `pixel_skia_gold_test` type). This includes
information such as how Skia Gold works, how to triage images produced by these
tests, and working on Gold itself.
This page describes various extra details of the Skia Gold service
that the GPU pixel tests use. For information on running the tests locally, see
[this section][local pixel testing]. For common information on triaging,
modification, or general pixel wrangling, see [GPU Pixel Wrangling] or these
sections ([1][pixel debugging], [2][pixel updating]) of the general GPU testing
documentation.
[local pixel testing]: gpu_testing.md#Running-the-pixel-tests-locally
[GPU Pixel Wrangling]: pixel_wrangling.md
[pixel debugging]: gpu_testing.md#Debugging-Pixel-Test-Failures-on-the-GPU-Bots
[pixel updating]: gpu_testing.md#Updating-and-Adding-New-Pixel-Tests-to-the-GPU-Bots
[TOC]
......@@ -111,45 +118,13 @@ the CL was merged as. In the above example, if the CL was merged as commit
`ffff`, then both `abcd` and `abef` would be approved images on master from
`ffff` onward.
## Triaging Failures
### Standard Workflow
The current approach to getting the triage links for new images is:
1. Navigate to the failed build.
2. Search for the failed step for the `pixel_skia_gold_test` test.
3. Look for either:
1. One or more links named `gold_triage_link for <test name>`. This will be
the case if there are fewer than 10 links.
2. A single link named
`Too many artifacts produced to link individually, click for links`. This
will be the case if there are 10 or more links.
4. In either case, follow the link(s) to the triage page for the image the
failing test produced.
5. Triage images on those pages (typically by approving them, but you can also
mark them as a negative if it is an image that should not be produced). In the
case of a negative image, a bug should be filed on [crbug] to investigate and
fix the cause of the that particular image being produced, as future
occurrences of it will cause the test to fail. Such bugs should include the
`Internals>GPU>Testing` component and whatever component is suitable for the
type of failing test (likely `Blink>WebGL` or `Blink>Canvas`). The test should
also be marked as failing or skipped in
`//content/test/gpu/gpu_tests/test_expectations/pixel_expectations.txt` so that
the test failure doesn't show up as a builder failure. If the failure is
consistent, prefer to skip instead of mark as failing so that the failure links
don't pile up. If the failure occurs on the trybots, include the change to the
expectations in your CL.
[crbug]: https://crbug.com
In the future, a comment will also be automatically posted to CLs that produce
new images with a link to triage them.
In addition, you can see all currently untriaged images that are currently
being produced on ToT on the [GPU Gold instance's main page][gpu gold instance]
and currently untriaged images for a CL by substituting the Gerrit CL number
into
## Triaging Less Common Failures
### Triaging Images Without A Specific Build
You can see all currently untriaged images that are currently being produced on
ToT on the [GPU Gold instance's main page][gpu gold instance] and currently
untriaged images for a CL by substituting the Gerrit CL number into
`https://chrome-gpu-gold.skia.org/search?issue=[CL Number]&unt=true&master=true`.
[gpu gold instance]: https://chrome-gpu-gold.skia.org
......@@ -197,38 +172,6 @@ the triage link for the problematic image.
[skia crbug]: https://bugs.chromium.org/p/skia
## Running Locally
Normally, `goldctl` uploads images and image metadata to the Gold server when
used. This is not desirable when running locally for a couple reasons:
1. Uploading requires the user to be whitelisted on the server, and whitelisting
everyone who wants to run the tests locally is not a viable solution.
2. Images produced during local runs are usually slightly different from those
that are produced on the bots due to hardware/software differences. Thus, most
images uploaded to Gold from local runs would likely only ever actually be used
by tests run on the machine that initially generated those images, which just
adds noise to the list of approved images.
Additionally, the tests normally rely on the Gold server for viewing images
produced by a test run. This does not work if the data is not actually uploaded.
In order to get around both of these issues, simply pass the `--local-run` flag
to the tests. This will disable uploading, but otherwise go through the same
steps as a test normally would. Each test will also print out a `file://` URL to
the image it produces and a link to all approved images for that test in Gold.
Because the image produced by the test locally is likely slightly different from
any of the approved images in Gold, local test runs are likely to fail during
the comparison step. In order to cut down on the amount of noise, you can also
pass the `--no-skia-gold-failure` flag to not fail the test on a failed image
comparison. When using `--no-skia-gold-failure`, you'll also need to pass the
`--passthrough` flag in order to actually see the link output.
Example usage:
`run_gpu_integration_test.py pixel --no-skia-gold-failure --local-run
--passthrough --build-revision aabbccdd --test-machine-name local`
## Working On Gold
### Modifying Gold And goldctl
......
docs/gpu/gpu_testing.md
View file @ c4650ad0
......@@ -189,12 +189,15 @@ argument. The tests this script can invoke are
in `src/content/test/gpu/gpu_tests/`. For example:
* `run_gpu_integration_test.py context_lost --browser=release`
* `run_gpu_integration_test.py pixel --browser=release`
* `run_gpu_integration_test.py webgl_conformance --browser=release --webgl-conformance-version=1.0.2`
* `run_gpu_integration_test.py maps --browser=release`
* `run_gpu_integration_test.py screenshot_sync --browser=release`
* `run_gpu_integration_test.py trace_test --browser=release`
The pixel tests are a bit special. See
[the section on running them locally](#Running-the-pixel-tests-locally) for
details.
If you're testing on Android and have built and deployed
`ChromePublic.apk` to the device, use `--browser=android-chromium` to
invoke it.
......@@ -246,49 +249,50 @@ for documentation on setting this up.
### Running the pixel tests locally
The pixel tests run in a few different modes:
* The waterfall bots generate reference images into cloud storage, and pass
the `--upload-refimg-to-cloud-storage` command line argument.
* The trybots use the reference images that were generated by the waterfall
bots. They pass the `--download-refimg-from-cloud-storage` command line
argument, as well as other needed ones like `--refimg-cloud-storage-bucket`
and `--os-type`.
* When run locally, the first time the pixel tests are run, generated
*reference* images are placed into
`src/content/test/data/gpu/gpu_reference/`. The second and subsequent times,
if tests fail, failure images will be placed into
`src/content/test/data/gpu/generated`.
The pixel tests are a special case because they use an external Skia service
called Gold to handle image approval and storage. See
[GPU Pixel Testing With Gold] for specifics.
It's possible to make your local pixel tests download the reference images from
cloud storage, if your workstation has the same OS and GPU type as one of the
bots on the waterfall, and you pass the `--download-refimg-from-cloud-storage`,
`--refimg-cloud-storage-bucket`, `--os-type` and `--build-revision` command line
arguments.
[GPU Pixel Testing With Gold]: gpu_pixel_testing_with_gold.md
Example command line for running the pixel tests locally on a desktop
platform, where the Chrome build is in out/Release:
TL;DR is that the pixel tests use a binary called `goldctl` to download and
upload data when running pixel tests.
* `run_gpu_integration_test.py pixel --browser=release`
Normally, `goldctl` uploads images and image metadata to the Gold server when
used. This is not desirable when running locally for a couple reasons:
Running against a connected Android device where ChromePublic.apk has
already been deployed:
1. Uploading requires the user to be whitelisted on the server, and whitelisting
everyone who wants to run the tests locally is not a viable solution.
2. Images produced during local runs are usually slightly different from those
that are produced on the bots due to hardware/software differences. Thus, most
images uploaded to Gold from local runs would likely only ever actually be used
by tests run on the machine that initially generated those images, which just
adds noise to the list of approved images.
* `run_gpu_integration_test.py pixel --browser=android-chromium`
Additionally, the tests normally rely on the Gold server for viewing images
produced by a test run. This does not work if the data is not actually uploaded.
You can run a subset of the pixel tests via the --test-filter argument, which
takes a regex:
In order to get around both of these issues, simply pass the `--local-run` flag
to the tests. This will disable uploading, but otherwise go through the same
steps as a test normally would. Each test will also print out a `file://` URL to
the image it produces and a link to all approved images for that test in Gold.
* `run_gpu_integration_test.py pixel --browser=release --test-filter=Pixel_WebGL`
* `run_gpu_integration_test.py pixel --browser=release --test-filter=\(Pixel_WebGL2\|Pixel_GpuRasterization_BlueBox\)`
Because the image produced by the test locally is likely slightly different from
any of the approved images in Gold, local test runs are likely to fail during
the comparison step. In order to cut down on the amount of noise, you can also
pass the `--no-skia-gold-failure` flag to not fail the test on a failed image
comparison. When using `--no-skia-gold-failure`, you'll also need to pass the
`--passthrough` flag in order to actually see the link output.
More complete example command line for Android:
Example usage:
`run_gpu_integration_test.py pixel --no-skia-gold-failure --local-run
--passthrough --build-revision aabbccdd`
* `run_gpu_integration_test.py pixel --show-stdout --browser=android-chromium
-v --passthrough --extra-browser-args='--enable-logging=stderr
--js-flags=--expose-gc' --refimg-cloud-storage-bucket
chromium-gpu-archive/reference-images --os-type android
--download-refimg-from-cloud-storage`
Note that `aabbccdd` must be replaced with an actual Chromium src revision
(typically whatever revision origin/master is currently synced to) in order for
the tests to work. This can be done automatically using:
``run_gpu_integration_test.py pixel --no-skia-gold-failure --local-run
--passthrough --build-revision `git rev-parse origin/master` ``
## Running Binaries from the Bots Locally
......@@ -495,83 +499,32 @@ break the FYI bots.
## Debugging Pixel Test Failures on the GPU Bots
If pixel tests fail on the bots, the stdout will contain text like:
If pixel tests fail on the bots, the build step will contain either one or more
links titled `gold_triage_link for <test name>` or a single link titled
`Too many artifacts produced to link individually, click for links`, which
itself will contain links. In either case, these links will direct to Gold
pages showing the image produced by the image and the approved image that most
closely matches it.
`See http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?[HASH]`
Note that for the tests which programatically check colors in certain regions of
the image (tests with `expected_colors` fields in [pixel_test_pages]), there
likely won't be a closest approved image since those tests only upload data to
Gold in the event of a failure.
This link contains all of the failing tests' generated and reference
images, and is useful for figuring out exactly what went wrong. [Issue
898649](http://crbug.com/898649) tracks improving this user interface,
so that the failures can be surfaced directly in the build logs rather
than having to dig through stdout.
[pixel_test_pages]: https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/pixel_test_pages.py
## Updating and Adding New Pixel Tests to the GPU Bots
Adding new pixel tests which require reference images is a slightly more
complex process than adding other kinds of tests which can validate their own
correctness. There are a few reasons for this.
* Reference image based pixel tests require different golden images for
different combinations of operating system, GPU, driver version, OS
version, and occasionally other variables.
* The reference images must be generated by the main waterfall. The try
servers are not allowed to produce new reference images, only consume them.
The reason for this is that a patch sent to the try servers might cause an
incorrect reference image to be generated. For this reason, the main
waterfall bots upload reference images to cloud storage, and the try
servers download them and verify their results against them.
* The try servers will fail if they run a pixel test requiring a reference
image that doesn't exist in cloud storage. This is deliberate, but needs
more thought; see [Issue 349262](http://crbug.com/349262).
If a reference image based pixel test's result is going to change because of a
change in a third party repository (e.g. in ANGLE), updating the reference
images is a slightly tricky process. Here's how to do it:
* Mark the pixel test as failing **without platform condition** in the
[pixel test]'s [test expectations]
* Commit the change to the third party repository, etc. which will change the
test's results
* Note that without the failure expectation, this commit would turn some bots
red, e.g. an ANGLE change will turn the chromium.gpu.fyi bots red
* Wait for the third party repository to roll into Chromium
* Commit a change incrementing the revision number associated with the test
in the [test pages]
* Commit a second change removing the failure expectation, once all of the
bots on the main waterfall have generated new reference images. This change
should go through the commit queue cleanly.
When adding a brand new pixel test that uses a reference image, the steps are
similar, but simpler:
* In the same commit which introduces the new test, mark the pixel test as
failing **without platform condition** in the [pixel test]'s [test
expectations]
* Wait for the reference images to be produced by all of the GPU bots on the
waterfalls (see [chromium-gpu-archive/reference-images])
* Commit a change un-marking the test as failing
When making a Chromium-side (including Blink which is now in the same Chromium
repository) change which changes the pixel tests' results:
* In your CL, both mark the pixel test as failing **without platform
condition** in the [pixel test]'s [test expectations] and increment the
test's version number associated with the test in the [test pages]
* After your CL lands, land another CL removing the failure expectations. If
this second CL goes through the commit queue cleanly, you know reference
images were generated properly.
In general, when adding a new pixel test, it's better to spot check a few
pixels in the rendered image rather than using a reference image per platform.
The [GPU rasterization test] is a good example of a recently added test which
performs such spot checks.
[pixel test]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/pixel_test_pages.py
[test expectations]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations/pixel_expectations.txt
[test pages]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/pixel_test_pages.py
[cloud storage bucket]: https://console.developers.google.com/storage/chromium-gpu-archive/reference-images
<!-- XXX: old link -->
[GPU rasterization test]: http://src.chromium.org/viewvc/chrome/trunk/src/content/test/gpu/gpu_tests/gpu_rasterization.py
If your CL adds a new pixel test or modifies existing ones, it's likely that
you will have to approve new images. Simply run your CL through the CQ and
follow the steps outline [here][pixel wrangling triage] under the "Check if any
pixel test failures are actual failures or need to be rebaselined." step.
[pixel wrangling triage]: pixel_wrangling.md#How-to-Keep-the-Bots-Green
Once your CL passes the CQ, you should be mostly good to go, although you should
keep an eye on the waterfall bots for a short period after your CL lands in case
any configurations not covered by the CQ need to have images approved, as well.
## Stamping out Flakiness
......
docs/gpu/pixel_wrangling.md
View file @ c4650ad0
......@@ -10,11 +10,6 @@ majority of the bots on the Chromium waterfall.
[slides]: https://docs.google.com/presentation/d/1sZjyNe2apUhwr5sinRfPs7eTzH-3zO0VQ-Cj-8DlEDQ/edit?usp=sharing
For information on pixel wrangling the Skia Gold version of the pixel tests
(`pixel_skia_gold_tests` test suite), see [this documentation](gold_doc).
[gold_doc]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/gpu/gpu_pixel_testing_with_gold.md
[TOC]
## Fleet Status
......@@ -205,24 +200,32 @@ shift, and a calendar appointment.
main waterfalls.
1. Check if any pixel test failures are actual failures or need to be
rebaselined.
1. For a given build failing the pixel tests, click the "stdio" link of
the "pixel" step.
1. The output will contain a link of the form
<http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?242523_Linux_Release_Intel__telemetry>
* For pixel_skia_gold_tests the output link will be of the form
<https://chrome-gpu-gold.skia.org/detail?test=Pixel_BackgroundImage&digest=13c038c8d426720ae575f4cb9f0bf8da>
1. Visit the link to see whether the generated or reference images look
incorrect.
1. All of the reference images for all of the bots are stored in cloud
storage under [chromium-gpu-archive/reference-images]. They are indexed
by version number, OS, GPU vendor, GPU device, and whether or not
antialiasing is enabled in that configuration. You can download the
reference images individually to examine them in detail.
1. Rebaseline pixel test reference images if necessary.
1. Follow the [instructions on the GPU testing page].
1. Alternatively, if absolutely necessary, you can use the [Chrome
Internal GPU Pixel Wrangling Instructions] to delete just the broken
reference images for a particular configuration.
1. For a given build failing the pixel tests, look for either:
1. One or more links named `gold_triage_link for <test name>`. This will
be the case if there are fewer than 10 links.
1. A single link named
`Too many artifacts produced to link individually, click for links`.
This will be the case if there are 10 or more links.
1. In either case, follow the link(s) to the triage page for the image the
failing test produced.
1. Ensure you are signed in to the Gold server the links take you to (both
@google.com and @chromium.org accounts work).
1. Triage images on those pages (typically by approving them, but you can
mark them as negative if it is an image that should not be produced). In
the case of a negative image, a bug should be filed on
[crbug](https://crbug.com) to investigate and fix the cause of that
particular image being produced, as future occurrences of it will cause
the test to fail. Such bugs should include the `Internals>GPU>Testing`
component and whatever component is suitable for the type of failing
test (likely `Blink>WebGL` or `Blink>Canvas`). The test should also be
marked as failing or skipped(see the item below on updating the
Telemetry-based test expectations) so that the test failure doesn't show
up as a builder failure. If the failure is consistent, prefer to skip
instead of mark as failing so that the failure links don't pile up. If
the failure occurs on the trybots, include the change to the
expectations in your CL.
1. Additional, less common triage steps for the pixel tests can be found in
[this section][gold less common failures] of the GPU Gold documentation.
1. Update Telemetry-based test expectations if necessary.
1. Most of the GPU tests are run inside a full Chromium browser, launched
by Telemetry, rather than a Gtest harness. The tests and their
......@@ -260,8 +263,7 @@ https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7-rel
[Chromium Try Flakes]: http://chromium-try-flakes.appspot.com/
<!-- TODO(kainino): link doesn't work, but is still included from chromium-swarm homepage so not removing it now -->
[Swarming Server Stats]: https://chromium-swarm.appspot.com/stats
[chromium-gpu-archive/reference-images]: https://console.developers.google.com/storage/chromium-gpu-archive/reference-images
[instructions on the GPU testing page]: https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/gpu_testing.md
[gold less common failures]: gpu_pixel_testing_with_gold.md#Triaging-Less-Common-Failures
[Chrome Internal GPU Pixel Wrangling Instructions]: https://sites.google.com/a/google.com/client3d/documents/chrome-internal-gpu-pixel-wrangling-instructions
[src/content/test/gpu/gpu_tests/test_expectations]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations
[webgl_conformance_expectations.txt]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations/webgl_conformance_expectations.txt
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment