Commit c4650ad0 authored by Brian Sheedy's avatar Brian Sheedy Committed by Commit Bot

Merge GPU pixel documentation

Merges various parts of the GPU Gold documentation from
gpu_pixel_testing_with_gold.md into the other parts of the GPU
documentation. Also removes references to the legacy pixel tests and
their triage process since the code for them has been removed.

Bug: 985007
Change-Id: I19d5480de77f1066046ef8fe0f076264cff01ee7
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1721523
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Auto-Submit: Brian Sheedy <bsheedy@chromium.org>
Reviewed-by: default avatarKenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#681868}
parent 0cc71129
# GPU Pixel Testing With Gold # GPU Pixel Testing With Gold
This page describes in detail the GPU pixel tests that utilize Skia Gold for This page describes various extra details of the Skia Gold service
storing and comparing images (the `pixel_skia_gold_test` type). This includes that the GPU pixel tests use. For information on running the tests locally, see
information such as how Skia Gold works, how to triage images produced by these [this section][local pixel testing]. For common information on triaging,
tests, and working on Gold itself. modification, or general pixel wrangling, see [GPU Pixel Wrangling] or these
sections ([1][pixel debugging], [2][pixel updating]) of the general GPU testing
documentation.
[local pixel testing]: gpu_testing.md#Running-the-pixel-tests-locally
[GPU Pixel Wrangling]: pixel_wrangling.md
[pixel debugging]: gpu_testing.md#Debugging-Pixel-Test-Failures-on-the-GPU-Bots
[pixel updating]: gpu_testing.md#Updating-and-Adding-New-Pixel-Tests-to-the-GPU-Bots
[TOC] [TOC]
...@@ -111,45 +118,13 @@ the CL was merged as. In the above example, if the CL was merged as commit ...@@ -111,45 +118,13 @@ the CL was merged as. In the above example, if the CL was merged as commit
`ffff`, then both `abcd` and `abef` would be approved images on master from `ffff`, then both `abcd` and `abef` would be approved images on master from
`ffff` onward. `ffff` onward.
## Triaging Failures ## Triaging Less Common Failures
### Standard Workflow ### Triaging Images Without A Specific Build
The current approach to getting the triage links for new images is: You can see all currently untriaged images that are currently being produced on
ToT on the [GPU Gold instance's main page][gpu gold instance] and currently
1. Navigate to the failed build. untriaged images for a CL by substituting the Gerrit CL number into
2. Search for the failed step for the `pixel_skia_gold_test` test.
3. Look for either:
1. One or more links named `gold_triage_link for <test name>`. This will be
the case if there are fewer than 10 links.
2. A single link named
`Too many artifacts produced to link individually, click for links`. This
will be the case if there are 10 or more links.
4. In either case, follow the link(s) to the triage page for the image the
failing test produced.
5. Triage images on those pages (typically by approving them, but you can also
mark them as a negative if it is an image that should not be produced). In the
case of a negative image, a bug should be filed on [crbug] to investigate and
fix the cause of the that particular image being produced, as future
occurrences of it will cause the test to fail. Such bugs should include the
`Internals>GPU>Testing` component and whatever component is suitable for the
type of failing test (likely `Blink>WebGL` or `Blink>Canvas`). The test should
also be marked as failing or skipped in
`//content/test/gpu/gpu_tests/test_expectations/pixel_expectations.txt` so that
the test failure doesn't show up as a builder failure. If the failure is
consistent, prefer to skip instead of mark as failing so that the failure links
don't pile up. If the failure occurs on the trybots, include the change to the
expectations in your CL.
[crbug]: https://crbug.com
In the future, a comment will also be automatically posted to CLs that produce
new images with a link to triage them.
In addition, you can see all currently untriaged images that are currently
being produced on ToT on the [GPU Gold instance's main page][gpu gold instance]
and currently untriaged images for a CL by substituting the Gerrit CL number
into
`https://chrome-gpu-gold.skia.org/search?issue=[CL Number]&unt=true&master=true`. `https://chrome-gpu-gold.skia.org/search?issue=[CL Number]&unt=true&master=true`.
[gpu gold instance]: https://chrome-gpu-gold.skia.org [gpu gold instance]: https://chrome-gpu-gold.skia.org
...@@ -197,38 +172,6 @@ the triage link for the problematic image. ...@@ -197,38 +172,6 @@ the triage link for the problematic image.
[skia crbug]: https://bugs.chromium.org/p/skia [skia crbug]: https://bugs.chromium.org/p/skia
## Running Locally
Normally, `goldctl` uploads images and image metadata to the Gold server when
used. This is not desirable when running locally for a couple reasons:
1. Uploading requires the user to be whitelisted on the server, and whitelisting
everyone who wants to run the tests locally is not a viable solution.
2. Images produced during local runs are usually slightly different from those
that are produced on the bots due to hardware/software differences. Thus, most
images uploaded to Gold from local runs would likely only ever actually be used
by tests run on the machine that initially generated those images, which just
adds noise to the list of approved images.
Additionally, the tests normally rely on the Gold server for viewing images
produced by a test run. This does not work if the data is not actually uploaded.
In order to get around both of these issues, simply pass the `--local-run` flag
to the tests. This will disable uploading, but otherwise go through the same
steps as a test normally would. Each test will also print out a `file://` URL to
the image it produces and a link to all approved images for that test in Gold.
Because the image produced by the test locally is likely slightly different from
any of the approved images in Gold, local test runs are likely to fail during
the comparison step. In order to cut down on the amount of noise, you can also
pass the `--no-skia-gold-failure` flag to not fail the test on a failed image
comparison. When using `--no-skia-gold-failure`, you'll also need to pass the
`--passthrough` flag in order to actually see the link output.
Example usage:
`run_gpu_integration_test.py pixel --no-skia-gold-failure --local-run
--passthrough --build-revision aabbccdd --test-machine-name local`
## Working On Gold ## Working On Gold
### Modifying Gold And goldctl ### Modifying Gold And goldctl
......
This diff is collapsed.
...@@ -10,11 +10,6 @@ majority of the bots on the Chromium waterfall. ...@@ -10,11 +10,6 @@ majority of the bots on the Chromium waterfall.
[slides]: https://docs.google.com/presentation/d/1sZjyNe2apUhwr5sinRfPs7eTzH-3zO0VQ-Cj-8DlEDQ/edit?usp=sharing [slides]: https://docs.google.com/presentation/d/1sZjyNe2apUhwr5sinRfPs7eTzH-3zO0VQ-Cj-8DlEDQ/edit?usp=sharing
For information on pixel wrangling the Skia Gold version of the pixel tests
(`pixel_skia_gold_tests` test suite), see [this documentation](gold_doc).
[gold_doc]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/gpu/gpu_pixel_testing_with_gold.md
[TOC] [TOC]
## Fleet Status ## Fleet Status
...@@ -205,24 +200,32 @@ shift, and a calendar appointment. ...@@ -205,24 +200,32 @@ shift, and a calendar appointment.
main waterfalls. main waterfalls.
1. Check if any pixel test failures are actual failures or need to be 1. Check if any pixel test failures are actual failures or need to be
rebaselined. rebaselined.
1. For a given build failing the pixel tests, click the "stdio" link of 1. For a given build failing the pixel tests, look for either:
the "pixel" step. 1. One or more links named `gold_triage_link for <test name>`. This will
1. The output will contain a link of the form be the case if there are fewer than 10 links.
<http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?242523_Linux_Release_Intel__telemetry> 1. A single link named
* For pixel_skia_gold_tests the output link will be of the form `Too many artifacts produced to link individually, click for links`.
<https://chrome-gpu-gold.skia.org/detail?test=Pixel_BackgroundImage&digest=13c038c8d426720ae575f4cb9f0bf8da> This will be the case if there are 10 or more links.
1. Visit the link to see whether the generated or reference images look 1. In either case, follow the link(s) to the triage page for the image the
incorrect. failing test produced.
1. All of the reference images for all of the bots are stored in cloud 1. Ensure you are signed in to the Gold server the links take you to (both
storage under [chromium-gpu-archive/reference-images]. They are indexed @google.com and @chromium.org accounts work).
by version number, OS, GPU vendor, GPU device, and whether or not 1. Triage images on those pages (typically by approving them, but you can
antialiasing is enabled in that configuration. You can download the mark them as negative if it is an image that should not be produced). In
reference images individually to examine them in detail. the case of a negative image, a bug should be filed on
1. Rebaseline pixel test reference images if necessary. [crbug](https://crbug.com) to investigate and fix the cause of that
1. Follow the [instructions on the GPU testing page]. particular image being produced, as future occurrences of it will cause
1. Alternatively, if absolutely necessary, you can use the [Chrome the test to fail. Such bugs should include the `Internals>GPU>Testing`
Internal GPU Pixel Wrangling Instructions] to delete just the broken component and whatever component is suitable for the type of failing
reference images for a particular configuration. test (likely `Blink>WebGL` or `Blink>Canvas`). The test should also be
marked as failing or skipped(see the item below on updating the
Telemetry-based test expectations) so that the test failure doesn't show
up as a builder failure. If the failure is consistent, prefer to skip
instead of mark as failing so that the failure links don't pile up. If
the failure occurs on the trybots, include the change to the
expectations in your CL.
1. Additional, less common triage steps for the pixel tests can be found in
[this section][gold less common failures] of the GPU Gold documentation.
1. Update Telemetry-based test expectations if necessary. 1. Update Telemetry-based test expectations if necessary.
1. Most of the GPU tests are run inside a full Chromium browser, launched 1. Most of the GPU tests are run inside a full Chromium browser, launched
by Telemetry, rather than a Gtest harness. The tests and their by Telemetry, rather than a Gtest harness. The tests and their
...@@ -260,8 +263,7 @@ https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7-rel ...@@ -260,8 +263,7 @@ https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win7-rel
[Chromium Try Flakes]: http://chromium-try-flakes.appspot.com/ [Chromium Try Flakes]: http://chromium-try-flakes.appspot.com/
<!-- TODO(kainino): link doesn't work, but is still included from chromium-swarm homepage so not removing it now --> <!-- TODO(kainino): link doesn't work, but is still included from chromium-swarm homepage so not removing it now -->
[Swarming Server Stats]: https://chromium-swarm.appspot.com/stats [Swarming Server Stats]: https://chromium-swarm.appspot.com/stats
[chromium-gpu-archive/reference-images]: https://console.developers.google.com/storage/chromium-gpu-archive/reference-images [gold less common failures]: gpu_pixel_testing_with_gold.md#Triaging-Less-Common-Failures
[instructions on the GPU testing page]: https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/gpu_testing.md
[Chrome Internal GPU Pixel Wrangling Instructions]: https://sites.google.com/a/google.com/client3d/documents/chrome-internal-gpu-pixel-wrangling-instructions [Chrome Internal GPU Pixel Wrangling Instructions]: https://sites.google.com/a/google.com/client3d/documents/chrome-internal-gpu-pixel-wrangling-instructions
[src/content/test/gpu/gpu_tests/test_expectations]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations [src/content/test/gpu/gpu_tests/test_expectations]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations
[webgl_conformance_expectations.txt]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations/webgl_conformance_expectations.txt [webgl_conformance_expectations.txt]: https://chromium.googlesource.com/chromium/src/+/master/content/test/gpu/gpu_tests/test_expectations/webgl_conformance_expectations.txt
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment