Commit b71d507f authored by Stephen McGruer's avatar Stephen McGruer Committed by Commit Bot

Add some documentation on reproducing/debugging flaky web_tests/

This also adds a small section for WPT specifically.

Bug: 1127369, 1127368
Change-Id: Ie74413558fe47bd02f1ae29665327c68bf82ef43
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2508433
Commit-Queue: Stephen McGruer <smcgruer@chromium.org>
Reviewed-by: default avatarNick Burris <nburris@chromium.org>
Reviewed-by: default avatarRobert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#824542}
parent 0aefefcf
# Addressing Flaky WPTs
This document provides tips and tricks for reproducing and debugging flakes in
[Web Platform Tests](web_platform_tests.md) (WPTs). As WPTs are a form of Web
Test, you may also wish to refer to the documentation on [Addressing Flaky Web
Tests](web_tests_addressing_flake.md).
## Debugging flaky WPTs
See also the documentation in [Addressing Flaky Web
Tests](web_tests_addressing_flake.md#Debugging-flaky-Web-Tests).
### Loading WPT tests directly in content\_shell
WPT tests have to be loaded from a server, `wptserve`, to run properly. In a
terminal, run:
```
./third_party/blink/tools/run_blink_wptserve.py
```
This will start the necessary server(s), and print the ports they are listening
on. Most tests can just be loaded over the main HTTP server (usually
`http://localhost:8001`), although some may require using the HTTPS server
instead.
To load a WPT test in content\_shell, run:
```
out/Default/content_shell http://localhost:8001/path/to/test.html
```
Here, the `path/to/test.html` is relative to the root of
`third_party/blink/web_tests/external/wpt`, e.g. `dom/historical.html`.
**Caveat**: As with all Web Tests, running `content_shell` like this is not
equivalent to what `run_web_tests.py` runs. See the same section in [Addressing
Flaky Web
Tests](web_tests_addressing_flake.md#Loading-the-test-directly-in-content_shell)
for more details and some suggestions. In addition to that list, some WPTs
(usually security-related) also expect a real domain and may behave differently
when loaded from localhost.
# Addressing Flaky Web Tests # Addressing Flaky Web Tests
This document provides tips and tricks for reproducing and debugging flakes in
[Web Tests](web_tests.md). If you are debugging a flaky Web Platform Test (WPT),
you may wish to check the specific [Addressing Flaky
WPTs](web_platform_tests_addressing_flake.md) documentation.
This document assumes you are familiar with running Web Tests via
`run_web_tests.py`; if you are not then [see
here](web_tests.md#Running-Web-Tests).
[TOC]
## Understanding builder results ## Understanding builder results
Often (e.g. by Flake Portal), you will be pointed to a particular build in which Often (e.g. by Flake Portal), you will be pointed to a particular build in which
...@@ -56,11 +67,57 @@ highlighted. ...@@ -56,11 +67,57 @@ highlighted.
## Reproducing Web Test flakes ## Reproducing Web Test flakes
>TODO: add documentation on reproducing Web Test flakes >TODO: document how to get the args.gn that the bot used
>TODO: document how to get the flags that the bot passed to `run_web_tests.py`
### Repeatedly running tests
Flakes are by definition non-deterministic, so it may be necessary to run the
test or set of tests repeatedly to reproduce the failure. Two flags to
`run_web_tests.py` can help with this:
* `--repeat-each=N` - repeats each test in the test set N times. Given a set of
tests A, B, and C, `--repeat-each=3` will run AAABBBCCC.
* `--iterations=N` - repeats the entire test set N times. Given a set of tests
A, B, and C, `--iterations=3` will run ABCABCABC.
## Debugging flaky Web Tests ## Debugging flaky Web Tests
>TODO: add documentation on debugging flaky Web Tests >TODO: document how to attach gdb
### Seeing logs from content\_shell
When debugging flaky tests, it can be useful to add `LOG` statements to your
code to quickly understand test state. In order to see these logs when using
`run_web_tests.py`, pass the `--driver-logging` flag:
```
./third_party/blink/tools/run_web_tests.py --driver-logging path/to/test.html
```
### Loading the test directly in content\_shell
When debugging a specific test, it can be useful to skip `run_web_tests.py` and
directly run the test under `content_shell` in an interactive session. For many
tests, one can just pass the test path to `content_shell`:
```
out/Default/content_shell third_party/blink/web_tests/path/to/test.html
```
**Caveat**: running tests like this is not equivalent to `run_web_tests.py`,
which passes the `--run-web-tests` flag to `content_shell`. The
`--run-web-tests` flag enables a lot of testing-only code in `content_shell`,
but also runs in a non-interactive mode.
Useful flags to pass to get `content_shell` closer to the `--run-web-tests` mode
include:
* `--enable-blink-test-features` - enables status=test and status=experimental
features from `runtime_enabled_features.json5`.
>TODO: document how to deal with tests that require a server to be running
[web_tests_blink_web_tests_step]: images/web_tests_blink_web_tests_step.png [web_tests_blink_web_tests_step]: images/web_tests_blink_web_tests_step.png
[web_tests_archive_blink_web_tests_step]: images/web_tests_archive_blink_web_tests_step.png [web_tests_archive_blink_web_tests_step]: images/web_tests_archive_blink_web_tests_step.png
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment