Commit 16f7943a authored by Stephen McGruer's avatar Stephen McGruer Committed by Commit Bot

[wptrunner] Add documentation on using wptrunner in Chromium

This adds some basic documentation laying out what the project is, how
to use it, and the MVP bots. It is intended as a user guide rather than
technical details.

Bug: None
Change-Id: I7d90f542d2aeb33eecfd836693f2741ad91588c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2451505
Commit-Queue: Stephen McGruer <smcgruer@chromium.org>
Reviewed-by: default avatarLuke Z <lpz@chromium.org>
Reviewed-by: default avatarRobert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#814646}
parent bffa4476
# Using upstream wptrunner in Chromium (experimental)
This page documents the *experimental* support for using the upstream
[`wptrunner`](https://github.com/web-platform-tests/wpt/tree/master/tools/wptrunner/)
tooling for running WPT tests in Chromium (vs the [current
approach](web_platform_tests.md#Running-tests) that uses `run_web_tests.py`).
It is written as a user guide. For technical details on the project, see the
[design doc](https://docs.google.com/document/d/1Pq5fxR1t2JzOVPynqeRpRS4bM4QO_Z1um0Q_RiR5ETA/edit).
[TOC]
## Differences versus `run_web_tests.py`
The main differences between `run_web_tests.py` and `wptrunner` are that:
1. `wptrunner` runs the full `chrome` binary, rather than the stripped-down
`content_shell`, and
1. `wptrunner` communicates with the binary via WebDriver (`chromedriver`),
instead of talking directly to the browser binary.
These differences mean that any feature that works on upstream WPT today (e.g.
print-reftests) should work in `wptrunner`, but conversely features available to
`run_web_tests.py` (e.g. the `internals` API) are not yet available to
`wptrunner`.
## Running tests locally
*** note
**NOTE**: Running locally is an area of active development, so the following
instructions may be out of date.
***
The runner script is
[`testing/scripts/run_wpt_tests.py`](https://source.chromium.org/chromium/chromium/src/+/master:testing/scripts/run_wpt_tests.py).
Before running the script, you must have built the necessary ninja targets:
```
autoninja -C out/Release wpt_tests_isolate
```
To run the script, you must enter the `testing/scripts/` directory before
executing it:
```
cd testing/scripts
./run_wpt_tests.py --isolated-script-test-output=../../out/Release/results.json [test list]
```
The `--isolated-script-test-output` flag is currently required (see [Known
Issues](#known-issues)), and the list of tests should be given relative to
`external/wpt/`, e.g. `webauthn/createcredential-timeout.https.html`.
Directories are also accepted. Omitting the test list will run all WPT tests.
Results from the run are currently placed relative to
`--isolated-script-test-output`, in a folder called `layout-test-results` (so
the above example would put results in `../../out/Release/layout-test-results/`).
Logs from the browser should be shown by the runner as it executes each test.
Useful flags:
* `-t/--target`: select which `src/out/` sub-directory to use, e.g. `-t Debug`.
Defaults to `Release`.
* `--help`: show the help text
## The MVP bots
As of Q4 2020, an MVP of wptrunner in Chromium is being run with two customer
teams: Web Payments and Web Identity. For these teams, two **Linux-only** bots
have been brought up:
* [linux-wpt-identity-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-identity-fyi-rel),
which runs tests under `external/wpt/webauthn/`.
* [linux-wpt-payments-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-payments-fyi-rel),
which runs tests under `external/wpt/payment-{handler, method-basic-card,
request}`.
These bots run on the waterfall, but can also be run on CLs by clicking the
`Choose Tryjobs` button in Gerrit followed by searching for the bot name in the
modal dialog that appears. One can also include the tag `Cq-Include-Trybots:
luci.chromium.try:linux-wpt-identity-fyi-rel` (or payments) in the description
for the CL, which will make the bot mandatory for that CL.
Results for the bots use the existing layout test results viewer
([example](https://test-results.appspot.com/data/layout_results/linux-wpt-identity-fyi-rel/201/wpt_tests_suite/layout-test-results/results.html)).
## Expectations and baseline files
[Similar to `run_web_tests.py`](web_test_expectations.md), `wptrunner` offers
the ability to add an expected status for a test or provide a baseline file that
codifies what the output result should be.
By default `wptrunner` will inherit expected statuses from `TestExpecations`.
This can currently be overridden by adding an entry to the
[`WPTOverrideExpectations`](https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/web_tests/WPTOverrideExpectations)
file when `wptrunner` has a different result than `run_web_tests.py`.
`WPTOverrideExpectations` is however [deprecated](https://crbug.com/1035911),
and the preferred method for specifying expected results for `wptrunner` is to
use baseline files (which will also override a TestExpectation entry for the
test).
Baseline files for `wptrunner` use a different filename and format than
the `-expected.txt` files used by `run_web_tests.py`. The ini-like baseline format is
[documented here](https://web-platform-tests.org/tools/wptrunner/docs/expectation.html),
and baseline files should be placed alongside the test with an `.ini` suffix:
```
external/wpt/folder/my-test.html
external/wpt/folder/my-test-expected.txt <-- run_web_tests.py baseline
external/wpt/folder/my-test.html.ini <-- wptrunner baseline
```
We currently do not support the full ini-like format that upstream WPT does;
most notably we have chosen not to support dynamic conditionals (such as
platform checks). Most `.ini` baseline files in Chromium should have the form:
```
[my-test.html]
expected: OK
[First subtest name]
expected: FAIL
message: The failure message, e.g. assert_false didn't work
[Second subtest name]
expected: PASS
```
*** note
**TODO**: Explain how .any.js and variants work in this world.
***
## Known issues
* The `--isolated-script-test-output` flag is currently non-optional for
`run_wpt_tests.py`. We intend to infer this from `--target` in the future.
* There is no debugging support in `run_wpt_tests.py` today. In the future, we
intend to allow pausing the browser after each test, and (long-term) intend to
support hooking up gdb to test runs.
* There is not yet support for non-Linux platforms. We would love for you to try
it on other operating systems and file bugs against us if it doesn't work!
Please [file bugs and feature requests](https://crbug.com/new) against
`Blink>Infra>Ecosystem`, tagging the title with `[wptrunner]`.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment