[wptrunner] Add documentation on using wptrunner in Chromium

This adds some basic documentation laying out what the project is, how
to use it, and the MVP bots. It is intended as a user guide rather than
technical details.

Bug: None
Change-Id: I7d90f542d2aeb33eecfd836693f2741ad91588c6
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2451505
Commit-Queue: Stephen McGruer <smcgruer@chromium.org>
Reviewed-by: Luke Z <lpz@chromium.org>
Reviewed-by: Robert Ma <robertma@chromium.org>
Cr-Commit-Position: refs/heads/master@{#814646}

docs/testing/web_platform_tests_wptrunner.md

+# Using upstream wptrunner in Chromium (experimental)
+
+This page documents the *experimental* support for using the upstream
+[`wptrunner`](https://github.com/web-platform-tests/wpt/tree/master/tools/wptrunner/)
+tooling for running WPT tests in Chromium (vs the [current
+approach](web_platform_tests.md#Running-tests) that uses `run_web_tests.py`).
+
+It is written as a user guide. For technical details on the project, see the
+[design doc](https://docs.google.com/document/d/1Pq5fxR1t2JzOVPynqeRpRS4bM4QO_Z1um0Q_RiR5ETA/edit).
+
+[TOC]
+
+## Differences versus `run_web_tests.py`
+
+The main differences between `run_web_tests.py` and `wptrunner` are that:
+
+1. `wptrunner` runs the full `chrome` binary, rather than the stripped-down
+   `content_shell`, and
+1. `wptrunner` communicates with the binary via WebDriver (`chromedriver`),
+   instead of talking directly to the browser binary.
+
+These differences mean that any feature that works on upstream WPT today (e.g.
+print-reftests) should work in `wptrunner`, but conversely features available to
+`run_web_tests.py` (e.g. the `internals` API) are not yet available to
+`wptrunner`.
+
+## Running tests locally
+
+*** note
+**NOTE**: Running locally is an area of active development, so the following
+instructions may be out of date.
+***
+
+The runner script is
+[`testing/scripts/run_wpt_tests.py`](https://source.chromium.org/chromium/chromium/src/+/master:testing/scripts/run_wpt_tests.py).
+Before running the script, you must have built the necessary ninja targets:
+
+```
+autoninja -C out/Release wpt_tests_isolate
+```
+
+To run the script, you must enter the `testing/scripts/` directory before
+executing it:
+
+```
+cd testing/scripts
+./run_wpt_tests.py --isolated-script-test-output=../../out/Release/results.json [test list]
+```
+
+The `--isolated-script-test-output` flag is currently required (see [Known
+Issues](#known-issues)), and the list of tests should be given relative to
+`external/wpt/`, e.g. `webauthn/createcredential-timeout.https.html`.
+Directories are also accepted. Omitting the test list will run all WPT tests.
+
+Results from the run are currently placed relative to
+`--isolated-script-test-output`, in a folder called `layout-test-results` (so
+the above example would put results in `../../out/Release/layout-test-results/`).
+Logs from the browser should be shown by the runner as it executes each test.
+
+Useful flags:
+
+* `-t/--target`: select which `src/out/` sub-directory to use, e.g. `-t Debug`.
+  Defaults to `Release`.
+* `--help`: show the help text
+
+## The MVP bots
+
+As of Q4 2020, an MVP of wptrunner in Chromium is being run with two customer
+teams: Web Payments and Web Identity. For these teams, two **Linux-only** bots
+have been brought up:
+
+* [linux-wpt-identity-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-identity-fyi-rel),
+  which runs tests under `external/wpt/webauthn/`.
+* [linux-wpt-payments-fyi-rel](https://ci.chromium.org/p/chromium/builders/ci/linux-wpt-payments-fyi-rel),
+  which runs tests under `external/wpt/payment-{handler, method-basic-card,
+  request}`.
+
+These bots run on the waterfall, but can also be run on CLs by clicking the
+`Choose Tryjobs` button in Gerrit followed by searching for the bot name in the
+modal dialog that appears. One can also include the tag `Cq-Include-Trybots:
+luci.chromium.try:linux-wpt-identity-fyi-rel` (or payments) in the description
+for the CL, which will make the bot mandatory for that CL.
+
+Results for the bots use the existing layout test results viewer
+([example](https://test-results.appspot.com/data/layout_results/linux-wpt-identity-fyi-rel/201/wpt_tests_suite/layout-test-results/results.html)).
+
+## Expectations and baseline files
+
+[Similar to `run_web_tests.py`](web_test_expectations.md), `wptrunner` offers
+the ability to add an expected status for a test or provide a baseline file that
+codifies what the output result should be.
+
+By default `wptrunner` will inherit expected statuses from `TestExpecations`.
+This can currently be overridden by adding an entry to the
+[`WPTOverrideExpectations`](https://source.chromium.org/chromium/chromium/src/+/master:third_party/blink/web_tests/WPTOverrideExpectations)
+file when `wptrunner` has a different result than `run_web_tests.py`.
+`WPTOverrideExpectations` is however [deprecated](https://crbug.com/1035911),
+and the preferred method for specifying expected results for `wptrunner` is to
+use baseline files (which will also override a TestExpectation entry for the
+test).
+
+Baseline files for `wptrunner` use a different filename and format than
+the `-expected.txt` files used by `run_web_tests.py`. The ini-like baseline format is
+[documented here](https://web-platform-tests.org/tools/wptrunner/docs/expectation.html),
+and baseline files should be placed alongside the test with an `.ini` suffix:
+
+```
+external/wpt/folder/my-test.html
+external/wpt/folder/my-test-expected.txt <-- run_web_tests.py baseline
+external/wpt/folder/my-test.html.ini <-- wptrunner baseline
+```
+
+We currently do not support the full ini-like format that upstream WPT does;
+most notably we have chosen not to support dynamic conditionals (such as
+platform checks). Most `.ini` baseline files in Chromium should have the form:
+
+```
+[my-test.html]
+  expected: OK
+
+  [First subtest name]
+    expected: FAIL
+    message: The failure message, e.g. assert_false didn't work
+
+  [Second subtest name]
+    expected: PASS
+```
+
+*** note
+**TODO**: Explain how .any.js and variants work in this world.
+***
+
+## Known issues
+
+* The `--isolated-script-test-output` flag is currently non-optional for
+  `run_wpt_tests.py`. We intend to infer this from `--target` in the future.
+* There is no debugging support in `run_wpt_tests.py` today. In the future, we
+  intend to allow pausing the browser after each test, and (long-term) intend to
+  support hooking up gdb to test runs.
+* There is not yet support for non-Linux platforms. We would love for you to try
+  it on other operating systems and file bugs against us if it doesn't work!
+
+Please [file bugs and feature requests](https://crbug.com/new) against
+`Blink>Infra>Ecosystem`, tagging the title with `[wptrunner]`.