Commit 878d2afa authored by Nate Fischer's avatar Nate Fischer Committed by Commit Bot

AW docs: improve public CTS docs

No change to logic, only docs. This expands on the existing CTS docs:

* Links to the AOSP source code
* Failures must be ReleaseBlock-Dev
* How to change test logic
* How to run tests locally

Fixed: 1138051
Test: Upload to gerrit > open file > click "gitiles"
Change-Id: I635718e5d1e4327a1964e617279282a157f14d72
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2542679
Auto-Submit: Nate Fischer <ntfschr@chromium.org>
Commit-Queue: Richard Coles <torne@chromium.org>
Reviewed-by: default avatarRichard Coles <torne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#828288}
parent f227b179
......@@ -57,22 +57,62 @@ please adjust them accordingly.
## Disabling/Skipping tests
Add entries to the `excludes` list for the respective apk under `test_runs`.
**CTS regressions are more serious than most test failures.** CTS failures block
Android vendors from shipping devices and prevent the WebView team from dropping
new Chrome and WebView APKs in the Android source tree. If you need to disable a
test, please file a P1 crbug with **ReleaseBlock-Dev** in the `Mobile>WebView`
component.
If you must disable a test, you can add an entry to the `excludes` list for the
correct apk (most tests belong to `CtsWebkitTestCases.apk`) under `test_runs`
for each OS level which is failing.
## Re-enabling skipped tests
Before re-enabling tests, make sure it's actually safe to enable the test again.
* The test source code lives in Android and line numbers vary between OS
versions. You can find test code for a particular CTS release by finding the
appropriate git branch in codesearch:
* Lollipop: [lollipop-mr1-cts-release]
* Marshmallow: [marshmallow-cts-release]
* Nougat: [nougat-cts-dev]
* Oreo: [oreo-cts-dev]
* Pie: [pie-cts-dev]
* Android 10 (Q): [android10-tests-dev]
* If the test was fixed on the Android side, the fix must be cherry-picked back
to the earliest applicable version. Ex. if the test was added in Android Oreo
(API 26), the fix should be picked back to `aosp/oreo-cts-dev`.
* **Note:** some OS levels are no longer supported by the CTS team, and will
to the earliest applicable version (see the git branches above). Ex. if the
test was added in Android Oreo (API 26), the fix should be picked back to
`aosp/oreo-cts-dev`.
* **Note:** some OS levels are no longer supported by the CTS team and will
no longer receive CTS releases. Unfortunately, if there was a test bug for
these OS levels, we must disable the test forever on that OS (and you
should cherry-pick the fix to the earliest supported CTS branch).
* If the failure was due to a chromium-side bug/regression, you can re-enable
the test as soon as the bug is fixed on trunk.
the test as soon as the bug is fixed on trunk. You can run CTS on a device or
emulator with [this guide](/android_webview/docs/test-instructions.md#cts).
Re-enabling the test case is as simple as removing it from the `excludes` for
the relevant OS levels. Please verify this change by adding the
`android-webview-*` trybots (not enabled by default).
## Changing CTS tests retroactively
Android generally has strict backward compatibility requirements, and this
extends to CTS. However, sometimes it's appropriate to change the test logic
rather than restoring the old chromium behavior, such as when the test logic is
responsible for flakiness or relies on an invalid assumption. Please reach out
to [the WebView team][1] quickly if you think a CTS test needs to change (the
failure is still considered **ReleaseBlock-Dev** until the test change actually
lands in Android).
Any CTS changes must be backward compatible. The original WebView version which
shipped on that OS version must continue to pass the revised CTS test.
[1]: https://groups.google.com/a/chromium.org/forum/#!forum/android-webview-dev
[lollipop-mr1-cts-release]: https://cs.android.com/android/platform/superproject/+/lollipop-mr1-cts-release:cts/tests/tests/webkit/src/android/webkit/cts/
[marshmallow-cts-release]: https://cs.android.com/android/platform/superproject/+/marshmallow-cts-release:cts/tests/tests/webkit/src/android/webkit/cts/
[nougat-cts-dev]: https://cs.android.com/android/platform/superproject/+/nougat-cts-dev:cts/tests/tests/webkit/src/android/webkit/cts/
[oreo-cts-dev]: https://cs.android.com/android/platform/superproject/+/oreo-cts-dev:cts/tests/tests/webkit/src/android/webkit/cts/
[pie-cts-dev]: https://cs.android.com/android/platform/superproject/+/pie-cts-dev:cts/tests/tests/webkit/src/android/webkit/cts/
[android10-tests-dev]: https://cs.android.com/android/platform/superproject/+/android10-tests-dev:cts/tests/tests/webkit/src/android/webkit/cts/
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment