Skip to content

GitLab

T
tangled
Commit bf9f5623 authored by bsheedy's avatar bsheedy Committed by Commit Bot
Browse files
Options

Update XR instrumentation test documentation 

Updates the XR instrumentation Markdown documentation to be consistent
with the recent refactor.

Bug: 863256
Change-Id: I21e2a492ab8e0a776010af9d624e04ef5631314a
Reviewed-on: https://chromium-review.googlesource.com/1152515Reviewed-by: default avatarAmirhossein Simjour <asimjour@chromium.org>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578752}
parent 17b91f53
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 453 additions and 325 deletions
chrome/android/javatests/src/org/chromium/chrome/browser/vr/adding_new_tests.md 0 → 100644
View file @ bf9f5623
# Adding New XR Instrumentation Tests
## Introduction
This is a brief overview of general steps to adding new XR instrumentation
tests. If you want to add tests as fast as possible, keep reading and glance
through some existing tests, which should give you enough information to start
writing your own.
If you want to better understand what's going on under the hood or why we do
certain things, take a look at
[`xr_instrumentation_deep_dive.md`][xr_instrumentation_deep_dive].
### An Overview Of XR Test Frameworks
Pretty much all XR instrumentation tests with the exception of some VR Browser
tests interact with asynchronous (Promise based) JavaScript code. This is where
the XR Test Frameworks come in, with test classes defining `mXyzTestFramework`
for testing feature Xyz. Together with some JavaScript imports in your test's
HTML file, these allow you to run tests as a series of synchronous steps that
alternate between JavaScript and Java.
For a concrete example, take a look at
[`WebXrVrTransitionTest`][webxr_vr_transition_test]'s
`testNonImmersiveStopsDuringImmersive` test and its corresponding HTML file
[test_non_immersive_stops_during_immersive.html][webxr_vr_transition_test_html].
The general flow in tests will be:
1. Load the HTML file with loadUrlAndAwaitInitialization - this ensures that any
pre-test setup in JavaScript is completed.
2. Run some code on Java's side.
3. Trigger some JavaScript code and wait for it to signal that it is finished.
These can be identified as the `*AndWait` methods, and stop blocking once the
JavaScript side calls `finishJavaScriptStep()`.
4. Repeat from 2 until done.
5. End the test.
## Adding Tests To Existing Test Classes
If you're adding a new test to an existing test class, all the per-class
boilerplate code should be around already, so you can get right to adding a new
test case using the following general components.
### Annotations
The following annotations can be applied before your test body to modify its
behavior.
#### @Test
Every test method must be annotated with the `@Test` annotation in order for the
test runner to identify it as an actual test.
#### Test Length
Every test method must also be annotated with a test length annotation,
typically `@MediumTest`. Eventually, the test length annotations should imply
the presence of `@Test`, but both must currently be present.
#### Supported Activities
Unless your test uses the VR Browser, you can use the `@XrActivityRestriction`
annotation to automatically run your test multiple times in different supported
activities. The currently supported activities are:
* ChromeTabbedActivity (regular Chrome)
* CustomTabActivity (used to open links in apps like GMail)
* WebappActivity (used for Progressive Webapps)
#### @Restriction
You can restrict your test or test class to only be run under certain
circumstances, such as only on Daydream-ready devices or only with the Daydream
View headset paired, using the `@Restriction` annotation.
#### Command Line Flags
You can add or remove command line flags that are set before the test runs using
`@CommandLineFlags.Add` and `@CommandLineFlags.Remove`. Note that if you want to
override a flag set by the test class on a per-test basis, you must remove and
re-add it.
### Test Body
#### HTML Test File
You will likely need an HTML file to load during your test, which should be
placed in `//chrome/test/data/xr/e2e_test_files/html`. The exact contents of
your file will depend on your test, but you will likely be importing some or all
of the following scripts from `//chrome/test/data/xr/e2e_test_files/resources`:
* `webxr_e2e.js`/`webvr_e2e.js` - Sets up the necessary code to communicate back
and forth between Java and JavaScript
* `webxr_boilerplate.js`/`webvr_boilerplate.js` - Handles the WebXR and WebVR
boilerplate code, such as getting an XRDevice and setting up a canvas.
Additionally, in order to use asserts in JavaScript, you must import
`//third_party/WebKit/LayoutTests/resources/testharness.js`.
#### Java Test Body
The exact contents of your test body are going to depend on the test you're
trying to write, so just keep the following guidelines in mind:
* Use the most specific version of a class as possible, e.g. use
`WebXrArTestFramework` for WebXR for AR testing instead of `WebXrTestFramework`.
* If you need to do something that involves the webpage/web contents, it's
likely available through your test framework.
* If you need to do something that doesn't involve the webpage/web contents,
it's likely available in one of the classes in `util/`.
## Adding A New Test Class
If you're adding a new test class instead of just adding a new test to an
existing class, there are a few additional bits of boilerplate code you will
need to add before being able to write your test.
### Test Parameterization
Test parameterization is how running a test multiple times in different
activities is handled. However, it adds some amount of overhead to test runtime,
so it's only enabled where it makes sense. If your new test class will only have
VR Browser-related tests, you can skip parameterization. Otherwise, you'll want
to enable it.
#### Non-Parameterized
See [`VrBrowserTransitionTest`][vr_browser_transition_test] for an example of a
non-parameterized class. The general things you will need to do are:
* Set the class' `@RunWith` to `ChromeJUnit4ClassRunner.class`.
* Declare `mTestRule` as a `ChromeTabbedActivityVrTestRule`, annotate it with
`@Rule`, and initialize it where it's declared.
* Declare `mVrBrowserTestFramework` as a `VrBrowserTestFramework` and initialize
it using `mTestRule` in a setup method annotated with `@Before`. You can do
the same with other types of test frameworks as necessary.
#### Parameterized
See [`WebXrVrTransitionTest`][webxr_vr_transition_test] for an example of a
parameterized class. The general things you will need to are:
* Set `@RunWith` to `ParameterizedRunner.class`.
* Add `@UseRunnerDelegate` and set it to `ChromeJUnit4RunnerDelegate.class`.
* Declare `sClassParams` as a static `List` of `ParameterSet`, annotate it with
`@ClassParameter`, and set it to the value returned by either
`XrTestRuleUtils.generateDefaultTestRuleParameters()` for AR tests or
`VrTestRuleUtils.generateDefaultTestRuleParameters()` for VR tests.
* Declare `mRuleChain` as a `RuleChain` and annotate it with `@Rule`.
* Declare `mTestRule` as a `ChromeActivityTestRule`.
* Declare any necessary test frameworks and initialize them using `mTestRule` in
a setup method annotated with `@Before`.
* Define a constructor for your test class that takes a
`Callable<ChromeActivityTestRule>`. This constructor must set `mVrTestRule` to
the `Callable`'s `call()` return value and set `mRuleChain` to the return
value of `XrTestRuleUtils.wrapRuleInXrActivityRestrictionRule(mTestRule)`
for AR tests or `VrTestRuleUtils.wrapRuleInXrActivityRestrictionRule
(mTestRule)` for VR tests.
### Add The New File
Add the new test class to [`//chrome/android/BUILD.gn`][build_gn]. If it is a VR
test class, it should be added to the `java_files` list of the
`chrome_test_vr_java` `android_library` target. If it is an AR test class, it
should be added to the `java_files` list of the `chrome_test_ar_java`
`android_library` target.
[xr_instrumentation_deep_dive]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/xr_instrumentation_deep_dive.md
[webxr_vr_transition_test]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/WebXrVrTransitionTest.java
[webxr_vr_transition_test_html]: https://chromium.googlesource.com/chromium/src/+/master/chrome/test/data/xr/e2e_test_files/html/test_non_immersive_stops_during_immersive.html
[vr_browser_transition_test]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/VrBrowserTransitionTest.java
[build_gn]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/BUILD.gn
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/rules/README.md deleted 100644 → 0
View file @ 17b91f53
# VR Instrumentation Rules
## Introduction
This directory contains all the Java files related to VR-specific JUnit4 rules.
For the most part, the rules are identical to the standard activity-specific
rules (e.g. `ChromeTabbedActivityTestRule` for `ChromeTabbedActivity`), but with
additional code to work with JUnit4 test parameterization to support running
a test case multiple times in different activities.
Usage of this feature is already covered in `../README.md`, so this
documentation concerns implementation details.
## How It Works
When a test is set up to use test parameterization (determined by whether the
test class is annotated with `@RunWith(ParameterizedRunner.class)` or not), the
test runner automatically runs each test case in the class once with every
`ParameterSet` in the list of `ParameterSet`s that is annotated with
`@ClassParameter`, passing each `ParameterSet`'s contents to the class constructor
each time.
In the case of VR tests, the `ParameterSet` list contains `Callable`s that
construct VR test rules for the three supported activity types
(`ChromeTabbedActivity`, `CustomTabActivity`, and `WebappActivity`). The
constructor for parameterized VR tests runs the provided `Callable`, effectively
running every test case once in each activity type.
However, if the class were to assign the `Callable` return value to its member
variable annotated with `@Rule`, then every test case in the class would always
run in all three activity types, which isn't desirable since some test cases
test scenarios that are only valid in some activities (e.g. tests that involve
the VR browser are currently only supported in ChromeTabbedActivity). To avoid
this, the `Rule` annotated with `@Rule` is actually a `RuleChain` that wraps the
generated VR test rule in a `XrActivityRestrictionRule`.
`XrActivityRestrictionRule` interacts with the `@XrActivityRestriction`
annotation and the VR test rule for the current test case run. If the activity
type of the current rule is contained in the list provided by
`@XrActivityRestriction` (or there is no restriction annotation and the activity
type is `ChromeTabbedActivity`), then the `XrActivityRestrictionRule` becomes a
no-op and the test case runs normally. Otherwise, `XrActivityRestrictionRule`
causes an assumption failure, which is interpreted by the test runner as a
signal to skip that particular test case/parameter combination.
Thus, the end result is that the same test case can be run multiple times in
different activities without having to duplicate any code for use in different
activity types, while still retaining the ability to not run every test in every
activity type if necessary.
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/vr_test_framework.md deleted 100644 → 0
View file @ 17b91f53
# VR Test Framework
## Introduction
This is an overview of how the VR test framework functions. It is split into two
parts, with Java code located in this directory in `VrTestFramework.java` and
JavaScript code located in `//chrome/test/data/vr/e2e_test_files/`.
It is primarily used for testing WebVR, but can also be useful for testing the
VR Browser.
A similar approach is used for testing VR on desktop, with the framework
re-implemented in C++ for use in browser tests and many of the same JavaScript
files used on both platforms. See the documentation in
`//chrome/browser/vr/test` for more information on this.
## Structure
Tests utilizing this framework are split into separate Java and JavaScript files
since WebVR interaction is done via JavaScript, but instrumentation tests are
Java-based. In general, the JavaScript code handles any interactions with the
WebVR API, and the Java code handles everything else (user gestures, controller
emulation, etc.).
In general, the flow of a test is:
* In Java, load the HTML test file, which:
* Loads the WebVR boilerplate code and test code
* Sets up any test steps that need to be triggered by Java as separate
functions
* Creates an asynchronous test (denoted t here)
* Repeat:
* Run any necessary Java-side code, e.g. trigger a user action
* Trigger the next JavaScript test step and wait for it to finish
* Finally, call t.done() in JavaScript and endTest in Java
### JavaScript
The JavaScript test code mainly makes use of testharness.js, which is also used
for layout tests. This allows the use of asserts in JavaScript code, and any
assert failures will propagate up to Java. There are four main test-specific
functions that you will need to use:
#### async\_test
This creates an asynchronous test from testharness.js which you will use
throughout the JavaScript code. It serves two purposes:
* By being an asynchronous test, it will prevent testharness.js from ending the
test run prematurely once the page is loaded. It will wait until all tests
are done before checking results.
* Enables you to use asserts in JavaScript. Asserts must be within a test to
do anything.
#### finishJavaScriptStep
This signals that the current portion of JavaScript code is done and that the
Java side can continue execution.
#### t.step
This defines a test step in an asynchronous test t. Any asserts must be within
a test step, so assertions will look along the lines of:
`t.step( () => {
assertTrue(someBool);
});`
#### t.done
This signals that the asynchronous test is done. Once all tests in a file are
completed (usually only one), testharness.js will check the results and
automatically call finishJavaScriptStep when done.
### Java
There are many Java-side functions that enable VR testing, but only the three
basic ones will be covered here. For specifics about less common functions, see
the JavaDoc comments in `VrTestFramework.java` and the utility classes in
`util/`.
#### loadUrlAndAwaitInitialization
This is similar to the standard loadUrl method in Chrome, but also waits until
all initialization steps are complete and the page is ready for testing. By
default, this only includes the page being loaded. Additionally, any files
that include the WebVR boilerplate script (generally any test for WebVR) will
wait until the promise returned by `navigator.getVRDisplays` has resolved or
rejected. Tests can add their own initialization steps by adding an entry to
the `initializationSteps` dictionary defined in
`//chrome/test/data/android/webvr_instrumentation/resources/webvr_e2e.js` with
the value set to `false`. Once the step is done, simply set the value to
`true`.
#### executeStepAndWait
This executes the given JavaScript and waits until finishJavaScriptStep is
called on the JavaScript side.
#### endTest
Performs any post-test checks after the JavaScript test code has finished
running. Since test failures are caught after each step, all this really does
is ensure that the the JavaScript code has actually finished running, throwing
an error if this is not the case.
Typically called at the very end of the test after everything else is
completed.
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/xr_instrumentation_deep_dive.md 0 → 100644
View file @ bf9f5623
# XR Instrumentation Test Deep Dive
## Introduction
This documentation aims to provide a more in-depth view of how various aspects
of the XR instrumentation tests work under the hood and explain why certain
design decisions were made. If you just want an overview on how to build/run the
tests or add new ones, see [`README.md`][readme] and
[`adding_new_tests.md`][adding_new_tests].
## Naming Convention
Classes, files, and variables are named using the following (hopefully
self-explanatory) naming convention:
* `XR`/`Xr` - The most general term. Used for things that are applicable to any
XR feature.
* `VR`/`Vr` - A subset of `XR`. Used for things that are applicable to any VR
feature.
* `AR`/`Ar` - A subset of `XR`. Used for things that are applicable to any AR
feature.
* `VR Browser`/`VrBrowser` - A subset of `VR`. Used for things that are only
applicable to the VR Browser feature of Chrome.
* `WebXR`/`WebXr` - A subset of `XR`. Used for things that are applicable to any
XR-related web APIs.
* `WebXR for AR`/`WebXrAr` - A subset of `WebXR` and `AR`. Used for things that
are applicable to the `WebXR Devices API` in its AR usecase.
* `WebXR for VR`/`WebXrVr` - A subset of `WebXR` and `VR`. Used for things that
are applicable to the `WebXR Devices API` in its VR usecase.
* `WebVR`/`WebVr` - A subset of `WebXrVr`. Used for things that are only
applicable to the older `WebVR` API.
## Test Framework Structure
### Hierarchy
Based on the above naming scheme, the various `TestFramework` classes are
structured in the following hierarchy:
* `XrTestFramework`
* `VrBrowserTestFramework`
* `WebXrTestFramework`
* `WebXrArTestFramework`
* `WebXrVrTestFramework`
* `WebVrTestFramework`
### Static vs. Non-Static Methods
Most methods in the `TestFramework` classes have both a static and non-static
version, with the non-static version simply calling the static one with the
framework's `mFirstTabWebContents` reference. This is because the vast majority
of use cases are interacting with the web page that is in the tab automatically
opened at the start of the test, but some rare cases require interacting with
other tabs. Thus, we need to provide a way of using the frameworks with
arbitrary tabs/`WebContents` (the static methods), but offering the non-static
versions cuts down the clutter of calling `getFirstTabWebContents()` everywhere.
## Parameterization
Parameterization is a JUnit4 concept. You can read the official guide about it
on the [JUnit4 GitHub Wiki][junit4_wiki_parameterization], but the TL;DR is that
it allows a test method or class to be automatically run multiple times with
varying inputs.
When used in XR tests, the `List` of `ParameterSet`s annotated with
`@ClassParameter` is what will be iterated over. Specifically,
`XrTestRuleUtils.generateDefaultTestRuleParameters` or
`VrTestRuleUtils.generateDefaultTestRuleParameters` will generate a `List` of
`ParamaterSet`s each containing a single `Callable` whose `call()` returns a
`ChromeActivityTestRule`. Each `ParameterSet` corresponds to one of the activity
types that XR features are supported in. This is why constructors of
parameterized test classes must accept a `Callable<ChromeActivityTestRule>` -
they will be called once per element of the `ParameterSet` `List` and passed the
contents of that element.
`Callable` is used instead of `ChromeActivityTestRule` directly due to an
implementation detail in `ParameterSet` ([source code][parameter_set_source]).
`ParameterSet` only accepts primitives, `Callable`s, and a certain set of
`Class`es such as `String`, which means we can't pass a `ChromeActivityTestRule`
directly to a `ParameterSet`.
## Rules
Rules are another JUnit4 concept. You can read a fairly comprehensive guide
about them on the [JUnit4 GitHub Wiki][junit4_wiki_rules], but the TL;DR of them
is that they allow pre- and post-test code to be easily shared across multiple
test classes. Any rule annotated with `@Rule` will be automatically applied to
the test in an arbitrary order.
### Activity Restriction Rule
`XrActivityRestrictionRule` and the `@XrActivityRestriction` annotation are what
allow tests to only be run in the activities they support, as otherwise
parameterization would cause all tests to be run in all activities. The method
for this is simple - if the activity type for the current
`ChromeActivityTestRule` is in the list of supported activities provided by
`@XrActivityRestriction`, continue running the test as normal. Otherwise, don't
run the test, and instead, generate a statement that throws an assumption
failure, which the test runner treats as a signal that the test was skipped.
#### RuleChain
Every place where `XrActivityRestrictionRule` is used makes use of a `RuleChain`
and `XrTestRuleUtils.wrapRuleInXrActivityRestrictionRule()`. The reason for this
is simply optimization. By using a `RuleChain` to wrap a given
`ChromeActivityTestRule` in an `XrActivityRestrictionRule`, we can ensure that
the decision to skip a test due to being unsupported in an activity is made
before we go through the (slow) process of starting said activity.
### XR And VR Test Rules
XR instrumentation tests use special versions of `ChromeActivityTestRule` that
implement `XrTestRule` or `VrTestRule`.
Rules that implement `XrTestRule` simply open the specified activity type to a
blank page and implement a method that allows it to work with
`XrActivityRestrictionRule`.
Rules that implement `VrTestRule` do the same, but also perform some additional
VR-specific setup such as ensuring that the test is not started in VR and
allowing the use of the experimental/broken VrCore head tracking service.
## VR Controller Input
There are currently two ways of injecting Daydream controller input into tests,
each with their own pros and cons.
### EmulatedVrController
The `EmulatedVrController` class is the older of the two approaches and works by
setting VrCore to accept Android `Intent`s as controller input instead of using
an actual controller.
The main benefit of this is that it allows complete end-to-end
testing of controller-related Chrome code, as it still receives controller input
from VrCore the same as if a real controller was in use. It also allows the use
of the home button to recenter the view or go to Daydream Home.
A downside to this downside is that `Intent`s do not allow for precise timing,
leading to flakiness. Additionally, since we're essentially sending raw
quaternions to VrCore to turn into the controller's orientation, pointing at
specific UI elements is tedious and prone to breaking if the UI changes.
### NativeUiUtils
The methods in the `NativeUiUtils` class work by causing Chrome to start reading
controller input from a test-only queue instead of getting data from VrCore.
The downsides to this are that it causes Chrome to use some non-production code
and doesn't test the controller-related interaction with VrCore.
This approach does have quite a few benefits though. First it should not be
flaky while also being faster since everything is handled Chrome-side.
Additionally, it allows interaction with specific UI elements by name, which is
both easier and less prone to breaking than specifying a position in space.
[readme]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/README.md
[adding_new_tests]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/adding_new_tests.md
[junit4_wiki_parameterization]: https://github.com/junit-team/junit4/wiki/parameterized-tests
[parameter_set_source]: https://chromium.googlesource.com/chromium/src/+/master/base/test/android/javatests/src/org/chromium/base/test/params/ParameterSet.java
[junit4_wiki_rules]: https://github.com/junit-team/junit4/wiki/rules
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment