Skip to content

GitLab

T
tangled
Commit bf9f5623 authored by bsheedy's avatar bsheedy Committed by Commit Bot
Browse files
Options

Update XR instrumentation test documentation 

Updates the XR instrumentation Markdown documentation to be consistent
with the recent refactor.

Bug: 863256
Change-Id: I21e2a492ab8e0a776010af9d624e04ef5631314a
Reviewed-on: https://chromium-review.googlesource.com/1152515Reviewed-by: default avatarAmirhossein Simjour <asimjour@chromium.org>
Commit-Queue: Brian Sheedy <bsheedy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#578752}
parent 17b91f53
Hide whitespace changes
Inline Side-by-side
Showing with 453 additions and 325 deletions
chrome/android/javatests/src/org/chromium/chrome/browser/vr/README.md
View file @ bf9f5623
# VR Instrumentation Tests
# XR Instrumentation Tests
## Introduction
This directory contains all the Java-side infrastructure for running
instrumentation tests on the two virtual reality (VR) features currently
in Chrome:
instrumentation tests for XR (VR/Virtual Reality and AR/Augmented Reality)
features currently in Chrome:
* [WebVR](https://webvr.info/) - Experience VR content on the web
* [WebXR](https://immersive-web.github.io/webxr-samples/explainer.html) -
Successor to WebVR, experience VR and AR content on the web
* VR Browser - Browse the 2D web from a VR headset
These tests are integration/end-to-end tests run in the full Chromium browser on
actual Android devices.
## Directories
These are the files and directories that are relevant to VR instrumentation
testing. Additional information on files in other directories can be found in
those directories' README.md files (if the files as a whole warrant special
documentation) and the JavaDoc comments in each file.
These are the files and directories that are relevant to XR instrumentation
testing.
### Subdirectories
* `mock/` - Contains all the classes for mock implementations of VR classes
* `mock/` - Contains all the classes for mock implementations of XR classes.
* `nfc_apk/` - Contains the code for the standalone APK for NFC simulation. Used
by Telemetry tests, not instrumentation tests, but kept here since it uses
code from util/
* `rules/` - Contains all the VR-specific JUnit4 rules
* `util/` - Contains utility classes with code that's used by multiple test
classes
by Telemetry tests, not instrumentation tests, but kept here since it uses code
from `util/`.
* `rules/` - Contains all the XR-specific JUnit4 rules for handling
functionality such as running tests multiple times in different activities and
handling the fake VR pose tracker service.
* `util/` - Contains utility classes with code that is used by multiple test
classes and that does not make sense to include in the core test framework.
### Other Directories
* `//chrome/android/shared_preference_files/test/` - Contains the VrCore settings
files for running VR instrumentation tests (see the "Building and Running"
section for more information).
* `//chrome/test/data/vr/e2e_test_files/` - Contains the JavaScript
and HTML files for VR instrumentation tests.
* `//third_party/gvr-android-sdk/test-apks/` - Contains the VrCore and Daydream
Home APKs for testing. You must have `DOWNLOAD_VR_TEST_APKS` set as an
environment variable when you run gclient sync in order to actually download
these from storage.
* `//third_party/gvr-android-sdk/test-libraries/` - Contains third party VR
testing libraries. Currently, only has the Daydream controller test library.
## Building and Running
* [`//chrome/android/shared_preferences_files/test`][shared_prefs_dir] -
Contains the VrCore settings files for running VR instrumentation tests (see the
"Building and Running" section for more information).
* [`//chrome/test/data/xr/e2e_test_files/`][html_dir] - Contains the JavaScript
and HTML files for XR instrumentation tests.
* [`//third_party/gvr-android-sdk/test-apks`][vr_test_apks] - Contains the VR
APKs used for testing, such as VrCore. You must have `DOWNLOAD_VR_TEST_APKS` set
as an environment variable when you run gclient runhooks in order to actually
download these from storage.
* [`//third_party/gvr-android-sdk/test-libraries`][vr_test_libraries] - Contains
third party VR testing libraries. Currently, only has the Daydream controller
test library used for sending controller events to VrCore using intents.
* [`//third_party/arcore-android-sdk/test-apks`][ar_test_apks] - Contains the AR
APKs used for testing, such as ArCore. You must have `DOWNLOAD_VR_TEST_APKS` set
as an environment variable when you run gclient runhooks in order to actually
download these from storage.
## Building
### AR
The AR instrumentation tests can be built with the
`monochrome_public_test_ar_apk` target, which will also build
`monochrome_public_apk` to test with.
### VR
The VR instrumentation tests can be built with the `chrome_public_test_vr_apk`
target, which will also build `chrome_public_apk` to test with.
Once both are built, the tests can be run by executing
`run_chrome_public_test_vr_apk` in your output directory's `bin/` directory.
However, unless you happen to have everything set up properly on the device, the
tests will most likely fail. To fix this, you just need to pass a few extra
arguments to `run_chrome_public_test_vr_apk`.
## Running
**NOTE** These tests can only be run on rooted devices.
Both the VR and AR tests are run using the generated script in your build output
directory's `bin/` directory, e.g. `out/foo/bin/run_chrome_public_test_vr_apk`
to run the VR tests. You will likely need to pass some or all of the following
arguments in order for the tests to run properly, though.
### Standard Arguments
**NOTE** The instrumentation tests can only be run on rooted devices.
These are the arguments that you'll pretty much always want to pass in order to
ensure that your device is set up properly for testing.
### Common Arguments
#### additional-apk
`--additional-apk path/to/apk/to/install`
All this does is install the specified APK before running tests. You'll
generally want to install the current version of VrCore by passing it
Installs the specified APK before running the tests. No-ops if the provided APK
is already installed on the device and their versions match.
`--additional-apk
third_party/gvr-android-sdk/test-apks/vr_services/vr_services_current.apk`
For VR tests, you'll likely want to use `--additional-apk
third_party/gvr-android-sdk/test-apks/vr_services/vr_services_current.apk` to
ensure that the VrCore version used is the one used for automated testing at
whatever your current git revision is.
Just make sure that the APK is up to date and on your system by running gclient
sync with the `DOWNLOAD_VR_TEST_APKS` environment variable set. This argument can
be ommitted if you're fine with using whatever version of VrCore is already
installed on the device.
For AR tests, you'll likely want to use `--additional-apk
third_party/arcore-android-sdk/test-apks/arcore/arcore_current.apk` to ensure
that the ArCore version used is the one used for automated testing at whatever
your current git revision is.
**NOTE** This will fail on most Pixel devices, as VrCore is pre-installed as a
system app. You can get around this in two ways.
**NOTE** Using this argument for VR on most Pixel devices will fail, as VrCore
is pre-installed as a system app. This can be dealt with in the following ways:
* Remove VrCore as a system app by following the instructions
[here](go/vrcore/building-and-running). This will permanently solve the issue
......@@ -83,157 +101,84 @@ system app. You can get around this in two ways.
com.google.vr.vrcore,//third_party/gvr-android-sdk/test-apks/vr_services/vr_services_current.apk`
instead. This will take significantly longer, as it requires rebooting, and
must be done every time you run the tests.
#### shared-prefs-file
`--shared-prefs-file path/to/preference/json/file`
This will configure VrCore according to the provided file, e.g. changing the
paired headset. The two most common files to use are:
* `//chrome/android/shared_preference_files/test/vr_cardboard_skipdon_setupcomplete.json`
This will pair the device with a Cardboard headset and disabled controller
emulation.
* `//chrome/android/shared_preference_files/test/vr_ddview_skipdon_setupcomplete.json`
This will pair the device with a Daydream View headset, set the DON flow to be
skipped, and enable controller emulation. **NOTE** This will prevent you from
using a real controller outside of tests. To fix this, you can either
reinstall VrCore or apply the Cardboard settings file (there isn't currently a
way to manually re-enable real controller use from within the VrCore developer
settings)
### Other Useful Arguments
* Skip this argument entirely and just ensure that the VrCore version on the
device is up to date via the Play Store.
#### test-filter
`--test-filter TestClass#TestCase`
If you only have interest in a particular test case or class, such as when you
add a new test or are trying to reproduce a failure, you can significantly cut
down on the test runtime by limiting the tests that are run. You can either
specify a specific test case to restrict to, or use \* to restrict to all test
cases within a test class.
## Creating New Tests
### Adding A Test Case To An Existing Test Class
If you're new to adding VR instrumentation tests or instrumentation tests in
general, it's a good idea to take a look at existing tests to get a feel for
what's going on. In general, these are the things that you need to be aware of
when adding a new test:
#### @Test
Allows you to limit the set of tests run to a particular test class or subset of
tests within a test class. Use of the `*` wildcard is supported, e.g.
`--test-filter VrBrowserTransitionTest#*` will run all tests in the
VrBrowserTransitionTest class.
Every test case must be annotated with the `@Test` annotation in order to be
identified as a test.
#### local-output/json-results-file
#### Test Length
`--local-output --json-results-file output.json`
Every test case must also have a test length annotation, typically
`@MediumTest`. Eventually, the test length annotations should imply the presence
of `@Test` as well, but for now, both must be present.
Sets the test runner to generate a local results summary after running all tests
and print out a file URL pointing to the summary. This allows you to view both
logcat output for a particular test and its post-failure screenshot.
#### VR Test Framework
#### num-retries
Most test classes will define a `VrTestFramework` member as `mVrTestFramework`,
which contains functions that are critical for WebVR testing, and still useful
for VR Browser testing. Examples include:
`--num-retries <#>`
* Retrieving HTML test files
* Accessing the WebContents of whatever tab the test started
on, which is where the vast majority of testing takes place
* Waiting on JavaScript test steps and running arbitrary JavaScript code
Sets the test runner to retry failed tests a certain number of times. The
default is 2, resulting in a max of 3 test runs. Usually used as `--num-retries
0` when debugging to reduce test runtime and make flakiness more visible.
If you are writing a test that uses WebVR, you will likely also need to add an
HTML file to `//chrome/test/data/vr/e2e_test_files/html/` that
handles the JavaScript portion of the test. In general, a WebVR test will have a
number of steps as functions on the JavaScript side. The Java side of the test
will load the file and start a JavaScript step, blocking until it finishes. The
JavaScript code will execute and call `finishJavaScriptStep()` when done, which
will signal the Java code to continue. This repeats until the test is done. For
more specifics, see `vr_test_framework.md`.
#### repeat
#### VR Test Rule
`--repeat <#>`
Every test class will have a `mVrTestRule` member that inherits from
`ChromeActivityTestRule`. It does some important setup under the hood, but can
also be used to interact with Chrome during a test. If you need to interact with
Chrome and `VrTestFramework` does not have a way of doing that, then there is
likely a way to do it using `mVrTestRule`.
Sets the test runner to repeat the tests a certain number of times. The default
is 0, resulting in only one iteration. Usually used to repeat a test many times
in order to check for or reproduce flakiness.
#### Test Parameterization
### VR-Specific Arguments
If a test class is annotated with `@RunWith(ParameterizedRunner.class)`, then it
has support for test parameterization. While parameterization has many potential
uses, the current use in VR is to allow a test case to be automatically run in
multiple different activity types. For specifics, see
`rules/README.md`.
If a class has test parameterization enabled, you have three options when adding
a new test:
#### shared-prefs-file
* Do nothing special. This will default to only running the test in
ChromeTabbedActivity, which is the normal Chrome browser.
* Add the `@XrActivityRestriction` annotation with
`XrActivityRestriction.SupportedActivity.ALL` as its value, which will run the
test in all activities that support VR
* Add the `@XrActivityRestriction` annotation, manually specifying the
activities you want the test to run in
`--shared-prefs-file path/to/preference/json/file`
### Adding A New Test Class
Configures VrCore according to the provided file, e.g. changing the paired
headset. The two most common files used are:
Adding a new test class is similar to adding a test case to an existing class,
but with a few extra steps.
* `//chrome/android/shared_preference_files/test/vr_cardboard_skipdon_setupcomplete.json`
This will pair the device with a Cardboard headset and disable controller
emulation.
* `//chrome/android/shared_preference_files/test/vr_ddview_skipdon_setupcomplete.json`
This will pair the device with a Daydream View headset, set the DON flow to be
skipped, and enable controller emulation. **NOTE** This will prevent you from
using a real controller outside of tests. To fix this, you can either
reinstall VrCore or apply the Cardboard settings file (there isn't currently a
way to manually re-enable real controller use from within the VrCore developer
settings)
#### Determine Whether To Enable Test Parameterization
#### don-enabled
Adding test parameterization will allow tests in the class to be automatically
run in multiple activities that support VR. However, enabling it on a test class
adds a non-trivial amount of runtime overhead, so you should only enable it in
classes where it actually makes sense. In general, this boils down to whether
your test class will have WebVR tests in it or not - WebVR is supported in
multiple activity types, but VR Browsing is only supported in
ChromeTabbedActivity.
`--don-enabled --annotation=Restriction=DON_Enabled`
Whether test parameterization is enabled or not only affects the boilerplate
code in the test class - test cases themselves do not care whether the feature
is enabled or not.
Tells the test runner to allow the running of tests that only work with the DON
flow enabled and limits the set of tests run to those that work with the DON
flow enabled.
##### Non-Parameterized
This should only be used when `--shared-prefs-file` is passed
`//chrome/android/shared_preference_files/test/vr_ddview_don_setupcomplete.json`
as otherwise the DON flow will be disabled.
See `VrShellNavigationTest.java` for an example of how to set up non-parameterized
test classes. In general, you will need to:
## Adding New Tests
* Set `@RunWith` to `ChromeJUnit4ClassRunner.class`
* Define `mVrTestRule` as a `ChromeTabbedActivityXrTestRule`, annotate it with
`@Rule`, and initialize it where it's defined
* Define `mVrTestFramework` as a VrTestFramework and initialize it using
`mVrTestRule` in a setup function annotated with `@Before`
See [adding_new_tests.md][adding_new_tests].
Initializing this in the setup function is necessary since the test framework
needs `mVrTestRule` to be applied, which is guaranteed to have occurred by the
time the setup function is run.
##### Parameterized
See `WebVrTransitionTest.java` for an example of how to set up parameterized
test classes. In general, you will need to:
* Set `@RunWith` to `ParameterizedRunner.class`
* Add `@UseRunnerDelegate` and set it to `ChromeJUnit4RunnerDelegate.class`
* Define `sClassParams`, annotate it with `@ClassParameter`, and set it to the
value returned by `XrTestRuleUtils.generateDefaultTestRuleParameters()`
* Define `mRuleChain` as a `RuleChain` and annotate it with `@Rule`
* Define `mVrTestRule` as a `ChromeActivityTestRule`
* Define `mVrTestFramework` as a `VrTestFramework` and initialize it using
`mVrTestRule` in a setup function annotated with `@Before`
* Define a constructor for your test class that takes a
`Callable<ChromeActivityTestRule>`. This constructor must set `mVrTestRule` to
the callable's `call()` return value and set `mRuleChain` to the return value
of `XrTestRuleUtils.wrapRuleInXrActivityRestrictionRule(mVrTestRule)`
#### Include New Class In Build
Add the new test class to the `java_files` list of the `chrome_test_vr_java`
target in `//chrome/android/BUILD.gn`
[shared_prefs_dir]:
https://chromium.googlesource.com/chromium/src/+/master/chrome/android/shared_preference_files/test
[html_dir]: https://chromium.googlesource.com/chromium/src/+/master/chrome/test/data/xr/e2e_test_files
[vr_test_apks]: https://chromium.googlesource.com/chromium/src/+/master/third_party/gvr-android-sdk/test-apks
[vr_test_libraries]: https://chromium.googlesource.com/chromium/src/+/master/third_party/gvr-android-sdk/test-libraries
[ar_test_apks]: https://chromium.googlesource.com/chromium/src/+/master/third_party/arcore-android-sdk/test-apks
[adding_new_tests]:
https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/adding_new_tests.md
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/adding_new_tests.md 0 → 100644
View file @ bf9f5623
# Adding New XR Instrumentation Tests
## Introduction
This is a brief overview of general steps to adding new XR instrumentation
tests. If you want to add tests as fast as possible, keep reading and glance
through some existing tests, which should give you enough information to start
writing your own.
If you want to better understand what's going on under the hood or why we do
certain things, take a look at
[`xr_instrumentation_deep_dive.md`][xr_instrumentation_deep_dive].
### An Overview Of XR Test Frameworks
Pretty much all XR instrumentation tests with the exception of some VR Browser
tests interact with asynchronous (Promise based) JavaScript code. This is where
the XR Test Frameworks come in, with test classes defining `mXyzTestFramework`
for testing feature Xyz. Together with some JavaScript imports in your test's
HTML file, these allow you to run tests as a series of synchronous steps that
alternate between JavaScript and Java.
For a concrete example, take a look at
[`WebXrVrTransitionTest`][webxr_vr_transition_test]'s
`testNonImmersiveStopsDuringImmersive` test and its corresponding HTML file
[test_non_immersive_stops_during_immersive.html][webxr_vr_transition_test_html].
The general flow in tests will be:
1. Load the HTML file with loadUrlAndAwaitInitialization - this ensures that any
pre-test setup in JavaScript is completed.
2. Run some code on Java's side.
3. Trigger some JavaScript code and wait for it to signal that it is finished.
These can be identified as the `*AndWait` methods, and stop blocking once the
JavaScript side calls `finishJavaScriptStep()`.
4. Repeat from 2 until done.
5. End the test.
## Adding Tests To Existing Test Classes
If you're adding a new test to an existing test class, all the per-class
boilerplate code should be around already, so you can get right to adding a new
test case using the following general components.
### Annotations
The following annotations can be applied before your test body to modify its
behavior.
#### @Test
Every test method must be annotated with the `@Test` annotation in order for the
test runner to identify it as an actual test.
#### Test Length
Every test method must also be annotated with a test length annotation,
typically `@MediumTest`. Eventually, the test length annotations should imply
the presence of `@Test`, but both must currently be present.
#### Supported Activities
Unless your test uses the VR Browser, you can use the `@XrActivityRestriction`
annotation to automatically run your test multiple times in different supported
activities. The currently supported activities are:
* ChromeTabbedActivity (regular Chrome)
* CustomTabActivity (used to open links in apps like GMail)
* WebappActivity (used for Progressive Webapps)
#### @Restriction
You can restrict your test or test class to only be run under certain
circumstances, such as only on Daydream-ready devices or only with the Daydream
View headset paired, using the `@Restriction` annotation.
#### Command Line Flags
You can add or remove command line flags that are set before the test runs using
`@CommandLineFlags.Add` and `@CommandLineFlags.Remove`. Note that if you want to
override a flag set by the test class on a per-test basis, you must remove and
re-add it.
### Test Body
#### HTML Test File
You will likely need an HTML file to load during your test, which should be
placed in `//chrome/test/data/xr/e2e_test_files/html`. The exact contents of
your file will depend on your test, but you will likely be importing some or all
of the following scripts from `//chrome/test/data/xr/e2e_test_files/resources`:
* `webxr_e2e.js`/`webvr_e2e.js` - Sets up the necessary code to communicate back
and forth between Java and JavaScript
* `webxr_boilerplate.js`/`webvr_boilerplate.js` - Handles the WebXR and WebVR
boilerplate code, such as getting an XRDevice and setting up a canvas.
Additionally, in order to use asserts in JavaScript, you must import
`//third_party/WebKit/LayoutTests/resources/testharness.js`.
#### Java Test Body
The exact contents of your test body are going to depend on the test you're
trying to write, so just keep the following guidelines in mind:
* Use the most specific version of a class as possible, e.g. use
`WebXrArTestFramework` for WebXR for AR testing instead of `WebXrTestFramework`.
* If you need to do something that involves the webpage/web contents, it's
likely available through your test framework.
* If you need to do something that doesn't involve the webpage/web contents,
it's likely available in one of the classes in `util/`.
## Adding A New Test Class
If you're adding a new test class instead of just adding a new test to an
existing class, there are a few additional bits of boilerplate code you will
need to add before being able to write your test.
### Test Parameterization
Test parameterization is how running a test multiple times in different
activities is handled. However, it adds some amount of overhead to test runtime,
so it's only enabled where it makes sense. If your new test class will only have
VR Browser-related tests, you can skip parameterization. Otherwise, you'll want
to enable it.
#### Non-Parameterized
See [`VrBrowserTransitionTest`][vr_browser_transition_test] for an example of a
non-parameterized class. The general things you will need to do are:
* Set the class' `@RunWith` to `ChromeJUnit4ClassRunner.class`.
* Declare `mTestRule` as a `ChromeTabbedActivityVrTestRule`, annotate it with
`@Rule`, and initialize it where it's declared.
* Declare `mVrBrowserTestFramework` as a `VrBrowserTestFramework` and initialize
it using `mTestRule` in a setup method annotated with `@Before`. You can do
the same with other types of test frameworks as necessary.
#### Parameterized
See [`WebXrVrTransitionTest`][webxr_vr_transition_test] for an example of a
parameterized class. The general things you will need to are:
* Set `@RunWith` to `ParameterizedRunner.class`.
* Add `@UseRunnerDelegate` and set it to `ChromeJUnit4RunnerDelegate.class`.
* Declare `sClassParams` as a static `List` of `ParameterSet`, annotate it with
`@ClassParameter`, and set it to the value returned by either
`XrTestRuleUtils.generateDefaultTestRuleParameters()` for AR tests or
`VrTestRuleUtils.generateDefaultTestRuleParameters()` for VR tests.
* Declare `mRuleChain` as a `RuleChain` and annotate it with `@Rule`.
* Declare `mTestRule` as a `ChromeActivityTestRule`.
* Declare any necessary test frameworks and initialize them using `mTestRule` in
a setup method annotated with `@Before`.
* Define a constructor for your test class that takes a
`Callable<ChromeActivityTestRule>`. This constructor must set `mVrTestRule` to
the `Callable`'s `call()` return value and set `mRuleChain` to the return
value of `XrTestRuleUtils.wrapRuleInXrActivityRestrictionRule(mTestRule)`
for AR tests or `VrTestRuleUtils.wrapRuleInXrActivityRestrictionRule
(mTestRule)` for VR tests.
### Add The New File
Add the new test class to [`//chrome/android/BUILD.gn`][build_gn]. If it is a VR
test class, it should be added to the `java_files` list of the
`chrome_test_vr_java` `android_library` target. If it is an AR test class, it
should be added to the `java_files` list of the `chrome_test_ar_java`
`android_library` target.
[xr_instrumentation_deep_dive]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/xr_instrumentation_deep_dive.md
[webxr_vr_transition_test]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/WebXrVrTransitionTest.java
[webxr_vr_transition_test_html]: https://chromium.googlesource.com/chromium/src/+/master/chrome/test/data/xr/e2e_test_files/html/test_non_immersive_stops_during_immersive.html
[vr_browser_transition_test]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/VrBrowserTransitionTest.java
[build_gn]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/BUILD.gn
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/rules/README.md deleted 100644 → 0
View file @ 17b91f53
# VR Instrumentation Rules
## Introduction
This directory contains all the Java files related to VR-specific JUnit4 rules.
For the most part, the rules are identical to the standard activity-specific
rules (e.g. `ChromeTabbedActivityTestRule` for `ChromeTabbedActivity`), but with
additional code to work with JUnit4 test parameterization to support running
a test case multiple times in different activities.
Usage of this feature is already covered in `../README.md`, so this
documentation concerns implementation details.
## How It Works
When a test is set up to use test parameterization (determined by whether the
test class is annotated with `@RunWith(ParameterizedRunner.class)` or not), the
test runner automatically runs each test case in the class once with every
`ParameterSet` in the list of `ParameterSet`s that is annotated with
`@ClassParameter`, passing each `ParameterSet`'s contents to the class constructor
each time.
In the case of VR tests, the `ParameterSet` list contains `Callable`s that
construct VR test rules for the three supported activity types
(`ChromeTabbedActivity`, `CustomTabActivity`, and `WebappActivity`). The
constructor for parameterized VR tests runs the provided `Callable`, effectively
running every test case once in each activity type.
However, if the class were to assign the `Callable` return value to its member
variable annotated with `@Rule`, then every test case in the class would always
run in all three activity types, which isn't desirable since some test cases
test scenarios that are only valid in some activities (e.g. tests that involve
the VR browser are currently only supported in ChromeTabbedActivity). To avoid
this, the `Rule` annotated with `@Rule` is actually a `RuleChain` that wraps the
generated VR test rule in a `XrActivityRestrictionRule`.
`XrActivityRestrictionRule` interacts with the `@XrActivityRestriction`
annotation and the VR test rule for the current test case run. If the activity
type of the current rule is contained in the list provided by
`@XrActivityRestriction` (or there is no restriction annotation and the activity
type is `ChromeTabbedActivity`), then the `XrActivityRestrictionRule` becomes a
no-op and the test case runs normally. Otherwise, `XrActivityRestrictionRule`
causes an assumption failure, which is interpreted by the test runner as a
signal to skip that particular test case/parameter combination.
Thus, the end result is that the same test case can be run multiple times in
different activities without having to duplicate any code for use in different
activity types, while still retaining the ability to not run every test in every
activity type if necessary.
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/vr_test_framework.md deleted 100644 → 0
View file @ 17b91f53
# VR Test Framework
## Introduction
This is an overview of how the VR test framework functions. It is split into two
parts, with Java code located in this directory in `VrTestFramework.java` and
JavaScript code located in `//chrome/test/data/vr/e2e_test_files/`.
It is primarily used for testing WebVR, but can also be useful for testing the
VR Browser.
A similar approach is used for testing VR on desktop, with the framework
re-implemented in C++ for use in browser tests and many of the same JavaScript
files used on both platforms. See the documentation in
`//chrome/browser/vr/test` for more information on this.
## Structure
Tests utilizing this framework are split into separate Java and JavaScript files
since WebVR interaction is done via JavaScript, but instrumentation tests are
Java-based. In general, the JavaScript code handles any interactions with the
WebVR API, and the Java code handles everything else (user gestures, controller
emulation, etc.).
In general, the flow of a test is:
* In Java, load the HTML test file, which:
* Loads the WebVR boilerplate code and test code
* Sets up any test steps that need to be triggered by Java as separate
functions
* Creates an asynchronous test (denoted t here)
* Repeat:
* Run any necessary Java-side code, e.g. trigger a user action
* Trigger the next JavaScript test step and wait for it to finish
* Finally, call t.done() in JavaScript and endTest in Java
### JavaScript
The JavaScript test code mainly makes use of testharness.js, which is also used
for layout tests. This allows the use of asserts in JavaScript code, and any
assert failures will propagate up to Java. There are four main test-specific
functions that you will need to use:
#### async\_test
This creates an asynchronous test from testharness.js which you will use
throughout the JavaScript code. It serves two purposes:
* By being an asynchronous test, it will prevent testharness.js from ending the
test run prematurely once the page is loaded. It will wait until all tests
are done before checking results.
* Enables you to use asserts in JavaScript. Asserts must be within a test to
do anything.
#### finishJavaScriptStep
This signals that the current portion of JavaScript code is done and that the
Java side can continue execution.
#### t.step
This defines a test step in an asynchronous test t. Any asserts must be within
a test step, so assertions will look along the lines of:
`t.step( () => {
assertTrue(someBool);
});`
#### t.done
This signals that the asynchronous test is done. Once all tests in a file are
completed (usually only one), testharness.js will check the results and
automatically call finishJavaScriptStep when done.
### Java
There are many Java-side functions that enable VR testing, but only the three
basic ones will be covered here. For specifics about less common functions, see
the JavaDoc comments in `VrTestFramework.java` and the utility classes in
`util/`.
#### loadUrlAndAwaitInitialization
This is similar to the standard loadUrl method in Chrome, but also waits until
all initialization steps are complete and the page is ready for testing. By
default, this only includes the page being loaded. Additionally, any files
that include the WebVR boilerplate script (generally any test for WebVR) will
wait until the promise returned by `navigator.getVRDisplays` has resolved or
rejected. Tests can add their own initialization steps by adding an entry to
the `initializationSteps` dictionary defined in
`//chrome/test/data/android/webvr_instrumentation/resources/webvr_e2e.js` with
the value set to `false`. Once the step is done, simply set the value to
`true`.
#### executeStepAndWait
This executes the given JavaScript and waits until finishJavaScriptStep is
called on the JavaScript side.
#### endTest
Performs any post-test checks after the JavaScript test code has finished
running. Since test failures are caught after each step, all this really does
is ensure that the the JavaScript code has actually finished running, throwing
an error if this is not the case.
Typically called at the very end of the test after everything else is
completed.
\ No newline at end of file
chrome/android/javatests/src/org/chromium/chrome/browser/vr/xr_instrumentation_deep_dive.md 0 → 100644
View file @ bf9f5623
# XR Instrumentation Test Deep Dive
## Introduction
This documentation aims to provide a more in-depth view of how various aspects
of the XR instrumentation tests work under the hood and explain why certain
design decisions were made. If you just want an overview on how to build/run the
tests or add new ones, see [`README.md`][readme] and
[`adding_new_tests.md`][adding_new_tests].
## Naming Convention
Classes, files, and variables are named using the following (hopefully
self-explanatory) naming convention:
* `XR`/`Xr` - The most general term. Used for things that are applicable to any
XR feature.
* `VR`/`Vr` - A subset of `XR`. Used for things that are applicable to any VR
feature.
* `AR`/`Ar` - A subset of `XR`. Used for things that are applicable to any AR
feature.
* `VR Browser`/`VrBrowser` - A subset of `VR`. Used for things that are only
applicable to the VR Browser feature of Chrome.
* `WebXR`/`WebXr` - A subset of `XR`. Used for things that are applicable to any
XR-related web APIs.
* `WebXR for AR`/`WebXrAr` - A subset of `WebXR` and `AR`. Used for things that
are applicable to the `WebXR Devices API` in its AR usecase.
* `WebXR for VR`/`WebXrVr` - A subset of `WebXR` and `VR`. Used for things that
are applicable to the `WebXR Devices API` in its VR usecase.
* `WebVR`/`WebVr` - A subset of `WebXrVr`. Used for things that are only
applicable to the older `WebVR` API.
## Test Framework Structure
### Hierarchy
Based on the above naming scheme, the various `TestFramework` classes are
structured in the following hierarchy:
* `XrTestFramework`
* `VrBrowserTestFramework`
* `WebXrTestFramework`
* `WebXrArTestFramework`
* `WebXrVrTestFramework`
* `WebVrTestFramework`
### Static vs. Non-Static Methods
Most methods in the `TestFramework` classes have both a static and non-static
version, with the non-static version simply calling the static one with the
framework's `mFirstTabWebContents` reference. This is because the vast majority
of use cases are interacting with the web page that is in the tab automatically
opened at the start of the test, but some rare cases require interacting with
other tabs. Thus, we need to provide a way of using the frameworks with
arbitrary tabs/`WebContents` (the static methods), but offering the non-static
versions cuts down the clutter of calling `getFirstTabWebContents()` everywhere.
## Parameterization
Parameterization is a JUnit4 concept. You can read the official guide about it
on the [JUnit4 GitHub Wiki][junit4_wiki_parameterization], but the TL;DR is that
it allows a test method or class to be automatically run multiple times with
varying inputs.
When used in XR tests, the `List` of `ParameterSet`s annotated with
`@ClassParameter` is what will be iterated over. Specifically,
`XrTestRuleUtils.generateDefaultTestRuleParameters` or
`VrTestRuleUtils.generateDefaultTestRuleParameters` will generate a `List` of
`ParamaterSet`s each containing a single `Callable` whose `call()` returns a
`ChromeActivityTestRule`. Each `ParameterSet` corresponds to one of the activity
types that XR features are supported in. This is why constructors of
parameterized test classes must accept a `Callable<ChromeActivityTestRule>` -
they will be called once per element of the `ParameterSet` `List` and passed the
contents of that element.
`Callable` is used instead of `ChromeActivityTestRule` directly due to an
implementation detail in `ParameterSet` ([source code][parameter_set_source]).
`ParameterSet` only accepts primitives, `Callable`s, and a certain set of
`Class`es such as `String`, which means we can't pass a `ChromeActivityTestRule`
directly to a `ParameterSet`.
## Rules
Rules are another JUnit4 concept. You can read a fairly comprehensive guide
about them on the [JUnit4 GitHub Wiki][junit4_wiki_rules], but the TL;DR of them
is that they allow pre- and post-test code to be easily shared across multiple
test classes. Any rule annotated with `@Rule` will be automatically applied to
the test in an arbitrary order.
### Activity Restriction Rule
`XrActivityRestrictionRule` and the `@XrActivityRestriction` annotation are what
allow tests to only be run in the activities they support, as otherwise
parameterization would cause all tests to be run in all activities. The method
for this is simple - if the activity type for the current
`ChromeActivityTestRule` is in the list of supported activities provided by
`@XrActivityRestriction`, continue running the test as normal. Otherwise, don't
run the test, and instead, generate a statement that throws an assumption
failure, which the test runner treats as a signal that the test was skipped.
#### RuleChain
Every place where `XrActivityRestrictionRule` is used makes use of a `RuleChain`
and `XrTestRuleUtils.wrapRuleInXrActivityRestrictionRule()`. The reason for this
is simply optimization. By using a `RuleChain` to wrap a given
`ChromeActivityTestRule` in an `XrActivityRestrictionRule`, we can ensure that
the decision to skip a test due to being unsupported in an activity is made
before we go through the (slow) process of starting said activity.
### XR And VR Test Rules
XR instrumentation tests use special versions of `ChromeActivityTestRule` that
implement `XrTestRule` or `VrTestRule`.
Rules that implement `XrTestRule` simply open the specified activity type to a
blank page and implement a method that allows it to work with
`XrActivityRestrictionRule`.
Rules that implement `VrTestRule` do the same, but also perform some additional
VR-specific setup such as ensuring that the test is not started in VR and
allowing the use of the experimental/broken VrCore head tracking service.
## VR Controller Input
There are currently two ways of injecting Daydream controller input into tests,
each with their own pros and cons.
### EmulatedVrController
The `EmulatedVrController` class is the older of the two approaches and works by
setting VrCore to accept Android `Intent`s as controller input instead of using
an actual controller.
The main benefit of this is that it allows complete end-to-end
testing of controller-related Chrome code, as it still receives controller input
from VrCore the same as if a real controller was in use. It also allows the use
of the home button to recenter the view or go to Daydream Home.
A downside to this downside is that `Intent`s do not allow for precise timing,
leading to flakiness. Additionally, since we're essentially sending raw
quaternions to VrCore to turn into the controller's orientation, pointing at
specific UI elements is tedious and prone to breaking if the UI changes.
### NativeUiUtils
The methods in the `NativeUiUtils` class work by causing Chrome to start reading
controller input from a test-only queue instead of getting data from VrCore.
The downsides to this are that it causes Chrome to use some non-production code
and doesn't test the controller-related interaction with VrCore.
This approach does have quite a few benefits though. First it should not be
flaky while also being faster since everything is handled Chrome-side.
Additionally, it allows interaction with specific UI elements by name, which is
both easier and less prone to breaking than specifying a position in space.
[readme]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/README.md
[adding_new_tests]: https://chromium.googlesource.com/chromium/src/+/master/chrome/android/javatests/src/org/chromium/chrome/browser/vr/adding_new_tests.md
[junit4_wiki_parameterization]: https://github.com/junit-team/junit4/wiki/parameterized-tests
[parameter_set_source]: https://chromium.googlesource.com/chromium/src/+/master/base/test/android/javatests/src/org/chromium/base/test/params/ParameterSet.java
[junit4_wiki_rules]: https://github.com/junit-team/junit4/wiki/rules
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment