chromeos: Add a doc for debugging test failures on Chrome OS.

Add a link to that doc in Tast test failure log snippets.

Will make failure snippets look like:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8904924613768986736/+/steps/chrome_all_tast_tests__with_patch__on_ChromeOS/0/logs/Deterministic_failure:_session.LogoutCleanup__status_FAILURE_/0

Bug: 923426
Change-Id: Id7d19fa94126a013763e6671cfc0cd8f6263c71b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1759234
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Mitsuru Oshima <oshima@chromium.org>
Cr-Commit-Position: refs/heads/master@{#692275}

build/chromeos/test_runner.py

@@ -50,6 +50,8 @@ SANITY_TEST_TARGET = 'cros_browser_sanity_test'
 # depending on which lab machine you're on.
 LAB_DUT_HOSTNAME = 'variable_chromeos_device_hostname'
 
+TAST_DEBUG_DOC = 'https://bit.ly/2LgvIXz'
+
 
 class TestFormatError(Exception):
   pass
@@ -349,6 +351,9 @@ class TastTest(RemoteTest):
         # https://godoc.org/chromium.googlesource.com/chromiumos/platform/tast.git/src/chromiumos/tast/testing#Error
         for err in errors:
           error_log += str(err['stack']) + '\n'
+      error_log += (
+           "\nIf you're unsure why this test failed, consult the steps "
+           'outlined in\n%s\n' % TAST_DEBUG_DOC)
       base_result = base_test_result.BaseTestResult(
           test['name'], result, duration=duration_ms, log=error_log)
       suite_results.AddResult(base_result)

docs/testing/chromeos_debugging_tips.md

+# Chrome OS Debugging Instructions
+Chrome on Chrome OS is tested using a handful of frameworks, each of which
+you'll find running on Chrome's CQ and waterfalls. If you're investigating
+failures in these tests, below are some tips for debugging and identifying the
+cause.
+
+*** note
+
+This doc outlines tests running in true Chrome OS environments (ie: on virtual
+machines or real devices). [linux-chromeos] tests, on the other hand, can be
+debugged like any other linux test.
+***
+
+## Tast
+
+[Tast] is Chrome OS's integration testing framework. Since Chrome itself is
+instrumental to the Chrome OS system, it's equally important that we run some
+of these integration tests on Chrome's waterfalls. If you find one of these
+tests failing (likely in the `chrome_all_tast_tests` step), you can:
+
+- **Inspect the failed test's log snippet**: There should be a log link for
+each failed test with failure information. eg: For this [failed build], opening
+the [platform.Histograms] log link contains stack traces and error messages.
+
+- **View browser & system logs**: A common cause of failure on Chrome's builders
+are browser crashes. When this happens, each test's log snippets will simply
+contain warnings like "[Chrome probably crashed]". To debug these crashes,
+navigate to the test's Isolated output, most likely listed in the build under
+the test step's [shard #0 isolated out] link. From there, view the various
+`log/chrome/...` or `log/ui/...` text files and you should find some with
+browser [crashes and stack traces].
+
+To disable a test on Chrome's builders, the preferred method is to add the
+`informational` attribute to the test's definition (see [Tast attributes] for
+more info). Note that this requires a full Chrome OS checkout. If that's not an
+option, or if it needs to be disabled ASAP, you can add it to the list of
+disabled tests for the step's GN target. For example, to disable a test in the
+`chrome_all_tast_tests` step, add it to [this list].
+
+## Telemetry
+
+>TODO: Add instructions for debugging telemetry failures.
+
+## GTest
+
+>TODO: Add instructions for debugging GTest failures.
+
+## Rerunning these tests locally
+
+>TODO: Add instructions for rerunning these tests locally.
+
+
+[linux-chromeos]: https://chromium.googlesource.com/chromium/src/+/master/docs/chromeos_build_instructions.md
+[Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md
+[failed build]: https://ci.chromium.org/p/chromium/builders/ci/chromeos-kevin-rel/14102
+[platform.Histograms]: https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8904949911599004400/+/steps/chrome_all_tast_tests_on_ChromeOS/0/logs/Deterministic_failure:_platform.Histograms__status_FAILURE_/0
+[Chrome probably crashed]: https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8905974915785988832/+/steps/chrome_all_tast_tests__retry_shards_with_patch__on_ChromeOS/0/logs/Deterministic_failure:_ui.ChromeLogin__status_FAILURE_/0
+[shard #0 isolated out]: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=fd1f6d76b076f07cc98fa7b2e0c0097f35c51cd0
+[crashes and stack traces]: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=993d58ff48bb08071d951bd8e103fa5a3c03efb1&as=chrome_20190805-044653
+[Tast attributes]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md
+[this list]: https://codesearch.chromium.org/chromium/src/chromeos/BUILD.gn?rcl=7b0393a9091fd02edc9ae773739124f7be5a0782&l=242

docs/testing/testing_in_chromium.md

@@ -34,6 +34,7 @@ for one platform.
     It allows you to perform arbitrary actions on a set of web pages and
     report metrics about it.
 *   **[Fuzzer Tests]** is used to uncover potential security & stability problems in Chromium.
+*   **[Tast]** is a test framework for system integration tests on Chrome OS.
 
 
 The following table shows which types of test works on which platforms.
@@ -44,10 +45,11 @@ The following table shows which types of test works on which platforms.
 | Junit(Java)                 |         |         |         | √ |         |         |
 | Browser Tests(C++)          | √ | √ | √ | √ |         |         |
 | Web Tests(HTML, JS)         | √ | √ | √ |         |         |         |
-| Telemetry(Python)           | √ | √ | √ | √ |         | ?   |
+| Telemetry(Python)           | √ | √ | √ | √ |         | √ |
 | Instrumentation Tests(Java) |         |         |         | √ |         |         |
 | EarlGrey                    |         |         |         |         | √ |         |
 | Fuzzer Tests(C++)           | √ | √ | √ | √ |         | √ |
+| Tast(Golang)                |         |         |         |         |         | √ |
 
 *** note
 **Browser Tests Note**
@@ -68,19 +70,6 @@ Web Tests were enabled on Android K before, but it is disabled on Android platfo
 see [this thread](https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/338WKwWPbPI/discussion) for more context.
 ***
 
-*** note
-**Telemetry**
-
-Telemetry works on CrOS, but it is not officially supported yet.
-***
-
-*** note
-**CrOS Note**
-
-There is an on going effort to enable existing Chromium tests on CrOS VM hosted on Linux,
-right now only unit tests are enabled on CrOS VM.
-***
-
 ## General Principles
 
 *   All the tests in Chromium running on CQ and main waterfall should be hermetic and stable.
@@ -154,6 +143,7 @@ Go to [code coverage dashboard](https://analysis.chromium.org/p/chromium/coverag
 
 ## How to debug tests
 *  [Android Debugging Instructions]
+*  [Chrome OS Debugging Tips]
 *  [Debugging Web Tests]
 
 ## How to deal with flaky tests
@@ -173,6 +163,7 @@ then fix it later. [How do I disable a flaky test] is the instruction about how
 [EarlGrey]: https://github.com/google/EarlGrey
 [Telemetry]: https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md
 [Fuzzer Tests]: https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/README.md
+[Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md
 [Web Tests]: ./web_tests.md
 [crbug/611756]: https://bugs.chromium.org/p/chromium/issues/detail?id=611756
 [Flaky portal]: https://analysis.chromium.org/p/chromium/flake-portal
@@ -180,6 +171,7 @@ then fix it later. [How do I disable a flaky test] is the instruction about how
 [Telemetry: Run benchmarks locally]: https://chromium.googlesource.com/catapult/+/HEAD/telemetry/docs/run_benchmarks_locally.md
 [Run fuzz target locally]: https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/getting_started.md#build-and-run-fuzz-target-locally
 [Android Debugging Instructions]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/android_debugging_instructions.md
+[Chrome OS Debugging Tips]: ./chromeos_debugging_tips.md
 [Debugging Web Tests]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/web_tests.md#Debugging-Web-Tests
 [code coverage dashboard]: https://analysis.chromium.org/p/chromium/coverage
 [How do I disable a flaky test]: https://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-How-do-I-disable-a-flaky-test-