Commit 532b3f34 authored by Dirk Pranke's avatar Dirk Pranke Committed by Commit Bot

Clarify location of test_artifacts in json_test_results_format.

The existing documentation for where to put any artifacts produced by a
test wasn't completely clear, so this adds some additional text to
hopefully make it so. Namely, paths should be specified relative to the
location of the results file, which normally is in $ISOLATED_OUTDIR,
meaning that the paths will also be relative to $ISOLATED_OUTDIR).


Change-Id: I7f03b9f31e7141051a4c783c383547e9dd27d286
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2426975
Commit-Queue: Dirk Pranke <dpranke@google.com>
Reviewed-by: default avatarDirk Pranke <dpranke@google.com>
Reviewed-by: default avatarStephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#810021}
parent 3ff4a0bc
......@@ -112,7 +112,7 @@ then is retried and passes, both `is_regression` and `is_unexpected` will be fal
|-------------|-----------|-------------|
| `actual` | string | **Required.** An ordered space-separated list of the results the test actually produced. `FAIL PASS` means that a test was run twice, failed the first time, and then passed when it was retried. If a test produces multiple different results, then it was actually flaky during the run. |
| `expected` | string | **Required.** An unordered space-separated list of the result types expected for the test, e.g. `FAIL PASS` means that a test is expected to either pass or fail. A test that contains multiple values is expected to be flaky. |
| `artifacts` | dict | **Optional.** A dictionary describing test artifacts generated by the execution of the test. The dictionary maps the name of the artifact (`screenshot`, `crash_log`) to a list of relative locations of the artifact (`screenshot/page.png`, `logs/crash.txt`). Any '/' characters in the file paths are meant to be platform agnostic; tools will replace them with the appropriate per platform path separators. There is one entry in the list per test execution. If `artifact_permanent_location` is specified, then this location is relative to that path. Otherwise, the path is assumed to be relative to the location of the json file which contains this. |
| `artifacts` | dict | **Optional.** A dictionary describing test artifacts generated by the execution of the test. The dictionary maps the name of the artifact (`screenshot`, `crash_log`) to a list of relative locations of the artifact (`screenshot/page.png`, `logs/crash.txt`). Any '/' characters in the file paths are meant to be platform agnostic; tools will replace them with the appropriate per platform path separators. There is one entry in the list per test execution. If `artifact_permanent_location` is specified, then this location is relative to that path. Otherwise, the path is assumed to be relative to the location of the json file which contains this (i.e., `$ISOLATED_OUTDIR`). The actual locations of artifacts are implementation-defined by the test program and can follow any convention, since these entries will allow them to be looked up easily. |
| `bugs` | string | **Optional.** A comma-separated list of URLs to bug database entries associated with each test. |
| `shard` | int | **Optional.** The 0-based index of the shard that the test ran on, if the test suite was sharded across multiple bots. |
| `is_flaky` | bool | **Optional.** If present and true, the test was run multiple times and produced more than one kind of result. If false (or if the key is not present at all), the test either only ran once or produced the same result every time. |
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment