Commit def1cd0d authored by zhaoyangli's avatar zhaoyangli Committed by Commit Bot

[iOS][test runner] Do not report cancelled statuses as failed tests in test results.

When TESTS_DID_NOT_START or BUILD_INTERRUPTED happened in the an attempt
and some tests didn't run due to these statuses, the tests will be
retried in next attempt if there is one attempt in our test runner.

These cancellations are listed as failed tests in the attempt result
and will be finally copied into standard test result json, even though
all affected tests succeed in following attempts. This will cause a
failure in Chromium recipe. See the bug for details.

Since this change, we don't mark these in test results. Tests affected
by these statuses are already marked as timeout in standard test
results so this won't let go any failures.

Bug: 1073491
Change-Id: Id031d3fa1b7ed61bf61db2887ed8037df4244d0c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2161930Reviewed-by: default avatarJustin Cohen <justincohen@chromium.org>
Commit-Queue: Zhaoyang Li <zhaoyangli@chromium.org>
Cr-Commit-Position: refs/heads/master@{#761993}
parent ff8690eb
......@@ -216,16 +216,25 @@ class LaunchCommand(object):
LOGGER.info('Failure for passed tests %s: %s' % (status, failure))
break
self._log_parser.copy_screenshots(outdir_attempt)
# If tests are not completed(interrupted or did not start)
# re-run them with the same number of shards,
# otherwise re-run with shards=1 and exclude passed tests.
cancelled_attempt = cancelled_statuses.intersection(
self.test_results['attempts'][-1]['failed'].keys())
# Item in cancelled_statuses is used to config for next attempt. The usage
# should be confined in this method. Real tests affected by these statuses
# will be marked timeout in results.
for status in cancelled_statuses:
self.test_results['attempts'][-1]['failed'].pop(status, None)
if (not cancelled_attempt
# If need to re-run less than 20 tests, 1 shard should be enough.
or (len(running_tests) - len(self.egtests_app.excluded_tests)
<= MAXIMUM_TESTS_PER_SHARD_FOR_RERUN)):
shards = 1
self.summary_log()
return {
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment