Commit 0dd83ea7 authored by Stephen McGruer's avatar Stephen McGruer Committed by Chromium LUCI CQ

[blinkpy] Limit WPT manifest generation to <= 25 processes on Windows

We are seeing issues on Windows + Python 3 where a high number of
processes passed to multiprocessing.Pool causes the Pool to crash and
manifest generation to hang. (Very similar or identical to
https://bugs.python.org/issue40263).

This is a temporary local patch to Blink's copy of WPT tooling, in order
to land a workaround as quickly as possible. A proper upstream fix in
WPT should be landed soon, so that we do not have to carry this patch
for too long. Care should be taken in the meantime not to override this
fix if/when we roll Blink tooling.

Bug: 1160108
Change-Id: I8eda8d65caa8cd295ae444676f4628832225f55d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2601399Reviewed-by: default avatarThomas Guilbert <tguilbert@chromium.org>
Reviewed-by: default avatarRakib Hasan <rmhasan@google.com>
Reviewed-by: default avatarDale Curtis <dalecurtis@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Auto-Submit: Stephen McGruer <smcgruer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#838996}
parent 79a4138f
import io
import itertools
import os
import sys
from atomicwrites import atomic_write
from copy import deepcopy
from multiprocessing import Pool, cpu_count
......@@ -224,10 +225,21 @@ class Manifest(object):
changed = True
if parallel and len(to_update) > 25 and cpu_count() > 1:
# On Windows desktops with high cpu count, we are seeing issues
# with high CPU counts and multiprocessing. This is a temporary
# workaround until we add logic to WPT to select a max number of
# processes. See https://crbug.com/1160108
#
# 25 was experimentally determined to work in
# https://crbug.com/1160108#c22
processes = cpu_count()
if sys.platform == "win32" and processes > 25:
processes = 25
# 25 derived experimentally (2020-01) to be approximately
# the point at which it is quicker to create Pool and
# parallelize this
pool = Pool()
pool = Pool(processes)
# chunksize set > 1 when more than 10000 tests, because
# chunking is a net-gain once we get to very large numbers
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment