Commit 0dd83ea7 authored by Stephen McGruer's avatar Stephen McGruer Committed by Chromium LUCI CQ

[blinkpy] Limit WPT manifest generation to <= 25 processes on Windows

We are seeing issues on Windows + Python 3 where a high number of
processes passed to multiprocessing.Pool causes the Pool to crash and
manifest generation to hang. (Very similar or identical to
https://bugs.python.org/issue40263).

This is a temporary local patch to Blink's copy of WPT tooling, in order
to land a workaround as quickly as possible. A proper upstream fix in
WPT should be landed soon, so that we do not have to carry this patch
for too long. Care should be taken in the meantime not to override this
fix if/when we roll Blink tooling.

Bug: 1160108
Change-Id: I8eda8d65caa8cd295ae444676f4628832225f55d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2601399Reviewed-by: default avatarThomas Guilbert <tguilbert@chromium.org>
Reviewed-by: default avatarRakib Hasan <rmhasan@google.com>
Reviewed-by: default avatarDale Curtis <dalecurtis@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Auto-Submit: Stephen McGruer <smcgruer@chromium.org>
Cr-Commit-Position: refs/heads/master@{#838996}
parent 79a4138f
import io import io
import itertools import itertools
import os import os
import sys
from atomicwrites import atomic_write from atomicwrites import atomic_write
from copy import deepcopy from copy import deepcopy
from multiprocessing import Pool, cpu_count from multiprocessing import Pool, cpu_count
...@@ -224,10 +225,21 @@ class Manifest(object): ...@@ -224,10 +225,21 @@ class Manifest(object):
changed = True changed = True
if parallel and len(to_update) > 25 and cpu_count() > 1: if parallel and len(to_update) > 25 and cpu_count() > 1:
# On Windows desktops with high cpu count, we are seeing issues
# with high CPU counts and multiprocessing. This is a temporary
# workaround until we add logic to WPT to select a max number of
# processes. See https://crbug.com/1160108
#
# 25 was experimentally determined to work in
# https://crbug.com/1160108#c22
processes = cpu_count()
if sys.platform == "win32" and processes > 25:
processes = 25
# 25 derived experimentally (2020-01) to be approximately # 25 derived experimentally (2020-01) to be approximately
# the point at which it is quicker to create Pool and # the point at which it is quicker to create Pool and
# parallelize this # parallelize this
pool = Pool() pool = Pool(processes)
# chunksize set > 1 when more than 10000 tests, because # chunksize set > 1 when more than 10000 tests, because
# chunking is a net-gain once we get to very large numbers # chunking is a net-gain once we get to very large numbers
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment