Commit 95e22d39 authored by Stephen Martinis's avatar Stephen Martinis Committed by Commit Bot

Reland "Parallelize fetching telemetry benchmark deps"

This is a reland of 8ab0b856

Original change's description:
> Parallelize fetching telemetry benchmark deps
> 
> The fetching script currently fetches about 2 GB (on my linux workstation)
> from cloud storage serially. This can (and is) trivially parallelizable, and
> should speed up the fetching script significantly.
> 
> This script has been taking 20 minutes on mac machines for a long time. This
> will hopefully significantly speed the mac builders up.
> 
> Bug: 1102597
> Change-Id: I93d2fa2b1f4b4ef6e27e9c894a9dd4c0d0e12d7b
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2285503
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Reviewed-by: Wenbin Zhang <wenbinzhang@google.com>
> Reviewed-by: John Chen <johnchen@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#786521}

Bug: 1102597
Change-Id: I37d0055fadb26bf4eae2b0b5fe0ecc4732edde3d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2288740
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: default avatarJohn Chen <johnchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#789736}
parent 6a7ffec3
...@@ -7,10 +7,11 @@ ...@@ -7,10 +7,11 @@
import argparse import argparse
import json import json
import logging
import multiprocessing
import optparse import optparse
import os import os
import sys import sys
import logging
from chrome_telemetry_build import chromium_config from chrome_telemetry_build import chromium_config
from core import benchmark_finders from core import benchmark_finders
...@@ -121,8 +122,12 @@ def main(args): ...@@ -121,8 +122,12 @@ def main(args):
raw_input( raw_input(
'No benchmark name is specified. Fetching all benchmark deps. ' 'No benchmark name is specified. Fetching all benchmark deps. '
'Press enter to continue...') 'Press enter to continue...')
for b in benchmark_finders.GetOfficialBenchmarks(): benchmarks = benchmark_finders.GetOfficialBenchmarks()
deps[b.Name()] = _FetchDepsForBenchmark(b) p = multiprocessing.Pool()
results = p.map(_FetchDepsForBenchmark, benchmarks)
p.terminate()
for benchmark, result in zip(benchmarks, results):
deps[benchmark.Name()] = result
if options.output_deps: if options.output_deps:
with open(options.output_deps, 'w') as outfile: with open(options.output_deps, 'w') as outfile:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment