Commit 81dfeb7b authored by brucedawson's avatar brucedawson Committed by Commit bot

Change ExecLinkWrapper to not buffer all tool output

/verbose linking of chrome.dll creates over one GB of output. This
causes ExecLinkWrapper to consume over two GB of memory which leads to
an OOM failure in the 32-bit depot_tools python, and the loss of all
of the valuable output. This change modifies ExecLinkWrapper to
process the output one line at a time, thus avoiding the OOM.

I've tested that this handles the 1.1 GB of output which the previous
version of this function failed on and I've visually confirmed that the
output looks the same - no extraneous blank lines, for instance, when
displaying warnings, errors, or 1.9 million lines of verbose output.

I also verified that the script stays idle when waiting for output -
blocking on .readline().

This fixes the previous attempt which omitted the vital link.wait() call
which meant that error codes were not propagated.

R=scottmg@chromium.org
BUG=672182,672841

Review-Url: https://codereview.chromium.org/2568563002
Cr-Commit-Position: refs/heads/master@{#437696}
parent 9b3a179c
......@@ -133,13 +133,14 @@ class WinTool(object):
# non-Windows don't do that there.
link = subprocess.Popen(args, shell=sys.platform == 'win32', env=env,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, _ = link.communicate()
for line in out.splitlines():
# Read output one line at a time as it shows up to avoid OOM failures when
# GBs of output is produced.
for line in link.stdout:
if (not line.startswith(' Creating library ') and
not line.startswith('Generating code') and
not line.startswith('Finished generating code')):
print line
return link.returncode
print line,
return link.wait()
def ExecLinkWithManifests(self, arch, embed_manifest, out, ldcmd, resname,
mt, rc, intermediate_manifest, *manifests):
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment