Add tool to help analyze binary size

BUG=

Review URL: https://codereview.chromium.org/119083006

git-svn-id: svn://svn.chromium.org/chrome/trunk/src@245405 0039d316-1c4b-4281-b951-d872f2087c98
parent 21580a91
...@@ -183,5 +183,13 @@ ...@@ -183,5 +183,13 @@
'dependencies': [ 'dependencies': [
], ],
}, },
{
# Build the java portions of the binary size analysis tool.
'target_name': 'binary_size_tool',
'type': 'none',
'dependencies': [
'../tools/binary_size/binary_size.gyp:binary_size_java',
],
},
], # targets ], # targets
} }
================================================================================
__________ .__
\______ \ |__| ____ _____ _______ ___.__.
| | _/ | | / \ \__ \ \_ __ \ < | |
| | \ | | | | \ / __ \_ | | \/ \___ |
|______ / |__| |___| / (____ / |__| / ____|
\/ \/ \/ \/
_________ .__ ___________ .__
/ _____/ |__| ________ ____ \__ ___/ ____ ____ | |
\_____ \ | | \___ / _/ __ \ | | / _ \ / _ \ | |
/ \ | | / / \ ___/ | | ( <_> ) ( <_> ) | |__
/_______ / |__| /_____ \ \___ > |____| \____/ \____/ |____/
\/ \/ \/
================================================================================
--------------------------------------------------------------------------------
Introduction
--------------------------------------------------------------------------------
The ever-increasing size of binaries is a problem for everybody. Increased
binary size means longer download times and a bigger on-disk footprint after
installation. Mobile devices suffer the worst, as they frequently have
sub-optimal connectivity and limited storage capacity. Developers currently
have almost no visibility into how the space in the existing binaries is
divided nor how their contributions change the space within those binaries.
The first step to reducing the size of binaries is to make the size information
accessible to everyone so that developers can take action.
The Binary Size Tool does the following:
1. Runs "nm" on a specified binary to dump the symbol table
2. Runs a parallel pool of "addr2line" processes to map the symbols from the
symbol table back to source code (way faster than running "nm -l")
3. Creates (and optionally saves) an intermediate file that accurately mimcs
(binary compatible with) the "nm" output format, with all the source code
mappings present
4. Parses, sorts and analyzes the results
5. Generates an HTML-based report in the specified output directory
--------------------------------------------------------------------------------
How to Run
--------------------------------------------------------------------------------
Running the tool is fairly simply. For the sake of this example we will
pretend that you are building the Content Shell APK for Android.
1. Build your product as you normally would, e.g.:
ninja -C out/Release -j 100 content_shell_apk
2. Build the "binary_size_tool" target from ../../build/all_android.gyp, e.g.:
ninja -C out/Release binary_size_tool
3. Run the tool specifying the library and the output report directory.
This command will run the analysis on the Content Shell native library for
Android using the nm/addr2line binaries from the Android NDK for ARM,
producing an HTMl report in /tmp/report:
tools/binary_size/run_binary_size_analysis.py \
--library out/Release/lib/libcontent_shell_content_view.so \
--destdir /tmp/report \
--arch android-arm
Of course, there are additional options that you can see by running the tool
with "--help".
This whole process takes about an hour on a modern (circa 2014) quad-core
machine. If you have LOTS of RAM, you can use the "--jobs" argument to
add more addr2line workers; doing so will *greatly* reduce the processing time
but will devour system memory. If you've got the horsepower, 10 workers can
thrash through the binary in about 5 minutes at a cost of around 60 GB of RAM.
--------------------------------------------------------------------------------
Analyzing the Output
--------------------------------------------------------------------------------
When the tool finishes its work you'll find an HTML report in the output
directory that you specified with "--destdir". Open the index.html file in your
*cough* browser of choice *cough* and have a look around. The index.html page
is likely to evolve over time, but will always be your starting point for
investigation. From here you'll find links to various views of the data such
as treemap visualizations, overall statistics and "top n" lists of various
kinds.
The report is completely standalone. No external resources are required, so the
report may be saved and viewed offline with no problems.
--------------------------------------------------------------------------------
Caveats
--------------------------------------------------------------------------------
The tool is not perfect and has several shortcomings:
* Not all space in the binary is accounted for. The cause is still under
investigation.
* The tool is partly written in Java, temporarily tying it to Android
purely and solely because it needs Java build logic which is only defined
in the Android part of the build system. The Java portions need to be
rewritten in Python so we can decouple from Android, or we need to find
an alternative (readelf, linker maps, etc) to running nm/addr2line.
* The Java code assumes that the library file is within a Chromium release
directory. This limits it to Chromium-based binaries only.
* The Java code has a hack to accurately identify the source of ICU data
within the Chromium source tree due to missing symbols in the ICU ASM
output.
* The Python script assumes that arm-based and mips-based nm/addr2line
binaries exist in ../../third_party. This is true only when dealing with
Android and again limits the tool to Chromium-based binaries.
* The Python script uses build system variables to construct the classpath
for running the Java code.
* The Javascript code in the HTML report Assumes code lives in Chromium for
generated hyperlinks and will not hyperlink any file that starts with the
substring "out".
--------------------------------------------------------------------------------
Feature Requests and Bug Reports
--------------------------------------------------------------------------------
Please file bugs and feature requests here, making sure to use the label
"Binary-Size-Tool":
https://code.google.com/p/chromium/issues/entry?labels=Binary-Size-Tool
View all open issues here:
https://code.google.com/p/chromium/issues/list?can=2&q=label:Binary-Size-Tool
\ No newline at end of file
# Copyright 2014 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
{
'targets': [
{
'target_name': 'binary_size_java',
'type': 'none',
'variables': {
'java_in_dir': '../binary_size/java',
},
'includes': [ '../../build/java.gypi' ],
},
],
}
// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
package org.chromium.tools.binary_size;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.nio.charset.Charset;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Queue;
import java.util.Random;
import java.util.Set;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
/**
* A pool of workers that run 'addr2line', looking up source locations for
* addresses that are fed into the pool via a queue. As address lookups
* complete, records are placed onto an output queue.
*/
class Addr2LineWorkerPool {
private static final Charset sAscii = Charset.forName("US-ASCII");
private final Addr2LineWorker[] mWorkers;
private final ArrayBlockingQueue<Record> mRecordsIn = new ArrayBlockingQueue<Record>(1000);
private final Queue<Record> mRecordsOut = new ConcurrentLinkedQueue<Record>();
private final CountDownLatch mCompletionLatch;
private final String mAddr2linePath;
private final String mLibraryPath;
private final boolean mDisambiguate;
private final boolean mDedupe;
private final String stripLocation;
private final ConcurrentMap<Long, Record> mAddressesSeen =
new ConcurrentHashMap<Long, Record>(100000, 0.75f, 32);
private volatile Map<String,String> mFileLookupTable = null;
private final AtomicInteger mDisambiguationSuccessCount = new AtomicInteger(0);
private final AtomicInteger mDisambiguationFailureCount = new AtomicInteger(0);
private final AtomicInteger mDedupeCount = new AtomicInteger(0);
/**
* These are the suffixes of files that we are potentially interested in
* searching during symbol lookups when disambiguation is enabled. Anything
* that could theoretically end up being linked into the library file
* should be listed here.
* <p>
* IMPORTANT: All of these should be lowercase. When doing comparisons,
* always lowercase the file name before attempting the match.
*/
private static final String[] INTERESTING_FILE_ENDINGS = new String[]{
".c", ".cc", ".h", ".cp", ".cpp", ".cxx", ".c++", ".asm", ".inc", ".s", ".hxx"
};
Addr2LineWorkerPool(final int size,
final String addr2linePath, final String libraryPath,
final boolean disambiguate, final boolean dedupe)
throws IOException {
this.mAddr2linePath = addr2linePath;
this.mLibraryPath = libraryPath;
this.mDisambiguate = disambiguate;
this.mDedupe = dedupe;
// Prepare disambiguation table if necessary. This must be done
// before launching the threads in the processing pool for visibility.
if (disambiguate) {
try {
createDisambiguationTable();
} catch (IOException e) {
throw new RuntimeException("Can't create lookup table", e);
}
}
// The library is in, e.g.: src/out/Release
// Convert all output paths to be relative to src.
// Strip everything up to and including "src/".
String canonical = new File(libraryPath).getCanonicalPath();
int end = canonical.lastIndexOf("/src/");
if (end < 0) {
// Shouldn't happen if the library exists.
throw new RuntimeException("Bad library path: " + libraryPath +
". Library is expected to be within a build directory.");
}
stripLocation = canonical.substring(0, end + "/src/".length());
mWorkers = new Addr2LineWorker[size];
mCompletionLatch = new CountDownLatch(size);
for (int x = 0; x < mWorkers.length; x++) {
mWorkers[x] = new Addr2LineWorker();
}
}
void submit(Record record) throws InterruptedException {
mRecordsIn.put(record);
}
void allRecordsSubmitted() {
for (Addr2LineWorker worker : mWorkers) {
worker.stopIfQueueIsEmpty = true;
}
}
boolean await(int amount, TimeUnit unit) throws InterruptedException {
return mCompletionLatch.await(amount, unit);
}
/**
* @param value the base value
* @param percent absolute percentage to jitter by (in the range [0,100])
* @return a value that is on average uniformly distributed within
* plus or minus <em>percent</em> of the base value.
*/
private static int jitter(final int value, final int percent) {
Random r = new Random();
int delta = (r.nextBoolean() ? 1 : -1) * r.nextInt((percent * value) / 100);
return value + delta;
}
/**
* A class that encapsulates an addr2line process and the work that it
* needs to do.
*/
private class Addr2LineWorker {
// Our work queues
private final AtomicReference<Process> processRef = new AtomicReference<Process>();
private final Thread workerThread;
private volatile boolean stopIfQueueIsEmpty = false;
/**
* After this many addresses have been processed, the addr2line process
* itself will be recycled. This prevents the addr2line process from
* getting too huge, which in turn allows more parallel addr2line
* processes to run. There is a balance to be achieved here, as
* creating a new addr2line process is very costly. A value of
* approximately 2000 appears, empirically, to be a good tradeoff
* on a modern machine; memory usage stays tolerable, and good
* throughput can be achieved. The value is jittered by +/- 10% so that
* the processes don't all restart at once.
*/
private final int processRecycleThreshold = jitter(2000, 10);
private Addr2LineWorker() throws IOException {
this.processRef.set(createAddr2LineProcess());
workerThread = new Thread(new Addr2LineTask(), "Addr2Line Worker");
workerThread.setDaemon(true);
workerThread.start();
}
/**
* Builds a new addr2line process for use in this worker.
* @return the process
* @throws IOException if unable to create the process for any reason
*/
private Process createAddr2LineProcess()
throws IOException {
ProcessBuilder builder = new ProcessBuilder(mAddr2linePath, "-e", mLibraryPath, "-f");
Process process = builder.start();
return process;
}
/**
* Reads records from the input queue and pipes addresses into
* addr2line, using the output to complete the record and pushing
* the record into the output queue.
*/
private class Addr2LineTask implements Runnable {
@Override
public void run() {
int processTaskCounter = 0;
InputStream inStream = processRef.get().getInputStream();
Reader isr = new InputStreamReader(inStream);
BufferedReader reader = new BufferedReader(isr);
try {
while (true) {
// Check for a task.
final Record record = mRecordsIn.poll(1, TimeUnit.SECONDS);
if (record == null) {
if (stopIfQueueIsEmpty) {
// All tasks have been submitted, so if the
// queue is empty then there is nothing left
// to do and it's ready to shut down
return;
}
continue; // else, queue starvation. Try again.
}
// Avoid reprocessing previously-seen symbols if
// deduping is enabled
if (tryDedupe(record)) continue;
// Create a new reader if the addr2line process is new
// or has been recycled. A single reader will be used
// for the entirety of the addr2line process' lifetime.
final Process process = processRef.get();
if (inStream == null) {
inStream = process.getInputStream();
isr = new InputStreamReader(inStream);
reader = new BufferedReader(isr);
}
// Write the address to stdin of addr2line
process.getOutputStream().write(record.address.getBytes(sAscii));
process.getOutputStream().write('\n');
process.getOutputStream().flush();
// Read the answer from addr2line. Example:
// ABGRToYRow_C
// /src/out/Release/../../third_party/libyuv/source/row_common.cc:293
final String name = reader.readLine();
if (name == null || name.isEmpty()) {
stopIfQueueIsEmpty = true;
continue;
}
String location = reader.readLine();
if (location == null || location.isEmpty()) {
stopIfQueueIsEmpty = true;
continue;
}
record.resolvedSuccessfully = !(
name.equals("??") && location.equals("??:0"));
if (record.resolvedSuccessfully) {
// Keep the name from the initial NM dump.
// Some addr2line impls, such as the one for Android
// on ARM, seem to lose data here.
// Note that the location may also include a line
// discriminator, which maps to a part of a line.
// Not currently used.
record.location = processLocation(location);;
}
// Check if there is more input on the stream.
// If there is then it is a serious processing
// error, and reading anything else would de-sync
// the input queue from the results being read.
if (inStream.available() > 0) {
throw new IllegalStateException(
"Alignment mismatch in output from address " + record.address);
}
// Track stats and move record to output queue
processTaskCounter++;
mRecordsOut.add(record);
// If the addr2line process has done too much work,
// kill it and start a new one to reduce memory
// pressure created by the pool.
if (processTaskCounter >= processRecycleThreshold) {
// Out with the old...
try {
processRef.get().destroy();
} catch (Exception e) {
System.err.println("WARNING: zombie process");
e.printStackTrace();
}
// ... and in with the new!
try {
processRef.set(createAddr2LineProcess());
} catch (IOException e) {
e.printStackTrace();
}
processTaskCounter = 0;
inStream = null; // New readers need to be created next iteration
}
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
// Make an attempt to clean up. If we fail, there is
// nothing we can do beyond this.
processRef.get().destroy();
} catch (Exception e) {
// There's nothing more we can do here.
}
mCompletionLatch.countDown();
}
}
}
/**
* Examines the location from a record and attempts to canonicalize
* it and strip off the common source root.
* @param location the location to process
* @return the canonicalized, stripped location or the original input
* string if the location cannot be canonicalized
*/
private String processLocation(String location) {
if (location.startsWith("/")) {
try {
location = new File(location).getCanonicalPath();
} catch (IOException e) {
System.err.println("Unable to canonicalize path: " + location);
}
} else if (mDisambiguate) {
// Ambiguous path (only has the file name)
// Try dictionary lookup if that's enabled
final int indexOfColon = location.lastIndexOf(':');
final String key;
final String line;
if (indexOfColon != -1) {
key = location.substring(0, indexOfColon);
line = location.substring(indexOfColon + 1);
} else {
key = location;
line = null;
}
final String found = mFileLookupTable.get(key);
if (found != null) {
mDisambiguationSuccessCount.incrementAndGet();
location = found;
if (line != null) location = location + ":" + line;
} else {
mDisambiguationFailureCount.incrementAndGet();
}
}
if (location.startsWith(stripLocation)) {
location = location.substring(stripLocation.length());
}
return location;
}
/**
* Attempts to dedupe a record using a cache of previously-seen
* addresses if and only if deduping is enabled.
* @param record the record to attempt deduping
* @return true if deduplication is enabled and the record references
* an address that has already been seen; otherwise false
*/
private boolean tryDedupe(Record record) {
if (mDedupe) {
long addressLong = Long.parseLong(record.address, 16);
Record existing = mAddressesSeen.get(addressLong);
if (existing != null) {
if (!existing.size.equals(record.size)) {
System.err.println("WARNING: Deduped address " +
record.address + " has a size mismatch, " +
existing.size + " != " + record.size);
}
mDedupeCount.incrementAndGet();
return true;
}
if (mAddressesSeen.putIfAbsent(addressLong, record) != null) {
// putIfAbsent is used to ensure that we have
// an accurate dedupeCount; otherwise, two
// workers could insert the same record in
// parallel without realizing that one of them
// was actually a duplicate.
mDedupeCount.incrementAndGet();
return true;
}
}
return false;
}
}
// TODO(andrewhayden): Make this less Android-specific
private void createDisambiguationTable() throws IOException {
// Convert /src/out/Release/lib/*.so -> /src/out/Release
final File libraryOutputDirectory = new File(mLibraryPath)
.getParentFile().getParentFile().getCanonicalFile();
// Convert /src/out/Release -> /src
final File root = libraryOutputDirectory
.getParentFile().getParentFile().getCanonicalFile();
// There is no code at the root of Chromium.
// Ignore all the 'out' directories.
mFileLookupTable = new HashMap<String, String>();
Set<String> dupes = new HashSet<String>();
for (File file : root.listFiles()) {
if (file.isDirectory()) {
String name = file.getName();
if (name.startsWith("out")) {
if (new File(file, "Release").exists() || new File(file, "Debug").exists()) {
// It's an output directory, skip it - except for the
// 'obj' and 'gen' subdirectories that are siblings
// to the library file's parent dir, which is needed.
// Include those at the very end, since they are known.
continue;
}
} else if (name.startsWith(".")) {
// Skip dot directories: .git, .svn, etcetera.
continue;
}
findInterestingFiles(file, dupes);
}
}
// Include directories that contain generated resources we are likely
// to encounter in the symbol table.
findInterestingFiles(new File(libraryOutputDirectory, "gen"), dupes);
findInterestingFiles(new File(libraryOutputDirectory, "obj"), dupes);
// Any duplicates in the filesystem can't be used for disambiguation
// because it is not obvious which of the duplicates is the true source.
// Therefore, discard all files that have duplicate names.
for (String dupe : dupes) {
mFileLookupTable.remove(dupe);
}
}
// TODO(andrewhayden): Could integrate with build system to know EXACTLY
// what is out there. This would avoid the need for the dupes set, which
// would make it possible to do much better deduping.
private void findInterestingFiles(File directory, Set<String> dupes) {
for (File file : directory.listFiles()) {
if (file.isDirectory() && file.canRead()) {
if (!file.getName().startsWith(".")) {
findInterestingFiles(file, dupes);
}
} else {
String name = file.getName();
String normalized = name.toLowerCase();
for (String ending : INTERESTING_FILE_ENDINGS) {
if (normalized.endsWith(ending)) {
String other = mFileLookupTable.put(
name, file.getAbsolutePath());
if (other != null) dupes.add(name);
}
}
}
}
}
/**
* Polls the output queue for the next record.
* @return the next record
*/
Record poll() {
return mRecordsOut.poll();
}
/**
* @return the number of ambiguous paths successfully disambiguated
*/
int getDisambiguationSuccessCount() {
return mDisambiguationSuccessCount.get();
}
/**
* @return the number of ambiguous paths that couldn't be disambiguated
*/
int getDisambiguationFailureCount() {
return mDisambiguationFailureCount.get();
}
/**
* @return the number of symbols deduped
*/
int getDedupeCount() {
return mDedupeCount.get();
}
}
\ No newline at end of file
// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
package org.chromium.tools.binary_size;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.PrintWriter;
/**
* Converts records to a format that simulates output from 'nm'.
*/
class NmDumper {
private final String mOutPath;
private final String mSkipPath;
private final String mFailPath;
private final Output mOutput;
/**
* Create a new dumper that will output to the specified paths.
* @param outPath where to write all records and lines, including lines
* that are skipped or records that failed to resolve
* @param failPath if not null, a path to which <em>only</em> records that
* failed to resolve will be written
* @param skipPath if not null, a path to which <em>only</em> lines that
* were skipped will be written
*/
NmDumper(final String outPath, final String failPath, final String skipPath) {
mOutPath = outPath;
mFailPath = failPath;
mSkipPath = skipPath;
mOutput = new Output();
}
/**
* Close all output files.
*/
void close() {
mOutput.closeAll();
}
/**
* Output a line that was skipped.
* @param line the line
*/
void skipped(String line) {
mOutput.printSkip(line);
}
/**
* Output a record that failed to resolve.
* @param record the record
*/
void failed(Record record) {
mOutput.printFail(simulateNmOutput(record));
}
/**
* Output a record that successfully resolved.
* @param record the record
*/
void succeeded(Record record) {
mOutput.print(simulateNmOutput(record));
}
/**
* Generate a string that looks like output from nm for a given record.
* @param record the record
* @return nm-like output
*/
private static final String simulateNmOutput(final Record record) {
StringBuilder builder = new StringBuilder(record.address);
builder.append(' ');
builder.append(record.size);
builder.append(' ');
builder.append(record.symbolType);
builder.append(' ');
builder.append(record.symbolName != null ? record.symbolName : "unknown");
if (record.location != null) {
builder.append('\t');
builder.append(record.location);
}
return builder.toString();
}
private class Output {
private final PrintWriter skipWriter;
private final PrintWriter failWriter;
private final PrintWriter outWriter;
private Output() {
try {
new File(mOutPath).getParentFile().mkdirs();
outWriter = new PrintWriter(mOutPath);
} catch (FileNotFoundException e) {
throw new RuntimeException("Can't open output file: " + mOutPath, e);
}
if (mFailPath != null) {
try {
new File(mFailPath).getParentFile().mkdirs();
failWriter = new PrintWriter(mFailPath);
} catch (FileNotFoundException e) {
throw new RuntimeException("Can't open fail file: " + mFailPath, e);
}
} else {
failWriter = null;
}
if (mSkipPath != null) {
try {
new File(mSkipPath).getParentFile().mkdirs();
skipWriter = new PrintWriter(mSkipPath);
} catch (IOException e) {
throw new RuntimeException("Can't open skip file: " + mSkipPath, e);
}
} else {
skipWriter = null;
}
}
private synchronized void println(PrintWriter writer, String string) {
if (writer != null) {
writer.println(string);
writer.flush();
}
}
synchronized void print(String string) {
println(outWriter, string);
}
synchronized void printSkip(String string) {
println(skipWriter, string);
println(outWriter, string);
}
synchronized void printFail(String string) {
println(failWriter, string);
println(outWriter, string);
}
private void closeAll() {
for (PrintWriter writer : new PrintWriter[]{outWriter, failWriter, skipWriter}) {
if (writer != null) {
try {
writer.flush(); }
catch (Exception ignored) {
// Nothing to be done.
}
try {
writer.close(); }
catch (Exception ignored) {
// Nothing to be done.
}
}
}
}
}
}
\ No newline at end of file
// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
package org.chromium.tools.binary_size;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* A tool for parallelizing "addr2line" against a given binary.
* The tool runs "nm" to dump the symbols from a library, then spawns a pool
* of addr2name workers that resolve addresses to name in parallel.
* <p>
* This tool is intentionally written to be standalone so that it can be
* compiled without reliance upon any other libraries. All that is required
* is a vanilla installation of the Java Runtime Environment, 1.5 or later.
*/
// TODO(andrewhayden): Rewrite entire tool in Python
public class ParallelAddress2Line {
private final AtomicBoolean mStillEnqueuing = new AtomicBoolean(true);
private final AtomicInteger mEnqueuedCount =
new AtomicInteger(Integer.MAX_VALUE);
private final AtomicInteger mDoneCount = new AtomicInteger(0);
private final AtomicInteger mSuccessCount = new AtomicInteger(0);
private final AtomicInteger mAddressSkipCount = new AtomicInteger(0);
private final String mLibraryPath;
private final String mNmPath;
private final String mNmInPath;
private final String mAddr2linePath;
private final boolean mVerbose;
private final boolean mNoProgress;
private Addr2LineWorkerPool mPool;
private final boolean mNoDedupe;
private final boolean mDisambiguate;
private final NmDumper mNmDumper;
private static final String USAGE =
"--addr2line [ARG] instead of the 'addr2line' in $PATH, use this (e.g.,\n" +
" arch-specific binary) (optional)\n" +
"--disambiguate create a listing of all source files which can be used\n" +
" to disambiguate some percentage of ambiguous source\n" +
" references; only useful on some architectures and adds\n" +
" significant startup cost (optional)\n" +
"--failfile [ARG] output symbols from failed lookups to the specified\n" +
" file (optional)\n" +
"--library [ARG] path to the library to process, e.g.\n" +
" out/Release/lib/libchromeview.so (required)\n" +
"--nm [ARG] instead of the 'nm' in $PATH, use this (e.g.,\n" +
" arch-specific binary) (optional)\n" +
"--nm-infile [ARG] instead of running nm on the specified library,\n" +
" ingest the specified nm file. (optional)\n" +
"--no-dedupe don't de-dupe symbols that live at the same address;\n" +
" deduping more accurately describes the use of space\n" +
" within the binary; if spatial analysis is your goal,\n" +
" leave deduplication on. (optional)\n" +
"--no-progress don't output periodic progress reports (optional)\n" +
"--outfile [ARG] output results into the specified file (required)\n" +
"--skipfile [ARG] output skipped symbols to the specified file (optional)\n" +
"--threads [ARG] number of parallel worker threads to create. Start low\n" +
" and watch your memory, defaults to 1 (optional)\n" +
"--verbose be verbose (optional)\n";
// Regex for parsing "nm" output. A sample line looks like this:
// 0167b39c 00000018 t ACCESS_DESCRIPTION_free /path/file.c:95
//
// The fields are: address, size, type, name, source location
// Regular expression explained ( see also: https://xkcd.com/208 ):
// ([0-9a-f]{8}+) The address
// [\\s]+ Whitespace separator
// ([0-9a-f]{8}+) The size. From here on out it's all optional.
// [\\s]+ Whitespace separator
// (\\S?) The symbol type, which is any non-whitespace char
// [\\s*] Whitespace separator
// ([^\\t]*) Symbol name, any non-tab character (spaces ok!)
// [\\t]? Tab separator
// (.*) The location (filename[:linennum|?][ (discriminator n)]
private static final Pattern sNmPattern = Pattern.compile(
"([0-9a-f]{8}+)[\\s]+([0-9a-f]{8}+)[\\s]*(\\S?)[\\s*]([^\\t]*)[\\t]?(.*)");
private ParallelAddress2Line(
final String libraryPath,
final String nmPath,
final String nmInPath,
final String addr2linePath,
final String outPath,
final String skipPath,
final String failPath,
final boolean verbose,
final boolean noProgress,
final boolean noDedupe,
final boolean disambiguate) {
this.mLibraryPath = libraryPath;
this.mNmPath = nmPath;
this.mNmInPath = nmInPath;
this.mAddr2linePath = addr2linePath;
this.mVerbose = verbose;
this.mNoProgress = noProgress;
this.mNoDedupe = noDedupe;
this.mDisambiguate = disambiguate;
this.mNmDumper = new NmDumper(outPath, failPath, skipPath);
final File libraryFile = new File(libraryPath);
if (!(libraryFile.exists() && libraryFile.canRead())) {
throw new IllegalStateException("Can't read library file: " + libraryPath);
}
}
private static final File findFile(File directory, String target) {
for (File file : directory.listFiles()) {
if (file.isDirectory() && file.canRead()) {
File result = findFile(file, target);
if (result != null) return result;
} else {
if (target.equals(file.getName())) return file;
}
}
return null;
}
private void run(final int addr2linePoolSize) throws InterruptedException {
try {
runInternal(addr2linePoolSize);
} finally {
mNmDumper.close();
}
}
private void runInternal(final int addr2linePoolSize) throws InterruptedException {
// Step 1: Dump symbols with nm
final String nmOutputPath;
if (mNmInPath == null) {
// Generate nm output with nm binary
logVerbose("Running nm to dump symbols from " + mLibraryPath);
try {
nmOutputPath = dumpSymbols();
} catch (Exception e) {
throw new RuntimeException("nm failed", e);
}
} else {
// Use user-supplied nm output
logVerbose("Using user-supplied nm file: " + mNmInPath);
nmOutputPath = mNmInPath;
}
// Step 2: Prepare addr2line worker pool to process nm output
try {
logVerbose("Creating " + addr2linePoolSize + " workers for " + mAddr2linePath);
mPool = new Addr2LineWorkerPool(addr2linePoolSize,
mAddr2linePath, mLibraryPath, mDisambiguate, !mNoDedupe);
} catch (IOException e) {
throw new RuntimeException("Couldn't initialize name2address pool!", e);
}
// Step 3: Spool symbol-processing tasks to workers
final long startTime = System.currentTimeMillis();
Timer timer = null;
if (!mNoProgress) {
timer = startTaskMonitor(startTime);
}
final int queued = spoolTasks(nmOutputPath);
// All tasks have been enqueued.
mEnqueuedCount.set(queued);
mStillEnqueuing.set(false);
mPool.allRecordsSubmitted();
float percentAddressesSkipped = 100f * (mAddressSkipCount.floatValue()
/ (queued + mAddressSkipCount.get()));
float percentAddressesQueued = 100f - percentAddressesSkipped;
int totalAddresses = mAddressSkipCount.get() + queued;
logVerbose("All addresses have been enqueued (total " + queued + ").");
// Remember that the queue to which the addresses was enqueued is of a
// small fixed size; by the time this code executes, there is very
// little work left to do. Await the termination of the pool with a
// reasonable timeout for safety purposes.
boolean timedOut = !mPool.await(5, TimeUnit.MINUTES);
if (timedOut) {
throw new RuntimeException("Worker pool did not terminate!");
}
if (!mNoProgress) timer.cancel();
log(totalAddresses + " addresses discovered; " +
queued + " queued for processing (" +
String.format("%.2f", percentAddressesQueued) + "%), " +
mAddressSkipCount.get() + " skipped (" +
String.format("%.2f", percentAddressesSkipped) + "%)");
dumpStats(startTime);
log("Done.");
}
/**
* Monitors the pool periodically printing status updates to stdout.
* @param addressProcessingStartTime the time address processing began
* @return the daemon timer that is generating the status updates
*/
private final Timer startTaskMonitor(
final long addressProcessingStartTime) {
Runnable monitorTask = new OutputSpooler();
Thread monitor = new Thread(monitorTask, "progress monitor");
monitor.setDaemon(true);
monitor.start();
TimerTask task = new TimerTask() {
@Override
public void run() {
dumpStats(addressProcessingStartTime);
}
};
Timer timer = new Timer(true);
timer.schedule(task, 1000L, 1000L);
return timer;
}
/**
* Spools address-lookup tasks to the addr2line workers.
* This method will block until most of (or possibly all of) the tasks
* have been spooled.
* If a skip path is set, any line in the input file that doesn't have
* an address will be copied into the skip file.
*
* @param inputPath the path to the dump produced by nm
* @return the number of tasks spooled
*/
private final int spoolTasks(final String inputPath) {
FileReader inputReader = null;
try {
inputReader = new FileReader(inputPath);
} catch (IOException e) {
throw new RuntimeException("Can't open input file: " + inputPath, e);
}
final BufferedReader bufferedReader = new BufferedReader(inputReader);
String currentLine = null;
int numSpooled = 0;
try {
while ((currentLine = bufferedReader.readLine()) != null) {
try {
final Matcher matcher = sNmPattern.matcher(currentLine);
if (!matcher.matches()) {
// HACK: Special case for ICU data.
// This thing is HUGE (5+ megabytes) and is currently
// missed because there is no size information.
// torne@ has volunteered to patch the generation code
// so that the generated ASM includes a size attribute
// so that this hard-coding can go away in the future.
if (currentLine.endsWith("icudt46_dat")) {
Record record = getIcuRecord(currentLine);
if (record != null) {
numSpooled++;
mPool.submit(record);
continue;
}
}
mNmDumper.skipped(currentLine);
mAddressSkipCount.incrementAndGet();
continue;
}
final Record record = new Record();
record.address = matcher.group(1);
record.size = matcher.group(2);
if (matcher.groupCount() >= 3) {
record.symbolType = matcher.group(3).charAt(0);
}
if (matcher.groupCount() >= 4) {
// May or may not be present
record.symbolName = matcher.group(4);
}
numSpooled++;
mPool.submit(record);
} catch (Exception e) {
throw new RuntimeException("Error processing line: '" + currentLine + "'", e);
}
}
} catch (Exception e) {
throw new RuntimeException("Input processing failed", e);
} finally {
try {
bufferedReader.close();
} catch (Exception ignored) {
// Nothing to be done
}
try {
inputReader.close();
} catch (Exception ignored) {
// Nothing to be done
}
}
return numSpooled;
}
private Record getIcuRecord(String line) throws IOException {
// Line looks like this:
// 01c9ee00 r icudt46_dat
String[] parts = line.split("\\s");
if (parts.length != 3) return null;
// Convert /src/out/Release/lib/[libraryfile] -> /src/out/Release
final File libraryOutputDirectory = new File(mLibraryPath)
.getParentFile().getParentFile().getCanonicalFile();
final File icuDir = new File(
libraryOutputDirectory.getAbsolutePath() +
"/obj/third_party/icu");
final File icuFile = findFile(icuDir, "icudata.icudt46l_dat.o");
if (!icuFile.exists()) return null;
final Record record = new Record();
record.address = parts[0];
record.symbolType = parts[1].charAt(0);
record.symbolName = parts[2];
record.size = Integer.toHexString((int) icuFile.length());
record.location = icuFile.getCanonicalPath() + ":0";
record.resolvedSuccessfully = true;
while (record.size.length() < 8) {
record.size = "0" + record.size;
}
return record;
}
/**
* @return the path to the file that nm wrote
* @throws Exception
* @throws FileNotFoundException
* @throws InterruptedException
*/
private String dumpSymbols() throws Exception, FileNotFoundException, InterruptedException {
final Process process = createNmProcess();
final File tempFile = File.createTempFile("ParallelAddress2Line", "nm");
tempFile.deleteOnExit();
final CountDownLatch completionLatch = sink(
process.getInputStream(), new FileOutputStream(tempFile), true);
sink(process.getErrorStream(), System.err, false);
logVerbose("Dumping symbols to: " + tempFile.getAbsolutePath());
final int nmRc = process.waitFor();
if (nmRc != 0) {
throw new RuntimeException("nm process returned " + nmRc);
}
completionLatch.await(); // wait for output to be done
return tempFile.getAbsolutePath();
}
private void dumpStats(final long startTime) {
long successful = mSuccessCount.get();
long doneNow = mDoneCount.get();
long unsuccessful = doneNow - successful;
float successPercent = doneNow == 0 ? 100f : 100f * ((float)successful / (float)doneNow);
long elapsedMillis = System.currentTimeMillis() - startTime;
float elapsedSeconds = elapsedMillis / 1000f;
long throughput = doneNow / (elapsedMillis / 1000);
final int mapLookupSuccess = mPool.getDisambiguationSuccessCount();
final int mapLookupFailure = mPool.getDisambiguationFailureCount();
final int mapLookupTotal = mapLookupSuccess + mapLookupFailure;
float mapLookupSuccessPercent = 0f;
if (mapLookupTotal != 0 && mapLookupSuccess != 0) {
mapLookupSuccessPercent = 100f *
((float) mapLookupSuccess / (float) mapLookupTotal);
}
log(doneNow + " addresses processed (" +
mSuccessCount.get() + " ok, " + unsuccessful + " failed)" +
", avg " + throughput + " addresses/sec, " +
String.format("%.2f", successPercent) + "% success" +
", " + mapLookupTotal + " ambiguous path" +
(!mDisambiguate ? "" :
", (" + String.format("%.2f", mapLookupSuccessPercent) + "% disambiguated)") +
(mNoDedupe ? "" : ", " + mPool.getDedupeCount() + " deduped") +
", elapsed time " + String.format("%.3f", elapsedSeconds) + " seconds");
}
private Process createNmProcess() throws Exception {
ProcessBuilder builder = new ProcessBuilder(
mNmPath,
"-C", // demangle (for the humans)
"-S", // print size
mLibraryPath);
logVerbose("Creating process: " + builder.command());
return builder.start();
}
/**
* Make a pipe to drain the specified input stream into the specified
* output stream asynchronously.
* @param in read from here
* @param out and write to here
* @param closeWhenDone whether or not to close the target output stream
* when the input stream terminates
* @return a latch that can be used to await the final write to the
* output stream, which occurs when either of the streams closes
*/
private static final CountDownLatch sink(final InputStream in,
final OutputStream out, final boolean closeWhenDone) {
final CountDownLatch latch = new CountDownLatch(1);
final Runnable task = new Runnable() {
@Override
public void run() {
byte[] buffer = new byte[4096];
try {
int numRead = 0;
do {
numRead = in.read(buffer);
if (numRead > 0) {
out.write(buffer, 0, numRead);
out.flush();
}
} while (numRead >= 0);
} catch (Exception e) {
e.printStackTrace();
} finally {
try { out.flush(); } catch (Exception ignored) {
// Nothing to be done
}
if (closeWhenDone) {
try { out.close(); } catch (Exception ignored) {
// Nothing to be done
}
}
latch.countDown();
}
}
};
final Thread worker = new Thread(task, "pipe " + in + "->" + out);
worker.setDaemon(true);
worker.start();
return latch;
}
private final class OutputSpooler implements Runnable {
@Override
public void run() {
do {
readRecord();
} while (mStillEnqueuing.get() || (mDoneCount.get() < mEnqueuedCount.get()));
}
/**
* Read a record and process it.
*/
private void readRecord() {
Record record = mPool.poll();
if (record != null) {
mDoneCount.incrementAndGet();
if (record.resolvedSuccessfully) {
mSuccessCount.incrementAndGet();
mNmDumper.succeeded(record);
} else {
mNmDumper.failed(record);
}
} else {
try {
// wait to keep going
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
/**
* Log a message to the console.
* @param message the message to log
*/
private final void log(String message) {
System.out.println(message);
}
/**
* Log a message to the console iff verbose logging is enabled.
* @param message the message to log
*/
private final void logVerbose(String message) {
if (mVerbose) log(message);
}
/**
* Runs the tool. Run with --help for limited help.
* @param args
* @throws Exception if anything explodes
*/
public static void main(String[] args) throws Exception {
ParallelAddress2Line tool = new ParallelAddress2Line(
getArg(args, "--library"),
getArg(args, "--nm", "nm"),
getArg(args, "--nm-infile", null),
getArg(args, "--addr2line", "addr2line"),
getArg(args, "--outfile"),
getArg(args, "--skipfile", null),
getArg(args, "--failfile", null),
hasFlag(args, "--verbose"),
hasFlag(args, "--no-progress"),
hasFlag(args, "--no-dedupe"),
hasFlag(args, "--disambiguate"));
tool.run(Integer.parseInt(getArg(args, "--threads", "1")));
}
private static boolean hasFlag(String[] args, String name) {
for (int x = 0; x < args.length; x++) if (name.equals(args[x])) return true;
return false;
}
private static String getArg(String[] args, String name, String defaultValue) {
for (int x = 0; x < args.length; x++) {
if (name.equals(args[x])) {
if (x < args.length - 1) return args[x + 1];
throw new RuntimeException(name + " is missing a value\n" + USAGE);
}
}
return defaultValue;
}
private static String getArg(String[] args, String name) {
String result = getArg(args, name, null);
if (result == null) throw new RuntimeException(name + " is required\n" + USAGE);
return result;
}
}
\ No newline at end of file
// Copyright 2014 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
package org.chromium.tools.binary_size;
/**
* A record that is filled in partially by nm and partially by addr2line,
* along with tracking information about whether or not the lookup in
* addr2line was successful.
*/
class Record {
/**
* The base-16 address, as a string.
*/
String address;
/**
* The symbol type.
*/
char symbolType;
/**
* The name of the symbol. Note that this may include whitespace, but
* not tabs.
*/
String symbolName;
/**
* The base-10 size in bytes, as a String.
*/
String size;
/**
* The location, if available; may include a file name and, optionally,
* a colon separator character followed by a line number or a
* question mark.
*/
String location;
/**
* Whether or not the record was successfully resolved. Records that are
* successfully resolved should have a non-null location.
*/
boolean resolvedSuccessfully;
}
\ No newline at end of file
#!/usr/bin/python
# Copyright 2014 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
"""Generate a spatial analysis against an arbitrary library.
To use, build the 'binary_size_tool' target. Then run this tool, passing
in the location of the library to be analyzed along with any other options
you desire.
"""
import collections
import fileinput
import json
import optparse
import os
import pprint
import re
import shutil
import subprocess
import sys
import tempfile
def FormatBytes(bytes):
"""Pretty-print a number of bytes."""
if bytes > 1e6:
bytes = bytes / 1.0e6
return '%.1fm' % bytes
if bytes > 1e3:
bytes = bytes / 1.0e3
return '%.1fk' % bytes
return str(bytes)
def SymbolTypeToHuman(type):
"""Convert a symbol type as printed by nm into a human-readable name."""
return {'b': 'bss',
'd': 'data',
'r': 'read-only data',
't': 'code',
'w': 'weak symbol',
'v': 'weak symbol'}[type]
def ParseNm(input):
"""Parse nm output.
Argument: an iterable over lines of nm output.
Yields: (symbol name, symbol type, symbol size, source file path).
Path may be None if nm couldn't figure out the source file.
"""
# Match lines with size, symbol, optional location, optional discriminator
sym_re = re.compile(r'^[0-9a-f]{8} ' # address (8 hex digits)
'([0-9a-f]{8}) ' # size (8 hex digits)
'(.) ' # symbol type, one character
'([^\t]+)' # symbol name, separated from next by tab
'(?:\t(.*):[\d\?]+)?.*$') # location
# Match lines with addr but no size.
addr_re = re.compile(r'^[0-9a-f]{8} (.) ([^\t]+)(?:\t.*)?$')
# Match lines that don't have an address at all -- typically external symbols.
noaddr_re = re.compile(r'^ {8} (.) (.*)$')
for line in input:
line = line.rstrip()
match = sym_re.match(line)
if match:
size, type, sym = match.groups()[0:3]
size = int(size, 16)
type = type.lower()
if type == 'v':
type = 'w' # just call them all weak
if type == 'b':
continue # skip all BSS for now
path = match.group(4)
yield sym, type, size, path
continue
match = addr_re.match(line)
if match:
type, sym = match.groups()[0:2]
# No size == we don't care.
continue
match = noaddr_re.match(line)
if match:
type, sym = match.groups()
if type in ('U', 'w'):
# external or weak symbol
continue
print >>sys.stderr, 'unparsed:', repr(line)
def TreeifySymbols(symbols):
"""Convert symbols into a path-based tree, calculating size information
along the way.
The result is a dictionary that contains two kinds of nodes:
1. Leaf nodes, representing source code locations (e.g., c++ files)
These nodes have the following dictionary entries:
sizes: a dictionary whose keys are categories (such as code, data,
vtable, etceteras) and whose values are the size, in bytes, of
those categories;
size: the total size, in bytes, of all the entries in the sizes dict
2. Non-leaf nodes, representing directories
These nodes have the following dictionary entries:
children: a dictionary whose keys are names (path entries; either
directory or file names) and whose values are other nodes;
size: the total size, in bytes, of all the leaf nodes that are
contained within the children dict (recursively expanded)
The result object is itself a dictionary that represents the common ancestor
of all child nodes, e.g. a path to which all other nodes beneath it are
relative. The 'size' attribute of this dict yields the sum of the size of all
leaf nodes within the data structure.
"""
dirs = {'children': {}, 'size': 0}
for sym, type, size, path in symbols:
dirs['size'] += size
if path:
path = os.path.normpath(path)
if path.startswith('/'):
path = path[1:]
parts = None
if path:
parts = path.split('/')
if parts:
assert path
file_key = parts.pop()
tree = dirs
try:
# Traverse the tree to the parent of the file node, creating as needed
for part in parts:
assert part != ''
if part not in tree['children']:
tree['children'][part] = {'children': {}, 'size': 0}
tree = tree['children'][part]
tree['size'] += size
# Get (creating if necessary) the node for the file
# This node doesn't have a 'children' attribute
if file_key not in tree['children']:
tree['children'][file_key] = {'sizes': collections.defaultdict(int),
'size': 0}
tree = tree['children'][file_key]
tree['size'] += size
# Accumulate size into a bucket within the file
type = type.lower()
if 'vtable for ' in sym:
tree['sizes']['[vtable]'] += size
elif 'r' == type:
tree['sizes']['[rodata]'] += size
elif 'd' == type:
tree['sizes']['[data]'] += size
elif 'b' == type:
tree['sizes']['[bss]'] += size
elif 't' == type:
# 'text' in binary parlance means 'code'.
tree['sizes']['[code]'] += size
elif 'w' == type:
tree['sizes']['[weak]'] += size
else:
tree['sizes']['[other]'] += size
except:
print >>sys.stderr, sym, parts, key
raise
else:
key = 'symbols without paths'
if key not in dirs['children']:
dirs['children'][key] = {'sizes': collections.defaultdict(int),
'size': 0}
tree = dirs['children'][key]
subkey = 'misc'
if (sym.endswith('::__FUNCTION__') or
sym.endswith('::__PRETTY_FUNCTION__')):
subkey = '__FUNCTION__'
elif sym.startswith('CSWTCH.'):
subkey = 'CSWTCH'
elif '::' in sym:
subkey = sym[0:sym.find('::') + 2]
tree['sizes'][subkey] = tree['sizes'].get(subkey, 0) + size
tree['size'] += size
return dirs
def JsonifyTree(tree, name):
"""Convert TreeifySymbols output to a JSON treemap.
The format is very similar, with the notable exceptions being
lists of children instead of maps and some different attribute names."""
children = []
css_class_map = {
'[vtable]': 'vtable',
'[rodata]': 'read-only_data',
'[data]': 'data',
'[bss]': 'bss',
'[code]': 'code',
'[weak]': 'weak_symbol'
}
if 'children' in tree:
# Non-leaf node. Recurse.
for child_name, child in tree['children'].iteritems():
children.append(JsonifyTree(child, child_name))
else:
# Leaf node; dump per-file stats as entries in the treemap
for kind, size in tree['sizes'].iteritems():
child_json = {'name': kind + ' (' + FormatBytes(size) + ')',
'data': { '$area': size }}
css_class = css_class_map.get(kind)
if css_class is not None: child_json['data']['$symbol'] = css_class
children.append(child_json)
# Sort children by size, largest to smallest.
children.sort(key=lambda child: -child['data']['$area'])
# For leaf nodes, the 'size' attribute is the size of the leaf;
# Non-leaf nodes don't really have a size, but their 'size' attribute is
# the sum of the sizes of all their children.
return {'name': name + ' (' + FormatBytes(tree['size']) + ')',
'data': { '$area': tree['size'] },
'children': children }
def DumpTreemap(symbols, outfile):
dirs = TreeifySymbols(symbols)
out = open(outfile, 'w')
try:
out.write('var kTree = ' + json.dumps(JsonifyTree(dirs, '/')))
finally:
out.flush()
out.close()
def DumpLargestSymbols(symbols, outfile, n):
# a list of (sym, type, size, path); sort by size.
symbols = sorted(symbols, key=lambda x: -x[2])
dumped = 0
out = open(outfile, 'w')
try:
out.write('var largestSymbols = [\n')
for sym, type, size, path in symbols:
if type in ('b', 'w'):
continue # skip bss and weak symbols
if path is None:
path = ''
entry = {'size': FormatBytes(size),
'symbol': sym,
'type': SymbolTypeToHuman(type),
'location': path }
out.write(json.dumps(entry))
out.write(',\n')
dumped += 1
if dumped >= n:
return
finally:
out.write('];\n')
out.flush()
out.close()
def MakeSourceMap(symbols):
sources = {}
for sym, type, size, path in symbols:
key = None
if path:
key = os.path.normpath(path)
else:
key = '[no path]'
if key not in sources:
sources[key] = {'path': path, 'symbol_count': 0, 'size': 0}
record = sources[key]
record['size'] += size
record['symbol_count'] += 1
return sources
def DumpLargestSources(symbols, outfile, n):
map = MakeSourceMap(symbols)
sources = sorted(map.values(), key=lambda x: -x['size'])
dumped = 0
out = open(outfile, 'w')
try:
out.write('var largestSources = [\n')
for record in sources:
entry = {'size': FormatBytes(record['size']),
'symbol_count': str(record['symbol_count']),
'location': record['path']}
out.write(json.dumps(entry))
out.write(',\n')
dumped += 1
if dumped >= n:
return
finally:
out.write('];\n')
out.flush()
out.close()
def DumpLargestVTables(symbols, outfile, n):
vtables = []
for symbol, type, size, path in symbols:
if 'vtable for ' in symbol:
vtables.append({'symbol': symbol, 'path': path, 'size': size})
vtables = sorted(vtables, key=lambda x: -x['size'])
dumped = 0
out = open(outfile, 'w')
try:
out.write('var largestVTables = [\n')
for record in vtables:
entry = {'size': FormatBytes(record['size']),
'symbol': record['symbol'],
'location': record['path']}
out.write(json.dumps(entry))
out.write(',\n')
dumped += 1
if dumped >= n:
return
finally:
out.write('];\n')
out.flush()
out.close()
def RunParallelAddress2Line(outfile, library, arch, jobs, verbose):
"""Run a parallel addr2line processing engine to dump and resolve symbols."""
out_dir = os.getenv('CHROMIUM_OUT_DIR', 'out')
build_type = os.getenv('BUILDTYPE', 'Release')
classpath = os.path.join(out_dir, build_type, 'lib.java',
'binary_size_java.jar')
cmd = ['java',
'-classpath', classpath,
'org.chromium.tools.binary_size.ParallelAddress2Line',
'--disambiguate',
'--outfile', outfile,
'--library', library,
'--threads', jobs]
if verbose is True:
cmd.append('--verbose')
prefix = os.path.join('third_party', 'android_tools', 'ndk', 'toolchains')
if arch == 'android-arm':
prefix = os.path.join(prefix, 'arm-linux-androideabi-4.7', 'prebuilt',
'linux-x86_64', 'bin', 'arm-linux-androideabi-')
cmd.extend(['--nm', prefix + 'nm', '--addr2line', prefix + 'addr2line'])
elif arch == 'android-mips':
prefix = os.path.join(prefix, 'mipsel-linux-android-4.7', 'prebuilt',
'linux-x86_64', 'bin', 'mipsel-linux-android-')
cmd.extend(['--nm', prefix + 'nm', '--addr2line', prefix + 'addr2line'])
elif arch == 'android-x86':
prefix = os.path.join(prefix, 'x86-4.7', 'prebuilt',
'linux-x86_64', 'bin', 'i686-linux-android-')
cmd.extend(['--nm', prefix + 'nm', '--addr2line', prefix + 'addr2line'])
# else, use whatever is in PATH (don't pass --nm or --addr2line)
if verbose:
print cmd
return_code = subprocess.call(cmd)
if return_code:
raise RuntimeError('Failed to run ParallelAddress2Line: returned ' +
str(return_code))
def GetNmSymbols(infile, outfile, library, arch, jobs, verbose):
if infile is None:
if outfile is None:
infile = tempfile.NamedTemporaryFile(delete=False).name
else:
infile = outfile
if verbose:
print 'Running parallel addr2line, dumping symbols to ' + infile;
RunParallelAddress2Line(outfile=infile, library=library, arch=arch,
jobs=jobs, verbose=verbose)
elif verbose:
print 'Using nm input from ' + infile
with file(infile, 'r') as infile:
return list(ParseNm(infile))
def main():
usage="""%prog [options]
Runs a spatial analysis on a given library, looking up the source locations
of its symbols and calculating how much space each directory, source file,
and so on is taking. The result is a report that can be used to pinpoint
sources of large portions of the binary, etceteras.
Under normal circumstances, you only need to pass two arguments, thusly:
%prog --library /path/to/library --destdir /path/to/output
In this mode, the program will dump the symbols from the specified library
and map those symbols back to source locations, producing a web-based
report in the specified output directory.
Other options are available via '--help'.
"""
parser = optparse.OptionParser(usage=usage)
parser.add_option('--nm-in', metavar='PATH',
help='if specified, use nm input from <path> instead of '
'generating it. Note that source locations should be '
'present in the file; i.e., no addr2line symbol lookups '
'will be performed when this option is specified. '
'Mutually exclusive with --library.')
parser.add_option('--destdir', metavar='PATH',
help='write output to the specified directory. An HTML '
'report is generated here along with supporting files; '
'any existing report will be overwritten.')
parser.add_option('--library', metavar='PATH',
help='if specified, process symbols in the library at '
'the specified path. Mutually exclusive with --nm-in.')
parser.add_option('--arch',
help='the architecture that the library is targeted to. '
'Determines which nm/addr2line binaries are used. When '
'\'host-native\' is chosen, the program will use whichever '
'nm/addr2line binaries are on the PATH. This is '
'appropriate when you are analyzing a binary by and for '
'your computer. '
'This argument is only valid when using --library. '
'Default is \'host-native\'.',
choices=['host-native', 'android-arm',
'android-mips', 'android-x86'],)
parser.add_option('--jobs',
help='number of jobs to use for the parallel '
'addr2line processing pool; defaults to 1. More '
'jobs greatly improve throughput but eat RAM like '
'popcorn, and take several gigabytes each. Start low '
'and ramp this number up until your machine begins to '
'struggle with RAM. '
'This argument is only valid when using --library.')
parser.add_option('-v', dest='verbose', action='store_true',
help='be verbose, printing lots of status information.')
parser.add_option('--nm-out', metavar='PATH',
help='keep the nm output file, and store it at the '
'specified path. This is useful if you want to see the '
'fully processed nm output after the symbols have been '
'mapped to source locations. By default, a tempfile is '
'used and is deleted when the program terminates.'
'This argument is only valid when using --library.')
opts, args = parser.parse_args()
if ((not opts.library) and (not opts.nm_in)) or (opts.library and opts.nm_in):
parser.error('exactly one of --library or --nm-in is required')
if (opts.nm_in):
if opts.jobs:
print >> sys.stderr, ('WARNING: --jobs has no effect '
'when used with --nm-in')
if opts.arch:
print >> sys.stderr, ('WARNING: --arch has no effect '
'when used with --nm-in')
if not opts.destdir:
parser.error('--destdir is required argument')
if not opts.jobs:
opts.jobs = '1'
if not opts.arch:
opts.arch = 'host-native'
symbols = GetNmSymbols(opts.nm_in, opts.nm_out, opts.library, opts.arch,
opts.jobs, opts.verbose is True)
if not os.path.exists(opts.destdir):
os.makedirs(opts.destdir, 0755)
DumpTreemap(symbols, os.path.join(opts.destdir, 'treemap-dump.js'))
DumpLargestSymbols(symbols,
os.path.join(opts.destdir, 'largest-symbols.js'), 100)
DumpLargestSources(symbols,
os.path.join(opts.destdir, 'largest-sources.js'), 100)
DumpLargestVTables(symbols,
os.path.join(opts.destdir, 'largest-vtables.js'), 100)
# TODO(andrewhayden): Switch to D3 for greater flexibility
treemap_out = os.path.join(opts.destdir, 'webtreemap')
if not os.path.exists(treemap_out):
os.makedirs(treemap_out, 0755)
treemap_src = os.path.join('third_party', 'webtreemap', 'src',
'webtreemap-gh-pages')
shutil.copy(os.path.join(treemap_src, 'COPYING'), treemap_out)
shutil.copy(os.path.join(treemap_src, 'webtreemap.js'), treemap_out)
shutil.copy(os.path.join(treemap_src, 'webtreemap.css'), treemap_out)
shutil.copy(os.path.join('tools', 'binary_size', 'template', 'index.html'),
opts.destdir)
if opts.verbose:
print 'Report saved to ' + opts.destdir + '/index.html'
if __name__ == '__main__':
sys.exit(main())
\ No newline at end of file
<!DOCTYPE html>
<!--
Copyright 2014 The Chromium Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.
-->
<html>
<head>
<title>Binary Size Analysis</title>
<link rel='stylesheet' href='webtreemap/webtreemap.css'>
<style>
body { font-family: sans-serif; }
tt, pre { font-family: WebKitWorkaround, monospace; }
#map {
margin: 0 auto;
position: relative;
cursor: pointer;
-webkit-user-select: none;
}
#table {
border: 1px solid black;
}
.treemaplegend {
margin: 0 auto;
position: relative;
}
.webtreemap-symbol-vtable {
background: #FFFFAA;
}
.webtreemap-node:hover {
border-color: red;
background: linear-gradient(rgb(240,240,200), rgb(180,180,200));
}
</style>
<script src='webtreemap/webtreemap.js'></script>
<script src='treemap-dump.js'></script>
<script src='largest-symbols.js'></script>
<script src='largest-sources.js'></script>
<script src='largest-vtables.js'></script>
</head>
<body onload='show_report_treemap()'>
<div style='text-align: center; margin-bottom: 2em;'>
<h1>Binary Size Analysis</h1>
<a href='#' onclick='show_report_treemap()'>Spatial Treemap</a>
~
<a href='#' onclick='show_report_largest_sources()'>Largest Sources</a>
~
<a href='#' onclick='show_report_largest_symbols()'>Largest Symbols</a>
~
<a href='#' onclick='show_report_largest_vtables()'>Largest VTables</a>
</div>
<div id='report'></div>
<script>
function escape(str) {
return str.replace(/&/g, '&amp;')
.replace(/"/g, '&quot;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
}
var treemap_width = 800;
var treemap_height = 450;
function show_report_treemap() {
console.log('displaying treemap')
var div = document.getElementById('report');
var w = window.treemap_width;
var h = window.treemap_height;
div.innerHTML = '<div style=\'text-align: center;\'>' +
'<button onclick=\'zoomInTreemap()\'>Zoom In</button>' +
', <button onclick=\'zoomOutTreemap()\'>Zoom Out</button>' +
' or resize to: ' +
'<input type=text size=5 id=treemap_width value=' + w + '>x' +
'<input type=text size=5 id=treemap_height value=' + h + '>' +
'<button onclick=\'resizeTreemap()\'>Go</button>' +
'<br><em>Click on a box to zoom in. ' +
'Click on the outermost box to zoom out.</em>' +
'<br>Legend: <table border=1 class=\'treemaplegend\' cellborder=1><tr>' +
'<td class=\'webtreemap-symbol-bss\'>BSS</td>' +
'<td class=\'webtreemap-symbol-data\'>Data</td>' +
'<td class=\'webtreemap-symbol-code\'>Code</td>' +
'<td class=\'webtreemap-symbol-read-only_data\'>RO Data</td>' +
'<td class=\'webtreemap-symbol-weak_symbol\'>Weak</td>' +
'<td class=\'webtreemap-symbol-vtable\'>VTable</td>' +
'</tr></table>' +
'<br>' +
'<div id=\'map\' ' +
'style=\'width: ' + w + 'px; height: ' + h + 'px;\'>' +
'</div></div>';
var map = document.getElementById('map');
appendTreemap(map, kTree);
}
function zoomInTreemap() {
window.treemap_width = Math.round(window.treemap_width * 1.25);
window.treemap_height = Math.round(window.treemap_height * 1.25);
show_report_treemap();
}
function zoomOutTreemap() {
window.treemap_width = Math.round(window.treemap_width / 1.25);
window.treemap_height = Math.round(window.treemap_height / 1.25);
show_report_treemap();
}
function resizeTreemap() {
window.treemap_width = document.getElementById('treemap_width').value;
window.treemap_height = document.getElementById('treemap_height').value;
show_report_treemap();
}
function show_report_largest_symbols() {
console.log('displaying largest-symbols report')
var div = document.getElementById('report');
div.innerHTML = '<div><table id=\'list\' border=1><tr>' +
'<th>Rank</th><th>Size</th><th>Type</th><th>Source</th>' +
'</tr></table>';
var list = document.getElementById('list');
for (var i = 0; i < largestSymbols.length; i++) {
var record = largestSymbols[i];
var link;
if (record.location.indexOf('out') == 0) {
link = record.location;
} else {
link = '<a href="https://code.google.com/p/chromium/codesearch#chromium/src/'
+ record.location + '">' + escape(record.location) + '</a>';
}
list.innerHTML += '<tr>'
+ '<td>' + (i+1) + '</td>'
+ '<td>' + escape(record.size) + '</td>'
+ '<td style=\'white-space: nowrap;\'>' + escape(record.type) + '</td>'
+ '<td>' + link + ':<br>'
+ escape(record.symbol) + '</td>'
+ '</tr>';
}
}
function show_report_largest_sources() {
console.log('displaying largest-sources report')
var div = document.getElementById('report');
div.innerHTML = '<div><table id=\'list\' border=1><tr>' +
'<th>Rank</th><th>Size</th><th>Symbol Count</th><th>Source</th>' +
'</tr></table>';
var list = document.getElementById('list');
for (var i = 0; i < largestSources.length; i++) {
var record = largestSources[i];
var link;
if (record.location.indexOf('out') == 0) {
link = record.location;
} else {
link = '<a href="https://code.google.com/p/chromium/codesearch#chromium/src/'
+ record.location + '">' + escape(record.location) + '</a>';
}
list.innerHTML += '<tr>'
+ '<td>' + (i+1) + '</td>'
+ '<td>' + escape(record.size) + '</td>'
+ '<td>' + escape(record.symbol_count) + '</td>'
+ '<td>' + link + '</td>'
+ '</tr>';
}
}
function show_report_largest_vtables() {
console.log('displaying largest-vtables report')
var div = document.getElementById('report');
div.innerHTML = '<div><table id=\'list\' border=1><tr>' +
'<th>Rank</th><th>Size</th><th>Symbol</th><th>Source</th>' +
'</tr></table>';
var list = document.getElementById('list');
for (var i = 0; i < largestVTables.length; i++) {
var record = largestVTables[i];
var link;
if (record.location.indexOf('out') == 0) {
link = record.location;
} else {
link = '<a href="https://code.google.com/p/chromium/codesearch#chromium/src/'
+ record.location + '">' + escape(record.location) + '</a>';
}
list.innerHTML += '<tr>'
+ '<td>' + (i+1) + '</td>'
+ '<td>' + escape(record.size) + '</td>'
+ '<td>' + escape(record.symbol) + '</td>'
+ '<td>' + link + '</td>'
+ '</tr>';
}
}
</script>
</body>
</html>
\ No newline at end of file
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment