Use the actual Chrome OS AFDO profiles on Android

(This is also the third attempt roll to a newer AFDO profile, which
we're currently using internally.)

While the gs bucket that holds these profiles is restricted to Googlers
only, our Chrome OS friends jump through hoops specifically to make
these AFDO profiles world-readable. So, all of the crazy redaction and
such that we're currently doing is entirely unnecessary. Woo!

As noted in the script, due to the permissions setup, actually accessing
these profiles is a bit awkward. But it's workable.

also ran update_afdo_profile.py on a few profiles from a non-Google
machine, and it succeeded on the ones I tried. Also ran
build/install-build-deps-android.sh.

Bug: 804479
Test: `gclient runhooks` and built for Android with/without clank. I
Change-Id: I20d973b62251ebe249e80fad6051bd3a6805aa12
Reviewed-on: https://chromium-review.googlesource.com/879849
Commit-Queue: George Burgess <gbiv@chromium.org>
Reviewed-by: agrieve <agrieve@chromium.org>
Reviewed-by: Tommy Nyquist <nyquist@chromium.org>
Cr-Commit-Position: refs/heads/master@{#531368}

.gitignore

@@ -68,7 +68,8 @@ vs-chromium-project.txt
 /c
 /cdm
 /ceee/internal/
-/chrome/android/profiles/chrome-profile-*.prof
+/chrome/android/profiles/afdo.prof
+/chrome/android/profiles/local.txt
 /chrome/angle_unittests_run.xml
 /chrome/build/chrome.x64.orderfile
 /chrome/build/chrome.x86.orderfile

DEPS

@@ -1222,9 +1222,14 @@ hooks = [
     'action': ['src/build/cipd/cipd_wrapper.py',
                '--chromium-root', 'src',
                '--ensure-file', 'src/build/cipd/android/android.ensure',
-               '--ensure-file', 'src/chrome/android/android.ensure',
     ],
   },
+  {
+    'name': 'Fetch Android AFDO profile',
+    'pattern': '.',
+    'condition': 'checkout_android',
+    'action': ['vpython', 'src/chrome/android/profiles/update_afdo_profile.py'],
+  },
   {
     # This downloads SDK extras and puts them in the
     # third_party/android_tools/sdk/extras directory.

build/config/android/config.gni

@@ -39,16 +39,16 @@ if (is_android) {
       # Android SDK release. Currently, only "o_mr1" is publicly supported.
       android_sdk_release = default_android_sdk_release
     }
+  }
 
-    # This is the result of profiling Chromium. Blindly applying it to arbitrary
-    # downstream projects and hoping it'll grant a speedup seems fragile. (Not
-    # to mention that it resides in a Chromium-specific directory.)
-    if (build_with_chromium) {
-      # The default AFDO profile used by public builds. Value may differ in
-      # internal builds.
-      clang_default_afdo_profile =
-          rebase_path("//chrome/android/profiles/chrome-profile-3309-text.prof")
-    }
+  # This is the result of profiling Chromium. Blindly applying it to arbitrary
+  # downstream projects and hoping it'll grant a speedup seems fragile. (Not
+  # to mention that it resides in a Chromium-specific directory.)
+  # FIXME(gbiv): remove the !defined() check after removing custom profiling
+  # logic from //clank.
+  if (!defined(clang_default_afdo_profile) && build_with_chromium) {
+    clang_default_afdo_profile =
+        rebase_path("//chrome/android/profiles/afdo.prof")
   }
 
   if (!defined(extra_chrome_shared_library_configs)) {

build/install-build-deps-android.sh

@@ -37,6 +37,9 @@ sudo apt-get -y install libncurses5:i386 libstdc++6:i386 zlib1g:i386
 # Required for apk-patch-size-estimator
 sudo apt-get -y install bsdiff
 
+# Needed to unpack the profiles we pull with `gclient runhooks`
+sudo apt-get -y install bzip2
+
 # Do our own error handling for java.
 set +e
 

chrome/android/profiles/cipd.yaml

-# Copyright 2018 The Chromium Authors. All rights reserved.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-# To create a CIPD package, run the following command.
-# cipd create --pkg-def cipd.yaml -tag version:version-of-afdo-profile
-package: chromium/afdo/profiles/android
-description: AFDO profiles collected for Chromium on Android
-
-# FIXME(gbiv): When we can specify dependencies on the profile in the build
-# system, settle on a profile name that doesn't include the version.
-data:
-  - file: chrome-profile-3309-text.prof

chrome/android/profiles/newest.txt

+chromeos-chrome-amd64-65.0.3322.0_rc-r1.afdo.bz2

chrome/android/profiles/update_afdo_profile.py

+#!/usr/bin/python
+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""This script is used to update our local AFDO profiles.
+
+This uses profiles of Chrome provided by our friends from Chrome OS. Though the
+profiles are available externally, the bucket they sit in is otherwise
+unreadable by non-Googlers. Gsutil usage with this bucket is therefore quite
+awkward: you can't do anything but `cp` certain files with an external account,
+and you can't even do that if you're not yet authenticated.
+
+No authentication is necessary if you pull these profiles directly over
+https."""
+
+import argparse
+import os
+import subprocess
+import sys
+import urllib
+
+GS_BASE_URL = 'https://storage.googleapis.com/chromeos-prebuilt/afdo-job/llvm'
+PROFILE_DIRECTORY = os.path.abspath(os.path.dirname(__file__))
+LOCAL_PROFILE_PATH = os.path.join(PROFILE_DIRECTORY, 'afdo.prof')
+
+# We use these to track the local profile; newest.txt is owned by git and tracks
+# the name of the newest profile we should pull, and local.txt is the most
+# recent profile we've successfully pulled.
+NEWEST_PROFILE_NAME_PATH = os.path.join(PROFILE_DIRECTORY, 'newest.txt')
+LOCAL_PROFILE_NAME_PATH = os.path.join(PROFILE_DIRECTORY, 'local.txt')
+
+
+def ReadUpToDateProfileName():
+  with open(NEWEST_PROFILE_NAME_PATH) as f:
+    return f.read().strip()
+
+
+def ReadLocalProfileName():
+  try:
+    with open(LOCAL_PROFILE_NAME_PATH) as f:
+      return f.read().strip()
+  except IOError:
+    # Assume it either didn't exist, or we couldn't read it. In either case, we
+    # should probably grab a new profile (and, in doing so, make this file sane
+    # again)
+    return None
+
+
+def WriteLocalProfileName(name):
+  with open(LOCAL_PROFILE_NAME_PATH, 'w') as f:
+    f.write(name)
+
+
+def CheckCallOrExit(cmd):
+  proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+  stdout, stderr = proc.communicate()
+  exit_code = proc.wait()
+  if not exit_code:
+    return
+
+  complaint_lines = [
+      '## %s failed with exit code %d' % (cmd[0], exit_code),
+      '## Full command: %s' % cmd,
+      '## Stdout:\n' + stdout,
+      '## Stderr:\n' + stderr,
+  ]
+  print >>sys.stderr, '\n'.join(complaint_lines)
+  sys.exit(1)
+
+
+def RetrieveProfile(desired_profile_name, out_path):
+  # vpython is > python 2.7.9, so we can expect urllib to validate HTTPS certs
+  # properly.
+  compressed_path = out_path + '.bz2'
+  profile = urllib.urlretrieve(GS_BASE_URL + '/' + desired_profile_name,
+                               filename=compressed_path)
+
+  # NOTE: we can't use Python's bzip module, since it doesn't support
+  # multi-stream bzip files. It will silently succeed and give us a garbage
+  # profile.
+  # bzip2 removes the compressed file on success.
+  CheckCallOrExit(['bzip2', '-d', compressed_path])
+
+
+def CleanProfilesDirectory():
+  # Start with a clean slate, removing old profiles/downloads/etc.
+  old_artifacts = (p for p in os.listdir(PROFILE_DIRECTORY) if
+                   p.startswith('chromeos-chrome-'))
+  for artifact in old_artifacts:
+    os.remove(os.path.join(PROFILE_DIRECTORY, artifact))
+
+
+def main():
+  parser = argparse.ArgumentParser('Downloads profiles provided by Chrome OS')
+  parser.add_argument('-f', '--force', action='store_true',
+                      help='Fetch a profile even if the local one is current')
+  args = parser.parse_args()
+
+  up_to_date_profile = ReadUpToDateProfileName()
+  if not args.force:
+    local_profile_name = ReadLocalProfileName()
+    # In a perfect world, the local profile should always exist if we
+    # successfully read local_profile_name. If it's gone, though, the user
+    # probably removed it as a way to get us to download it again.
+    if local_profile_name == up_to_date_profile \
+        and os.path.exists(LOCAL_PROFILE_PATH):
+      return 0
+
+  CleanProfilesDirectory()
+
+  new_tmpfile = LOCAL_PROFILE_PATH + '.new'
+  RetrieveProfile(up_to_date_profile, new_tmpfile)
+  os.rename(new_tmpfile, LOCAL_PROFILE_PATH)
+  WriteLocalProfileName(up_to_date_profile)
+
+
+if __name__ == '__main__':
+  sys.exit(main())