Reland "win: write a deterministic-ish timestamp into the PE/COFF header...

Reland "win: write a deterministic-ish timestamp into the PE/COFF header instead of the current time"

This reverts commit 2df0c83a.

Reason for revert: official android should be fine with this after #583420

Original change's description:
> Revert "win: write a deterministic-ish timestamp into the PE/COFF header instead of the current time"
> 
> This reverts commit ef36dc19.
> 
> Reason for revert: This turns out to break the official android build, which apparently was relying on a broken aspect of the build that this fixed :(. See https://crbug.com/871173.
> 
> Original change's description:
> > win: write a deterministic-ish timestamp into the PE/COFF header instead of the current time
> >
> > We used to set the timestamp to a hash of the binary, similar to
> > https://blogs.msdn.microsoft.com/oldnewthing/20180103-00/?p=97705
> > However, that caused an appcompat warning on Windows 7 to appear, which
> > interpreted the hash as a timestamp. (It's possible that https://llvm.org/PR38429
> > could help with that, but my guess it won't have an effect on Windows 7,
> > which likely always believes that the the coff timestamp field always stores
> > a timestamp).
> >
> > So currently we write the current time during linking in that field, but that's
> > bad for build determinism and that in turn is bad for swarming test result cachability.
> >
> > build/write_build_date_header.py already creates a deterministic BUILD_DATE
> > with several tradeoffs. Cachability wants this to change infrequently, but
> > things like HSTS need a "real" build date and want this to change frequently.
> > The compromise is: The date changes once per day in official builds, and
> > once a month in regular builds.
> >
> > (We could use /Brepro in ldflags instead of /TIMESTAMP for unofficial builds to get
> > the binary hash in the timestamp, but having the header timestamp match the BUILD_DATE
> > define seems nice.)
> >
> > So let's use that same time as timestamp in the PE/COFF header. lld-link has a
> > /TIMESTAMP: flag we can use to pass in an explicit timestamp.
> >
> > Since tools can't have deps, we need to compute the timestamp at gn time,
> > so split write_build_date_header.py in two pieces: build/compute_build_timestamp.py
> > that just prints the timestamp we want to use, and the old write_build_date_header.py, which
> > now takes that timestamp and writes the header file.
> >
> > Call compute_build_timestamp.py at gn time so that we can pass it in ldflags, and
> > pass the resultl to write_build_date_header.py which keeps running as an action
> > during build time (so that we at least don't need to write a file at gn time).
> >
> > An additional wrinkle here is that the PE/COFF timestamp is used as one of just two
> > keys per binary for uploading PE binaries to the symbol server, the other being file size.
> > https://bugs.llvm.org/show_bug.cgi?id=35914#c0 has a good description of this, and
> > tools/symsrc/img_fingerprint.py's GetImgFingerprint() is our implementation of it.
> > But since we only upload binaries with symbols for official chrome builds to the symbol server,
> > a timestamp that changes once a day should be still enough. (32-bit and 64-bit chromes
> > have the same filename, and we might rarely build canary and beta and stable all on the
> > same day, but them all being the same size seems highly unlikely.)
> >
> > Bug: 843199,804926,330260
> > Change-Id: I1d4193cc537ae0c4b2d6ac9281fad29de754dd6c
> > Reviewed-on: https://chromium-review.googlesource.com/1161104
> > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > Reviewed-by: Hans Wennborg <hans@chromium.org>
> > Commit-Queue: Nico Weber <thakis@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#580585}
> 
> TBR=thakis@chromium.org,hans@chromium.org,dpranke@chromium.org
> NOTRY=true
> 
> # Not skipping CQ checks because original CL landed > 1 day ago.
> 
> Bug: 843199, 804926, 330260
> Change-Id: Ib93697a82f8a9d3fb303b763609e82e0612887cd
> Reviewed-on: https://chromium-review.googlesource.com/1166203
> Commit-Queue: Hans Wennborg <hans@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#581485}

TBR=thakis@chromium.org,hans@chromium.org,dpranke@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 843199, 804926, 330260
Change-Id: I136e405c84eba3f61a4ac96b2017a34ade0cfba6
Reviewed-on: https://chromium-review.googlesource.com/1179281
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583945}

base/BUILD.gn

@@ -29,7 +29,7 @@ import("//build/config/nacl/config.gni")
 import("//build/config/sysroot.gni")
 import("//build/config/ui.gni")
 import("//build/nocompile.gni")
-import("//build/util/lastchange.gni")
+import("//build/timestamp.gni")
 import("//testing/libfuzzer/fuzzer_test.gni")
 import("//testing/test.gni")
 
@@ -2770,22 +2770,14 @@ test("base_unittests") {
 action("build_date") {
   script = "//build/write_build_date_header.py"
 
-  # Force recalculation if there's been a change.
-  inputs = [
-    lastchange_file,
-  ]
   outputs = [
     "$target_gen_dir/generated_build_date.h",
   ]
 
-  args =
-      [ rebase_path("$target_gen_dir/generated_build_date.h", root_build_dir) ]
-
-  if (is_official_build) {
-    args += [ "official" ]
-  } else {
-    args += [ "default" ]
-  }
+  args = [
+    rebase_path("$target_gen_dir/generated_build_date.h", root_build_dir),
+    build_timestamp,
+  ]
 }
 
 if (enable_nocompile_tests) {

build/compute_build_timestamp.py

+#!/usr/bin/env python
+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+"""Returns a timestamp that approximates the build date.
+
+build_type impacts the timestamp generated, both relative to the date of the
+last recent commit:
+- default: the build date is set to the most recent first Sunday of a month at
+  5:00am. The reason is that it is a time where invalidating the build cache
+  shouldn't have major reprecussions (due to lower load).
+- official: the build date is set to the current date at 5:00am, or the day
+  before if the current time is before 5:00am.
+Either way, it is guaranteed to be in the past and always in UTC.
+"""
+
+# The requirements for the timestamp:
+# (1) for the purposes of continuous integration, longer duration
+#     between cache invalidation is better, but >=1mo is preferable.
+# (2) for security purposes, timebombs would ideally be as close to
+#     the actual time of the build as possible. It must be in the past.
+# (3) HSTS certificate pinning is valid for 70 days. To make CI builds enforce
+#     HTST pinning, <=1mo is preferable.
+#
+# On Windows, the timestamp is also written in the PE/COFF file header of
+# executables of dlls.  That timestamp and the executable's file size are
+# the only two pieces of information that identify a given executable on
+# the symbol server, so rarely changing timestamps can cause conflicts there
+# as well. We only upload symbols for official builds to the symbol server.
+
+import argparse
+import calendar
+import datetime
+import doctest
+import os
+import sys
+
+
+THIS_DIR = os.path.abspath(os.path.dirname(__file__))
+
+
+def GetFirstSundayOfMonth(year, month):
+  """Returns the first sunday of the given month of the given year.
+
+  >>> GetFirstSundayOfMonth(2016, 2)
+  7
+  >>> GetFirstSundayOfMonth(2016, 3)
+  6
+  >>> GetFirstSundayOfMonth(2000, 1)
+  2
+  """
+  weeks = calendar.Calendar().monthdays2calendar(year, month)
+  # Return the first day in the first week that is a Sunday.
+  return [date_day[0] for date_day in weeks[0] if date_day[1] == 6][0]
+
+
+def GetBuildDate(build_type, utc_now):
+  """Gets the approximate build date given the specific build type.
+
+  >>> GetBuildDate('default', datetime.datetime(2016, 2, 6, 1, 2, 3))
+  datetime.datetime(2016, 1, 3, 1, 2, 3)
+  >>> GetBuildDate('default', datetime.datetime(2016, 2, 7, 5))
+  datetime.datetime(2016, 2, 7, 5, 0)
+  >>> GetBuildDate('default', datetime.datetime(2016, 2, 8, 5))
+  datetime.datetime(2016, 2, 7, 5, 0)
+  >>> GetBuildDate('official', datetime.datetime(2016, 2, 8, 5))
+  datetime.datetime(2016, 2, 8, 5, 0)
+  """
+  day = utc_now.day
+  month = utc_now.month
+  year = utc_now.year
+  if build_type != 'official':
+    first_sunday = GetFirstSundayOfMonth(year, month)
+    # If our build is after the first Sunday, we've already refreshed our build
+    # cache on a quiet day, so just use that day.
+    # Otherwise, take the first Sunday of the previous month.
+    if day >= first_sunday:
+      day = first_sunday
+    else:
+      month -= 1
+      if month == 0:
+        month = 12
+        year -= 1
+      day = GetFirstSundayOfMonth(year, month)
+  return datetime.datetime(
+      year, month, day, utc_now.hour, utc_now.minute, utc_now.second)
+
+
+def main():
+  if doctest.testmod()[0]:
+    return 1
+  argument_parser = argparse.ArgumentParser()
+  argument_parser.add_argument(
+      'build_type', help='The type of build', choices=('official', 'default'))
+  args = argument_parser.parse_args()
+
+  # The mtime of the revision in build/util/LASTCHANGE is stored in a file
+  # next to it. Read it, to get a deterministic time close to "now".
+  # That date is then modified as described at the top of the file so that
+  # it changes less frequently than with every commit.
+  # This intentionally always uses build/util/LASTCHANGE's commit time even if
+  # use_dummy_lastchange is set.
+  lastchange_file = os.path.join(THIS_DIR, 'util', 'LASTCHANGE.committime')
+  last_commit_timestamp = int(open(lastchange_file).read())
+  now = datetime.datetime.utcfromtimestamp(last_commit_timestamp)
+
+  if now.hour < 5:
+    # The time is locked at 5:00 am in UTC to cause the build cache
+    # invalidation to not happen exactly at midnight. Use the same calculation
+    # as the day before.
+    # See //base/build_time.cc.
+    now = now - datetime.timedelta(days=1)
+  now = datetime.datetime(now.year, now.month, now.day, 5, 0, 0)
+  build_date = GetBuildDate(args.build_type, now)
+  print int(calendar.timegm(build_date.utctimetuple()))
+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main())

build/config/win/BUILD.gn

@@ -8,6 +8,7 @@ import("//build/config/clang/clang.gni")
 import("//build/config/compiler/compiler.gni")
 import("//build/config/sanitizers/sanitizers.gni")
 import("//build/config/win/visual_studio_version.gni")
+import("//build/timestamp.gni")
 import("//build/toolchain/goma.gni")
 import("//build/toolchain/toolchain.gni")
 
@@ -106,12 +107,23 @@ config("compiler") {
     cflags += [ "/Brepro" ]
   }
 
+  ldflags = []
+
+  if (use_lld) {
+    # lld defaults to writing the current time in the pe/coff header.
+    # For build reproducibility, pass an explicit timestamp. See
+    # build/compute_build_timestamp.py for how the timestamp is chosen.
+    # (link.exe also writes the current time, but it doesn't have a flag to
+    # override that behavior.)
+    ldflags += [ "/TIMESTAMP:" + build_timestamp ]
+  }
+
   if (!is_debug && !is_component_build) {
     # Enable standard linker optimizations like GC (/OPT:REF) and ICF in static
     # release builds. These are implied by /PROFILE below, but /PROFILE is
     # incompatible with /debug:fastlink and LLD ignores it as of this writing.
     # Release builds always want these optimizations, so enable them explicitly.
-    ldflags = [
+    ldflags += [
       "/OPT:REF",
       "/OPT:ICF",
       "/INCREMENTAL:NO",

build/dotfile_settings.gni

@@ -24,6 +24,7 @@ build_dotfile_settings = {
     "//build/config/sysroot.gni",
     "//build/config/win/BUILD.gn",
     "//build/config/win/visual_studio_version.gni",
+    "//build/timestamp.gni",
     "//build/toolchain/BUILD.gn",
     "//build/toolchain/concurrent_links.gni",
     "//build/toolchain/mac/BUILD.gn",

build/timestamp.gni

+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+#
+# Defines the build_timestamp variable.
+
+import("//build/util/lastchange.gni")
+
+if (is_official_build) {
+  official_name = "official"
+} else {
+  official_name = "default"
+}
+
+# This will return a timestamp that's different each day (official builds)
+# of each month (regular builds).  Just rely on gn rerunning due to other
+# changes to keep this up to date.  (Bots run gn on each build, and for devs
+# the timestamp being 100% accurate doesn't matter.)
+# See compute_build_timestamp.py for tradeoffs for picking the timestamp.
+build_timestamp = exec_script("compute_build_timestamp.py",
+                              [ official_name ],
+                              "trim string",
+                              [ lastchange_file ])

build/write_build_date_header.py

@@ -2,110 +2,25 @@
 # Copyright (c) 2016 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
-"""Writes a file that contains a define that approximates the build date.
-
-build_type impacts the timestamp generated:
-- default: the build date is set to the most recent first Sunday of a month at
-  5:00am. The reason is that it is a time where invalidating the build cache
-  shouldn't have major repercussions (due to lower load).
-- official: the build date is set to the current date at 5:00am, or the day
-  before if the current time is before 5:00am.
-Either way, it is guaranteed to be in the past and always in UTC.
-
-It is also possible to explicitly set a build date to be used.
-"""
+"""Takes a timestamp and writes it in as readable text to a .h file."""
 
 import argparse
-import calendar
 import datetime
-import doctest
 import os
 import sys
 
 
-THIS_DIR = os.path.abspath(os.path.dirname(__file__))
-
-
-def GetFirstSundayOfMonth(year, month):
-  """Returns the first sunday of the given month of the given year.
-
-  >>> GetFirstSundayOfMonth(2016, 2)
-  7
-  >>> GetFirstSundayOfMonth(2016, 3)
-  6
-  >>> GetFirstSundayOfMonth(2000, 1)
-  2
-  """
-  weeks = calendar.Calendar().monthdays2calendar(year, month)
-  # Return the first day in the first week that is a Sunday.
-  return [date_day[0] for date_day in weeks[0] if date_day[1] == 6][0]
-
-
-def GetBuildDate(build_type, utc_now):
-  """Gets the approximate build date given the specific build type.
-
-  >>> GetBuildDate('default', datetime.datetime(2016, 2, 6, 1, 2, 3))
-  'Jan 03 2016 01:02:03'
-  >>> GetBuildDate('default', datetime.datetime(2016, 2, 7, 5))
-  'Feb 07 2016 05:00:00'
-  >>> GetBuildDate('default', datetime.datetime(2016, 2, 8, 5))
-  'Feb 07 2016 05:00:00'
-  """
-  day = utc_now.day
-  month = utc_now.month
-  year = utc_now.year
-  if build_type != 'official':
-    first_sunday = GetFirstSundayOfMonth(year, month)
-    # If our build is after the first Sunday, we've already refreshed our build
-    # cache on a quiet day, so just use that day.
-    # Otherwise, take the first Sunday of the previous month.
-    if day >= first_sunday:
-      day = first_sunday
-    else:
-      month -= 1
-      if month == 0:
-        month = 12
-        year -= 1
-      day = GetFirstSundayOfMonth(year, month)
-  now = datetime.datetime(
-      year, month, day, utc_now.hour, utc_now.minute, utc_now.second)
-  return '{:%b %d %Y %H:%M:%S}'.format(now)
-
-
 def main():
-  if doctest.testmod()[0]:
-    return 1
-  argument_parser = argparse.ArgumentParser(
-      description=sys.modules[__name__].__doc__,
-      formatter_class=argparse.RawDescriptionHelpFormatter)
+  argument_parser = argparse.ArgumentParser()
   argument_parser.add_argument('output_file', help='The file to write to')
-  argument_parser.add_argument(
-      'build_type', help='The type of build', choices=('official', 'default'))
+  argument_parser.add_argument('timestamp')
   args = argument_parser.parse_args()
 
-  # The mtime of the revision in build/util/LASTCHANGE is stored in a file
-  # next to it. Read it, to get a deterministic time close to "now".
-  # That date is then modified as described at the top of the file so that
-  # it changes less frequently than with every commit.
-  # This intentionally always uses build/util/LASTCHANGE's commit time even if
-  # use_dummy_lastchange is set.
-  lastchange_file = os.path.join(THIS_DIR, 'util', 'LASTCHANGE.committime')
-  last_commit_timestamp = int(open(lastchange_file).read())
-  now = datetime.datetime.utcfromtimestamp(last_commit_timestamp)
-
-  if now.hour < 5:
-    # The time is locked at 5:00 am in UTC to cause the build cache
-    # invalidation to not happen exactly at midnight. Use the same calculation
-    # as the day before.
-    # See //base/build_time.cc.
-    now = now - datetime.timedelta(days=1)
-  now = datetime.datetime(now.year, now.month, now.day, 5, 0, 0)
-  build_date = GetBuildDate(args.build_type, now)
-
+  date = datetime.datetime.utcfromtimestamp(int(args.timestamp))
   output = ('// Generated by //build/write_build_date_header.py\n'
            '#ifndef BUILD_DATE\n'
-           '#define BUILD_DATE "{}"\n'
-           '#endif // BUILD_DATE\n'.format(build_date))
+           '#define BUILD_DATE "{:%b %d %Y %H:%M:%S}"\n'
+           '#endif // BUILD_DATE\n'.format(date))
 
   current_contents = ''
   if os.path.isfile(args.output_file):