win: write a deterministic-ish timestamp into the PE/COFF header instead of the current time

We used to set the timestamp to a hash of the binary, similar to
https://blogs.msdn.microsoft.com/oldnewthing/20180103-00/?p=97705
However, that caused an appcompat warning on Windows 7 to appear, which
interpreted the hash as a timestamp. (It's possible that https://llvm.org/PR38429
could help with that, but my guess it won't have an effect on Windows 7,
which likely always believes that the the coff timestamp field always stores
a timestamp).

So currently we write the current time during linking in that field, but that's
bad for build determinism and that in turn is bad for swarming test result cachability.

build/write_build_date_header.py already creates a deterministic BUILD_DATE
with several tradeoffs. Cachability wants this to change infrequently, but
things like HSTS need a "real" build date and want this to change frequently.
The compromise is: The date changes once per day in official builds, and
once a month in regular builds.

(We could use /Brepro in ldflags instead of /TIMESTAMP for unofficial builds to get
the binary hash in the timestamp, but having the header timestamp match the BUILD_DATE
define seems nice.)

So let's use that same time as timestamp in the PE/COFF header. lld-link has a
/TIMESTAMP: flag we can use to pass in an explicit timestamp.

Since tools can't have deps, we need to compute the timestamp at gn time,
so split write_build_date_header.py in two pieces: build/compute_build_timestamp.py
that just prints the timestamp we want to use, and the old write_build_date_header.py, which
now takes that timestamp and writes the header file.

Call compute_build_timestamp.py at gn time so that we can pass it in ldflags, and
pass the resultl to write_build_date_header.py which keeps running as an action
during build time (so that we at least don't need to write a file at gn time).

An additional wrinkle here is that the PE/COFF timestamp is used as one of just two
keys per binary for uploading PE binaries to the symbol server, the other being file size.
https://bugs.llvm.org/show_bug.cgi?id=35914#c0 has a good description of this, and
tools/symsrc/img_fingerprint.py's GetImgFingerprint() is our implementation of it.
But since we only upload binaries with symbols for official chrome builds to the symbol server,
a timestamp that changes once a day should be still enough. (32-bit and 64-bit chromes
have the same filename, and we might rarely build canary and beta and stable all on the
same day, but them all being the same size seems highly unlikely.)

Bug: 843199,804926,330260
Change-Id: I1d4193cc537ae0c4b2d6ac9281fad29de754dd6c
Reviewed-on: https://chromium-review.googlesource.com/1161104Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#580585}

base/BUILD.gn

@@ -29,6 +29,7 @@ import("//build/config/nacl/config.gni")
 import("//build/config/sysroot.gni")
 import("//build/config/ui.gni")
 import("//build/nocompile.gni")
+import("//build/timestamp.gni")
 import("//build/util/lastchange.gni")
 import("//testing/libfuzzer/fuzzer_test.gni")
 import("//testing/test.gni")
@@ -2738,14 +2739,10 @@ action("build_date") {
     "$target_gen_dir/generated_build_date.h",
   ]
 
-  args =
-      [ rebase_path("$target_gen_dir/generated_build_date.h", root_build_dir) ]
-
-  if (is_official_build) {
-    args += [ "official" ]
-  } else {
-    args += [ "default" ]
-  }
+  args = [
+    rebase_path("$target_gen_dir/generated_build_date.h", root_build_dir),
+    build_timestamp,
+  ]
 }
 
 if (enable_nocompile_tests) {

build/compute_build_timestamp.py

+#!/usr/bin/env python
+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+"""Returns a timestamp that approximates the build date.
+
+build_type impacts the timestamp generated:
+- default: the build date is set to the most recent first Sunday of a month at
+  5:00am. The reason is that it is a time where invalidating the build cache
+  shouldn't have major reprecussions (due to lower load).
+- official: the build date is set to the current date at 5:00am, or the day
+  before if the current time is before 5:00am.
+Either way, it is guaranteed to be in the past and always in UTC.
+"""
+
+# The requirements for the timestamp:
+# (1) for the purposes of continuous integration, longer duration
+#     between cache invalidation is better, but >=1mo is preferable.
+# (2) for security purposes, timebombs would ideally be as close to
+#     the actual time of the build as possible. It must be in the past.
+# (3) HSTS certificate pinning is valid for 70 days. To make CI builds enforce
+#     HTST pinning, <=1mo is preferable.
+#
+# On Windows, the timestamp is also written in the PE/COFF file header of
+# executables of dlls.  That timestamp and the executable's file size are
+# the only two pieces of information that identify a given executable on
+# the symbol server, so rarely changing timestamps can cause conflicts there
+# as well. We only upload symbols for official builds to the symbol server.
+
+import argparse
+import calendar
+import datetime
+import doctest
+import sys
+
+
+def GetFirstSundayOfMonth(year, month):
+  """Returns the first sunday of the given month of the given year.
+
+  >>> GetFirstSundayOfMonth(2016, 2)
+  7
+  >>> GetFirstSundayOfMonth(2016, 3)
+  6
+  >>> GetFirstSundayOfMonth(2000, 1)
+  2
+  """
+  weeks = calendar.Calendar().monthdays2calendar(year, month)
+  # Return the first day in the first week that is a Sunday.
+  return [date_day[0] for date_day in weeks[0] if date_day[1] == 6][0]
+
+
+def GetBuildDate(build_type, utc_now):
+  """Gets the approximate build date given the specific build type.
+
+  >>> GetBuildDate('default', datetime.datetime(2016, 2, 6, 1, 2, 3))
+  datetime.datetime(2016, 1, 3, 1, 2, 3)
+  >>> GetBuildDate('default', datetime.datetime(2016, 2, 7, 5))
+  datetime.datetime(2016, 2, 7, 5, 0)
+  >>> GetBuildDate('default', datetime.datetime(2016, 2, 8, 5))
+  datetime.datetime(2016, 2, 7, 5, 0)
+  >>> GetBuildDate('official', datetime.datetime(2016, 2, 8, 5))
+  datetime.datetime(2016, 2, 8, 5, 0)
+  """
+  day = utc_now.day
+  month = utc_now.month
+  year = utc_now.year
+  if build_type != 'official':
+    first_sunday = GetFirstSundayOfMonth(year, month)
+    # If our build is after the first Sunday, we've already refreshed our build
+    # cache on a quiet day, so just use that day.
+    # Otherwise, take the first Sunday of the previous month.
+    if day >= first_sunday:
+      day = first_sunday
+    else:
+      month -= 1
+      if month == 0:
+        month = 12
+        year -= 1
+      day = GetFirstSundayOfMonth(year, month)
+  return datetime.datetime(
+      year, month, day, utc_now.hour, utc_now.minute, utc_now.second)
+
+
+def main():
+  if doctest.testmod()[0]:
+    return 1
+  argument_parser = argparse.ArgumentParser()
+  argument_parser.add_argument(
+      'build_type', help='The type of build', choices=('official', 'default'))
+  args = argument_parser.parse_args()
+
+  now = datetime.datetime.utcnow()
+  if now.hour < 5:
+    # The time is locked at 5:00 am in UTC to cause the build cache
+    # invalidation to not happen exactly at midnight. Use the same calculation
+    # as the day before.
+    # See //base/build_time.cc.
+    now = now - datetime.timedelta(days=1)
+  now = datetime.datetime(now.year, now.month, now.day, 5, 0, 0)
+  build_date = GetBuildDate(args.build_type, now)
+  print int(calendar.timegm(build_date.utctimetuple()))
+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main())

build/config/win/BUILD.gn

@@ -8,6 +8,7 @@ import("//build/config/clang/clang.gni")
 import("//build/config/compiler/compiler.gni")
 import("//build/config/sanitizers/sanitizers.gni")
 import("//build/config/win/visual_studio_version.gni")
+import("//build/timestamp.gni")
 import("//build/toolchain/goma.gni")
 import("//build/toolchain/toolchain.gni")
 
@@ -106,12 +107,23 @@ config("compiler") {
     cflags += [ "/Brepro" ]
   }
 
+  ldflags = []
+
+  if (use_lld) {
+    # lld defaults to writing the current time in the pe/coff header.
+    # For build reproducibility, pass an explicit timestamp. See
+    # build/compute_build_timestamp.py for how the timestamp is chosen.
+    # (link.exe also writes the current time, but it doesn't have a flag to
+    # override that behavior.)
+    ldflags += [ "/TIMESTAMP:" + build_timestamp ]
+  }
+
   if (!is_debug && !is_component_build) {
     # Enable standard linker optimizations like GC (/OPT:REF) and ICF in static
     # release builds. These are implied by /PROFILE below, but /PROFILE is
     # incompatible with /debug:fastlink and LLD ignores it as of this writing.
     # Release builds always want these optimizations, so enable them explicitly.
-    ldflags = [
+    ldflags += [
       "/OPT:REF",
       "/OPT:ICF",
       "/INCREMENTAL:NO",

build/dotfile_settings.gni

@@ -24,6 +24,7 @@ build_dotfile_settings = {
     "//build/config/sysroot.gni",
     "//build/config/win/BUILD.gn",
     "//build/config/win/visual_studio_version.gni",
+    "//build/timestamp.gni",
     "//build/toolchain/BUILD.gn",
     "//build/toolchain/concurrent_links.gni",
     "//build/toolchain/mac/BUILD.gn",

build/timestamp.gni

+# Copyright 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+#
+# Defines the build_timestamp variable.
+
+if (is_official_build) {
+  official_name = "official"
+} else {
+  official_name = "default"
+}
+
+# This will return a timestamp that's different each day (official builds)
+# of each month (regular builds).  Just rely on gn rerunning due to other
+# changes to keep this up to date.  (Bots run gn on each build, and for devs
+# the timestamp being 100% accurate doesn't matter.)
+# See compute_build_timestamp.py for tradeoffs for picking the timestamp.
+build_timestamp =
+    exec_script("compute_build_timestamp.py", [ official_name ], "trim string")

build/write_build_date_header.py

@@ -2,98 +2,25 @@
 # Copyright (c) 2016 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
-"""Writes a file that contains a define that approximates the build date.
-
-build_type impacts the timestamp generated:
-- default: the build date is set to the most recent first Sunday of a month at
-  5:00am. The reason is that it is a time where invalidating the build cache
-  shouldn't have major reprecussions (due to lower load).
-- official: the build date is set to the current date at 5:00am, or the day
-  before if the current time is before 5:00am.
-Either way, it is guaranteed to be in the past and always in UTC.
-
-It is also possible to explicitly set a build date to be used.
-"""
+"""Takes a timestamp and writes it in as readable text to a .h file."""
 
 import argparse
-import calendar
 import datetime
-import doctest
 import os
 import sys
 
 
-def GetFirstSundayOfMonth(year, month):
-  """Returns the first sunday of the given month of the given year.
-
-  >>> GetFirstSundayOfMonth(2016, 2)
-  7
-  >>> GetFirstSundayOfMonth(2016, 3)
-  6
-  >>> GetFirstSundayOfMonth(2000, 1)
-  2
-  """
-  weeks = calendar.Calendar().monthdays2calendar(year, month)
-  # Return the first day in the first week that is a Sunday.
-  return [date_day[0] for date_day in weeks[0] if date_day[1] == 6][0]
-
-
-def GetBuildDate(build_type, utc_now):
-  """Gets the approximate build date given the specific build type.
-
-  >>> GetBuildDate('default', datetime.datetime(2016, 2, 6, 1, 2, 3))
-  'Jan 03 2016 01:02:03'
-  >>> GetBuildDate('default', datetime.datetime(2016, 2, 7, 5))
-  'Feb 07 2016 05:00:00'
-  >>> GetBuildDate('default', datetime.datetime(2016, 2, 8, 5))
-  'Feb 07 2016 05:00:00'
-  """
-  day = utc_now.day
-  month = utc_now.month
-  year = utc_now.year
-  if build_type != 'official':
-    first_sunday = GetFirstSundayOfMonth(year, month)
-    # If our build is after the first Sunday, we've already refreshed our build
-    # cache on a quiet day, so just use that day.
-    # Otherwise, take the first Sunday of the previous month.
-    if day >= first_sunday:
-      day = first_sunday
-    else:
-      month -= 1
-      if month == 0:
-        month = 12
-        year -= 1
-      day = GetFirstSundayOfMonth(year, month)
-  now = datetime.datetime(
-      year, month, day, utc_now.hour, utc_now.minute, utc_now.second)
-  return '{:%b %d %Y %H:%M:%S}'.format(now)
-
-
 def main():
-  if doctest.testmod()[0]:
-    return 1
-  argument_parser = argparse.ArgumentParser(
-      description=sys.modules[__name__].__doc__,
-      formatter_class=argparse.RawDescriptionHelpFormatter)
+  argument_parser = argparse.ArgumentParser()
   argument_parser.add_argument('output_file', help='The file to write to')
-  argument_parser.add_argument(
-      'build_type', help='The type of build', choices=('official', 'default'))
+  argument_parser.add_argument('timestamp')
   args = argument_parser.parse_args()
 
-  now = datetime.datetime.utcnow()
-  if now.hour < 5:
-    # The time is locked at 5:00 am in UTC to cause the build cache
-    # invalidation to not happen exactly at midnight. Use the same calculation
-    # as the day before.
-    # See //base/build_time.cc.
-    now = now - datetime.timedelta(days=1)
-  now = datetime.datetime(now.year, now.month, now.day, 5, 0, 0)
-  build_date = GetBuildDate(args.build_type, now)
-
+  date = datetime.datetime.utcfromtimestamp(int(args.timestamp))
   output = ('// Generated by //build/write_build_date_header.py\n'
            '#ifndef BUILD_DATE\n'
-           '#define BUILD_DATE "{}"\n'
-           '#endif // BUILD_DATE\n'.format(build_date))
+           '#define BUILD_DATE "{:%b %d %Y %H:%M:%S}"\n'
+           '#endif // BUILD_DATE\n'.format(date))
 
   current_contents = ''
   if os.path.isfile(args.output_file):