Add the ability to easily disable CQ experiments during an outage.

Change-Id: I9a01eb868ece2231f3219903ae6201b9c3b6b924
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2432307
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Auto-Submit: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>
Cr-Commit-Position: refs/heads/master@{#810847}

infra/config/README.md

@@ -36,6 +36,8 @@ directories are consumed by the configuration:
 * validators - Definitions of lucicfg generators that perform additional
   validation on the the LUCI configuration (e.g. ensure all builders are added
   to at least one console).
+* outages - Definitions of config settings for operations common when handling
+  outages.
 
 The configuration rooted at [dev.star](dev.star) defines the LUCI services
 configuration for the chromium project on the dev instance of LUCI. This

infra/config/generators/cq-builders-md.star

@@ -3,6 +3,7 @@
 # found in the LICENSE file.
 
 load("//lib/try.star", "DEFAULT_EXCLUDE_REGEXPS")
+load("//outages/config.star", outages_config = "config")
 
 _MD_HEADER = """\
 <!-- Auto-generated by lucicfg (via cq-builders-md.star). -->
@@ -37,7 +38,11 @@ _EXPERIMENTAL_HEADER = """\
 These builders are run on some percentage of builds. Their results are ignored
 by CQ. These are often used to test new configurations before they are added
 as required builders.
-"""
+""" + ("""\
+
+These builders are currently disabled due to the cq_disable_experiments outages
+setting. See //infra/config/outages/README.md for more information.
+""" if outages_config.disable_cq_experiments else "")
 
 _TRY_BUILDER_VIEW_URL = "https://ci.chromium.org/p/chromium/builders/try"
 

infra/config/main.star

@@ -150,3 +150,7 @@ exec("//generators/scheduler-noop-jobs.star")
 exec("//generators/sort-consoles.star")
 
 exec("//validators/builders-in-consoles.star")
+
+# Execute this file last so that any configuration changes needed for handling
+# outages gets final say
+exec("//outages/outages.star")

infra/config/outages/README.md

+This directory contains configuration for dealing with outages.
+
+In response to outages, it is occasionally necessary to disable or alter the
+behavior of some portions of the LUCI project. The files located within this
+directory provide facilities to easily make such changes without having to
+modify the definitions themselves.
+
+# Directory contents
+
+* **config.star**: The file containing the actual outages configuration. See
+  [Modifying the outage configuration](#modifying-the-outages-configuration) for
+  more information.
+* **outages.star**: The file containing the generators that implement the
+  outages behavior.
+
+# Modifying the outages configuration
+
+Within config.star, `config` is declared with the result of a function call. To
+change the outages configuration you can modify the parameters to the call. The
+available parameters are:
+
+* `disable_cq_experiments`: (default: False) Disable experimental tryjobs on the
+  chromium CQ. This can be used to save both builder and test capacity when the
+  CQ is capacity-constrained. The exact amount and configurations of capacity
+  that is freed up depends on the specifics of what builders have experimental
+  tryjobs and what experiment percentage is used, but disabling all CQ
+  experiments will quickly free up any capacity that might be instead used for
+  the CQ so that we can focus on trying to address the root cause or implement
+  other workarounds to improve the CQ experience for developers rather than
+  tracking down which experimental tryjobs would be using the capacity that is
+  needed for the particular outage.

infra/config/outages/config.star

+# Copyright 2020 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+def _outages_config(*, disable_cq_experiments = False):
+    return struct(
+        disable_cq_experiments = disable_cq_experiments,
+    )
+
+# See README.md for documentation on allowable configuration values
+config = _outages_config(
+)

infra/config/outages/outages.star

+# Copyright 2020 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+load("./config.star", "config")
+load("@stdlib//internal/luci/proto.star", "cq_pb")
+
+def _disable_cq_experiments(ctx):
+    if not config.disable_cq_experiments:
+        return
+
+    for c in ctx.output["commit-queue.cfg"].config_groups:
+        if c.verifiers.tryjob == cq_pb.Verifiers.Tryjob():
+            # Accessing the tryjob field where it wasn't set causes it to be set
+            # to an empty message and added to the output, setting to None
+            # prevents the change to the output
+            c.verifiers.tryjob = None
+            continue
+        for b in c.verifiers.tryjob.builders:
+            if not b.experiment_percentage:
+                continue
+            project, bucket, builder = b.name.split("/", 2)
+            if project == "chromium" and bucket in ("try", "try-m85"):
+                b.includable_only = True
+                b.experiment_percentage = 0
+                b.location_regexp.clear()
+                b.location_regexp_exclude.clear()
+
+lucicfg.generator(_disable_cq_experiments)