Organize the chromium config in subprojects.

This matches the convention used by the infra repo's starlark code and
puts files that are more closely related closer to each in other in the
hierarchy.

Change-Id: I49ce3def16bef580063b2bf8ffe95ee56505ed4c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2161622
Commit-Queue: Garrett Beaty <gbeaty@chromium.org>
Reviewed-by: Aaron Gable <agable@chromium.org>
Cr-Commit-Position: refs/heads/master@{#762100}

infra/config/README.md

@@ -28,25 +28,17 @@ not end in -dev.cfg as well as the markdown file
 [cq-builders.md](generated/cq-builders.md). Starlark files in the following
 directories are consumed by the configuration:
 
-*   buckets - Definitions of builders for the chromium project. There is a .star
-    file for each bucket in the chromium project where the bucket and the
-    builders for that bucket are defined.
-*   consoles - Definitions of milo consoles for the chromium project. There is a
-    .star for each console that defines the console.
+* lib - Utilities for defining LUCI entities.
+* subprojects - Definitions of LUCI entities.
 * generators - Definitions of lucicfg generators that do various things to
-    post-process the LUCI configuration before the output files are generated
+  post-process the LUCI configuration before the output files are generated.
   (e.g. generate no-op jobs to workaround limitations of our recipe config) or
   generate additional files (e.g. the CQ builders markdown document).
 * validators - Definitions of lucicfg generators that perform additional
   validation on the the LUCI configuration (e.g. ensure all builders are added
   to at least one console).
-*   versioned - Definitions of builders for the main waterfall for the chromium
-    project. Builders on the main waterfall are branched so that when a new
-    milestone is cut, ci and try buckets specific to that milestone are created
-    with an equivalent set of builders, schedulers, CQ groups, etc. to enable
-    CI/CQ on the new branch.
 
 The configuration rooted at [dev.star](dev.star) defines the LUCI services
 configuration for the chromium project on the dev instance of LUCI. This
-configuration is responsible for generating the raw configuration files ending
-in -dev.cfg.
+configuration consumes starlark files under the dev directory and is responsible
+for generating the raw configuration files ending in -dev.cfg.

infra/config/buckets/OWNERS

-per-file goma.star=file:../GOMA_OWNERS

infra/config/consoles/OWNERS

-per-file *goma*.star=file:../GOMA_OWNERS

infra/config/dev.star

@@ -54,6 +54,4 @@ luci.milo(
     logo = 'https://storage.googleapis.com/chrome-infra-public/logo/chromium.svg',
 )
 
-exec('//dev/buckets/ci.star')
-
-exec('//dev/consoles/chromium.swarm.star')
+exec('//dev/subprojects/chromium/main.star')

infra/config/dev/consoles/chromium.swarm.star → infra/config/dev/subprojects/chromium/consoles/chromium.swarm.star

 luci.console_view(
     name = 'chromium.dev',
-    header = '//dev/consoles/chromium-header.textpb',
+    header = '//dev/chromium-header.textpb',
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [
         luci.console_view_entry(builder = 'ci/android-kitkat-arm-rel-swarming'),
@@ -13,7 +13,7 @@ luci.console_view(
 
 luci.console_view(
     name = 'chromium.staging',
-    header = '//dev/consoles/chromium-header.textpb',
+    header = '//dev/chromium-header.textpb',
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [
         luci.console_view_entry(builder = 'ci/linux-rel-swarming-staging'),

infra/config/dev/subprojects/chromium/main.star

+exec('./ci.star')
+exec('./consoles/chromium.swarm.star')

infra/config/lib/ci.star

@@ -53,7 +53,7 @@ def declare_bucket(milestone_vars):
 
   ci.main_console_view(
       name = milestone_vars.main_console_name,
-      header = '//consoles/chromium-header.textpb',
+      header = '//chromium-header.textpb',
       repo = 'https://chromium.googlesource.com/chromium/src',
       refs = [milestone_vars.ref],
       title = milestone_vars.main_console_title,
@@ -88,7 +88,7 @@ def set_defaults(milestone_vars, **kwargs):
       cpu = builders.cpu.X86_64,
       executable = 'recipe:chromium',
       execution_timeout = 3 * time.hour,
-      header = '//consoles/chromium-header.textpb',
+      header = '//chromium-header.textpb',
       os = builders.os.LINUX_DEFAULT,
       pool = 'luci.chromium.ci',
       repo = 'https://chromium.googlesource.com/chromium/src',

infra/config/main.star

@@ -98,29 +98,13 @@ luci.milo(
 
 exec('//recipes.star')
 
-exec('//buckets/ci.star')
-exec('//buckets/findit.star')
-exec('//buckets/goma.star')
-exec('//buckets/gpu.try.star')
-exec('//buckets/swangle.try.star')
-exec('//buckets/try.star')
-exec('//buckets/webrtc.star')
-exec('//buckets/webrtc.fyi.star')
-
-exec('//consoles/android.packager.star')
-exec('//consoles/chromium.goma.star')
-exec('//consoles/chromium.goma.fyi.star')
-exec('//consoles/chromium.goma.migration.star')
-exec('//consoles/chromium.webrtc.star')
-exec('//consoles/chromium.webrtc.fyi.star')
-exec('//consoles/findit.star')
-exec('//consoles/goma.latest.star')
-exec('//consoles/luci.chromium.goma.star')
-exec('//consoles/luci.chromium.try.star')
-exec('//consoles/sheriff.ios.star')
-
 exec('//notifiers.star')
 
+exec('//subprojects/chromium/main.star')
+exec('//subprojects/findit/main.star')
+exec('//subprojects/goma/main.star')
+exec('//subprojects/webrtc/main.star')
+
 exec('//generators/cq-builders-md.star')
 # This should be exec'ed before exec'ing scheduler-noop-jobs.star because
 # attempting to read the buildbucket field that is not set for the noop jobs

infra/config/subprojects/README.md

+Definitions of LUCI entities for the chromium project.
+
+The following subprojects exist:
+
+* chromium - Builders that test the chromium/src codebase.
+* findit - Builders that are used by the Sheriff-o-Matic findit service.
+* goma - Builders that test the use of goma by chromium builders.
+* webrtc - Builders that test the integration of WebRTC with chromium.
+
+Each subproject contains a main.star that is the entry point for its
+configuration, which execs the starlark files that define the LUCI entities for
+the subproject.

infra/config/subprojects/chromium/README.md

+Definitions of LUCI entities that test the chromium/src codebase.
+
+* versioned - Builders for the main waterfall.
+* ci.star - Non-main waterfall builders that do post-submit testing.
+* try.star, gpu.try.star, swangle.try.star - Non-main waterfall builders that do
+  pre-submit testing.
+* consoles - Manually curated consoles for chromium subproject builders.

infra/config/buckets/ci.star → infra/config/subprojects/chromium/ci.star

 load('//lib/builders.star', 'cpu', 'goma', 'os', 'xcode_cache')
 load('//lib/ci.star', 'ci')
-load('//versioned/trunk/vars.star', 'vars')
+load('./versioned/trunk/vars.star', 'vars')
 
 # Execute the versioned files to define all of the per-branch entities
 # (bucket, builders, console, poller, etc.)
-exec('//versioned/trunk/buckets/ci.star')
-exec('//versioned/milestones/m81/buckets/ci.star')
-exec('//versioned/milestones/m83/buckets/ci.star')
+exec('./versioned/trunk/buckets/ci.star')
+exec('./versioned/m81/buckets/ci.star')
+exec('./versioned/m83/buckets/ci.star')
 
 
 ci.set_defaults(

infra/config/consoles/android.packager.star → infra/config/subprojects/chromium/consoles/android.packager.star

 luci.console_view(
     name = 'android.packager',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [
         luci.console_view_entry(

infra/config/consoles/sheriff.ios.star → infra/config/subprojects/chromium/consoles/sheriff.ios.star

 luci.console_view(
     name = 'sheriff.ios',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     repo = 'https://chromium.googlesource.com/chromium/src',
     title = 'iOS Sheriff Console',
     entries = [

infra/config/buckets/gpu.try.star → infra/config/subprojects/chromium/gpu.try.star

 load('//lib/builders.star', 'builder', 'cpu', 'defaults', 'goma', 'os')
 load('//lib/try.star', 'try_')
-load('//versioned/trunk/vars.star', 'vars')
+load('./versioned/trunk/vars.star', 'vars')
 
 
 try_.set_defaults(

infra/config/subprojects/chromium/main.star

+exec('./ci.star')
+exec('./gpu.try.star')
+exec('./swangle.try.star')
+exec('./try.star')
+exec('./consoles/android.packager.star')
+exec('./consoles/luci.chromium.try.star')
+exec('./consoles/sheriff.ios.star')

infra/config/buckets/swangle.try.star → infra/config/subprojects/chromium/swangle.try.star

 load('//lib/builders.star', 'builder', 'cpu', 'defaults', 'goma', 'os')
 load('//lib/try.star', 'try_')
-load('//versioned/trunk/vars.star', 'vars')
+load('./versioned/trunk/vars.star', 'vars')
 
 
 try_.set_defaults(

infra/config/buckets/try.star → infra/config/subprojects/chromium/try.star

 load('//lib/builders.star', 'cpu', 'goma', 'os', 'xcode_cache')
 load('//lib/try.star', 'try_')
-load('//versioned/trunk/vars.star', 'vars')
+load('./versioned/trunk/vars.star', 'vars')
 
 # Execute the versioned files to define all of the per-branch entities
 # (bucket, builders, console, cq_group, etc.)
-exec('//versioned/trunk/buckets/try.star')
-exec('//versioned/milestones/m81/buckets/try.star')
-exec('//versioned/milestones/m83/buckets/try.star')
+exec('./versioned/trunk/buckets/try.star')
+exec('./versioned/m81/buckets/try.star')
+exec('./versioned/m83/buckets/try.star')
 
 
 try_.set_defaults(

infra/config/subprojects/findit/README.md

+Definitions of LUCI entities that are used for the Sheriff-o-Matic findit
+service.
+
+* consoles - Manually curated consoles for findit subproject builders.
+* findit.star - Builders that are used by the Sheriff-o-Matic findit service.

infra/config/subprojects/findit/main.star

+exec('./findit.star')
+exec('./consoles/findit.star')

infra/config/subprojects/goma/README.md

+Definitions of LUCI entities that test the use of goma by chromium builders.
+
+* consoles - Manually curated consoles for goma subproject builders.
+* goma.star - Builders that test the use of goma by chromium builders.

infra/config/consoles/chromium.goma.fyi.star → infra/config/subprojects/goma/consoles/chromium.goma.fyi.star

 luci.console_view(
     name = 'chromium.goma.fyi',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     include_experimental_builds = True,
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [

infra/config/consoles/chromium.goma.migration.star → infra/config/subprojects/goma/consoles/chromium.goma.migration.star

@@ -2,7 +2,7 @@
 # Goma RBE (See crbug.com/950413).
 luci.console_view(
     name = 'chromium.goma.migration',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     include_experimental_builds = True,
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [

infra/config/consoles/chromium.goma.star → infra/config/subprojects/goma/consoles/chromium.goma.star

 luci.console_view(
     name = 'chromium.goma',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     include_experimental_builds = True,
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [

infra/config/consoles/goma.latest.star → infra/config/subprojects/goma/consoles/goma.latest.star

 luci.console_view(
     name = 'goma.latest',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [
         luci.console_view_entry(

infra/config/consoles/luci.chromium.goma.star → infra/config/subprojects/goma/consoles/luci.chromium.goma.star

 luci.console_view(
     name = 'luci.chromium.goma',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     include_experimental_builds = True,
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [

infra/config/subprojects/goma/main.star

+exec('./goma.star')
+exec('./consoles/chromium.goma.star')
+exec('./consoles/chromium.goma.fyi.star')
+exec('./consoles/chromium.goma.migration.star')
+exec('./consoles/goma.latest.star')
+exec('./consoles/luci.chromium.goma.star')

infra/config/subprojects/webrtc/README.md

+Definitions of LUCI entities that test the integration of WebRTC with chromium.
+
+* consoles - Manually curated consoles for webrtc subproject builders.
+* webrtc.star, webrtc.fyi.star - Builders that test the integration of WebRTC
+  with chromium.

infra/config/consoles/chromium.webrtc.fyi.star → infra/config/subprojects/webrtc/consoles/chromium.webrtc.fyi.star

 luci.console_view(
     name = 'chromium.webrtc.fyi',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     repo = 'https://webrtc.googlesource.com/src',
     entries = [
         luci.console_view_entry(

infra/config/consoles/chromium.webrtc.star → infra/config/subprojects/webrtc/consoles/chromium.webrtc.star

 luci.console_view(
     name = 'chromium.webrtc',
-    header = '//consoles/chromium-header.textpb',
+    header = '//chromium-header.textpb',
     repo = 'https://chromium.googlesource.com/chromium/src',
     entries = [
         luci.console_view_entry(

infra/config/subprojects/webrtc/main.star

+exec('./webrtc.star')
+exec('./webrtc.fyi.star')
+exec('./consoles/chromium.webrtc.star')
+exec('./consoles/chromium.webrtc.fyi.star')