Skip to content

GitLab

T
tangled
Commit da894ce9 authored by Jamie Madill's avatar Jamie Madill Committed by Commit Bot
Browse files
Options

Reland "Add new ANGLE Linux and Mac tester configs." 

This is a reland of fab66bae

Includes necessary change to luci-scheduler.cfg and updated docs.

Original change's description:
> Add new ANGLE Linux and Mac tester configs.
>
> These configs will replace linux_angle_rel_ng and mac_angle_rel_ng.
> They initially mirror the configuration but will soon be updated to
> run only the subsets of tests that are most relevant to ANGLE. This
> should reduce flakiness in ANGLE and reduce load for Chrome.
>
> Also updates related configuration files to include linux-angle-rel
> and mac-angle-rel.
>
> Bug: 822310
> Change-Id: I2b40ed45f5d9e105c088ab166bd2207c96280151
> Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1551463
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Jamie Madill <jmadill@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#647854}

Bug: 822310
Tbr: jbudorick@chromium.org
Change-Id: Iaeeffc7b089b05d379bb353d627d13b39fd43704
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/1554296
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: default avatarKenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#648727}
parent e9ae8d50
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 2790 additions and 52 deletions
docs/gpu/gpu_testing_bot_details.md
View file @ da894ce9
...@@ -535,63 +535,55 @@ chrome-infra team if this doesn't work as expected.) ...@@ -535,63 +535,55 @@ chrome-infra team if this doesn't work as expected.)
[go/chromecals]: http://go/chromecals [go/chromecals]: http://go/chromecals
### How to add a new "optional" try bot ### How to add a new try bot that runs a subset of tests or extra tests
TODO(kbr): the naming of the "optional" try bots is confusing and
unfortunate. They should probably be renamed to something like "extratests" or
"extra_tests", so perhaps a new naming convention of "gpu_win_extratests_rel" or
"win_gpu_extratests_rel". Unfortunately making this change at this point
requires touching tons of files across many workspaces and is unlikely to happen
unless someone highly motivated wants to pick up the task.
The "optional" GPU try bots are a concession to the reality that there are some
long-running GPU test suites that simply can not run against every Chromium CL.
They run some additional tests that are usually run only on the
chromium.gpu.fyi waterfall. Some of these tests, like the WebGL 2.0 conformance
suite, are intended to be run on the normal try bots once hardware capacity is
available. Some are not intended to ever run on the normal try bots.
The optional try bots are a little different because they mirror waterfall bots
that don't actually exist. The waterfall bots' specifications exist only to
tell the optional try bots which tests to run.
Let's say that you intended to add a new such optional try bot on Windows. Call
it `win_new_optional_tests_rel` for example. Now, if you wanted to just add
this GPU type to the existing `win_optional_gpu_tests_rel` try bot, you'd
just follow the instructions above
([How to start running tests on a new GPU type on an existing try bot](#How-to-start-running-tests-on-a-new-GPU-type-on-an-existing-try-bot)). The steps below describe how to spin up
an entire new optional try bot.
Several projects (ANGLE, Dawn) run custom tests using the Chromium recipes. They
use try bot bot configs that run subsets of Chromium or additional slower tests
that can't be run on the main CQ.
These try bots are a little different because they mirror waterfall bots that
don't actually exist. The waterfall bots' specifications exist only to tell
these try bots which tests to run.
Let's say that you intended to add a new such custom try bot on Windows. Call it
`win-myproject-rel` for example. You will need to add a "fake" mirror bot for
each GPU family the tests you will need to run. For a GPU type of
"CoolNewGPUType" in this example you could add a "fake" bot named "MyProject GPU
Win10 Release (CoolNewGPUType)".
1. Allocate new virtual machines for the bots as described in [How to set up
new virtual machine
instances](#How-to-set-up-new-virtual-machine-instances).
1. Make sure that you have some swarming capacity for the new GPU type. Since 1. Make sure that you have some swarming capacity for the new GPU type. Since
it's not running against all Chromium CLs you don't need the recommended 30 it's not running against all Chromium CLs you don't need the recommended 30
minimum bots, though ~10 would be good. minimum bots, though ~10 would be good.
1. Create a CL in the Chromium workspace: 1. Create a CL in the Chromium workspace the does the following. Here's an
1. Add your new bot (for example, "Optional Win7 Release [example CL](https://crrev.com/c/1554296).
1. Add your new bot (for example, "MyProject GPU Win10 Release
(CoolNewGPUType)") to the chromium.gpu.fyi waterfall in (CoolNewGPUType)") to the chromium.gpu.fyi waterfall in
[waterfalls.pyl]. (Note, this is a bad example: the [waterfalls.pyl].
"optional" bots have special semantics in this script. You'd probably 1. Re-run [`src/testing/buildbot/generate_buildbot_json.py`][generate_buildbot_json.py] to regenerate the JSON files.
want to define some new category of bot if you didn't intend to add 1. Update [`cr-buildbucket.cfg`][cr-buildbucket.cfg] to add `win-myproject-rel`.
this to `win_optional_gpu_tests_rel`.) 1. Update [`luci-milo.cfg`][luci-milo.cfg] to include `win-myproject-rel`.
1. Re-run the script to regenerate the JSON files. 1. Update [`luci-scheduler.cfg`][luci-scheduler.cfg] to include "MyProject GPU Win10 Release
1. Land the above CL. (CoolNewGPUType)".
1. Create a CL in the tools/build workspace: 1. Update [`src/tools/mb/mb_config.pyl`][mb_config.pyl] to include `win-myproject-rel`.
1. Modify `masters/master.tryserver.chromium.win`'s [master.cfg] and 1. Also add your fake bot to [`src/testing/buildbot/generate_buildbot_json.py`][generate_buildbot_json.py] in the list of `get_bots_that_do_not_actually_exist` section.
[slaves.cfg] to add the new tryserver. Follow the pattern for the 1. *After* the Chromium-side CL lands and the bot is on the console, create a CL
existing `win_optional_gpu_tests_rel` tryserver. Namely, add the new in the [`tools/build`][tools/build] workspace which does the
entry to master.cfg, and add the new tryserver to the following. Here's an [example CL](https://crrev.com/c/1554272).
`optional_builders` list in `slaves.cfg`. 1. Adds "MyProject GPU Win10 Release
1. Modify [`chromium_gpu_fyi.py`][chromium_gpu_fyi.py] to add the new (CoolNewGPUType)" to [`chromium_gpu_fyi.py`][chromium_gpu_fyi.py] in
"Optional Win7 Release (CoolNewGPUType)" entry. `scripts/slave/recipe_modules/chromium_tests/`. You can copy a similar
1. Modify [`trybots.py`][trybots.py] to add step.
the new `win_new_optional_tests_rel` try bot, mirroring "Optional 1. Adds `win-myproject-rel` to [`trybots.py`][trybots.py] in the same folder.
Win7 Release (CoolNewGPUType)". This is where you associate "MyProject GPU Win10 Release
1. Land the above CL and request an off-hours restart of the (CoolNewGPUType)" with `win-myproject-rel`. See the sample CL for an example.
tryserver.chromium.win waterfall. 1. Get this reviewed and landed. This step tells the Chromium recipe about
1. Now you can send CLs to the new bot with: the newly-deployed waterfall bot, so it knows which JSON file to load
`git cl try -m tryserver.chromium.win -b win_new_optional_tests_rel` out of src/testing/buildbot and which entry to look at.
1. After your CLs land you should be able to find and run `win-myproject-rel` on CLs
[master.cfg]: https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.tryserver.chromium.win/master.cfg using Choose Trybots in Gerrit.
[slaves.cfg]: https://chromium.googlesource.com/chromium/tools/build/+/master/masters/master.tryserver.chromium.win/slaves.cfg
### How to test and deploy a driver update ### How to test and deploy a driver update
......
infra/config/cr-buildbucket.cfg
View file @ da894ce9
...@@ -3506,6 +3506,7 @@ buckets { ...@@ -3506,6 +3506,7 @@ buckets {
name: "gpu-manual-try-linux-nvidia-tsn" name: "gpu-manual-try-linux-nvidia-tsn"
} }
builders { mixins: "linux-try" name: "leak_detection_linux" } builders { mixins: "linux-try" name: "leak_detection_linux" }
builders { mixins: "linux-angle-try" name: "linux-angle-rel" }
builders { mixins: "linux-angle-try" name: "linux_angle_compile_dbg_ng" } builders { mixins: "linux-angle-try" name: "linux_angle_compile_dbg_ng" }
builders { mixins: "linux-angle-try" name: "linux_angle_dbg_ng" } builders { mixins: "linux-angle-try" name: "linux_angle_dbg_ng" }
builders { mixins: "linux-angle-try" name: "linux_angle_deqp_rel_ng" } builders { mixins: "linux-angle-try" name: "linux_angle_deqp_rel_ng" }
...@@ -3625,6 +3626,7 @@ buckets { ...@@ -3625,6 +3626,7 @@ buckets {
builders { mixins: "ios-try" name: "ios-simulator-eg" } builders { mixins: "ios-try" name: "ios-simulator-eg" }
builders { mixins: "ios-try" name: "ios-simulator-xcode-clang" } builders { mixins: "ios-try" name: "ios-simulator-xcode-clang" }
builders { mixins: "ios-try" name: "ios-slimnav" } builders { mixins: "ios-try" name: "ios-slimnav" }
builders { mixins: "mac-angle-try" name: "mac-angle-rel" }
builders { mixins: "mac-angle-try" name: "mac_angle_compile_dbg_ng" } builders { mixins: "mac-angle-try" name: "mac_angle_compile_dbg_ng" }
builders { mixins: "mac-angle-try" name: "mac_angle_dbg_ng" } builders { mixins: "mac-angle-try" name: "mac_angle_dbg_ng" }
builders { mixins: "mac-angle-try" name: "mac_angle_rel_ng" } builders { mixins: "mac-angle-try" name: "mac_angle_rel_ng" }
......
infra/config/luci-milo.cfg
View file @ da894ce9
...@@ -4477,6 +4477,9 @@ consoles { ...@@ -4477,6 +4477,9 @@ consoles {
builders { builders {
name: "buildbucket/luci.chromium.try/android_angle_vk64_rel_ng" name: "buildbucket/luci.chromium.try/android_angle_vk64_rel_ng"
} }
builders {
name: "buildbucket/luci.chromium.try/linux-angle-rel"
}
builders { builders {
name: "buildbucket/luci.chromium.try/linux_angle_dbg_ng" name: "buildbucket/luci.chromium.try/linux_angle_dbg_ng"
} }
...@@ -4489,6 +4492,9 @@ consoles { ...@@ -4489,6 +4492,9 @@ consoles {
builders { builders {
name: "buildbucket/luci.chromium.try/linux_angle_rel_ng" name: "buildbucket/luci.chromium.try/linux_angle_rel_ng"
} }
builders {
name: "buildbucket/luci.chromium.try/mac-angle-rel"
}
builders { builders {
name: "buildbucket/luci.chromium.try/mac_angle_dbg_ng" name: "buildbucket/luci.chromium.try/mac_angle_dbg_ng"
} }
...@@ -4718,6 +4724,9 @@ consoles { ...@@ -4718,6 +4724,9 @@ consoles {
builders { builders {
name: "buildbucket/luci.chromium.try/gpu-manual-try-win10-nvidia-rel" name: "buildbucket/luci.chromium.try/gpu-manual-try-win10-nvidia-rel"
} }
builders {
name: "buildbucket/luci.chromium.try/linux-angle-rel"
}
builders { builders {
name: "buildbucket/luci.chromium.try/linux-blink-heap-concurrent-marking-tsan-rel" name: "buildbucket/luci.chromium.try/linux-blink-heap-concurrent-marking-tsan-rel"
} }
...@@ -4857,6 +4866,9 @@ consoles { ...@@ -4857,6 +4866,9 @@ consoles {
builders { builders {
name: "buildbucket/luci.chromium.try/ios12-sdk-simulator" name: "buildbucket/luci.chromium.try/ios12-sdk-simulator"
} }
builders {
name: "buildbucket/luci.chromium.try/mac-angle-rel"
}
builders { builders {
name: "buildbucket/luci.chromium.try/mac-jumbo-rel" name: "buildbucket/luci.chromium.try/mac-jumbo-rel"
} }
......
infra/config/luci-scheduler.cfg
View file @ da894ce9
...@@ -1311,6 +1311,22 @@ job { ...@@ -1311,6 +1311,22 @@ job {
# builder-Builders. # builder-Builders.
################################################################################ ################################################################################
job {
id: "ANGLE GPU Linux Release (Intel HD 630)"
# Triggered by "GPU FYI Linux Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "ANGLE GPU Linux Release (NVIDIA)"
# Triggered by "GPU FYI Linux Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job { job {
id: "Cast Audio Linux" id: "Cast Audio Linux"
acl_sets: "default" acl_sets: "default"
...@@ -1984,6 +2000,30 @@ job { ...@@ -1984,6 +2000,30 @@ job {
# Mac Builders. Sorted alphabetically. # Mac Builders. Sorted alphabetically.
################################################################################ ################################################################################
job {
id: "ANGLE GPU Mac Release (Intel)"
# Triggered by "GPU FYI Mac Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "ANGLE GPU Mac Retina Release (NVIDIA)"
# Triggered by "GPU FYI Mac Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job {
id: "ANGLE GPU Mac Retina Release (AMD)"
# Triggered by "GPU FYI Mac Builder".
acl_sets: "triggered-by-parent-builders"
# This bot doesn't actually exist, so it's noop'ed out. crbug.com/819899
noop: {}
}
job { job {
id: "Dawn GPU Mac Release (Intel)" id: "Dawn GPU Mac Release (Intel)"
# Triggered by "GPU FYI Mac Builder". # Triggered by "GPU FYI Mac Builder".
......
testing/buildbot/chromium.gpu.fyi.json
View file @ da894ce9
This diff is collapsed.
testing/buildbot/generate_buildbot_json.py
View file @ da894ce9
...@@ -920,6 +920,11 @@ class BBJSONGenerator(object): ...@@ -920,6 +920,11 @@ class BBJSONGenerator(object):
# are defined only to be mirrored into trybots, and don't actually # are defined only to be mirrored into trybots, and don't actually
# exist on any of the waterfalls or consoles. # exist on any of the waterfalls or consoles.
return [ return [
'ANGLE GPU Linux Release (Intel HD 630)',
'ANGLE GPU Linux Release (NVIDIA)',
'ANGLE GPU Mac Release (Intel)',
'ANGLE GPU Mac Retina Release (AMD)',
'ANGLE GPU Mac Retina Release (NVIDIA)',
'ANGLE GPU Win10 Release (Intel HD 630)', 'ANGLE GPU Win10 Release (Intel HD 630)',
'ANGLE GPU Win10 Release (NVIDIA)', 'ANGLE GPU Win10 Release (NVIDIA)',
'Dawn GPU Linux Release (Intel HD 630)', 'Dawn GPU Linux Release (Intel HD 630)',
......
testing/buildbot/waterfalls.pyl
View file @ da894ce9
...@@ -2070,6 +2070,67 @@ ...@@ -2070,6 +2070,67 @@
'name': 'chromium.gpu.fyi', 'name': 'chromium.gpu.fyi',
'machines': { 'machines': {
# BEGIN Fake builder used as mirror targets for ANGLE's GPU tryservers # BEGIN Fake builder used as mirror targets for ANGLE's GPU tryservers
'ANGLE GPU Linux Release (Intel HD 630)': {
'os_type': 'linux',
'browser_config': 'release',
'mixins': [
'linux_intel_hd_630',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_linux_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_linux_intel_and_nvidia_release_telemetry_tests',
},
},
'ANGLE GPU Linux Release (NVIDIA)': {
'os_type': 'linux',
'browser_config': 'release',
'mixins': [
'linux_nvidia_quadro_p400',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_linux_release_gtests',
'isolated_scripts': 'gpu_angle_perftests',
'gpu_telemetry_tests': 'gpu_fyi_linux_intel_and_nvidia_release_telemetry_tests',
},
},
'ANGLE GPU Mac Release (Intel)': {
'os_type': 'mac',
'browser_config': 'release',
'mixins': [
'mac_mini_intel_gpu',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_mac_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_mac_release_telemetry_tests',
},
},
'ANGLE GPU Mac Retina Release (AMD)': {
'os_type': 'mac',
'browser_config': 'release',
'mixins': [
'mac_retina_amd_gpu',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_mac_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_mac_release_telemetry_tests',
},
},
'ANGLE GPU Mac Retina Release (NVIDIA)': {
'os_type': 'mac',
'browser_config': 'release',
'mixins': [
'mac_retina_nvidia_gpu',
],
'test_suites': {
# TODO(jmadill): Use custom test lists. crbug.com/822310
'gtest_tests': 'gpu_fyi_mac_release_gtests',
'gpu_telemetry_tests': 'gpu_fyi_mac_release_telemetry_tests',
},
},
'ANGLE GPU Win10 Release (Intel HD 630)': { 'ANGLE GPU Win10 Release (Intel HD 630)': {
'os_type': 'win', 'os_type': 'win',
'browser_config': 'release', 'browser_config': 'release',
......
tools/mb/mb_config.pyl
View file @ da894ce9
...@@ -669,10 +669,12 @@ ...@@ -669,10 +669,12 @@
'android_angle_deqp_rel_ng': 'deqp_android_release_trybot_arm64', 'android_angle_deqp_rel_ng': 'deqp_android_release_trybot_arm64',
'android_angle_vk32_deqp_rel_ng': 'deqp_android_vulkan_release_trybot', 'android_angle_vk32_deqp_rel_ng': 'deqp_android_vulkan_release_trybot',
'android_angle_vk64_deqp_rel_ng': 'deqp_android_vulkan_release_trybot_arm64', 'android_angle_vk64_deqp_rel_ng': 'deqp_android_vulkan_release_trybot_arm64',
'linux-angle-rel': 'gpu_fyi_tests_release_trybot',
'linux_angle_ozone_rel_ng': 'gpu_fyi_tests_ozone_linux_system_gbm_libdrm_release_trybot', 'linux_angle_ozone_rel_ng': 'gpu_fyi_tests_ozone_linux_system_gbm_libdrm_release_trybot',
'linux_angle_dbg_ng': 'gpu_fyi_tests_debug_trybot', 'linux_angle_dbg_ng': 'gpu_fyi_tests_debug_trybot',
'linux_angle_deqp_rel_ng': 'deqp_release_trybot', 'linux_angle_deqp_rel_ng': 'deqp_release_trybot',
'linux_angle_rel_ng': 'gpu_fyi_tests_release_trybot', 'linux_angle_rel_ng': 'gpu_fyi_tests_release_trybot',
'mac-angle-rel': 'gpu_fyi_tests_release_trybot',
'mac_angle_dbg_ng': 'gpu_fyi_tests_debug_trybot', 'mac_angle_dbg_ng': 'gpu_fyi_tests_debug_trybot',
'mac_angle_rel_ng': 'gpu_fyi_tests_release_trybot', 'mac_angle_rel_ng': 'gpu_fyi_tests_release_trybot',
'win-angle-rel': 'gpu_fyi_tests_release_trybot_x86', 'win-angle-rel': 'gpu_fyi_tests_release_trybot_x86',
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment