GitLab

I found that cooling down between runs helps a lot with the noise.
Applying the usual threshold of 35Celsius, identical to the one used by
story_runner.

My testing methodology is simple: connect a N5X and run this with bash:

COMMON_LOG="logs/noise"
for ((i=0;i<10;i++)); do
  OUTPUT="$COMMON_LOG/old$i"
  mkdir -p "$OUTPUT"
  CHROMIUM_OUTPUT_DIR=gn_android/ReleaseOfficial \
      tools/perf/run_benchmark -v \
      experimental.startup.android.coldish \
      --browser=android-chrome --output-dir="$OUTPUT" 2>&1 \
      | tee "$OUTPUT/run.log"

  OUTPUT="$COMMON_LOG/new$i"
  mkdir -p "$OUTPUT"
  CHROMIUM_OUTPUT_DIR=gn_android/ReleaseOfficial \
      tools/perf/run_benchmark -v \
      experimental.startup.mobile \
      --browser=android-chrome --output-dir="$OUTPUT" 2>&1 \
      | tee "$OUTPUT/run.log"
done

On my N5X we appear to have to wait ~2 times per 10 Chrome starts to
cool down the device, which is a lot of waiting. My desk is not the
coldest across offices though, close to 25C right now.

Noise is still an issue for 2-3 last measurements out of every batch of
10. These last runs are typically 200-300ms slower than the first 5+
runs (for messageloop start). I suspect there are still effects of
overheating.  The battery cutoff at 35C is probably not providing
sufficient time to cool the CPU down in the new benchmark.

But why is it 'good enough' for the old benchmark? I think the old way
performs more dummy intermediate steps (reinitializing commandline/trace
file, restarting forwarder/WPR, etc.) to allow the device to cool down.

Frankly, upwards trend is present on the old benchmark as well, but more
around +50ms over 10 runs. While it is not too bad at the current stage,
it may become unpleasant at bisect with more iterations, or in a room
that is slightly hotter. So I'll probably take a look at limiting
the clock frequency for big cores, like we do for N5.

Bug: 760498
Change-Id: Id050cf54c6ea21c51acdfbb1c06f20023d909454
Reviewed-on: https://chromium-review.googlesource.com/1180202Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Egor Pasko <pasko@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584742}