benchmarks/startup_mobile: cool down between runs
I found that cooling down between runs helps a lot with the noise. Applying the usual threshold of 35Celsius, identical to the one used by story_runner. My testing methodology is simple: connect a N5X and run this with bash: COMMON_LOG="logs/noise" for ((i=0;i<10;i++)); do OUTPUT="$COMMON_LOG/old$i" mkdir -p "$OUTPUT" CHROMIUM_OUTPUT_DIR=gn_android/ReleaseOfficial \ tools/perf/run_benchmark -v \ experimental.startup.android.coldish \ --browser=android-chrome --output-dir="$OUTPUT" 2>&1 \ | tee "$OUTPUT/run.log" OUTPUT="$COMMON_LOG/new$i" mkdir -p "$OUTPUT" CHROMIUM_OUTPUT_DIR=gn_android/ReleaseOfficial \ tools/perf/run_benchmark -v \ experimental.startup.mobile \ --browser=android-chrome --output-dir="$OUTPUT" 2>&1 \ | tee "$OUTPUT/run.log" done On my N5X we appear to have to wait ~2 times per 10 Chrome starts to cool down the device, which is a lot of waiting. My desk is not the coldest across offices though, close to 25C right now. Noise is still an issue for 2-3 last measurements out of every batch of 10. These last runs are typically 200-300ms slower than the first 5+ runs (for messageloop start). I suspect there are still effects of overheating. The battery cutoff at 35C is probably not providing sufficient time to cool the CPU down in the new benchmark. But why is it 'good enough' for the old benchmark? I think the old way performs more dummy intermediate steps (reinitializing commandline/trace file, restarting forwarder/WPR, etc.) to allow the device to cool down. Frankly, upwards trend is present on the old benchmark as well, but more around +50ms over 10 runs. While it is not too bad at the current stage, it may become unpleasant at bisect with more iterations, or in a room that is slightly hotter. So I'll probably take a look at limiting the clock frequency for big cores, like we do for N5. Bug: 760498 Change-Id: Id050cf54c6ea21c51acdfbb1c06f20023d909454 Reviewed-on: https://chromium-review.googlesource.com/1180202Reviewed-by:Juan Antonio Navarro Pérez <perezju@chromium.org> Reviewed-by:
Ned Nguyen <nednguyen@google.com> Commit-Queue: Egor Pasko <pasko@chromium.org> Cr-Commit-Position: refs/heads/master@{#584742}
Showing
Please register or sign in to comment