| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 1 | # chromium.perf Waterfall |
| 2 | |
| 3 | ## Overview |
| 4 | |
| Juan Antonio Navarro Perez | c410eef | 2019-10-29 11:41:22 | [diff] [blame] | 5 | The [chrome.perf waterfall](https://ci.chromium.org/p/chrome/g/chrome.perf/console) |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 6 | continuously builds and runs our performance tests on real Android, Windows, |
| Juan Antonio Navarro Perez | c410eef | 2019-10-29 11:41:22 | [diff] [blame] | 7 | Mac, and Linux hardware; see [list of platforms](perf_lab_platforms.md). |
| 8 | Results are reported to the |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 9 | [Performance Dashboard](https://chromeperf.appspot.com/) for analysis. The |
| Andrew Shulaev | 6e2d6a7 | 2022-08-04 15:30:30 | [diff] [blame] | 10 | [Perfbot Health Sheriffing Rotation](http://goto.google.com/perf-bot-health-sheriffs) ensures that the benchmarks stay green. The [Perf Sheriff Rotation](perf_regression_sheriffing.md) ensures that any regressions detected by those benchmarks are addressed quickly. Together, these rotations maintain |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 11 | [Chrome's Core Principles](https://www.chromium.org/developers/core-principles) |
| 12 | of speed: |
| 13 | |
| 14 | > "If you make a change that regresses measured performance, you will be |
| 15 | > required to fix it or revert". |
| 16 | |
| Ryan Heise | 1eb5728 | 2020-10-28 17:20:17 | [diff] [blame] | 17 | ## How It Works |
| 18 | |
| 19 | The perf waterfall is split into three stages - builders, testers, and processors. |
| 20 | |
| 21 | ### Builders |
| 22 | |
| 23 | For each commit to the Chromium repo, a Builder is invoked. The builder builds |
| 24 | Google Chrome, acquires test assets, and bundles everything up to be passed |
| 25 | along to Testers. We execute a builder for each platform we support. To ensure |
| 26 | we can keep up with the rapid flow of commits, we have a set of builders per |
| 27 | platform, each building a different commit. |
| 28 | |
| 29 | ### Testers |
| 30 | |
| 31 | For each platform, we have a single tester job running continuously. Each run of |
| 32 | a tester covers the set of commits from the end commit of the previous |
| 33 | invocation to the latest _built_ commit. Each tester has one or more subdevices |
| 34 | (shards). The tester delegates a subset of tests to each shard and aggregates |
| 35 | high-level results (pass/fail, runtimes). |
| 36 | |
| 37 | ### Processors |
| 38 | |
| 39 | Processors analyze the raw data generated by each Tester and convert it into a form that can be utilized |
| 40 | by the [Chrome Performance Dashboard](https://chromeperf.appspot.com/report). |
| 41 | For some jobs, this work is executed by the tester instead. |
| 42 | |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 43 | ## Contact |
| 44 | |
| 45 | * You can reach the Chromium performance sheriffs at [email protected]. |
| 46 | * Bugs on waterfall issues should have Component: |
| 47 | [Speed>Benchmarks>Waterfall](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3ASpeed%3EBenchmarks%3EWaterfall+&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids). |
| Caleb Rouleau | 0b366c2 | 2019-08-21 17:52:33 | [diff] [blame] | 48 | * For domain knowledge for a specific benchmark, consider reaching out to |
| 49 | benchmark owners listed in |
| 50 | [benchmark.csv](https://docs.google.com/spreadsheets/d/1xaAo0_SU3iDfGdqDJZX_jRV0QtkufwHUKH3kQKF3YQs/edit#gid=0) |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 51 | |
| 52 | ## Links |
| 53 | |
| Charlie Andrews | 01ce72e | 2018-05-18 20:19:43 | [diff] [blame] | 54 | * [Perf Sheriff Rotation](perf_regression_sheriffing.md) |
| Andrew Shulaev | 6e2d6a7 | 2022-08-04 15:30:30 | [diff] [blame] | 55 | * [Perfbot Health Sheriffing Rotation](http://goto.google.com/perf-bot-health-sheriffs) |
| John Palmer | 046f987 | 2021-05-24 01:24:56 | [diff] [blame] | 56 | * [How to SSH to Bots in Lab](https://chrome-internal.googlesource.com/infra/infra_internal/+/main/doc/ssh.md) |
| Caleb Rouleau | 0b366c2 | 2019-08-21 17:52:33 | [diff] [blame] | 57 | (googlers only!) |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 58 | * TODO: Page on how to repro failures locally |
| sullivan | d37f9c5 | 2017-06-17 01:25:50 | [diff] [blame] | 59 | * TODO: Page on how to hack on buildbot/recipe code |
| Charlie Andrews | 01ce72e | 2018-05-18 20:19:43 | [diff] [blame] | 60 | * TODO: Page on bringing up new hardware |
