GitLab

Rational: if bad canary is pushed, causing build failures, CQ would
false reject ~13% of CLs (and on retry, CL is highly likely to land).


How is 13% figure obtained? Monte-carlo simulation of 1M CLs!

    def simulate_cl(builders=31, canary=0.1, max_retry_total=3):
        """True means OK, False means CQ falsely rejects CL."""
        import random
        bad = sum(random.random() < canary for _ in xrange(builders))
        if bad > max_retry_total:
            return False
        still_fail = sum(random.random() < canary for _ in xrange(bad))
        return still_fail == 0

    def monte_carlo_cq_false_rejection_rate(times=10**6, **kwargs):
        fp = float(sum(simulate_cl(**kwargs) is False for _ in xrange(times)) * 100.0) / times
        return '%0.1f%% false rejection rate' % fp

    monte_carlo_cq_false_rejection_rate(builders=31, canary=0.01, max_retry_total=3)
    # '0.3% false rejection rate'

    monte_carlo_cq_false_rejection_rate(builders=31, canary=0.05, max_retry_total=3)
    # '12.8% false rejection rate'

    monte_carlo_cq_false_rejection_rate(builders=31, canary=0.10, max_retry_total=3)
    # '49.5% false rejection rate'

R=nodir@chromium.org

No-Try: True
Change-Id: I04bbdbb50bbeb0e9aea70fd5ca8aa89fd22d3fcf
Reviewed-on: https://chromium-review.googlesource.com/1045501Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557984}