GitLab

This CL adds the robots rules that will be retrieved from Litepages, and
some support classes to parse and apply the rules.

Bug: 1144836
Change-Id: I4ae00f0b08ccbb0ddfe98007c18f041f64d8022b
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2518970
Commit-Queue: rajendrant <rajendrant@chromium.org>
Reviewed-by: Tarun Bansal <tbansal@chromium.org>
Reviewed-by: Michael Crouse <mcrouse@chromium.org>
Cr-Commit-Position: refs/heads/master@{#826670}