GitLab

This CL changes how resource URL check for robots rules is returned. If
results are immediately available they are returned, otherwise nullopt
is returned and callback is invoked with the result when available.

This CL also introduces an enum to differentiate whether resource was
disallowed by robots rules due to rule fetch already timedout or due to
new timeout.

Bug: 1144836
Change-Id: I55fa7c57b749dd53907c9deb3e045beafd74463d
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2583846
Commit-Queue: rajendrant <rajendrant@chromium.org>
Reviewed-by: Michael Crouse <mcrouse@chromium.org>
Cr-Commit-Position: refs/heads/master@{#835774}