grobotstxt is a native Go port of Google's robots.txt parser and matcher library that enables...