grobotstxt

https://pkg.go.dev/github.com/jimsmart/grobotstxt

grobotstxt is a native Go port of Google's robots.txt parser and matcher, enabling developers to...

Tokens:5,632
Snippets:92
Trust Score:9.6
Update:1 month ago
Tokens:
Raw