Robots Exclusion Tools

https://github.com/turnersoftware/robotsexclusiontools

A comprehensive .NET library for parsing and querying robots.txt files, following the NoRobots and...

Tokens:5,403
Snippets:28
Trust Score:7.4
License:MIT
Update:3 months ago
Tokens:
Raw