ai.robots.txt

https://github.com/ai-robots-txt/ai.robots.txt

An open list of AI-related web crawlers with configuration files for robots.txt, Apache, Nginx,...

Tokens:10,286
Snippets:210
Trust Score:6.3
License:MIT
Update:1 month ago
Tokens:
Raw