### Install Python Dependencies Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/README.md Install project dependencies using pip. Ensure Python 3 is installed. ```console pip install -r requirements.txt ``` -------------------------------- ### Run Project Tests Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/README.md Execute project tests using the provided Python script. This requires Python 3 and installed dependencies. ```console code/tests.py ``` -------------------------------- ### Create an ai.txt File Source: https://github.com/ai-robots-txt/ai.robots.txt/wiki/Additional-resources Use this file to block specific AI crawlers from using your content. Configure User-Agent and Disallow directives as needed. ```text # Spawning AI # Prevent datasets from using the following file types User-Agent: * Disallow: / Disallow: * ``` -------------------------------- ### Docker Compose for Robots.txt Service Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/docs/traefik-manual-setup.md Defines a lightweight Nginx service to serve a static /robots.txt file. Ensure the './static/' directory exists and contains your robots.txt. Traefik labels configure routing and TLS. ```yaml services: robots: image: nginx:alpine container_name: robots-server volumes: - ./static/:/usr/share/nginx/html/:ro labels: - "traefik.enable=true" # Router for all /robots.txt requests - "traefik.http.routers.robots.rule=Path(`/robots.txt`)" - "traefik.http.routers.robots.entrypoints=web,websecure" - "traefik.http.routers.robots.priority=3000" - "traefik.http.routers.robots.service=robots" - "traefik.http.routers.robots.tls.certresolver=letsencrypt" - "traefik.http.services.robots.loadbalancer.server.port=80" networks: - external_network networks: external_network: name: traefik_external_network external: true ``` -------------------------------- ### HAProxy ACL Configuration Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/README.md Configure HAProxy to block AI bots by specifying a User-Agent substring match against a file list. Ensure the file path is correct for your environment. ```haproxy acl ai_robot hdr_sub(user-agent) -i -f /etc/haproxy/haproxy-block-ai-bots.txt http-request deny if ai_robot ``` -------------------------------- ### Set HTTP Header for Bing AI Opt-Out Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/docs/additional-steps/bing.md Alternatively, set the `X-Robots-Tag` HTTP header to `noarchive` in your server's response. This achieves the same opt-out from AI training as the metatag and is useful for non-HTML content. ```http X-Robots-Tag: noarchive ``` -------------------------------- ### Add Metatag for Bing AI Opt-Out Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/docs/additional-steps/bing.md Include this metatag in the `` section of your HTML to prevent Bing from using your content for AI training. This method is supported by Bing and does not affect your site's search engine visibility. ```html ``` -------------------------------- ### RSS Feed URL Source: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/README.md Subscribe to project updates via RSS/Atom feed for new releases. This URL can be used with various feed readers. ```url https://github.com/ai-robots-txt/ai.robots.txt/releases.atom ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.