### Robots.txt Directives for Bot Access

Source: https://context7_llms

Defines access rules for web crawlers using standard robots.txt directives. This section specifies which user agents are allowed to crawl specific parts of the site. It includes rules for specific bots and a catch-all rule for others.

```APIDOC
Robots.txt Configuration:

  Purpose:
    To inform web crawlers which parts of a website they are allowed or disallowed to access.

  Directives:
    User-agent: Specifies the user agent (bot) the following rules apply to.
    Allow: Specifies a URL path that the user agent is allowed to crawl.
    Disallow: Specifies a URL path that the user agent is not allowed to crawl.

  Rules:

    User-agent: GPTBot
      Allow: /
      Description: Allows the GPTBot crawler to access all paths on the website.

    User-agent: ClaudeBot
      Allow: /
      Description: Allows the ClaudeBot crawler to access all paths on the website.

    User-agent: PerplexityBot
      Allow: /
      Description: Allows the PerplexityBot crawler to access all paths on the website.

    User-agent: *
      Allow: /
      Description: A wildcard rule that applies to all other user agents not explicitly listed. Allows all other bots to access all paths on the website.

  Limitations:
    Robots.txt is a convention, not a security measure. It relies on crawlers respecting the rules. Malicious bots may ignore these directives.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.