Context7 Crawler

How Context7-Crawler discovers, indexes, and retrieves technical documentation.

Product token:Context7-Crawler

Discover

Context7 crawls documentation through automated refreshes or user requests.

Index

Documentation is converted into searchable snippets and stored in Context7's retrieval index.

Retrieve

Developers and AI tools retrieve relevant snippets together with links to the original source pages.

Purpose

Context7 helps developers and AI assistants access up-to-date, version-specific documentation. It crawls documentation websites, converts their pages into searchable snippets, and stores those snippets in a retrieval index.

When a user or an automated workflow requests documentation, Context7 retrieves the most relevant snippets and returns them with links to the original source pages.

Identification

Context7-Crawler sends the following user agent. Its robots.txt product token isContext7-Crawler.

Context7-Crawler/1.0 (+https://context7.com/crawler)

A user-agent string can be copied, so Context7-Crawler also uses Ed25519 HTTP Message Signatures to provide cryptographic proof that supported verification services can check against Context7's published public key.

Cryptographic verification

Context7-Crawler signs its document requests using Web Bot Auth. The corresponding public key is available from the HTTPS key directory below; the private signing key is never published.

https://context7.com/.well-known/http-message-signatures-directory

Each signed request includes short-livedSignature,Signature-Input, andSignature-Agentheaders. Compatible providers can use the directory to confirm that a request was signed by Context7 rather than by a client copying the crawler's user agent.

Control crawling

Context7 checks robots.txt before fetching pages. Rules written specifically for Context7-Crawler take precedence. If none exist, the crawler follows the genericUser-agent: *group.

Block a path

User-agent: Context7-Crawler
Disallow: /private/

Block the crawler

User-agent: Context7-Crawler
Disallow: /

Context7 keeps one robots.txt policy for the duration of a crawl. Temporary failures are retried; if every attempt fails, that origin is skipped for the remainder of the crawl.

Control indexing

Robots.txt controls fetching. To prevent a fetched page from appearing in the Context7 index, return a noindex directive in the page or response headers.

HTML

<meta name="robots" content="noindex">

HTTP header

X-Robots-Tag: noindex

A noindex directive is applied when Context7 next fetches the page. For removal assistance, contact the Context7 team.

Crawler behavior

  • Respects crawler-specific and generic robots.txt rules.
  • Honors noindex meta tags and X-Robots-Tag headers.
  • Limits concurrent requests and responds to server rate limits.
  • Crawls for documentation search and retrieval, not model training.
  • Returns retrieved snippets with their source pages.
  • Does not submit forms during public website crawling.