Spatie Crawler

https://github.com/spatie/crawler
Admin
A PHP package that provides a powerful, easy-to-use class to crawl links on a website using...
Tokens:119,427
Snippets:926
Trust Score:10
License:MIT
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install spatie/crawler via Composer

Source: https://github.com/spatie/crawler/blob/main/docs/installation-setup.md

Run this command in your project root to install the package. No additional setup is needed; the crawler works out of the box.

```bash
composer require spatie/crawler
```

--------------------------------

### Installing a custom URL parser (JsonLdParser)

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/url-parsers.md

Demonstrates how to create and install a custom UrlParser implementation. The example JsonLdParser extracts URLs using a regex and returns ExtractedUrl objects.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\ExtractedUrl;
use Spatie\Crawler\UrlParsers\UrlParser;

class JsonLdParser implements UrlParser
{
    public function extractUrls(string $html, string $baseUrl): array
    {
        preg_match_all('#https?://[^\s"<>]+#', $html, $matches);

        return array_map(
            fn (string $u) => new ExtractedUrl(url: $u),
            $matches[0],
        );
    }
}

Crawler::create('https://example.com')
    ->urlParser(new JsonLdParser)
    ->start();
```

--------------------------------

### start() method signature

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Declares the public start() method that runs the crawl synchronously and returns a FinishReason.

```php
public function start(): FinishReason
```

--------------------------------

### Usage Examples

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-profiles.md

Examples of using the built-in profiles and creating a custom profile.

```APIDOC
## Usage Examples

### Using CrawlInternalUrls

```php
use Spatie\Crawler\Crawler;

// Same as ->internalOnly()
Crawler::create('https://example.com')
    ->crawlProfile(new CrawlInternalUrls('https://example.com', matchWww: true, includeSubdomains: true))
    ->start();
```

### Custom Profile

```php
use Spatie\Crawler\CrawlProfiles\CrawlProfile;
use Spatie\Crawler\Crawler;

class DocsProfile implements CrawlProfile
{
    public function shouldCrawl(string $url): bool
    {
        return str_contains($url, '/docs');
    }
}

Crawler::create('https://example.com')
    ->crawlProfile(new DocsProfile)
    ->start();
```
```

--------------------------------

### Using AdaptiveThrottle with Crawler

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/throttlers.md

Example of creating a Crawler with an AdaptiveThrottle, setting min and max delays to 100ms and 2000ms, and starting the crawl.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Throttlers\AdaptiveThrottle;

// Politely pace the crawl: between 100ms and 2s, adapting to server latency
Crawler::create('https://example.com')
    ->throttle(new AdaptiveThrottle(minDelayMs: 100, maxDelayMs: 2000))
    ->start();
```

--------------------------------

### Usage example with limits and finish reason

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Shows how to set total and per-execution URL limits and a total time limit, then start crawling and check the finish reason. If the crawl limit is reached, it prints the total crawled URL count.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Enums\FinishReason;

$crawler = Crawler::create('https://example.com')
    ->limit(50)          // at most 50 URLs total
    ->limitPerExecution(20)
    ->timeLimit(120);    // at most 120s total

$reason = $crawler->start();

if ($reason === FinishReason::CrawlLimitReached) {
    echo "URL limit reached; already crawled {$crawler->getTotalCrawlCount()} URLs\n";
}
```

--------------------------------

### Example using ArrayCrawlQueue with Crawler

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-queue.md

Shows how to create an ArrayCrawlQueue, attach it to a Crawler, start crawling, and iterate over pending URLs.

```php
use Spatie\Crawler\CrawlQueues\ArrayCrawlQueue;
use Spatie\Crawler\Crawler;

$queue = new ArrayCrawlQueue;

$crawler = Crawler::create('https://example.com')
    ->crawlQueue($queue)
    ->start();

echo "Found {$queue->getUrlCount()} URLs, {$queue->getPendingUrlCount()} still pending.\n";

// Iterate remaining pending URLs by fetching them:
while ($pending = $queue->getPendingUrl()) {
    echo $pending->url, PHP_EOL;
    $queue->markAsProcessed($pending);
}
```

--------------------------------

### Custom observer example with Crawler::create()->addObserver()->start()

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-observer.md

Shows a custom ReportObserver extending CrawlObserver that writes crawled URLs and a summary to files, then attaches it to a crawler and starts the crawl.

```php
use Spatie\Crawler\CrawlObservers\CrawlObserver;
use Spatie\Crawler\CrawlProgress;
use Spatie\Crawler\CrawlResponse;
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Enums\FinishReason;

class ReportObserver extends CrawlObserver
{
    public function crawled(string $url, CrawlResponse $response, CrawlProgress $progress): void
    {
        file_put_contents('results.txt', "{$response->status()} {$url}\n", FILE_APPEND);
    }

    public function finishedCrawling(FinishReason $reason, CrawlProgress $progress): void
    {
        file_put_contents('summary.txt',
            "{$reason->value}: crawled={$progress->urlsCrawled} failed={$progress->urlsFailed}\n");
    }
}

Crawler::create('https://example.com')->addObserver(new ReportObserver)->start();
```

--------------------------------

### New entry point for starting the crawler

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

The entry point changed from `startCrawling()` to `start()`. The URL is now passed to `Crawler::create()` instead of `startCrawling()`.

```php
// Before
Crawler::create()->startCrawling('https://example.com');

// After
Crawler::create('https://example.com')->start();
```

--------------------------------

### Handle FinishReason from start()

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/tracking-progress.md

Shows how to use the FinishReason enum returned by start() to determine why the crawl stopped, with a match expression for each possible reason.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Enums\FinishReason;

$reason = Crawler::create('https://example.com')
    ->limit(100)
    ->start();

match ($reason) {
    FinishReason::Completed => 'All URLs have been crawled',
    FinishReason::CrawlLimitReached => 'Stopped because the crawl limit was reached',
    FinishReason::TimeLimitReached => 'Stopped because the time limit was reached',
    FinishReason::Interrupted => 'Stopped by a signal (SIGINT/SIGTERM)',
};
```

--------------------------------

### Custom throttle example

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/throttlers.md

Example of implementing the Throttle interface to create a custom throttle with server-side rate-limit awareness.

```APIDOC
## Custom throttle example

### Description
Example of a custom throttle that implements the `Throttle` interface. It sleeps a fixed 250ms and records response times for observability without adjusting the delay.

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Throttlers\Throttle;

class RateLimitAwareThrottle implements Throttle
{
    public function sleep(): void { usleep(250_000); }

    public function recordResponseTime(float $seconds): void
    {
        // store latency for observability; no delay adjustment
    }
}

Crawler::create('https://example.com')
    ->throttle(new RateLimitAwareThrottle)
    ->start();
```
```

--------------------------------

### Custom UrlParser Installation

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/url-parsers.md

Shows how to install a custom URL parser by implementing the UrlParser interface and passing it to the Crawler via the urlParser method.

```APIDOC
## Custom UrlParser Installation

### Description
Install a custom parser by implementing the `UrlParser` interface and passing it to the `Crawler` using the `urlParser` method.

### Interface
`Spatie\Crawler\UrlParsers\UrlParser`

### Method to Implement
`extractUrls(string $html, string $baseUrl): array`

### Example Implementation
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\ExtractedUrl;
use Spatie\Crawler\UrlParsers\UrlParser;

class JsonLdParser implements UrlParser
{
    public function extractUrls(string $html, string $baseUrl): array
    {
        preg_match_all('#https?://[^\s"<>]+#', $html, $matches);

        return array_map(
            fn (string $u) => new ExtractedUrl(url: $u),
            $matches[0],
        );
    }
}

Crawler::create('https://example.com')
    ->urlParser(new JsonLdParser)
    ->start();
```
```

--------------------------------

### Crawl with start() and check FinishReason

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Creates a crawler, registers an onCrawled callback, starts the crawl, and checks if the finish reason is TimeLimitReached.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlResponse;
use Spatie\Crawler\Enums\FinishReason;

$finishReason = Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response) {
        echo "{$url}: {$response->status()}\n";
    })
    ->start();

if ($finishReason === FinishReason::TimeLimitReached) {
    echo "Crawl stopped by time limit\n";
}
```

--------------------------------

### Example robots.txt with custom user agent

Source: https://github.com/spatie/crawler/blob/main/docs/configuring-the-crawler/respecting-robots-txt.md

Example robots.txt content showing a rule specific to the user agent 'my-agent'. When the crawler uses this user agent, it will not crawl any pages on the site.

```txt
User-agent: my-agent
Disallow: /
```

--------------------------------

### Example: Using CrawlInternalUrls with matchWww and includeSubdomains

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-profiles.md

Shows how to configure a crawler to use CrawlInternalUrls with both matchWww and includeSubdomains enabled, equivalent to ->internalOnly().

```php
use Spatie\Crawler\Crawler;

// Same as ->internalOnly()
Crawler::create('https://example.com')
    ->crawlProfile(new CrawlInternalUrls('https://example.com', matchWww: true, includeSubdomains: true))
    ->start();
```

--------------------------------

### Using the default crawl queue

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-queue.md

Creates a crawler, starts it, and then retrieves the queue to display counts of discovered, pending, and processed URLs.

```php
use Spatie\Crawler\Crawler;

$crawler = Crawler::create('https://example.com')
    ->internalOnly();

$crawler->start();

$queue = $crawler->getCrawlQueue();
echo "Discovered: {$queue->getUrlCount()}, pending: {$queue->getPendingUrlCount()}\n";
echo "Processed: {$queue->getProcessedUrlCount()}\n";
```

--------------------------------

### Using FixedDelayThrottle with Crawler

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/throttlers.md

Example of creating a Crawler with a FixedDelayThrottle of 500ms, equivalent to ->delay(500).

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Throttlers\FixedDelayThrottle;

// Fixed 500ms between requests — equivalent to ->delay(500)
Crawler::create('https://example.com')
    ->throttle(new FixedDelayThrottle(500))
    ->start();
```

--------------------------------

### FinishReason enum

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/tracking-progress.md

The start() method returns a FinishReason enum indicating why the crawl stopped.

```APIDOC
## FinishReason

### Description
The `FinishReason` enum is returned by `start()` and passed to `onFinished` callback to indicate why the crawl stopped.

### Values
- **Completed** - All URLs have been crawled.
- **CrawlLimitReached** - Stopped because the crawl limit was reached.
- **TimeLimitReached** - Stopped because the time limit was reached.
- **Interrupted** - Stopped by a signal (SIGINT/SIGTERM).

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Enums\FinishReason;

$reason = Crawler::create('https://example.com')
    ->limit(100)
    ->start();

match ($reason) {
    FinishReason::Completed => 'All URLs have been crawled',
    FinishReason::CrawlLimitReached => 'Stopped because the crawl limit was reached',
    FinishReason::TimeLimitReached => 'Stopped because the time limit was reached',
    FinishReason::Interrupted => 'Stopped by a signal (SIGINT/SIGTERM)',
};
```
```

--------------------------------

### urlParser(UrlParser $urlParser) / parseSitemaps(): self

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Installs a custom URL parser; parseSitemaps() installs SitemapUrlParser. The default is LinkUrlParser.

```APIDOC
## urlParser(UrlParser $urlParser) / parseSitemaps(): self

### Description
Installs a custom URL parser; `parseSitemaps()` installs `Spatie\Crawler\UrlParsers\SitemapUrlParser`. The default is `LinkUrlParser` (see [url-parsers.md](url-parsers.md)).

### Methods
- `urlParser(UrlParser $urlParser)` - Installs a custom URL parser.
- `parseSitemaps()` - Installs the sitemap URL parser.

### Parameters (urlParser)
- **$urlParser** (UrlParser) - Required - The URL parser instance.

### Returns
`self` - Returns `$this` for chaining.
```

--------------------------------

### Testing finish reasons with limit(1)

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/testing.md

Asserts that start() returns FinishReason::CrawlLimitReached when limit(1) is set.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Enums\FinishReason;

$reason = Crawler::create('https://example.com')
    ->fake([
        'https://example.com' => '<html><a href="/page">Page</a></html>',
        'https://example.com/page' => '<html>Page</html>',
    ])
    ->limit(1)
    ->start();

expect($reason)->toBe(FinishReason::CrawlLimitReached);
```

--------------------------------

### timeLimitPerExecution method signature

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum seconds for one start() call.

```php
public function timeLimitPerExecution(int $currentExecutionTimeLimitInSeconds): self
```

--------------------------------

### Catching MissingJavaScriptRenderer

Source: https://github.com/spatie/crawler/blob/main/_autodocs/errors.md

Shows how to catch the exception when calling executeJavaScript() without a renderer and spatie/browsershot is not installed. The comment suggests installing the package or providing a custom renderer.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Exceptions\MissingJavaScriptRenderer;

try {
    Crawler::create('https://example.com')->executeJavaScript()->start();
} catch (MissingJavaScriptRenderer $e) {
    // install spatie/browsershot or pass a custom JavaScriptRenderer
    $crawler->executeJavaScript(new MyRenderer())->start();
}
```

--------------------------------

### Check for interrupted crawl after start()

Source: https://github.com/spatie/crawler/blob/main/docs/advanced-usage/graceful-shutdown.md

Use this snippet to detect whether the crawl was interrupted by a signal. The start() method returns FinishReason::Interrupted when a SIGINT or SIGTERM is received. No configuration is needed; it works out of the box when the pcntl extension is loaded.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Enums\FinishReason;

$reason = Crawler::create('https://example.com')
    ->start();

if ($reason === FinishReason::Interrupted) {
    echo "Crawl was interrupted by a signal\n";
}
```

--------------------------------

### crawlProfile(CrawlProfile $crawlProfile)

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-scope.md

Installs a custom CrawlProfile implementation. Clears the scope mode.

```APIDOC
## crawlProfile(CrawlProfile $crawlProfile)

### Description
Installs a `CrawlProfile` implementation. Clears `$scopeMode`.

### Method
public function crawlProfile(CrawlProfile $crawlProfile): self

### Parameters
- **$crawlProfile** (CrawlProfile) - Required - A crawl profile implementation.

### Return Value
- `self` - Returns the crawler instance for chaining.
```

--------------------------------

### Matching www and non-www with matchWww

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/filtering-urls.md

Chain matchWww() after internalOnly() to treat example.com and www.example.com as equivalent hosts. This works in both directions, so starting from either host includes links to the other.

```php
use Spatie\Crawler\Crawler;

Crawler::create('https://example.com')
    ->internalOnly()
    ->matchWww()
    ->start();
```

--------------------------------

### Enable JavaScript rendering with executeJavaScript

Source: https://github.com/spatie/crawler/blob/main/docs/advanced-usage/rendering-javascript.md

Enables JavaScript rendering using the default BrowsershotRenderer, which requires spatie/browsershot to be installed; otherwise an exception is thrown.

```php
use Spatie\Crawler\Crawler;

Crawler::create('https://example.com')
    ->executeJavaScript()
    ->start();
```

--------------------------------

### onFinished callback with FinishReason

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/tracking-progress.md

Example of using the onFinished callback to receive the finish reason and final progress.

```APIDOC
## onFinished callback

### Description
Registers a callback that is invoked when the crawl finishes, providing the finish reason and final progress.

### Method
`onFinished`

### Parameters
- **callback** (callable) - Function that receives `FinishReason $reason` and `CrawlProgress $progress`.

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlProgress;
use Spatie\Crawler\Enums\FinishReason;

Crawler::create('https://example.com')
    ->limit(100)
    ->onFinished(function (FinishReason $reason, CrawlProgress $progress) {
        echo "Crawl finished: {$reason->value}\n";
        echo "Crawled {$progress->urlsCrawled} URLs, {$progress->urlsFailed} failed\n";
    })
    ->start();
```
```

--------------------------------

### JavaScript rendering driver-based migration (before/after)

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

Shows the change from setBrowsershot() and executeJavaScript() to executeJavaScript() optionally accepting a JavaScriptRenderer instance. Browsershot is still the default if installed.

```php
// Before
$crawler->setBrowsershot($browsershot);
$crawler->executeJavaScript();

// After (Browsershot is still the default if installed)
$crawler->executeJavaScript();

// Or with a custom renderer
$crawler->executeJavaScript(new BrowsershotRenderer($browsershot));
$crawler->executeJavaScript(new CloudflareRenderer($endpoint));
```

--------------------------------

### Use FinishReason in onFinished callback

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/tracking-progress.md

Example of using the onFinished callback to receive the FinishReason and CrawlProgress, printing the reason and crawl statistics.

```php
use Spatie\Crawler\CrawlProgress;
use Spatie\Crawler\Enums\FinishReason;

Crawler::create('https://example.com')
    ->limit(100)
    ->onFinished(function (FinishReason $reason, CrawlProgress $progress) {
        echo "Crawl finished: {$reason->value}\n";
        echo "Crawled {$progress->urlsCrawled} URLs, {$progress->urlsFailed} failed\n";
    })
    ->start();
```

--------------------------------

### Initial request: create queue and crawl first batch

Source: https://github.com/spatie/crawler/blob/main/docs/advanced-usage/crawling-across-requests.md

Creates an ArrayCrawlQueue (or custom queue), passes it to the crawler with limitPerExecution(10), starts the crawl, then serializes the queue for storage.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlQueues\ArrayCrawlQueue;

$queue = new ArrayCrawlQueue(); // or your custom queue

// Crawl the first batch of URLs
Crawler::create('https://example.com')
    ->crawlQueue($queue)
    ->limitPerExecution(10)
    ->start();

// Serialize and store the queue for the next request
$serializedQueue = serialize($queue);
```

--------------------------------

### Pass an observer to the crawler

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/using-observers.md

Create a crawler instance and add the observer using addObserver() before starting the crawl. This is the basic usage for using a single observer.

```php
Crawler::create('https://example.com')
    ->addObserver(new MyCrawlObserver())
    ->start();
```

--------------------------------

### Scope helpers: internalOnly and includeSubdomains

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/filtering-urls.md

Use internalOnly() to restrict crawling to the same host, and optionally includeSubdomains() to also crawl subdomains. These methods are chained before start().

```php
use Spatie\Crawler\Crawler;

// Only crawl URLs on the same host
Crawler::create('https://example.com')
    ->internalOnly()
    ->start();

// Crawl URLs on the same host and its subdomains
Crawler::create('https://example.com')
    ->internalOnly()
    ->includeSubdomains()
    ->start();
```

--------------------------------

### Create a Crawler with custom Guzzle options

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Example of using Crawler::create() with Guzzle request options. Passing null for a key removes the default option (e.g., CONNECT_TIMEOUT).

```php
use GuzzleHttp\RequestOptions;

$crawler = Crawler::create('https://example.com', [
    RequestOptions::TIMEOUT => 30,
    RequestOptions::CONNECT_TIMEOUT => null, // removes the default 10s connect timeout
]);
```

--------------------------------

### Testing depth limits with fake() and depth(1)

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/testing.md

Uses fake() with depth(1) to verify that only the start URL and level-1 are crawled. Expects 2 URLs.

```php
use Spatie\Crawler\Crawler;

$urls = Crawler::create('https://example.com')
    ->fake([
        'https://example.com' => '<html><a href="/level-1">Level 1</a></html>',
        'https://example.com/level-1' => '<html><a href="/level-2">Level 2</a></html>',
        'https://example.com/level-2' => '<html>Level 2</html>',
    ])
    ->depth(1)
    ->foundUrls();

// Only the start URL and level-1 will be crawled (depth 0 and 1)
expect($urls)->toHaveCount(2);
```

--------------------------------

### Using CloudflareRenderer with a remote endpoint

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/javascript-renderers.md

Example of configuring a CloudflareRenderer with a remote rendering service URL. No local browser is needed.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\JavaScriptRenderers\CloudflareRenderer;

Crawler::create('https://example.com')
    ->executeJavaScript(new CloudflareRenderer('https://renderer.example.internal/render'))
    ->start();
```

--------------------------------

### Custom JavaScriptRenderer

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/javascript-renderers.md

Example of creating a custom renderer by implementing the JavaScriptRenderer interface. The custom renderer can be passed to Crawler::executeJavaScript().

```APIDOC
## Custom JavaScriptRenderer

### Description
Any class implementing the JavaScriptRenderer interface can be used as a renderer. The example below shows a custom renderer that caches rendered HTML.

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\JavaScriptRenderers\JavaScriptRenderer;

class PageCacheRenderer implements JavaScriptRenderer
{
    public function getRenderedHtml(string $url): string
    {
        return cache()->remember("rendered:$url", 3600, fn () => /* render */ '');
    }
}

Crawler::create('https://example.com')
    ->executeJavaScript(new PageCacheRenderer)
    ->start();
```
```

--------------------------------

### timeLimit method signature

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum cumulative seconds spent crawling across executions. start() returns FinishReason::TimeLimitReached when the total execution time reaches this limit.

```php
public function timeLimit(int $totalExecutionTimeLimitInSeconds): self
```

--------------------------------

### executeJavaScript(?JavaScriptRenderer $renderer = null) / doNotExecuteJavaScript(): self

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Enables or disables JavaScript rendering. With no renderer, requires spatie/browsershot to be installed and throws an exception otherwise.

```APIDOC
## executeJavaScript(?JavaScriptRenderer $renderer = null) / doNotExecuteJavaScript(): self

### Description
Enables JavaScript rendering. With no renderer, requires `spatie/browsershot` to be installed and throws `Spatie\Crawler\Exceptions\MissingJavaScriptRenderer::browsershotNotInstalled()` otherwise. A custom `JavaScriptRenderer` (e.g. `CloudflareRenderer`) can be passed. See [javascript-renderers.md](javascript-renderers.md).

### Methods
- `executeJavaScript(?JavaScriptRenderer $renderer = null)` - Enables JavaScript rendering with an optional custom renderer.
- `doNotExecuteJavaScript()` - Disables JavaScript rendering.

### Parameters (executeJavaScript)
- **$renderer** (?JavaScriptRenderer) - Optional - A custom JavaScript renderer.

### Returns
`self` - Returns `$this` for chaining.
```

--------------------------------

### limit method signature

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum number of URLs to crawl in total across executions. start() returns FinishReason::CrawlLimitReached when the total crawl count reaches this limit.

```php
public function limit(int $totalCrawlLimit): self
```

--------------------------------

### limit

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum number of URLs to crawl in total across all executions. The start() method will return FinishReason::CrawlLimitReached when the total crawl count reaches this limit.

```APIDOC
## limit

### Description
Sets the maximum number of URLs to crawl in total across all executions. The `start()` method will return `FinishReason::CrawlLimitReached` when the total crawl count reaches this limit.

### Method
`public function limit(int $totalCrawlLimit): self`

### Parameters
- `$totalCrawlLimit` (int) - Required - The maximum number of URLs to crawl in total.

### Return Value
Returns the Crawler instance for method chaining.

### Example
```php
$crawler->limit(50);
```
```

--------------------------------

### foundUrls() for common use case

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

Use foundUrls() to get an array of CrawledUrl objects after crawling. This example restricts to internal URLs and limits depth to 3.

```php
$urls = Crawler::create('https://example.com')
    ->internalOnly()
    ->depth(3)
    ->foundUrls(); // Returns array<CrawledUrl>
```

--------------------------------

### throttle(Throttle $throttle): self

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Installs a Throttle implementing the throttle interface instead of a fixed delay. Response times are fed to the throttle via the Guzzle on_stats handler.

```APIDOC
## throttle(Throttle $throttle): self

### Description
Installs a `Throttle` implementing the throttle interface (`Spatie\Crawler\Throttlers\Throttle`) instead of a fixed delay. Response times are fed to the throttle via the Guzzle `on_stats` handler. See [throttlers.md](throttlers.md).

### Method
`throttle`

### Parameters
- **$throttle** (Throttle) - Required - The throttle instance.

### Returns
`self` - Returns `$this` for chaining.
```

--------------------------------

### start()

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Runs the crawl synchronously, blocking on the Guzzle Pool promise. It resets counters, builds the client, resolves scope, fetches robots.txt if configured, enqueues the base URL, and processes URLs until completion or a stop/limit condition. Returns a FinishReason indicating how the crawl ended.

```APIDOC
## start()

### Description
Runs the crawl synchronously (blocking on the Guzzle `Pool` promise). It resets counters, records the start time, normalizes the base URL, builds the Guzzle client, resolves the crawl scope, fetches and parses `robots.txt` when `respectRobots` is true, enqueues the base URL, registers signal handlers if `pcntl` is available, and loops over Guzzle `Pool`s of pending URLs until the queue drains or a limit/stop condition triggers. Finally, it dispatches `finishedCrawling` to observers and returns the `FinishReason`.

### Method
start

### Signature
```php
public function start(): FinishReason
```

### Return Value
Returns a `FinishReason` enum value indicating the reason the crawl finished:

| FinishReason | Condition |
|---|---|
| `FinishReason::Completed` | queue drained with no stop/limit |
| `FinishReason::Interrupted` | `$shouldStop` set by callback or signal |
| `FinishReason::CrawlLimitReached` | `limit()` / `limitPerExecution()` reached |
| `FinishReason::TimeLimitReached` | `timeLimit()` / `timeLimitPerExecution()` reached |

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlResponse;
use Spatie\Crawler\Enums\FinishReason;

$finishReason = Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response) {
        echo "{$url}: {$response->status()}\n";
    })
    ->start();

if ($finishReason === FinishReason::TimeLimitReached) {
    echo "Crawl stopped by time limit\n";
}
```
```

--------------------------------

### Using BrowsershotRenderer with custom node and npm binaries

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/javascript-renderers.md

Example of configuring a BrowsershotRenderer with a custom Browsershot instance that sets the node and npm binary paths. This is useful when the default binaries are not in the expected location.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\JavaScriptRenderers\BrowsershotRenderer;
use Spatie\Browsershot\Browsershot;

Crawler::create('https://example.com')
    ->executeJavaScript(new BrowsershotRenderer(
        (new Browsershot)->setNodeBinary('/usr/bin/node')->setNpmBinary('/usr/bin/npm'),
    ))
    ->start();
```

--------------------------------

### Combining matchWww and includeSubdomains

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/filtering-urls.md

Use both matchWww() and includeSubdomains() together. When includeSubdomains() is enabled, www is stripped from both hosts before the subdomain check, so blog.example.com matches a base URL of www.example.com.

```php
use Spatie\Crawler\Crawler;

Crawler::create('https://www.example.com')
    ->internalOnly()
    ->matchWww()
    ->includeSubdomains()
    ->start();
```

--------------------------------

### Custom failed handler example

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/handlers.md

Shows how to create a custom failed handler by extending CrawlRequestFailed and registering it with the crawler using the failedHandler method. The example uses a QuietFailedHandler class that could override the default behavior, such as bumping a metric.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Handlers\CrawlRequestFailed;

class QuietFailedHandler extends CrawlRequestFailed
{
    // e.g. bump a metric instead of the default flow
}

Crawler::create('https://example.com')
    ->failedHandler(QuietFailedHandler::class)
    ->start();
```

--------------------------------

### Basic crawler configuration with retry and token

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/configures-requests.md

Shows how to create a crawler with user agent, headers, request timeout, token, and retry settings. The retry method takes retry times and delay in milliseconds.

```php
use Spatie\Crawler\Crawler;
use GuzzleHttp\Middleware;

Crawler::create('https://example.com')
    ->userAgent('MyBot/1.0')
    ->headers(['X-Api-Key' => 'secret'])
    ->requestTimeout(30)
    ->token('abc123', 'Bearer')
    ->retry(3, 1000)
    ->start();
```

--------------------------------

### CrawlUrl constructor migration (before/after)

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

Shows the replacement of the static create() factory with a regular constructor. The constructor now accepts string URLs instead of UriInterface objects.

```php
// Before
CrawlUrl::create(new Uri('https://example.com'), new Uri('https://example.com/page'));

// After
new CrawlUrl('https://example.com', 'https://example.com/page');
```

--------------------------------

### foundUrls()

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

Convenience method to get all crawled URLs for the most common use case.

```APIDOC
## foundUrls()

### Description
Returns an array of CrawledUrl objects after the crawl completes.

### Usage
```php
$urls = Crawler::create('https://example.com')
    ->internalOnly()
    ->depth(3)
    ->foundUrls(); // Returns array<CrawledUrl>
```
```

--------------------------------

### Configuring BrowsershotRenderer

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

Shows how to configure Browsershot by passing a configured instance to BrowsershotRenderer, since setBrowsershot() and getBrowsershot() have been removed.

```php
$browsershot = (new Browsershot)->noSandbox()->waitUntilNetworkIdle();
$crawler->executeJavaScript(new BrowsershotRenderer($browsershot));
```

--------------------------------

### onCrawled callback with progress

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/tracking-progress.md

Example of using the onCrawled callback to log progress during a crawl.

```APIDOC
## onCrawled callback

### Description
Registers a callback that is invoked for each successfully crawled URL, providing the URL, response, and progress.

### Method
`onCrawled`

### Parameters
- **callback** (callable) - Function that receives `string $url`, `CrawlResponse $response`, and `CrawlProgress $progress`.

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlProgress;
use Spatie\Crawler\CrawlResponse;

Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response, CrawlProgress $progress) {
        echo "[{$progress->urlsProcessed}/{$progress->urlsFound}] {$url}\n";
    })
    ->start();
```
```

--------------------------------

### AdaptiveThrottle class constructor

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/throttlers.md

Constructor for AdaptiveThrottle with optional min and max delay in milliseconds. Defaults are 50ms and 5000ms.

```php
class AdaptiveThrottle implements Throttle
{
    public function __construct(
        protected int $minDelayMs = 50,
        protected int $maxDelayMs = 5000,
    ) {}
}
```

--------------------------------

### getCurrentCrawlCount

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Returns the number of URLs crawled in the current execution. Reset to 0 by start().

```APIDOC
## getCurrentCrawlCount

### Description
Returns the number of URLs crawled in the current execution. Reset to `0` by `start()`.

### Method
`public function getCurrentCrawlCount(): int`

### Return Value
Returns the current crawl count as an integer.

### Example
```php
$count = $crawler->getCurrentCrawlCount();
```
```

--------------------------------

### timeLimitPerExecution

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum seconds for one start() call. This limit is reset for each execution.

```APIDOC
## timeLimitPerExecution

### Description
Sets the maximum seconds for one `start()` call. This limit is reset for each execution.

### Method
`public function timeLimitPerExecution(int $currentExecutionTimeLimitInSeconds): self`

### Parameters
- `$currentExecutionTimeLimitInSeconds` (int) - Required - The maximum execution time in seconds for a single execution.

### Return Value
Returns the Crawler instance for method chaining.

### Example
```php
$crawler->timeLimitPerExecution(60);
```
```

--------------------------------

### MissingJavaScriptRenderer

Source: https://github.com/spatie/crawler/blob/main/_autodocs/errors.md

Thrown when `Crawler::executeJavaScript()` is called without a renderer and the spatie/browsershot package is not installed.

```APIDOC
## MissingJavaScriptRenderer

### Description
Thrown when `Crawler::executeJavaScript()` is called with `$renderer = null` while `Spatie\Browsershot\Browsershot` is not installed. The `spatie/browsershot` package is only a suggested dependency.

### Message
`To execute JavaScript, install spatie/browsershot or provide a custom JavaScriptRenderer.`

### Example
```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Exceptions\MissingJavaScriptRenderer;

try {
    Crawler::create('https://example.com')->executeJavaScript()->start();
} catch (MissingJavaScriptRenderer $e) {
    // install spatie/browsershot or pass a custom JavaScriptRenderer
    $crawler->executeJavaScript(new MyRenderer())->start();
}
```
```

--------------------------------

### Crawler factory and constructor signatures

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawler.md

Shows the public static create() method and the constructor signature. The create() method sets the base URL and accepts optional Guzzle client options.

```php
public static function create(string $url, array $clientOptions = []): self

public function __construct(array $clientOptions = [])
```

--------------------------------

### limitPerExecution method signature

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum number of URLs to crawl per start() call.

```php
public function limitPerExecution(int $currentCrawlLimit): self
```

--------------------------------

### Using Crawler::foundUrls() to iterate collected URLs

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-observer.md

Shows how to create a crawler and retrieve collected URLs via foundUrls(), then iterate over each CrawledUrl to print status, URL, and foundOnUrl.

```php
$urls = Crawler::create('https://example.com')->foundUrls();

foreach ($urls as $crawledUrl) {
    echo "{$crawledUrl->status} {$crawledUrl->url} (found on {$crawledUrl->foundOnUrl})\n";
}
```

--------------------------------

### shouldCrawl(Closure $closure)

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-scope.md

Installs a ClosureCrawlProfile wrapping the given closure. Clears the scope mode.

```APIDOC
## shouldCrawl(Closure $closure)

### Description
Installs a `ClosureCrawlProfile` wrapping `$closure` (invoked as `$closure($url)` returning `bool`). Clears `$scopeMode`.

### Method
public function shouldCrawl(Closure $closure): self

### Parameters
- **$closure** (Closure) - Required - A closure that takes a URL string and returns a boolean indicating whether to crawl it.

### Return Value
- `self` - Returns the crawler instance for chaining.
```

--------------------------------

### Run tests with composer

Source: https://github.com/spatie/crawler/blob/main/README.md

Command to run the package's test suite.

```bash
composer test
```

--------------------------------

### Getting the underlying PSR-7 response

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/handling-crawl-responses.md

Shows how to obtain the raw PSR-7 ResponseInterface from a CrawlResponse object.

```php
$response->toPsrResponse(); // Psr\Http\Message\ResponseInterface
```

--------------------------------

### Spatie\Crawler\JavaScriptRenderers\JavaScriptRenderer

Source: https://github.com/spatie/crawler/blob/main/_autodocs/types.md

Interface for rendering JavaScript to get the final HTML. Implement this to support JavaScript-heavy sites.

```APIDOC
## Interface: Spatie\Crawler\JavaScriptRenderers\JavaScriptRenderer

### Description
Renders JavaScript to obtain the final HTML of a page.

### Methods

#### getRenderedHtml(string $url): string
- **Parameters:**
  - `$url` (string) - The URL to render.
- **Returns:** `string` - The rendered HTML.

### Usage
- Consumed by `Crawler::executeJavaScript()`.
```

--------------------------------

### Building a fake CrawlResponse for Crawler::fake()

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-response.md

Shows how to create a fake CrawlResponse using the fake() factory and pass it to Crawler::fake() for testing without network access.

```php
use Spatie\Crawler\CrawlResponse;

$response = CrawlResponse::fake(
    body: '<html><a href="/about">About</a></html>',
    status: 200,
    headers: ['Content-Type' => 'text/html'],
);

$crawler = \Spatie\Crawler\Crawler::create('https://example.com')
    ->fake(['https://example.com' => $response]);
```

--------------------------------

### Extracting data with Symfony DomCrawler

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/handling-crawl-responses.md

Example of using the dom() method to filter for title and h1 elements and print them.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlResponse;

Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response) {
        $title = $response->dom()->filter('title')->text('');
        $h1 = $response->dom()->filter('h1')->text('');

        echo "{$url}: {$title}\n";
    })
    ->start();
```

--------------------------------

### Custom RateLimitAwareThrottle implementation

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/throttlers.md

Shows a custom throttle that sleeps 250ms and records response times without adjusting delay, then uses it with Crawler.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Throttlers\Throttle;

class RateLimitAwareThrottle implements Throttle
{
    public function sleep(): void { usleep(250_000); }

    public function recordResponseTime(float $seconds): void
    {
        // store latency for observability; no delay adjustment
    }
}

Crawler::create('https://example.com')
    ->throttle(new RateLimitAwareThrottle)
    ->start();
```

--------------------------------

### limitPerExecution

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/has-crawl-limits.md

Sets the maximum number of URLs to crawl per single start() call. This limit is reset for each execution.

```APIDOC
## limitPerExecution

### Description
Sets the maximum number of URLs to crawl per single `start()` call. This limit is reset for each execution.

### Method
`public function limitPerExecution(int $currentCrawlLimit): self`

### Parameters
- `$currentCrawlLimit` (int) - Required - The maximum number of URLs to crawl per execution.

### Return Value
Returns the Crawler instance for method chaining.

### Example
```php
$crawler->limitPerExecution(20);
```
```

--------------------------------

### HTTP Client Options

Source: https://github.com/spatie/crawler/blob/main/_autodocs/configuration.md

Fluent methods to configure the underlying HTTP client, including user agent, headers, timeouts, authentication, and more.

```APIDOC
## HTTP Client Options

### Description
These methods configure the HTTP client used by the crawler. All methods return `$this` for chaining.

### Methods

#### userAgent(string $userAgent)
- **Default**: `'*'`

#### headers(array $headers)
- **Default**: none (merged)

#### connectTimeout(int $connectTimeoutInSeconds)
- **Default**: `10` s

#### requestTimeout(int $requestTimeoutInSeconds)
- **Default**: `10` s

#### basicAuth(string $username, string $password)
- **Default**: none

#### token(string $token, string $type = 'Bearer')
- **Default**: none

#### withoutVerifying()
- **Default**: verify on

#### proxy(string $proxy)
- **Default**: none

#### cookies(array $cookies, string $domain)
- **Default**: Guzzle default jar

#### queryParameters(array $parameters)
- **Default**: none

#### middleware(callable $middleware, string $name = '')
- **Default**: none

#### retry(int $times = 2, int $delayInMs = 500)
- **Default**: disabled (`0`)

#### stream()
- **Default**: off

```

--------------------------------

### Change default scheme with defaultScheme()

Source: https://github.com/spatie/crawler/blob/main/docs/configuring-the-crawler/crawl-behavior.md

Sets the scheme for URLs without one. Default is https; this example uses http.

```php
use Spatie\Crawler\Crawler;

Crawler::create('example.com')
    ->defaultScheme('http')
    ->start();
```

--------------------------------

### Shorter method names for configuration

Source: https://github.com/spatie/crawler/blob/main/UPGRADING.md

Use shorter method names to configure the crawler: depth, concurrency, delay, userAgent, and limit.

```php
$crawler->depth(3);
$crawler->concurrency(10);
$crawler->delay(100);
$crawler->userAgent('Bot');
$crawler->limit(500);
```

--------------------------------

### Catching InvalidCrawlRequestHandler

Source: https://github.com/spatie/crawler/blob/main/_autodocs/errors.md

Shows how to catch the exception when setting an invalid fulfilled handler. The exception is thrown synchronously before the crawl starts.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\Exceptions\InvalidCrawlRequestHandler;

try {
    Crawler::create('https://example.com')
        ->fulfilledHandler(SomeUnrelatedClass::class);
} catch (InvalidCrawlRequestHandler $e) {
    echo $e->getMessage();
}
```

--------------------------------

### Custom renderer with in-process headless client

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/javascript-renderers.md

Example of a custom JavaScriptRenderer implementation that caches rendered HTML. The render logic is a placeholder; replace with actual rendering. Shows how to pass the custom renderer to executeJavaScript.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\JavaScriptRenderers\JavaScriptRenderer;

class PageCacheRenderer implements JavaScriptRenderer
{
    public function getRenderedHtml(string $url): string
    {
        return cache()->remember("rendered:$url", 3600, fn () => /* render */ '');
    }
}

Crawler::create('https://example.com')
    ->executeJavaScript(new PageCacheRenderer)
    ->start();
```

--------------------------------

### Use time limits with timeLimit() and timeLimitPerExecution()

Source: https://github.com/spatie/crawler/blob/main/docs/configuring-the-crawler/setting-crawl-limits.md

Sets maximum execution time in seconds. timeLimit() applies across all executions, while timeLimitPerExecution() applies to a single crawl. The example shows both methods; FinishReason::TimeLimitReached is returned if time runs out.

```php
use Spatie\Crawler\Crawler;

// Stop crawling after 60 seconds total
$reason = Crawler::create('https://example.com')
    ->timeLimit(60)
    ->start();

// $reason will be FinishReason::TimeLimitReached if time ran out

// Stop each execution after 30 seconds, but allow resuming
Crawler::create('https://example.com')
    ->crawlQueue($queue)
    ->timeLimitPerExecution(30)
    ->start();
```

--------------------------------

### Log progress during a crawl

Source: https://github.com/spatie/crawler/blob/main/docs/basic-usage/tracking-progress.md

Example that logs progress in the onCrawled callback, printing the processed and found URL counts along with the URL.

```php
use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlProgress;
use Spatie\Crawler\CrawlResponse;

Crawler::create('https://example.com')
    ->onCrawled(function (string $url, CrawlResponse $response, CrawlProgress $progress) {
        echo "[{$progress->urlsProcessed}/{$progress->urlsFound}] {$url}\n";
    })
    ->start();
```

--------------------------------

### CrawlObserverCollection::__construct

Source: https://github.com/spatie/crawler/blob/main/_autodocs/api-reference/crawl-observer.md

Initializes a new CrawlObserverCollection with an optional array of observers.

```APIDOC
## CrawlObserverCollection::__construct

### Description
Initializes a new CrawlObserverCollection with an optional array of observers.

### Method
Constructor

### Signature
```php
public function __construct(array $observers = [])
```

### Parameters
- **$observers** (array) - Optional - Initial array of CrawlObserver instances.
```
Tokens:
Raw