Laravel Robots Middleware

https://github.com/spatie/laravel-robots-middleware
Admin
A tiny, opinionated Laravel middleware package to enable or disable search engine indexing of your...
Tokens:8,997
Snippets:69
Trust Score:10
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Basic usage example - custom middleware and registration

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Shows how to extend RobotsMiddleware, override shouldIndex(), and register the subclass in the global middleware stack. The two files are part of the same setup: the middleware class and its registration in Kernel.php.

```php
// app/Http/Middleware/MyRobotsMiddleware.php
namespace App\Http\Middleware;

use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class MyRobotsMiddleware extends RobotsMiddleware
{
    protected function shouldIndex(Request $request): bool
    {
        return $request->segment(1) !== 'admin';
    }
}
```

```php
// app/Http/Kernel.php
class Kernel extends HttpKernel
{
    protected $middleware = [
        // ...
        \App\Http\Middleware\MyRobotsMiddleware::class,
    ];
}
```

--------------------------------

### Install via Composer

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/README.md

Install the package using Composer with the command `composer require spatie/laravel-robots-middleware`.

```bash
$ composer require spatie/laravel-robots-middleware
```

--------------------------------

### Non-trivial string rule example

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Return a string to use directives like 'noarchive' or 'none'. The string is written verbatim as the header value. This example returns 'noarchive' for archive, 'none' for drafts, and 'all' otherwise.

```php
class RobotsForPublishing extends RobotsMiddleware
{
    protected function shouldIndex(Request $request): string
    {
        if ($request->segment(1) === 'archive') {
            return 'noarchive';          // keep index, drop cache copies
        }

        if ($request->segment(1) === 'drafts') {
            return 'none';               // fully excluded
        }

        return 'all';
    }
}
```

--------------------------------

### Basic boolean rule example

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Override shouldIndex to return a boolean. Returning true writes 'all', false writes 'none'. This example excludes the 'admin' segment.

```php
class MyRobotsMiddleware extends RobotsMiddleware
{
    protected function shouldIndex(Request $request): bool
    {
        return $request->segment(1) !== 'admin';
    }
}
```

--------------------------------

### Create custom robots middleware class

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/README.md

Define a custom middleware class that extends Spatie\RobotsMiddleware\RobotsMiddleware and overrides the shouldIndex method to control indexing. In this example, pages under the 'admin' segment are not indexed.

```php
// app/Http/Middleware/MyRobotsMiddleware.php
namespace App\Http\Middleware;

use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class MyRobotsMiddleware extends RobotsMiddleware
{
    /**
     * @return string|bool
     */
    protected function shouldIndex(Request $request)
    {
        return $request->segment(1) !== 'admin';
    }
}
```

--------------------------------

### InvalidIndexRule

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/InvalidIndexRule.md

Documentation for the InvalidIndexRule exception class, including its static factory method and usage examples.

```APIDOC
## InvalidIndexRule

### Description
Exception thrown by `RobotsMiddleware::handle()` when an indexing rule returns a value that cannot be translated into a robots directive.

### Class
`Spatie\RobotsMiddleware\InvalidIndexRule`

### Extends
`\Exception`

### Message
`An indexing rule needs to return a boolean or a string.`

### Constructor
Inherited from `\Exception`; no own constructor is declared. Can be constructed directly with any `\Exception` arguments (e.g. `new InvalidIndexRule('Custom message', 0, $previous)`).

### Static Factory: `requiresBooleanOrString()`

#### Description
Factory that returns a fully-formed instance of the calling class (`new static`, late static binding).

#### Parameters
None.

#### Return
`InvalidIndexRule` (the calling class via `new static`) with the message `An indexing rule needs to return a boolean or a string.` and no code / no previous exception set.

#### Throws
None.

### Example — trigger via a bad rule

```php
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class InvalidTestMiddleware extends RobotsMiddleware
{
    protected function shouldIndex(Request $request): array
    {
        return []; // any non-bool, non-string return triggers InvalidIndexRule
    }
}
```

### Example — catching it in consumer code

```php
use Closure;
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\InvalidIndexRule;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class SafeRobotsMiddleware extends RobotsMiddleware
{
    public function handle(Request $request, Closure $next)
    {
        try {
            return parent::handle($request, $next);
        } catch (InvalidIndexRule $e) {
            // $this->response is already resolved at this point, so a
            // fallback directive can still be written.
            return $this->responseWithRobots('all');
        }
    }
}
```

Or render it through Laravel's exception handler (e.g. in `bootstrap/app.php`):

```php
->withExceptions(function (Exceptions $exceptions) {
    $exceptions->render(function (InvalidIndexRule $e) {
        return response('Blocked', 403);
    });
})
```
```

--------------------------------

### Advanced example - non-trivial rule and error handling

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

A subclass that returns a verbatim directive string from shouldIndex() and overrides handle() to catch InvalidIndexRule, falling back to 'none'. Includes the required imports and class definition.

```php
use Closure;
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\InvalidIndexRule;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class SeoRobotsMiddleware extends RobotsMiddleware
{
    public function handle(Request $request, Closure $next)
    {
        try {
            return parent::handle($request, $next);
        } catch (InvalidIndexRule $e) {
            // $this->response was already resolved before shouldIndex()
            // was called, so we can still return a decorated response.
            return $this->responseWithRobots('none');
        }
    }

    protected function shouldIndex(Request $request): string
    {
        if ($request->segment(1) === 'docs') {
            return 'noindex, follow';
        }

        return 'all';
    }
}
```

--------------------------------

### Catch InvalidIndexRule in consumer code

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/InvalidIndexRule.md

Example of wrapping parent::handle() in a try-catch block to catch InvalidIndexRule and return a fallback response. Also shows rendering the exception through Laravel's exception handler.

```php
use Closure;
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\InvalidIndexRule;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class SafeRobotsMiddleware extends RobotsMiddleware
{
    public function handle(Request $request, Closure $next)
    {
        try {
            return parent::handle($request, $next);
        } catch (InvalidIndexRule $e) {
            // $this->response is already resolved at this point, so a
            // fallback directive can still be written.
            return $this->responseWithRobots('all');
        }
    }
}
```

```php
->withExceptions(function (Exceptions $exceptions) {
    $exceptions->render(function (InvalidIndexRule $e) {
        return response('Blocked', 403);
    });
})
```

--------------------------------

### Trigger InvalidIndexRule via a bad rule

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/InvalidIndexRule.md

Example middleware that returns an array from shouldIndex(), which is neither a boolean nor a string, causing RobotsMiddleware::handle() to throw InvalidIndexRule::requiresBooleanOrString().

```php
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class InvalidTestMiddleware extends RobotsMiddleware
{
    protected function shouldIndex(Request $request): array
    {
        return []; // any non-bool, non-string return triggers InvalidIndexRule
    }
}
```

--------------------------------

### Run tests

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/README.md

Run the package's test suite using Composer.

```bash
$ composer test
```

--------------------------------

### handle(Request $request, Closure $next)

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Laravel middleware entry point that resolves the downstream response, determines the indexing directive via shouldIndex(), and writes the x-robots-tag header. Returns the response with the header applied, preserving redirects and Symfony responses.

```APIDOC
## handle(Request $request, Closure $next)

### Description
Laravel middleware entry point. Resolves the downstream response first, then asks `shouldIndex()` for the indexing directive and writes it to the response header.

### Method
Middleware handle method (invoked by Laravel's middleware pipeline)

### Endpoint
N/A - This is a PHP middleware method, not an HTTP endpoint.

### Parameters
#### Path Parameters
- **$request** (Illuminate\Http\Request) - Required - The inbound HTTP request being processed. Passed unchanged to `$next()` and to `shouldIndex()`.
- **$next** (Closure) - Required - The next middleware / terminal handler in the pipeline. Invoked as `$next($request)` to obtain the downstream response.

### Return
No declared return type. Effectively the value returned by `$next($request)` (the response), with the `x-robots-tag` header applied — either the same object returned by `responseWithRobots()`. The response object is never replaced, so redirects and non-Illuminate (`Symfony`) responses are preserved.

### Throws
- **Spatie\RobotsMiddleware\InvalidIndexRule** - Thrown when `shouldIndex($request)` returns a value that is neither `bool` nor `string` (e.g. `array`, `int`, `null`, `object`).

### Header decision logic
| `shouldIndex()` return | Header value written |
|---|---|
| `true` (boolean) | `all` |
| `false` (boolean) | `none` |
| any `string` | the string verbatim (`nofollow`, `noindex`, `noindex, follow`, …) |
| anything else | throws `InvalidIndexRule` |

The header is written with `replace = false`, so a directive already present on the response is kept and the new value is appended as an additional header value.

### Example — basic usage

Extend the class, override `shouldIndex()`, and register the subclass in the global middleware stack:

```php
// app/Http/Middleware/MyRobotsMiddleware.php
namespace App\Http\Middleware;

use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class MyRobotsMiddleware extends RobotsMiddleware
{
    protected function shouldIndex(Request $request): bool
    {
        return $request->segment(1) !== 'admin';
    }
}
```

```php
// app/Http/Kernel.php
class Kernel extends HttpKernel
{
    protected $middleware = [
        // ...
        \App\Http\Middleware\MyRobotsMiddleware::class,
    ];
}
```

Every response now carries `x-robots-tag: all` or `x-robots-tag: none`.

### Example — non-trivial rule and error handling

A subclass whose rule returns a verbatim directive string, plus a guard that falls back gracefully when the rule misbehaves:

```php
use Closure;
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\InvalidIndexRule;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class SeoRobotsMiddleware extends RobotsMiddleware
{
    public function handle(Request $request, Closure $next)
    {
        try {
            return parent::handle($request, $next);
        } catch (InvalidIndexRule $e) {
            // $this->response was already resolved before shouldIndex()
            // was called, so we can still return a decorated response.
            return $this->responseWithRobots('none');
        }
    }

    protected function shouldIndex(Request $request): string
    {
        if ($request->segment(1) === 'docs') {
            return 'noindex, follow';
        }

        return 'all';
    }
}
```

Alternatively, catch it globally in the Laravel exception handler, e.g. in `bootstrap/app.php` with `->withExceptions(fn ($exceptions) => $exceptions->render(function (InvalidIndexRule $e) { /* ... */ }))`.
```

--------------------------------

### Basic call inside an overridden handle

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Shows how to call responseWithRobots from a subclass's handle method when the response status is 404. The method is called with 'noindex' to set the x-robots-tag header.

```php
public function handle(Request $request, Closure $next)
{
    $this->response = $next($request);

    if ($this->response->getStatusCode() === 404) {
        return $this->responseWithRobots('noindex');
    }

    return parent::handle($request, $next);
}
```

--------------------------------

### responseWithRobots method implementation

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Protected method that sets the x-robots-tag header on the response and returns it. The third argument false prevents replacing existing header values. Throws a native PHP Error if called before the response is resolved.

```php
    protected function responseWithRobots(string $contents)
    {
        $this->response->headers->set('x-robots-tag', $contents, false);

        return $this->response;
    }
```

--------------------------------

### handle() method implementation

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

The core handle method that processes the request, resolves the downstream response, and applies the x-robots-tag header based on shouldIndex(). Throws InvalidIndexRule if shouldIndex returns a non-bool/string.

```php
    public function handle(Request $request, Closure $next)
    {
        $this->response = $next($request);

        $shouldIndex = $this->shouldIndex($request);

        if (is_bool($shouldIndex)) {
            return $this->responseWithRobots($shouldIndex ? 'all' : 'none');
        }

        if (is_string($shouldIndex)) {
            return $this->responseWithRobots($shouldIndex);
        }

        throw InvalidIndexRule::requiresBooleanOrString();
    }
```

--------------------------------

### Register middleware in Kernel.php

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/README.md

Register the custom middleware class in the $middleware property of the HTTP kernel to apply it globally to all requests.

```php
// app/Http/Kernel.php

class Kernel extends HttpKernel
{
    protected $middleware = [
        // ...
        \App\Http\Middleware\MyRobotsMiddleware::class,
    ];
    
    // ...
}
```

--------------------------------

### Default shouldIndex implementation

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

The default implementation returns true, meaning all requests are indexed. Override this method in a subclass to customize behavior per request.

```php
    /**
     * @return string|bool
     */
    protected function shouldIndex(Request $request)
    {
        return true;
    }
```

--------------------------------

### Class declaration for RobotsMiddleware

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Declares the RobotsMiddleware class. It is not abstract or final, and is meant to be extended in application code to override shouldIndex().

```php
class RobotsMiddleware
```

--------------------------------

### Preserving pre-set directives with responseWithRobots

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Demonstrates that responseWithRobots does not overwrite previously set x-robots-tag values because HeaderBag::set is called with false as the third argument. The response ends up with two values: ['nofollow', 'all'].

```php
// An upstream handler or middleware already set a directive:
return response('Hello world!')->header('x-robots-tag', 'nofollow');

// RobotsMiddleware::handle() then appends its own value instead of
// overwriting — the response now carries two x-robots-tag values:
//   ["nofollow", "all"]
```

--------------------------------

### Register render callback in bootstrap/app.php

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/errors.md

Register a render callback for InvalidIndexRule in Laravel 11+ exception handling configuration. This returns a 403 response when the exception is thrown. Because the exception extends \Exception, a generic handler will also intercept it.

```php
->withExceptions(function (Exceptions $exceptions) {
    $exceptions->render(function (InvalidIndexRule $e) {
        return response('Blocked by robots policy', 403);
    });
})
```

--------------------------------

### requiresBooleanOrString() method

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/InvalidIndexRule.md

Factory method that returns a fully-formed instance of the calling class via late static binding. It sets the message 'An indexing rule needs to return a boolean or a string.' with no code or previous exception.

```php
    public static function requiresBooleanOrString()
    {
        return new static('An indexing rule needs to return a boolean or a string.');
    }
```

--------------------------------

### RobotsMiddleware::shouldIndex()

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Overridable method that determines the value of the x-robots-tag header. Extend the class and override this method to customize the robots directive.

```APIDOC
## shouldIndex()

### Description
Determines the value of the `x-robots-tag` header. Override this method in a subclass to customize the robots directive.

### Method
`shouldIndex()`

### Signature
```php
public function shouldIndex(): bool
```

### Parameters
None.

### Return Value
- **bool** - `true` to allow indexing, `false` to disallow. The header value is derived from this result.

### Usage Example
```php
class MyRobotsMiddleware extends RobotsMiddleware
{
    public function shouldIndex(): bool
    {
        return false; // disallow indexing
    }
}
```

### Notes
- The method is not declared in the source, but the documentation states it is overridable and its return value decides the header value.
- The exact header value mapping is not specified in the source.
```

--------------------------------

### InvalidIndexRule class definition

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/InvalidIndexRule.md

The exception class extends \Exception and provides a static factory method requiresBooleanOrString() that returns an instance with the message 'An indexing rule needs to return a boolean or a string.'

```php
class InvalidIndexRule extends Exception
{
    public static function requiresBooleanOrString()
    {
        return new static('An indexing rule needs to return a boolean or a string.');
    }
}
```

--------------------------------

### Protected $response property declaration

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/api-reference/RobotsMiddleware.md

Declares the protected $response property, which holds the downstream HTTP response returned by $next($request) in handle(). It is assigned at the top of handle() and read by responseWithRobots().

```php
    protected $response;
```

--------------------------------

### Catch InvalidIndexRule in overridden handle()

Source: https://github.com/spatie/laravel-robots-middleware/blob/main/_autodocs/errors.md

Override handle() in a custom middleware to catch InvalidIndexRule and return a 'none' robots response. The exception is thrown when shouldIndex() returns a non-bool, non-string value. This snippet shows the required imports and class structure.

```php
use Closure;
use Illuminate\Http\Request;
use Spatie\RobotsMiddleware\InvalidIndexRule;
use Spatie\RobotsMiddleware\RobotsMiddleware;

class SafeRobotsMiddleware extends RobotsMiddleware
{
    public function handle(Request $request, Closure $next)
    {
        try {
            return parent::handle($request, $next);
        } catch (InvalidIndexRule $e) {
            return $this->responseWithRobots('none');
        }
    }
}
```
Tokens:
Raw