Robots.txt Parser

https://github.com/vipnytt/robotstxtparser
Admin
Robots.txt Parser is an easy-to-use, extensible robots.txt parser library with full support for...
Tokens:21,179
Snippets:337
Trust Score:6.3
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Robots.txt Parser with Composer

Source: https://github.com/vipnytt/robotstxtparser/blob/master/README.md

Add the robots.txt parser to your project's dependencies using Composer. This ensures you have the correct version installed.

```json
{
  "require": {
    "vipnytt/robotstxtparser": "^2.1"
  }
}
```

--------------------------------

### Initialize Delay Management

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Instantiate the DelayManage class using a Database object. This is the starting point for managing delay-related operations.

```php
<?php
$db = new \vipnytt\RobotsTxtParser\Database($pdo);
$delayManage = $db->delay();
```

--------------------------------

### Get Robots.txt Contents

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Fetch the raw content of the robots.txt file.

```php
$client->getContents();
```

--------------------------------

### CommentClient Get Method

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CommentClient.md

Retrieves comments for the matching user agent, ignoring the '*' user agent to prevent spam. Use this to get a list of comments.

```php
/**
 * @return string[]
 */
```

--------------------------------

### get

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CommentClient.md

Retrieves comments for the matching user-agent, ignoring the '*' user-agent.

```APIDOC
## get

### Description
Get any comments for the matching user-agent. Ignoring the `*` user-agent in case of spam.

### Method
[Method not specified, likely internal or part of a larger SDK]

### Return Value
- string[]: An array of strings, where each string is a comment.
```

--------------------------------

### Get User-Agents

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve a list of all declared User-agents found in the robots.txt file.

```php
$client->getUserAgents();
```

--------------------------------

### Get Valid Until Timestamp

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Get the timestamp until which the robots.txt file is considered valid.

```php
$client->validUntil();
```

--------------------------------

### Request-rate Directive Get Base URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the base URI for the Request-rate directive, intended for usage by a 3rd party Delay handler.

```APIDOC
## `Request-rate` directive get base URI

### Description
Intended for usage by an 3rd party Delay handler.

### Method
```php
$client->userAgent('myBot')->requestRate()->getBaseUri();
```
```

--------------------------------

### Get Base URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayClient.md

Retrieves the base URI to which the directive applies.

```php
@return string
```

--------------------------------

### Request-rate Directive Get User Agent

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the user agent for the Request-rate directive, intended for usage by a 3rd party Delay handler.

```APIDOC
## `Request-rate` directive get user agent

### Description
Intended for usage by an 3rd party Delay handler.

### Method
```php
$client->userAgent('myBot')->requestRate()->getUserAgent();
```
```

--------------------------------

### Get TxtClient Instance

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheBaseInterface.md

Returns an instance of TxtClient. This is used to interact with the robots.txt parser client.

```php
@return Client\TxtClient
```

--------------------------------

### Crawl-delay Directive Get User Agent

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the user agent for the Crawl-delay directive, intended for usage by a 3rd party Delay handler.

```APIDOC
## `Crawl-delay` directive get user agent

### Description
Intended for usage by an 3rd party Delay handler.

### Method
```php
$client->userAgent('myBot')->crawlDelay()->getUserAgent();
```
```

--------------------------------

### export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayClient.md

Get the numeric value of the directive.

```APIDOC
## export

### Description
Get the numeric value of the directive.

### Returns
- `float|int`: The numeric value of the delay directive.
```

--------------------------------

### Get Character Encoding

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Determine the character encoding of the robots.txt file.

```php
$client->getEncoding();
```

--------------------------------

### Export Allow Directive Rules

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Get an array of all rules associated with the Allow directive for a specific user-agent.

```php
$client->userAgent('myBot')->allow()->export();
```

--------------------------------

### Set Visit Time Window in robots.txt

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/Directives.md

The 'visit-time' directive suggests a time window during which robots should crawl the site. Specify the start and end times in HH:MM-HH:MM format (24-hour clock).

```robots.txt
visit-time: [time]-[time]
```

--------------------------------

### Get Effective URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Obtain the effective URI after any potential redirects have been resolved.

```php
$client->getEffectiveUri();
```

--------------------------------

### Get Base URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the base URI associated with the UriClient.

```php
$client->getBaseUri();
```

--------------------------------

### Get User-Agent from Cache-delay

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the user-agent associated with the Cache-delay directive. Intended for use by a third-party Delay handler.

```php
$client->userAgent('myBot')->cacheDelay()->getUserAgent();
```

--------------------------------

### Get Next-Update Timestamp

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the timestamp indicating when the robots.txt file should be updated next.

```php
$client->nextUpdate();
```

--------------------------------

### Comment Directive Get

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Exports a list of comments/messages/information for your user-agent only, which is spam-filtered and intended to be read.

```APIDOC
## `Comment` directive get

### Description
Exports an list of comments/messages/information that exists for your user-agent only. Spam-filtered and is intended to be read.

### Method
```php
$client->userAgent('myBot')->comment()->get();
```
```

--------------------------------

### Handling Request-rate Directive Check Queue

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the size of the current request queue in seconds for handling the Request-rate directive.

```APIDOC
## Handling of the `Request-rate` directive - check queue

### Description
Get the size of the current request queue in seconds.

### Method
```php
$client->userAgent('myBot')->requestRate()->handle($delayManager)->checkQueue();
```
```

--------------------------------

### Get Request-rate Value

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the specific request-rate value for the selected user-agent. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->requestRate()->getValue();
```

--------------------------------

### Initialize Database and Cache Base

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Instantiate the Database class and obtain a cache base instance for a given URI. Most parameters default to their documented values.

```php
<?php
$db = new \vipnytt\RobotsTxtParser\Database($pdo);
$cacheBase = $db->cache()->base('https://example.com');
```

--------------------------------

### Request-rate Directive Get Value

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the request-delay value for the Request-rate directive.

```APIDOC
## `Request-rate` directive get value

### Description
Get the request-delay value.

### Method
```php
$client->userAgent('myBot')->requestRate()->getValue();
```
```

--------------------------------

### Get Host with URI Fallback

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the main host declared by the Host directive, falling back to the host of the effective URI if the Host directive is not set.

```php
$client->host()->getWithUriFallback();
```

--------------------------------

### Handling Crawl-delay Directive Check Queue

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the size of the current request queue in seconds for handling the Crawl-delay directive.

```APIDOC
## Handling of the `Crawl-delay` directive - check queue

### Description
Get the size of the current request queue in seconds.

### Method
```php
$client->userAgent('myBot')->crawlDelay()->handle($delayManager)->checkQueue();
```
```

--------------------------------

### Crawl-delay Directive Get Value

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the request-delay value for the Crawl-delay directive.

```APIDOC
## `Crawl-delay` directive get value

### Description
Get the request-delay value.

### Method
```php
$client->userAgent('myBot')->crawlDelay()->getValue();
```
```

--------------------------------

### Initialize Database and Cache Manager

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Instantiate the Database class and obtain an instance of the cache manager. Most parameters default to their documented values.

```php
<?php
$db = new \vipnytt\RobotsTxtParser\Database($pdo);
$cacheManage = $db->cache();
```

--------------------------------

### Get Debug Data

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheBaseInterface.md

Retrieves the RAW data from the database for debugging purposes. Accepts a base URI as a parameter.

```php
@param string $base
@return array
```

--------------------------------

### Export Host Directive

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Get the main host declared by the Host directive. This method returns the explicitly set host.

```php
$client->host()->export();
```

--------------------------------

### Initialize TxtClient

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Instantiate the TxtClient with a URL, status code, and filename. Default parameters are often omitted for brevity; refer to specific class documentation for a full overview.

```php
<?php
$client = new \vipnytt\RobotsTxtParser\TxtClient('http://example.com', 200, 'robots.txt');
```

--------------------------------

### Get Cache Base Instance

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheManageInterface.md

Retrieve an instance of the BaseInterface for cache operations. This is used to interact with the underlying cache storage.

```php
@param string $baseUri
@return Client\Cache\BaseInterface
```

--------------------------------

### Get Crawl-delay Value

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the specific crawl-delay value for the selected user-agent. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->crawlDelay()->getValue();
```

--------------------------------

### __construct

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayBaseInterface.md

Initializes the Delay handler with database connection, base URI, user agent, and delay settings.

```APIDOC
## __construct

### Description
Initializes the Delay handler.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **pdo** (\PDO) - Required - The PDO database connection instance.
- **baseUri** (string) - Required - The base URI for the requests.
- **userAgent** (string) - Required - The user agent string.
- **delay** (float|int) - Required - The delay in seconds.
```

--------------------------------

### Handling Request-rate Directive Get Time Sleep Until

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the timestamp with microseconds until which to wait before sending the request for handling the Request-rate directive.

```APIDOC
## Handling of the `Request-rate` directive - get time sleep until

### Description
Get the timestamp w/microseconds you'll have to wait until before sending the request.

### Method
```php
$client->userAgent('myBot')->requestRate()->handle($delayManager)->getTimeSleepUntil();
```
```

--------------------------------

### Get Top Wait Times

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayManageInterface.md

Retrieves an array of hosts with the highest wait times. This includes hosts that are frequently requested or have high delay times specified in their robots.txt.

```php
@param int $limit
@param int $min
@return array
```

--------------------------------

### __construct

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheManageInterface.md

Initializes the Cache manager. Requires a PDO instance for database operations.

```APIDOC
## __construct

### Description
Initializes the Cache manager with a PDO database connection.

### Parameters
#### Path Parameters
- **pdo** (PDO) - Required - The PDO instance for database interaction.
```

--------------------------------

### Handling Crawl-delay Directive Get Time Sleep Until

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the timestamp with microseconds until which to wait before sending the request for handling the Crawl-delay directive.

```APIDOC
## Handling of the `Crawl-delay` directive - get time sleep until

### Description
Get the timestamp w/microseconds you'll have to wait until before sending the request.

### Method
```php
$client->userAgent('myBot')->crawlDelay()->handle($delayManager)->getTimeSleepUntil();
```
```

--------------------------------

### Get Top Wait Times Data

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Fetches the raw data of the top wait times from the database. This provides detailed insights into delay metrics.

```php
$delayManage->getTopWaitTimes();
```

--------------------------------

### Get Base URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RequestRateClient.md

Retrieves the base URI to which the request-rate directive applies. This helps in understanding the scope of the rate limit.

```php
/**
 * @return string
 */
```

--------------------------------

### Initialize Cache Manager with PDO

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheManageInterface.md

Instantiate the cache manager by passing a PDO object.

```php
@param PDO $pdo
```

--------------------------------

### Get User-Agent for Crawl-delay

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the user-agent string associated with the crawl-delay directive, intended for third-party delay handlers. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->crawlDelay()->getUserAgent();
```

--------------------------------

### Initialize Import Client

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Instantiate the Import client with an array of data. This is intended for debugging purposes to compare imported data with generated export data.

```php
<?php
$client = new \vipnytt\RobotsTxtParser\Import($array);
```

--------------------------------

### Get Comments for Your User-Agent

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves comments intended for your user-agent, which are spam-filtered and meant for direct reading. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->comment()->get();
```

--------------------------------

### handle

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayClient.md

Returns an instance of Client\Delay\ClientInterface.

```APIDOC
## handle

### Description
Returns an instance of [Client\Delay\ClientInterface](DelayBaseInterface.md).

### Parameters
#### Path Parameters
- `handler` (Client\Delay\ManageInterface) - Description: The handler to manage the delay.

### Returns
- `Client\Delay\BaseInterface`: An instance of the delay client interface.
```

--------------------------------

### TxtClient::__construct

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Initializes a new instance of the TxtClient class.

```APIDOC
## TxtClient::__construct

### Description
Initializes a new instance of the TxtClient class.

### Parameters
#### Path Parameters
- **baseUri** (string) - Description of baseUri
- **statusCode** (int) - Description of statusCode
- **content** (string) - Description of content
- **encoding** (string) - Description of encoding
- **effectiveUri** (string|null) - Description of effectiveUri
- **byteLimit** (int|null) - Description of byteLimit
```

--------------------------------

### Crawl-delay Directive Get Base URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Gets the base URI for the Crawl-delay directive, intended for usage by a 3rd party Delay handler.

```APIDOC
## `Crawl-delay` directive get base URI

### Description
Intended for usage by an 3rd party Delay handler.

### Method
```php
$client->userAgent('myBot')->crawlDelay()->getBaseUri();
```
```

--------------------------------

### Get User-Agent for Request-rate

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the user-agent string associated with the request-rate directive, intended for third-party delay handlers. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->requestRate()->getUserAgent();
```

--------------------------------

### Initialize UriClient

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Create a new UriClient instance for a given URL. Default parameters are often omitted for brevity; refer to specific class documentation for a full overview.

```php
<?php
$client = new \vipnytt\RobotsTxtParser\UriClient('http://example.com');
```

--------------------------------

### Get Base URI from Cache-delay

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the base URI associated with the Cache-delay directive. Intended for use by a third-party Delay handler.

```php
$client->userAgent('myBot')->cacheDelay()->getBaseUri();
```

--------------------------------

### Render Client Instance

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Returns an instance of RenderClient, likely used for rendering or outputting the parsed robots.txt content.

```php
@return RenderClient
```

--------------------------------

### Specify Main Host Directive

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/Directives.md

Use the 'host' directive to indicate the primary site when multiple mirrors exist.

```robots.txt
host: [host]
```

--------------------------------

### Initialize Delay Base Interfaces

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Sets up the TxtClient and retrieves different delay-related interfaces (crawlDelay, cacheDelay, requestRate). These interfaces are then handled by the DelayBase class.

```php
<?php
$txtClient = new \vipnytt\RobotsTxtParser\TxtClient('http://example.com', 200, 'robots.txt');

$delayInterface = $txtClient->userAgent('myBot')->crawlDelay();
// or
$delayInterface = $txtClient->userAgent('myBot')->cacheDelay();
// or
$delayInterface = $txtClient->userAgent('myBot')->requestRate();

$db = new \vipnytt\RobotsTxtParser\Database($pdo);
$delayBase = $delayInterface->handle($db->delay());
```

--------------------------------

### Create RobotsTxtParser Instance with TxtClient

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/README.md

Use TxtClient to initialize the parser with custom robots.txt content. This is useful for testing, or when you have the robots.txt content available directly, along with its HTTP status code.

```php
<?php
$robotsTxt = <<<TXT
User-agent: *
Disallow: /
Allow: /public
Crawl-delay: 5
TXT;
$client = new \vipnytt\RobotsTxtParser\TxtClient('http://example.com', 200, $robotsTxt);

```

--------------------------------

### Create RobotsTxtParser Instance with Import

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/README.md

Use the Import class to initialize the parser with an array of rules. This is intended for special cases, such as when importing rules previously exported by the TxtClient.

```php
<?php
$import = new \vipnytt\RobotsTxtParser\Import($array);

```

--------------------------------

### __construct

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayManageInterface.md

Initializes the Delay Manager. It requires a PDO connection to interact with the database.

```APIDOC
## __construct

### Description
Initializes the Delay Manager.

### Parameters
#### Path Parameters
- **pdo** (PDO) - Required - A PDO database connection instance.
```

--------------------------------

### client()

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheBaseInterface.md

Returns an instance of TxtClient.

```APIDOC
## client()

### Description
Returns an instance of [TxtClient](TxtClient.md).

### Method
client

### Return Value
- TxtClient - An instance of the TxtClient.
```

--------------------------------

### Create RobotsTxtParser Instance with Cache

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/README.md

Utilize the integrated caching system by providing a PDO database connection. This allows for efficient retrieval of robots.txt files by caching them after the first download.

```php
<?php
$pdo = new PDO('mysql:host=127.0.0.1;dbname=database', 'username', 'password');
$handler = new \vipnytt\RobotsTxtParser\Database($pdo);
$client = $handler->cache()->base('http://example.com')->client();

```

--------------------------------

### getTimeSleepUntil

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayBaseInterface.md

Gets the timestamp (with milliseconds) until which you will have to wait before sending the next request.

```APIDOC
## getTimeSleepUntil

### Description
Get the timestamp (with milliseconds) you'll have to wait until before you'll send the request.

### Method
GET (assumed, as it retrieves a timestamp)

### Endpoint
(Not specified, typically internal or part of a larger client operation)

### Parameters
None

### Response
#### Success Response (200)
- **float|int** - The timestamp (with milliseconds) to wait until.
```

--------------------------------

### Create Base Delay Client

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayManageInterface.md

Returns an instance of DelayInterface. Use this to configure the base URI, user agent, and default delay time.

```php
@param string $baseUri
@param string $userAgent
@param float|int $delay
@return Client\Delay\BaseInterface
```

--------------------------------

### TxtClient::render

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Returns an instance of RenderClient.

```APIDOC
## TxtClient::render

### Description
Returns an instance of [RenderClient](RenderClient.md).

### Returns
- **RenderClient** - An instance of RenderClient.
```

--------------------------------

### TxtClient::export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Returns a tree-formatted array containing every rule, suitable for external usage.

```APIDOC
## TxtClient::export

### Description
Returns an tree-formatted array containing every single rule. Perfect for external usage.

### Returns
- **array** - A tree-formatted array of all rules.
```

--------------------------------

### getValue

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayClient.md

Get the numeric value of the directive. For Cache-delay, it returns Crawl-delay if Cache-delay is not found.

```APIDOC
## getValue

### Description
Get the numeric value of the directive. When the value is requested but not found, the value of Crawl-delay is returned, to maintain compatibility.

### Returns
- `float|int`: The numeric value of the delay directive.
```

--------------------------------

### Import Class Constructor Parameters

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/Import.md

Defines the parameters accepted by the Import class constructor.

```php
@param array $export
@param string $baseUri
```

--------------------------------

### Initialize Database Cache Handler

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/sql/cache.md

Instantiate the database handler with a PDO connection to manage the robots.txt cache.

```php
<?php
$handler = new "vipnytt\RobotsTxtParser\Database"("$pdo");
$client = $handler->cache()->base('http://example.com');

```

--------------------------------

### Get Cache-delay Request-delay Value

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the specific request-delay value set by the Cache-delay directive.

```php
$client->userAgent('myBot')->cacheDelay()->getValue();
```

--------------------------------

### TxtClient Constructor Parameters

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Lists the parameters accepted by the TxtClient constructor for initializing the parser with custom content, status codes, and URIs.

```php
@param string $baseUri
@param int $statusCode
@param string $content
@param string $encoding
@param string|null $effectiveUri
@param int|null $byteLimit
```

--------------------------------

### Import Class Constructor

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/Import.md

Initializes the Import class. It accepts an array previously exported by TxtClient and an optional base URI.

```APIDOC
## __construct

### Description
Initializes the Import class with exported data and a base URI.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **export** (array) - Required - An array previously exported by the TxtClient.
* **baseUri** (string) - Optional - The base URI for the import.
```

--------------------------------

### DelayBaseInterface Constructor

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayBaseInterface.md

Initializes the DelayBaseInterface with database connection, base URI, user agent, and delay settings.

```php
@param \PDO $pdo
@param string $baseUri
@param string $userAgent
@param float|int $delay
```

--------------------------------

### TxtClient::sitemap

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Wrapper for the Sitemap directive. Returns an instance of SitemapClient.

```APIDOC
## TxtClient::sitemap

### Description
Wrapper for the [Sitemap](../Directives.md#sitemap) directive. Returns an instance of [SitemapClient](SitemapClient.md).

### Returns
- **SitemapClient** - An instance of SitemapClient.
```

--------------------------------

### Create TxtClient for Parsing

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Initializes a TxtClient instance for parsing robots.txt content from a given URL. This client is used for processing the robots.txt file itself.

```php
$client = $handler->client('http://example.com');
```

--------------------------------

### Get Raw Cache Data

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the raw data stored in the cache for debugging or inspection purposes.

```php
$cacheBase->debug();
```

--------------------------------

### Create Cache Base for a URI

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Initializes the cache base for a specific URI. This is a prerequisite for certain cache operations.

```php
$cacheManage->base('https://example.com');
```

--------------------------------

### Get HTTP/FTP Status Code

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve the HTTP or FTP status code returned when fetching the robots.txt file.

```php
$client->getStatusCode();
```

--------------------------------

### Render Robots.txt Compressed

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Render the robots.txt content in a compressed format, optimized for storage in databases.

```php
$client->render()->compressed();
```

--------------------------------

### Import Class Namespace

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/Import.md

Specifies the package for the Import class.

```php
@package vipnytt\RobotsTxtParser
```

--------------------------------

### Get Ignored Import Data

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve ignored import data from the Import client. This method is primarily for debugging.

```php
$client->getIgnoredImportData();
```

--------------------------------

### handle

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RequestRateClient.md

Returns an instance of Client\Delay\ClientInterface. Falls back to Crawl-delay if the value is not found.

```APIDOC
## handle

### Description
Returns an instance of [Client\Delay\ClientInterface](DelayBaseInterface.md).

When the value is requested but not found, the value of [``Crawl-delay``](../Directives.md#crawl-delay) is returned, to maintain compatibility.

### Parameters
#### Path Parameters
- **handler** (Client\Delay\ManageInterface) - Required - The handler to manage delays.

### Returns
- Client\Delay\BaseInterface: An instance of Client\Delay\ClientInterface.
```

--------------------------------

### RenderClient::normal

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RenderClient.md

Generates a standard-looking robots.txt file. Directives have uppercase first characters, whitespace is used between directives and values, and user-agent groups are separated by blank lines. This format is easy to edit.

```APIDOC
## normal

### Description
Generates an normal looking robots.txt.

Differences compared to [compressed](#compressed):
- The directives first character is uppercase
- Whitespace between the directive and it's value
- Easy to edit
- User-agent groups are separated with blank lines

### Parameters
#### Path Parameters
- **eol** (string) - Required - The end-of-line character to use.
```

--------------------------------

### Render Robots.txt in Compatibility Mode

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Render the robots.txt content in compatibility mode, optimized for parsing by third-party parsers that may not strictly adhere to standards.

```php
$client->render()->compatibility();
```

--------------------------------

### TxtClient::getUserAgents

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Lists all user-agents defined in the robots.txt file.

```APIDOC
## TxtClient::getUserAgents

### Description
List all user-agents defined in the `robots.txt`.

### Returns
- **string[]** - An array of user-agent strings.
```

--------------------------------

### SQL Table Creation for Robots.txt Cache

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/sql/cache.md

This SQL script defines the `robotstxt__cache1` table structure for storing robots.txt content and associated metadata in a MySQL database.

```SQL
CREATE TABLE `robotstxt__cache1` (
  `base`       VARCHAR(269)
               CHARACTER SET ascii
               COLLATE ascii_bin           NOT NULL,
  `content`    MEDIUMTEXT COLLATE utf8_bin NOT NULL,
  `statusCode` SMALLINT(3) UNSIGNED DEFAULT NULL,
  `validUntil` INT(10) UNSIGNED            NOT NULL,
  `nextUpdate` INT(10) UNSIGNED            NOT NULL,
  `effective`  VARCHAR(269)
               CHARACTER SET ascii
               COLLATE ascii_bin    DEFAULT NULL,
  `worker`     TINYINT(3) UNSIGNED  DEFAULT NULL,
  PRIMARY KEY (`base`),
  KEY `worker` (`worker`, `nextUpdate`)
)
  DEFAULT CHARSET = utf8
  COLLATE = utf8_bin

```

--------------------------------

### base

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayManageInterface.md

Creates and returns an instance of DelayInterface, configured with base URI, user agent, and delay time.

```APIDOC
## base

### Description
Returns an instance of [DelayInterface](DelayClient.md), configured with the provided parameters.

### Method
[Method Signature]

### Parameters
#### Path Parameters
- **baseUri** (string) - Required - The base URI for the client.
- **userAgent** (string) - Required - The user agent string.
- **delay** (float|int) - Required - The delay time in seconds.

### Returns
- **Client\Delay\BaseInterface** - An instance of the delay interface.
```

--------------------------------

### UserAgentClient Methods

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/UserAgentClient.md

This section details the public methods available on the UserAgentClient class for managing robots.txt directives.

```APIDOC
## UserAgentClient

### Public functions

- [allow](#allow)
- [cacheDelay](#cachedelay)
- [comment](#comment)
- [crawlDelay](#crawldelay)
- [disallow](#disallow)
- [export](#export)
- [isAllowed](#isallowed)
- [isDisallowed](#isdisallowed)
- [noIndex](#noindex)
- [requestRate](#requestrate)
- [robotVersion](#robotversion)
- [visitTime](#visittime)

### allow

Wrapper for the [Allow](../Directives.md#allow) directive.

Returns an instance of [AllowClient](AllowClient.md).

### cacheDelay

Wrapper for the [Cache-delay](../Directives.md#cache-delay) directive.

Returns an instance of [DelayClient](DelayClient.md).

### comment

Wrapper for the [Comment](../Directives.md#comment) directive.

Returns an instance of [CommentClient](CommentClient.md).

### crawlDelay

Wrapper for the [Crawl-delay](../Directives.md#crawl-delay) directive.

Returns an instance of [DelayClient](DelayClient.md).

### disallow

Wrapper for the [Disallow](../Directives.md#disallow) directive.

Returns an instance of [AllowClient](AllowClient.md).

### export

Export an array of rules for the current User-agent.

### isAllowed

Check if an URI is allowed to crawl.

@param string $uri
@return bool

### isDisallowed

Check if an URI is disallowed to crawl.

@param string $uri
@return bool

### noIndex

Wrapper for the [NoIndex](../Directives.md#noindex) directive.

Returns an instance of [AllowClient](AllowClient.md).

### requestRate

Wrapper for the [Request-rate](../Directives.md#request-rate) directive.

Returns an instance of [RequestRateClient](RequestRateClient.md).

### robotVersion

Wrapper for the [Robot-version](../Directives.md#robot-version) directive.

Returns an instance of [RobotVersionClient](RobotVersionClient.md).

### visitTime

Wrapper for the [Visit-time](../Directives.md#visit-time) directive.

Returns an instance of [VisitTimeClient](VisitTimeClient.md).
```

--------------------------------

### ImportClient

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

The Import class allows for the creation of a client from an array. It extends TxtClient and provides methods for debugging by comparing imported data with generated export data.

```APIDOC
## ImportClient

### Description
Allows for the creation of a client from an array. It extends TxtClient and provides methods for debugging by comparing imported data with generated export data.

### Example Usage
```php
<?php
$client = new \vipnytt\RobotsTxtParser\Import($array);
```

### Methods
- `getIgnoredImportData()`: Get the difference between the imported and the generated export array. Intended for debugging purposes only.
```

--------------------------------

### RenderClient::compatibility

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RenderClient.md

Generates a robots.txt optimized for parsing by custom 3rd party parsers that do not strictly follow the standards. Each User-agent is listed with its own separate rule set, potentially increasing file size but ensuring maximum compatibility with poorly written parsers.

```APIDOC
## compatibility

### Description
Generates an robots.txt optimized for parsing by custom 3rd party parsers, witch do not strictly follow the standards.

Differences compared to [normal](#normal):
- Each User-agent is listed with it's own separate rule set (even if it's equal to others)
- Byte consuming, may be multiple times larger (depending on the number of user-agents)
- Maximum compatibility, optimized for badly written 3rd party parsers with limited specification support

### Parameters
#### Path Parameters
- **eol** (string) - Required - The end-of-line character to use.
```

--------------------------------

### getValue

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RequestRateClient.md

Gets the rate for a given timestamp. If no timestamp is provided, the current timestamp is used. Falls back to Crawl-delay if the value is not found.

```APIDOC
## getValue

### Description
Get the rate for an given timestamp. If no timestamp integer is provided, current timestamp is used.

When the value is requested but not found, the value of [``Crawl-delay``](../Directives.md#crawl-delay) is returned, to maintain compatibility.

### Parameters
#### Path Parameters
- **timestamp** (int|null) - Optional - The timestamp for which to retrieve the rate. If null, the current timestamp is used.

### Returns
- float|int: The rate for the given timestamp.
```

--------------------------------

### UriClient Constructor

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/UriClient.md

Initializes a new instance of the UriClient class.

```APIDOC
## __construct

### Description
Initializes a new instance of the UriClient class.

### Parameters
#### Path Parameters
- **baseUri** (string) - Required - The base URI to fetch robots.txt from.
- **curlOptions** (array) - Optional - An array of cURL options to apply.
- **byteLimit** (int|null) - Optional - A limit on the number of bytes to read.
```

--------------------------------

### Host Directive Wrapper

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Provides a wrapper for the Host directive, returning an instance of HostClient for managing host rules.

```php
@return HostClient
```

--------------------------------

### export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CleanParamClient.md

Exports an array of parameters and their corresponding paths.

```APIDOC
## export

### Description
Export an array of parameters and their corresponding paths.

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)

### Response
#### Success Response
- **array** - An array where keys are parameters and values are their corresponding paths.
```

--------------------------------

### Get Sleep Until Timestamp

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the timestamp, including microseconds, indicating when the next request can be sent. This is crucial for implementing polite crawling strategies.

```php
$delayBase->getTimeSleepUntil();
```

--------------------------------

### Get Base URI for Request-rate

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the base URI associated with the request-rate directive, intended for third-party delay handlers. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->requestRate()->getBaseUri();
```

--------------------------------

### TxtClient::host

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Wrapper for the Host directive. Returns an instance of HostClient.

```APIDOC
## TxtClient::host

### Description
Wrapper for the [Host](../Directives.md#host) directive. Returns an instance of [HostClient](HostClient.md).

### Returns
- **HostClient** - An instance of HostClient.
```

--------------------------------

### Get Base URI for Crawl-delay

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieves the base URI associated with the crawl-delay directive, intended for third-party delay handlers. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->crawlDelay()->getBaseUri();
```

--------------------------------

### debug(string $base)

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheBaseInterface.md

Retrieves the raw data from the cache database.

```APIDOC
## debug(string $base)

### Description
Get the RAW data from the database.

### Method
debug

### Parameters
#### Path Parameters
- **base** (string) - Required - The base URL to retrieve debug data for.

### Return Value
- array - An array containing the raw cache data.
```

--------------------------------

### Render Robots.txt Normally

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Render the robots.txt content in a normal, human-readable format, suitable for easy modification.

```php
$client->render()->normal();
```

--------------------------------

### Export Sitemap Directives

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Retrieve a list of all declared sitemap URLs from the robots.txt file.

```php
$client->sitemap()->export();
```

--------------------------------

### Get Cache-delay Sleep Time

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Determine the timestamp until which processing must sleep or be delayed, based on the Cache-delay directive and a Delay manager. This is part of handling the Cache-delay directive.

```php
$client->userAgent('myBot')->cacheDelay()->handle($delayManager)->getTimeSleepUntil();
```

--------------------------------

### export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/AllowClient.md

Exports all the current directive rules as an array.

```APIDOC
## export

### Description
Export an array of the current directives rules.

### Returns
- array: An array containing the directive rules.
```

--------------------------------

### Get Time to Sleep Until for Request-rate

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Determines the timestamp (with microseconds) until which processing must wait before sending the next request, using a delay manager. This is part of handling the request-rate directive.

```php
$client->userAgent('myBot')->requestRate()->handle($delayManager)->getTimeSleepUntil();
```

--------------------------------

### TxtClient::userAgent

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Wrapper for the User-agent directive. Returns an instance of UserAgentClient.

```APIDOC
## TxtClient::userAgent

### Description
Wrapper for the [User-agent](../Directives.md#user-agent) directive. Returns an instance of [UserAgentClient](UserAgentClient.md).

### Parameters
#### Path Parameters
- **string** (string) - Description of string

### Returns
- **UserAgentClient** - An instance of UserAgentClient.
```

--------------------------------

### Get Time to Sleep Until with Delay Manager

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Determines the timestamp (with microseconds) until which processing must wait before sending the next request, using a delay manager. This is part of handling the crawl-delay directive.

```php
$client->userAgent('myBot')->crawlDelay()->handle($delayManager)->getTimeSleepUntil();
```

--------------------------------

### List All User Agents

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Retrieves a list of all user agents defined in the robots.txt file.

```php
@return string[]
```

--------------------------------

### Get Request Rate Value

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RequestRateClient.md

Retrieves the request rate for a given timestamp. If no timestamp is provided, the current timestamp is used. For compatibility, if the specific rate is not found, the Crawl-delay value is returned.

```php
/**
 * @param int|null $timestamp
 * @return float|int
 */
```

--------------------------------

### Create RobotsTxtParser Instance with UriClient

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/README.md

Use UriClient to automatically download and parse a robots.txt file from a given URL. This is suitable for standard use cases where the robots.txt file is publicly accessible.

```php
<?php
$client = new \vipnytt\RobotsTxtParser\UriClient('http://example.com');

```

--------------------------------

### SQL Script for Delay Table Creation

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/sql/delay.md

This SQL script creates the necessary table (`robotstxt__delay0`) for storing delay information, compatible with MySQL 5.6+ and similar databases.

```sql
CREATE TABLE `robotstxt__delay0` (
  `base`       VARCHAR(269)
               COLLATE ascii_bin   NOT NULL,
  `userAgent`  VARCHAR(63)
               COLLATE ascii_bin   NOT NULL,
  `delayUntil` BIGINT(20) UNSIGNED NOT NULL,
  `lastDelay`  BIGINT(20) UNSIGNED NOT NULL,
  PRIMARY KEY (`base`, `userAgent`),
  KEY `delayUntil` (`delayUntil`)
)
  DEFAULT CHARSET = ascii
  COLLATE = ascii_bin
```

--------------------------------

### Sitemap Directive Wrapper

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/TxtClient.md

Provides a wrapper for the Sitemap directive, returning an instance of SitemapClient for managing sitemap declarations.

```php
@return SitemapClient
```

--------------------------------

### RenderClient::compressed

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RenderClient.md

Generates a robots.txt compressed to an absolute minimum. This format is best suited for parsing purposes and storage in databases.

```APIDOC
## compressed

### Description
Generates an robots.txt compressed to a absolute minimum.

Best suited for parsing purposes and storage in databases.

### Parameters
#### Path Parameters
- **eol** (string) - Required - The end-of-line character to use.
```

--------------------------------

### TxtClient

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

The base TxtClient class provides core functionality for parsing robots.txt content. It includes methods for handling directives like Clean-param, Host, Allow, and Cache-delay, as well as exporting rules and user-agents.

```APIDOC
## TxtClient

### Description
Provides core functionality for parsing robots.txt content. It includes methods for handling directives like Clean-param, Host, Allow, and Cache-delay, as well as exporting rules and user-agents.

Note: _Most parameters available is set to their default values and is not shown. Refer to the documentation for the specific class, for a full overview._

### Example Usage
```php
<?php
$client = new \vipnytt\RobotsTxtParser\TxtClient('http://example.com', 200, 'robots.txt');
```

### Methods
- `cleanParam()`: Accesses methods related to the `Clean-param` directive.
  - `detect(string $uri)`: Array of dynamic URI parameters detected in a URI.
  - `detectWithCommon(string $uri)`: Array of dynamic URI parameters detected in a URI. This function also includes a list of generic dynamic parameters, as well as any (optional) custom parameters.
  - `export()`: List of dynamic URI parameters.
- `export()`: Export all rules as an array.
- `getUserAgents()`: Get a list of all declared User-agents.
- `host()`: Accesses methods related to the `Host` directive.
  - `export()`: Get the main host declared by the Host directive.
  - `getWithUriFallback()`: Get the main host declared by the Host directive. Falls back to the host of the effective URI if it isn't set.
  - `isPreferred()`: Find out whether the host of the current URI also is the preferred one.
- `render()`: Accesses methods for rendering robots.txt content.
  - `compatibility()`: Compatibility mode. Optimized for parsing by custom 3rd party parsers, which do not follow the standards strictly.
  - `compressed()`: Compressed to an absolute minimum. Optimized for storage in ex. databases.
  - `normal()`: Normal looking robots.txt. Optimized for human readability, it's also the easiest to modify.
- `sitemap()`: Accesses methods related to the `Sitemap` directive.
  - `export()`: Export a list of sitemaps.
- `userAgent(string $userAgent)`: Accesses methods related to user-agent specific directives.
  - `allow()`: Accesses methods related to the `Allow` directive.
    - `export()`: Export an array of the directives rules.
    - `isCovered(string $path)`: Check if the specified path is covered by this directive.
  - `cacheDelay()`: Accesses methods related to the `Cache-delay` directive.
    - `export()`: Export the value of the directive.
    - `getBaseUri()`: Intended for usage by a 3rd party Delay handler.
    - `getUserAgent()`: Intended for usage by a 3rd party Delay handler.
    - `getValue()`: Get the request-delay value.
    - `handle(DelayBaseInterface $delayManager)`: Handles the Cache-delay directive.
      - `checkQueue()`: Get the size of the current request queue in seconds.
      - `getTimeSleepUntil()`: Get the timestamp w/microseconds you'll have to wait until before sending the request.
      - `reset()`: Reset the queue for this host.
      - `sleep()`: Sleep or delay the php processing until it's your turn to send the request.
```

--------------------------------

### base

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CacheManageInterface.md

Returns an instance of BaseInterface for cache operations.

```APIDOC
## base

### Description
Retrieves an instance of the cache base interface, which can be used for cache operations.

### Parameters
#### Path Parameters
- **baseUri** (string) - Required - The base URI to associate with the cache instance.

### Returns
- **Client\Cache\BaseInterface** - An instance of the cache base interface.
```

--------------------------------

### Export Visit-time Directive List

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Exports a list of configured visit-times in UTC for the selected user-agent. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->visitTime()->export();
```

--------------------------------

### export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/HostClient.md

Retrieves the Host value specified by the 'Host' directive in the robots.txt file. Returns null if no Host directive is present.

```APIDOC
## export

### Description
Get the Host listed by the directive.

### Method
N/A (SDK Method)

### Endpoint
N/A (SDK Method)

### Returns
- string|null: The Host value or null if not set.
```

--------------------------------

### Handle Delay Management

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayClient.md

Returns an instance of Client\Delay\ClientInterface, utilizing a provided handler for delay management.

```php
@param Client\Delay\ManageInterface $handler
@return Client\Delay\BaseInterface
```

--------------------------------

### Export All Rules

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Generate an array representation of all rules defined in the robots.txt file.

```php
$client->export();
```

--------------------------------

### isPreferred

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/HostClient.md

Checks if the currently requested host matches the preferred host specified in the 'Host' directive.

```APIDOC
## isPreferred

### Description
Check if the requested host also is the preferred one.

### Method
N/A (SDK Method)

### Endpoint
N/A (SDK Method)

### Returns
- bool: True if the requested host is the preferred host, false otherwise.
```

--------------------------------

### Semi-Automated Delay Handling with GetTimeSleepUntil

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/sql/delay.md

Retrieve the exact timestamp for the next allowed crawl time, allowing for custom sleep logic. This provides more control over the pausing mechanism.

```php
<?php
$db = new \vipnytt\RobotsTxtParser\Database($pdo);
if ($client->userAgent('MyBot')->isAllowed('http://example.com/path/to/file')) {
    // Crawl allowed
    $timestamp = $client->userAgent('MyBot')->crawlDelay()->handle($db->delay())->getTimeSleepUntil();
    time_sleep_until($timestamp);
    // Put your crawling code here!
}
```

--------------------------------

### Export All Rules for Selected User-Agent

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Exports an array of all configured rules (directives) for the currently selected user-agent. Requires the user-agent to be set.

```php
$client->userAgent('myBot')->export();
```

--------------------------------

### detect

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CleanParamClient.md

Detects and lists dynamic URI parameters for a given URI.

```APIDOC
## detect

### Description
Detects and lists dynamic URI parameters for a given URI.

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)

### Parameters
#### Path Parameters
- **uri** (string) - Required - The URI to analyze.

### Response
#### Success Response
- **string[]** - An array of detected dynamic URI parameters.
```

--------------------------------

### getContents

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/UriClient.md

Retrieves the raw content of the robots.txt file.

```APIDOC
## getContents

### Description
Contents of the `robots.txt` file.

### Returns
- **string** - The content of the robots.txt file.
```

--------------------------------

### export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CommentClient.md

Exports an array of comments for a matching user-agent.

```APIDOC
## export

### Description
Export an array of comments for the matching user-agent.

### Method
[Method not specified, likely internal or part of a larger SDK]

### Return Value
- array: An array containing the comments.
```

--------------------------------

### getTopWaitTimes

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/DelayManageInterface.md

Retrieves a list of hosts with the highest wait times, indicating frequent requests or high delay settings in their robots.txt.

```APIDOC
## getTopWaitTimes

### Description
Retrieves an array of hosts that have the highest wait times. These hosts are either frequently requested or specify an unusually high delay time in their `robots.txt` file.

### Method
[Method Signature]

### Parameters
#### Path Parameters
- **limit** (int) - Required - The maximum number of hosts to return.
- **min** (int) - Required - The minimum wait time threshold.

### Returns
- **array** - An array containing hosts with the highest wait times.
```

--------------------------------

### SitemapClient::export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/SitemapClient.md

Exports all sitemap URLs found. This method returns an array of strings, where each string is a sitemap URL.

```APIDOC
## export

### Description
Export all sitemaps.

### Method
public

### Returns
`string[]` - An array of sitemap URLs.
```

--------------------------------

### Export Clean-param Directives

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/CheatSheet.md

Export a list of all detected dynamic URI parameters.

```php
$client->cleanParam()->export();
```

--------------------------------

### Export Visit-Times as UTC Array

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/VisitTimeClient.md

Use this method to retrieve all configured visit-times as an array of UTC timestamps. This is useful for scheduling or logging purposes.

```php
export(): array
```

--------------------------------

### export

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/VisitTimeClient.md

Exports an array of visit-times in UTC. This method is useful for retrieving all configured visit-time directives.

```APIDOC
## export

### Description
Export an array of visit-times in UTC.

### Method
N/A (This is a method within a class, not an HTTP endpoint)

### Endpoint
N/A

### Parameters
None

### Response
#### Success Response
- **array** - An array containing visit-time data in UTC.

### Response Example
```json
{
  "example": "[ \"0400-0500\", \"1200-1300\" ]"
}
```
```

--------------------------------

### Handle Request Rate

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/RequestRateClient.md

Returns an instance of Client\Delay\ClientInterface, managing the request rate. For compatibility, if the specific rate is not found, the Crawl-delay value is returned.

```php
/**
 * @param Client\Delay\ManageInterface $handler
 * @return Client\Delay\BaseInterface
 */
```

--------------------------------

### detectWithCommon

Source: https://github.com/vipnytt/robotstxtparser/blob/master/docs/methods/CleanParamClient.md

Identifies dynamic URI parameters, including common and custom ones.

```APIDOC
## detectWithCommon

### Description
Array of uri parameters witch are considered dynamic. This function includes generic dynamic parameters, in addition to those in the `robots.txt` and in the input parameter `$customParam` (optional).

### Method
Not applicable (SDK method)

### Endpoint
Not applicable (SDK method)

### Parameters
#### Path Parameters
- **uri** (string) - Required - The URI to analyze.
- **customParam** (string[]) - Optional - An array of custom dynamic parameters to consider.

### Response
#### Success Response
- **string[]** - An array of detected dynamic URI parameters, including common and custom ones.
```
Tokens:
Raw