### WriteToFile Pipeline Run Example

Source: https://hexdocs.pm/crawly/Crawly.Pipelines.WriteToFile

This example shows how to manually run the WriteToFile pipeline with a sample item and specific options. It illustrates the expected output, which includes the processed item and the file descriptor.

```elixir
iex> item = %{my: "item"}
iex> WriteToFile.run(item, %{}, folder: "/tmp", extension: "csv")
{ %{my: "item"} , %{write_to_file_fd: #PID<0.123.0>} }
```

--------------------------------

### WriteToFile Pipeline Declaration Example

Source: https://hexdocs.pm/crawly/Crawly.Pipelines.WriteToFile

This example demonstrates how to declare the WriteToFile pipeline within the 'pipelines' configuration. It shows how to specify custom folder and extension options.

```elixir
pipelines: [
Crawly.Pipelines.JSONEncoder,
{Crawly.Pipelines.WriteToFile, folder: "/tmp", extension: "csv"}
]
```

--------------------------------

### Start DataWorker Process

Source: https://hexdocs.pm/crawly/Crawly.DataStorage.Worker

Starts a new instance of the DataStorage.Worker process, typically linking it to the calling process. This function is used to initiate the worker and make it available for storing items and other operations.

```elixir
start_link(list)
```

--------------------------------

### Start a Spider

Source: https://hexdocs.pm/crawly/http_api

Initiates the execution of a specified Crawly spider.

```APIDOC
## POST /spiders/{spider_name}/schedule

### Description
Starts a given Crawly spider.

### Method
POST

### Endpoint
/spiders/{spider_name}/schedule

### Parameters
#### Path Parameters
- **spider_name** (string) - Required - The name of the spider to schedule.

### Request Example
```bash
curl -v localhost:4001/spiders/<spider_name>/schedule
```

### Response
#### Success Response (200)
Details about the spider scheduling status.

#### Response Example
```json
{
  "message": "Spider scheduled successfully"
}
```
```

--------------------------------

### Start a Spider with Crawly.Engine

Source: https://hexdocs.pm/crawly/Crawly.Engine

Starts a spider using Crawly.Engine. Options passed to this function are forwarded to the spider's init/1 callback. Reserved options include :crawl_id, :closespider_itemcount, :closespider_timeout, and :concurrent_requests_per_domain.

```elixir
Crawly.Engine.start_spider(spider_name, opts \\ [])
```

--------------------------------

### Basic Crawly Configuration

Source: https://hexdocs.pm/crawly/configuration

A fundamental example of how to configure the Crawly framework, specifying pipelines and middlewares.

```elixir
config :crawly,
  pipelines: [
    # my pipelines
  ],
  middlewares: [
    # my middlewares
  ]
```

--------------------------------

### Example of Request Options and Auto Cookies Manager Middleware

Source: https://hexdocs.pm/crawly/basic_concepts

This example demonstrates how to configure request options, such as timeout and receive timeout, along with enabling automatic cookie management for requests. These configurations are passed as a keyword list to the `Crawly.Middlewares.RequestOptions` and `Crawly.Middlewares.AutoCookiesManager` respectively.

```elixir
{
  Crawly.Middlewares.RequestOptions,
  [timeout: 30_000, recv_timeout: 15000]
},
Crawly.Middlewares.AutoCookiesManager
```

--------------------------------

### Crawly Retry Configuration Example

Source: https://hexdocs.pm/crawly/configuration

An example demonstrating how to configure the retry logic for failed requests in Crawly. This includes specifying retry codes, maximum retries, and ignored middlewares.

```elixir
retry:
    [
      retry_codes: [400],
      max_retries: 3,
      ignored_middlewares: [Crawly.Middlewares.UniqueRequest]
  ]
```

--------------------------------

### Generate Crawly Configuration (Mix Command)

Source: https://hexdocs.pm/crawly/readme

This Mix command generates an example configuration file for Crawly. This is useful for quickly setting up or reviewing the available configuration options.

```bash
mix crawly.gen.config
```

--------------------------------

### Start DataWorker Supervisor Specification

Source: https://hexdocs.pm/crawly/Crawly.DataStorage.Worker

Generates a child specification for starting the DataStorage.Worker module under a supervisor. This function is essential for integrating the worker into an Elixir application's supervision tree, ensuring fault tolerance and proper lifecycle management.

```elixir
child_spec(init_arg)
```

--------------------------------

### Start Crawler Manager Link (Elixir)

Source: https://hexdocs.pm/crawly/Crawly.Manager

Initiates the Crawly.Manager process, typically as part of a larger supervision tree. It takes a list of arguments for initialization.

```elixir
start_link(list) :: :ignore | {:error, term()} | {:ok, pid()}
```

--------------------------------

### Initialize Crawly.Engine Supervisor Spec

Source: https://hexdocs.pm/crawly/Crawly.Engine

Provides a child specification for starting the Crawly.Engine module under a supervisor. This is a standard Elixir pattern for supervised processes.

```elixir
Crawly.Engine.child_spec(init_arg)
```

--------------------------------

### Crawly.RequestsStorage: start_worker() function specification

Source: https://hexdocs.pm/crawly/Crawly.RequestsStorage

This function specification defines the return types for the `start_worker/2` function in Crawly.RequestsStorage. It attempts to start a worker for a given spider and crawl ID, returning {:ok, pid()} on success or an error tuple if the worker is already started.

```elixir
start_worker(Crawly.spider(), crawl_id :: String.t()) ::
  {:ok, pid()} | {:error, :already_started}
```

--------------------------------

### YML Spider Definition Example

Source: https://hexdocs.pm/crawly/spiders_in_yml

This is an example of a YML file used to define a spider in Crawly. It specifies the spider's name, base URL, starting URLs, fields to extract, and rules for following links. This structure allows for the creation of simple spiders without writing Python code.

```yaml
name: BooksSpiderForTest
base_url: "https://books.toscrape.com/"
start_urls:
 - "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
fields:
 - name: title
 selector: ".product_main"
 - name: price
 selector: ".product_main .price_color"
links_to_follow:
 - selector: "a"
 attribute: "href"
```

--------------------------------

### Elixir: Start Link for Request Storage

Source: https://hexdocs.pm/crawly/Crawly.RequestsStorage.Worker

Initiates the linking of the request storage worker process. This function is used to start the worker, likely within a supervision tree, and requires the spider name and crawl ID as arguments.

```elixir
start_link(spider_name, crawl_id)
```

--------------------------------

### Example Crawly Configuration with Pipelines

Source: https://hexdocs.pm/crawly/Crawly.Pipeline

Demonstrates how to declare pipelines within the Crawly configuration file (config.exs). It shows the inclusion of a JSONEncoder pipeline.

```elixir
# config.exs
:crawly,
  parsers: [
    # with options
    {Crawly.ExtractRequests, selector: "a" }
  ],
  middlewares: [
    Crawly.Middlewares.DomainFilter,
    Crawly.Middlewares.UniqueRequest,
    Crawly.Middlewares.RobotsTxt
  ],
  pipelines: [Crawly.Pipelines.JSONEncoder ]
```

--------------------------------

### Spider Initialization with Start URLs

Source: https://hexdocs.pm/crawly/Crawly.Spider

The `init/0` function is mandatory for a Crawly Spider. It must return a keyword list containing either `start_urls` or `start_requests`, which are the initial points for the spider's crawl. This function defines the entry points for the crawling process.

```elixir
def init() do
  [
    start_urls: ["http://example.com"]
  ]
end
```

--------------------------------

### Start a Crawly Spider

Source: https://hexdocs.pm/crawly/index

This command initiates a Crawly crawl using the `BooksToScrape` spider defined previously. It runs within an Elixir Mix project environment. The output of the crawl, in JSON Lines format, will be saved to `/tmp/BooksToScrape_<timestamp>.jl`.

```bash
iex -S mix run -e "Crawly.Engine.start_spider(BooksToScrape)"
```

--------------------------------

### Render URL with Headers using Crawly Render Server (cURL)

Source: https://hexdocs.pm/crawly/Crawly.Fetchers.CrawlyRenderServer

This example demonstrates how to use cURL to send a POST request to the Crawly Render Server's /render endpoint. It includes a JSON payload with the URL to render and custom headers, such as a User-Agent.

```bash
curl -X POST http://localhost:3000/render -H 'Content-Type: application/json' -d '{
 "url": "https://example.com",
 "headers": {"User-Agent": "Custom User Agent"}
}'
```

--------------------------------

### Custom Item Pipeline: Ecto Storage

Source: https://hexdocs.pm/crawly/basic_concepts

An example of a custom item pipeline that inserts scraped items into a database table using Ecto. It delegates the insertion to an application context function `MyApp.insert_with_ecto/1`.

```Elixir
defmodule MyApp.MyEctoPipeline do
  @impl Crawly.Pipeline
  def run(item, state, _opts \ []) do
    case MyApp.insert_with_ecto(item) do
      {:ok, _} ->
        # insert successful, carry on with pipeline
        {item, state}
      {:error, _} ->
        # insert not successful, drop from pipeline
        {false, state}
    end
  end
end
```

--------------------------------

### Get Job

Source: https://hexdocs.pm/crawly/Crawly.Models.Job

Retrieves a specific job's information from storage using its unique crawl ID.

```APIDOC
## GET /jobs/{crawl_id}

### Description
Retrieves the job with the given `crawl_id` from the SimpleStorage instance.

### Method
GET

### Endpoint
`/jobs/{crawl_id}`

### Parameters
#### Path Parameters
- **crawl_id** (binary) - Required - The unique ID of the job to retrieve.

### Response
#### Success Response (200)
- **job** (Crawly.Models.Job) - The retrieved job details.

#### Error Response (404)
- **error** (string) - A message indicating the job was not found.

### Response Example (Success)
```json
{
  "id": "my-crawl-id",
  "spider_name": "my-spider",
  "start": "2023-03-31T16:00:00Z",
  "end": null,
  "scraped_items": 0,
  "stop_reason": null
}
```

### Response Example (Error)
```json
{
  "error": "Job not found"
}
```
```

--------------------------------

### Generate Supervisor Child Specification (Elixir)

Source: https://hexdocs.pm/crawly/Crawly.Manager

Returns a process specification suitable for starting the Crawly.Manager module under a supervisor. This is a standard Elixir pattern for defining how modules should be supervised.

```elixir
child_spec(init_arg) :: supervisor:child_spec()
```

--------------------------------

### List All Crawly Jobs

Source: https://hexdocs.pm/crawly/Crawly.Models.Job

Lists all registered jobs currently stored in the SimpleStorage instance. This function provides a way to get an overview of all active or completed crawling jobs.

```elixir
iex> Crawly.Models.Job.list()
[job1, job2, ...]
```

--------------------------------

### Crawly File Logging Configuration

Source: https://hexdocs.pm/crawly/configuration

Example configuration for enabling file logging in Crawly. This involves setting up the logger backend and specifying the log directory and file logging status.

```elixir
config :logger,
  backends: [{LoggerFileBackend, :info_log}]

config :crawly,
  log_dir: "/tmp/spider_logs",
  log_to_file: true,
  ......
  other configurations
```

--------------------------------

### Get Spider Settings

Source: https://hexdocs.pm/crawly/Crawly.Utils

Retrieves a specific setting for a spider, merging global and spider-specific overrides. Spider settings take precedence. Supports a default value if the setting is not found.

```elixir
get_settings(setting_name, Crawly.spider(), default) :: result
when setting_name: atom(), default: term(), result: term()
```

--------------------------------

### Run JSONEncoder Pipeline

Source: https://hexdocs.pm/crawly/Crawly.Pipelines.JSONEncoder

This example demonstrates the usage of the JSONEncoder pipeline's `run` function. It takes a map as input and returns a tuple containing the JSON encoded string and an empty map.

```elixir
iex> JSONEncoder.run(%{my: "field"}, %{})
  {"{\"my\":\"field\"}", %{}}
```

--------------------------------

### Custom Request Middleware: Add Proxy

Source: https://hexdocs.pm/crawly/basic_concepts

An example of a custom request middleware that adds proxy and proxy authentication options to a request. It follows the `HTTPoison` documentation for proxy options.

```Elixir
defmodule MyApp.MyProxyMiddleware do
  @impl Crawly.Pipeline
  def run(request, state, opts \ []) do
    # Set default proxy and proxy_auth to nil
    opts = Enum.into(opts, %{proxy: nil, proxy_auth: nil})

    case opts.proxy do
      nil ->
        # do nothing
        {request, state}
      value ->
        old_options = request.options
        new_options = [proxy: opts.proxy, proxy_auth: opts.proxy_auth]
        new_request =  Map.put(request, :options, old_optoins ++ new_options)
        {new_request, state}
    end
  end
end
```

--------------------------------

### Pass Configuration Options to Pipeline

Source: https://hexdocs.pm/crawly/basic_concepts

Demonstrates how to pass configuration options to a pipeline using a tuple-based declaration. Options are received as a keyword list in the `opts` argument of the `run` function.

```Elixir
pipelines: [
  {MyCustomPipeline, my_option: "value"}
]
```

```Elixir
defmodule MyCustomPipeline do
  @impl Crawly.Pipeline
  def run(item, state, opts) do
    IO.inspect(opts)        # shows keyword list of  [ my_option: "value" ]
    # Do something
  end
end
```

--------------------------------

### Spider Initialization with Options

Source: https://hexdocs.pm/crawly/Crawly.Spider

The `init/1` function allows for initialization with a list of options passed from the Crawly Engine. Similar to `init/0`, it must return a keyword list with `start_urls` or `start_requests`. This provides a way to configure the spider dynamically.

```elixir
def init(options) do
  # Process options if needed
  [
    start_requests: [Crawly.Request.new("http://example.com")]
  ]
end
```

--------------------------------

### Initialize Crawly.Engine GenServer

Source: https://hexdocs.pm/crawly/Crawly.Engine

Callback implementation for `GenServer.init/1`. This function initializes the Crawly.Engine process state, returning {:ok, t()} where t() is the engine's state.

```elixir
init(args)
```

--------------------------------

### Initialize DataWorker GenServer

Source: https://hexdocs.pm/crawly/Crawly.DataStorage.Worker

Callback implementation for `GenServer.init/1`. This function is called when the GenServer process is initialized, setting up the initial state of the DataStorage.Worker. It takes an argument list to configure the worker upon startup.

```elixir
init(list)
```

--------------------------------

### Header and Setting Utilities

Source: https://hexdocs.pm/crawly/Crawly.Utils

Functions for retrieving header values and spider settings.

```APIDOC
## get_header(headers, key, default \\ nil)

### Description
Retrieves a header value from a list of key-value tuples or a map. This function searches for a header with the specified key in the given list of headers or map. If found, it returns the corresponding value; otherwise, it returns the provided default value if provided, otherwise `nil`.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `binary() | nil` - The value of the header if found, otherwise the default value if provided, otherwise `nil`.

#### Response Example
```
"application/json"
```

## get_settings(setting_name, spider_name \\ nil, default \\ nil)

### Description
A helper which allows to extract a given setting.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Get Spider Stats

Source: https://hexdocs.pm/crawly/http_api

Retrieves statistics for a specific Crawly spider, including scheduled requests and scraped items.

```APIDOC
## GET /spiders/{spider_name}/scheduled-requests

### Description
Gets the number of scheduled requests for a given spider.

### Method
GET

### Endpoint
/spiders/{spider_name}/scheduled-requests

### Parameters
#### Path Parameters
- **spider_name** (string) - Required - The name of the spider to get stats for.

### Request Example
```bash
curl -v localhost:4001/spiders/<spider_name>/scheduled-requests
```

### Response
#### Success Response (200)
Number of scheduled requests.

#### Response Example
```json
{
  "scheduled_requests": 150
}
```

## GET /spiders/{spider_name}/scraped-items

### Description
Gets the number of scraped items for a given spider.

### Method
GET

### Endpoint
/spiders/{spider_name}/scraped-items

### Parameters
#### Path Parameters
- **spider_name** (string) - Required - The name of the spider to get stats for.

### Request Example
```bash
curl -v localhost:4001/spiders/<spider_name>/scraped-items
```

### Response
#### Success Response (200)
Number of scraped items.

#### Response Example
```json
{
  "scraped_items": 500
}
```
```

--------------------------------

### Logging and Testing Utilities

Source: https://hexdocs.pm/crawly/Crawly.Utils

Utilities for generating log file paths and wrapping process communication for testing.

```APIDOC
## spider_log_path(spider_name, crawl_id)

### Description
Composes the log file path for a given spider and crawl ID.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `binary()` - The composed log file path.

#### Response Example
```
"/var/log/crawly/my_spider/crawl_123.log"
```

## send_after(pid, message, timeout)

### Description
A wrapper over `Process.send_after`. This wrapper should be used instead of `Process.send_after`, so it's possible to mock the last one. To avoid race conditions on worker's testing.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Get Registered Spiders

Source: https://hexdocs.pm/crawly/Crawly.Utils

Returns a list of all spiders that have been registered with the Crawly system. This provides a way to access spiders that are currently active or available.

```elixir
registered_spiders() :: [module()]
```

--------------------------------

### Crawly HTTP API Endpoints

Source: https://hexdocs.pm/crawly/Crawly.API.Router

This section details the available endpoints for interacting with the Crawly HTTP API, including scheduling, stopping, and getting statistics for spiders.

```APIDOC
## Crawly HTTP API

### Description
Provides endpoints to schedule, stop, and retrieve statistics for all running spiders.

### Method
GET, POST, DELETE (Specific methods depend on the underlying Plug functions)

### Endpoint
`/websites/hexdocs_pm_crawly/*` (Base path for Crawly API)

### Parameters

#### Path Parameters
None explicitly defined in the provided text.

#### Query Parameters
None explicitly defined in the provided text.

#### Request Body
None explicitly defined in the provided text.

### Request Example
(No specific examples provided for scheduling/stopping/stats)

### Response
#### Success Response (200)
(Response format depends on the specific operation, e.g., spider status, statistics)

#### Response Example
(No specific examples provided)

## Plug Callbacks

### Description
These are callback implementations for the Plug interface used by Crawly.

### Method
`call/2`, `init/1`

### Endpoint
Internal Plug endpoints.

### Parameters

#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```elixir
# Example for Plug.call/2
Crawly.API.Router.call(conn, opts)

# Example for Plug.init/1
Crawly.API.Router.init(opts)
```

### Response
#### Success Response (200)
(Varies based on Plug context)

#### Response Example
(No specific examples provided)

## Put Base Path

### Description
Callback implementation for setting the base path.

### Method
`put_base_path/2`

### Endpoint
Internal Plug endpoint.

### Parameters

#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```elixir
Crawly.API.Router.put_base_path(conn, opts)
```

### Response
#### Success Response (200)
(Varies based on Plug context)

#### Response Example
(No specific examples provided)

```

--------------------------------

### Module and Application Utilities

Source: https://hexdocs.pm/crawly/Crawly.Utils

Utilities for working with modules and applications.

```APIDOC
## ensure_loaded?(module)

### Description
Wrapper function for `Code.ensure_loaded?/1` to allow mocking.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `boolean()` - Returns `true` if the module is loaded, `false` otherwise.

#### Response Example
```
true
```

## get_modules_from_applications()

### Description
Retrieves all modules loaded from applications.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `list(module())` - A list of all loaded module names.

#### Response Example
```
[:elixir, :kernel, :my_app]
```

## unwrap_module_and_options(setting)

### Description
Function to get setting module in proper data structure.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Initialize Crawly.Loggers.SendToUiBackend

Source: https://hexdocs.pm/crawly/Crawly.Loggers.SendToUiBackend

Provides the init function for initializing the SendToUiBackend logger. This function is crucial for setting up the logger's initial state. It takes a single argument for configuration.

```Elixir
init(arg)
```

--------------------------------

### Crawly.Middlewares.SameDomainFilter

Source: https://hexdocs.pm/crawly/Crawly.Middlewares.SameDomainFilter

This middleware filters out requests that navigate outside the domain of the spider's initial start URL. It operates automatically without requiring any configuration options.

```APIDOC
## Crawly.Middlewares.SameDomainFilter

### Description
Filters out requests which are going outside of the crawled domain. The domain that is used to compare against the request url is obtained from the previous response, so it ends up being the spider's start_url. Spider's base_url is not evaluated.

### Method
N/A (This is a middleware configuration)

### Endpoint
N/A (This is a middleware configuration)

### Parameters
This middleware does not accept any options. Tuple-based configuration options will be ignored.

### Request Example
```elixir
middlewares: [
  Crawly.Middlewares.SameDomainFilter
]
```

### Response
This middleware does not directly return a response. It influences the request pipeline by filtering URLs.

#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Preview Utility

Source: https://hexdocs.pm/crawly/Crawly.Utils

A utility function to preview spider results based on YML configuration.

```APIDOC
## preview(yml)

### Description
A helper function that allows to preview spider results based on a given YML.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Configure Proxy Settings Middleware

Source: https://hexdocs.pm/crawly/Crawly.Middlewares.RequestOptions

Shows how to configure proxy settings within the RequestOptions middleware. This includes specifying the proxy host, port, and authentication details.

```elixir
middlewares: [
 {Crawly.Middlewares.RequestOptions, [proxy: {"https://my_host.com", 3000}, proxy_auth: {"my_user", "my_password"}]}
]
```

--------------------------------

### Configure Request Options Middleware

Source: https://hexdocs.pm/crawly/Crawly.Middlewares.RequestOptions

Demonstrates how to declare and configure the RequestOptions middleware in a Crawly project. This includes setting request and receive timeouts.

```elixir
middlewares: [
  {Crawly.Middlewares.RequestOptions, [timeout: 30_000, recv_timeout: 15000]}
]
```

--------------------------------

### Crawly.Models.Job Struct

Source: https://hexdocs.pm/crawly/Crawly.Models.Job

Defines the structure of a web scraping job, including its ID, spider name, start and end times, scraped items count, and stop reason.

```APIDOC
## Crawly.Models.Job Struct

### Description
The `Crawly.Models.Job` struct holds information about a web scraping job.

### Fields
- **id** (binary) - Unique identifier for the job.
- **spider_name** (binary) - The name of the spider used for the job.
- **start** (DateTime.t) - The time the job started.
- **end** (DateTime.t | nil) - The time the job ended. `nil` if not completed.
- **scraped_items** (integer) - The number of items scraped during the job.
- **stop_reason** (binary | nil) - The reason the job was stopped. `nil` if not stopped.

### Type Definition
```elixir
t() :: %Crawly.Models.Job{
  end: DateTime.t() | nil,
  id: binary(),
  scraped_items: integer(),
  spider_name: binary(),
  start: DateTime.t(),
  stop_reason: binary() | nil
}
```
```

--------------------------------

### Custom Item Pipeline: Key-Based Pattern Matching

Source: https://hexdocs.pm/crawly/basic_concepts

Shows how to use key-based pattern matching in an item pipeline to process items based on the presence of specific keys in a map. This is useful for processing related or multiple items.

```Elixir
defmodule MyApp.MyCustomPipeline do
  @impl Crawly.Pipeline
  def run(%{my_item: my_item} = item, state, _opts \ []) do
    # do something
  end
  # do nothing if it does not match
  def run(item, state, _opts), do: {item, state}
end
```

--------------------------------

### UserAgent Middleware Configuration

Source: https://hexdocs.pm/crawly/Crawly.Middlewares.UserAgent

This section details how to configure the UserAgent middleware, including setting custom user agents and understanding default behavior.

```APIDOC
## UserAgent Middleware

### Description

This middleware allows you to set and rotate user agents for your web crawling requests. User agents are read from `:crawly` and `:user_agents` sessions. The default user agent is 'Crawly Bot 1.0'. Rotation is handled randomly using `Enum.random/1`.

### Method

N/A (Configuration)

### Endpoint

N/A (Middleware Configuration)

### Parameters

#### Options

- **`:user_agents`** (list of strings) - Optional - A list of user agent strings to rotate. Defaults to `["Crawly Bot 1.0"]`.

### Request Example

```elixir
middlewares: [
  {UserAgent, user_agents: ["My Custom Bot/1.0", "Another Bot/2.0"] }
]
```

### Response

N/A (Middleware Configuration)

#### Success Response (N/A)

N/A

#### Response Example

N/A
```

--------------------------------

### Get Spider Manager PID with Crawly.Engine

Source: https://hexdocs.pm/crawly/Crawly.Engine

Retrieves the PID of the manager process for a given spider. Returns an error if the spider is not found. This can be useful for direct process interaction if needed.

```elixir
Crawly.Engine.get_manager(spider_name)
```

--------------------------------

### Configure Splash Fetcher in Crawly

Source: https://hexdocs.pm/crawly/Crawly.Fetchers.Splash

This configuration snippet shows how to set up the Splash fetcher within Crawly. It requires specifying the Splash base URL, which is typically 'http://localhost:8050/render.html' when running Splash locally via Docker.

```elixir
fetcher: {Crawly.Fetchers.Splash, [base_url: "http://localhost:8050/render.html"]}
```

--------------------------------

### URL Handling Utilities

Source: https://hexdocs.pm/crawly/Crawly.Utils

Functions for building absolute URLs from relative ones, both for single URLs and lists.

```APIDOC
## build_absolute_url(url, base_url)

### Description
A helper function which joins relative url with a base URL.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A

## build_absolute_urls(urls, base_url)

### Description
A helper function which joins relative url with a base URL for a list.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Elixir: Get Request Storage Statistics

Source: https://hexdocs.pm/crawly/Crawly.RequestsStorage.Worker

Fetches statistics about the requests stored. It takes a process ID (pid) and returns a tuple with the atom :stored_requests and a non-negative integer representing the count of stored requests.

```elixir
stats(pid()) :: {:stored_requests, non_neg_integer()}
```

--------------------------------

### Get Spider Crawl ID with Crawly.Engine

Source: https://hexdocs.pm/crawly/Crawly.Engine

Retrieves the crawl ID for a running spider. Returns an error if the spider is not running. The crawl ID is a binary identifier for a specific crawling session.

```elixir
Crawly.Engine.get_crawl_id(spider_name)
```

--------------------------------

### List All Jobs

Source: https://hexdocs.pm/crawly/Crawly.Models.Job

Retrieves a list of all registered web scraping jobs.

```APIDOC
## GET /jobs

### Description
Lists all registered jobs from the SimpleStorage instance.

### Method
GET

### Endpoint
`/jobs`

### Response
#### Success Response (200)
- **jobs** (list) - A list of `Crawly.Models.Job` structs representing all registered jobs.

### Response Example
```json
[
  {
    "id": "my-crawl-id-1",
    "spider_name": "spider-a",
    "start": "2023-03-31T16:00:00Z",
    "end": "2023-03-31T16:05:00Z",
    "scraped_items": 50,
    "stop_reason": "finished"
  },
  {
    "id": "my-crawl-id-2",
    "spider_name": "spider-b",
    "start": "2023-04-01T10:00:00Z",
    "end": null,
    "scraped_items": 0,
    "stop_reason": null
  }
]
```
```

--------------------------------

### Get Spider Information with Crawly.Engine

Source: https://hexdocs.pm/crawly/Crawly.Engine

Retrieves information about a specific spider, including its name, status, and PID if running. Returns nil if the spider is not found. This function is crucial for monitoring spider states.

```elixir
Crawly.Engine.get_spider_info(spider_name)
```

--------------------------------

### Add erlang-node-discovery Dependency

Source: https://hexdocs.pm/crawly/experimental_ui

Adds the erlang-node-discovery library as a project dependency. This is typically done in the `deps` section of the `mix.exs` file to facilitate Erlang cluster node discovery.

```elixir
{:erlang_node_discovery, git: "https://github.com/oltarasenko/erlang-node-discovery"}
```

--------------------------------

### Add Crawly Dependencies to mix.exs

Source: https://hexdocs.pm/crawly/index

This snippet shows how to add Crawly and Floki as dependencies in your Elixir project's `mix.exs` file. Ensure you have Elixir `~> 1.14` installed. This is a prerequisite for using Crawly.

```elixir
# mix.exs
defp deps do
    [
      {:crawly, "~> 0.17.2"},
      {:floki, "~> 0.33.0"}
    ]
end
```

--------------------------------

### Get Value from Storage - Elixir

Source: https://hexdocs.pm/crawly/Crawly.SimpleStorage

Retrieves the value associated with a given key from the specified table. It returns the value wrapped in an {:ok, value()} tuple on success, or {:error, :not_found} if the key does not exist, or another error tuple.

```elixir
Crawly.SimpleStorage.get(:spiders, Test)
```

```elixir
Crawly.SimpleStorage.get(:spiders, T)
```

--------------------------------

### Spider Management Utilities

Source: https://hexdocs.pm/crawly/Crawly.Utils

Utilities for managing dynamic spiders, including clearing, registering, and listing them.

```APIDOC
## clear_registered_spiders()

### Description
Remove all previously registered dynamic spiders.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `:ok` (:atom()) - Indicates successful clearing of spiders.

#### Response Example
```
:ok
```

## register_spider(name)

### Description
Register a given spider (so it's visible in the spiders list).

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A

## registered_spiders()

### Description
Return a list of registered spiders.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `list()` - A list of registered spider names.

#### Response Example
```
[:my_spider_1, :my_spider_2]
```

## list_spiders()

### Description
Returns a list of known modules which implements Crawly.Spider behaviour.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `list()` - A list of spider module names.

#### Response Example
```
[:my_spider_1, :my_spider_2]
```

## load_spiders()

### Description
Loads spiders from a given directory. Stores them in persistent term under :spiders.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Get Spider Configuration - Elixir

Source: https://hexdocs.pm/crawly/Crawly.Models.YMLSpider

Retrieves the YML content for a given spider from storage. It accepts the spider's name and returns a tuple containing :ok and the YML string on success, or an :error tuple with a reason for failure.

```Elixir
Crawly.Models.YMLSpider.get(spider_name)
```

--------------------------------

### POST /render

Source: https://hexdocs.pm/crawly/Crawly.Fetchers.CrawlyRenderServer

Renders JavaScript on incoming requests using the Crawly Render Server. This endpoint is part of the Crawly.Fetchers.CrawlyRenderServer fetcher.

```APIDOC
## POST /render

### Description
Renders JavaScript on incoming requests using the Crawly Render Server. This endpoint is part of the Crawly.Fetchers.CrawlyRenderServer fetcher.

### Method
POST

### Endpoint
/render

### Parameters
#### Request Body
- **url** (string) - Required - The URL to render.
- **headers** (object) - Optional - Custom headers to be sent with the request.
  - **User-Agent** (string) - Optional - Custom User-Agent header.

### Request Example
```json
{
  "url": "https://example.com",
  "headers": {"User-Agent": "Custom User Agent"}
}
```

### Response
#### Success Response (200)
- **html** (string) - The rendered HTML content.
- **status** (integer) - The HTTP status code of the rendered page.
- **headers** (object) - The headers returned by the rendered page.

#### Response Example
```json
{
  "html": "<html><body><h1>Hello World</h1></body></html>",
  "status": 200,
  "headers": {"Content-Type": "text/html"}
}
```
```

--------------------------------

### Initialize Storage - Elixir

Source: https://hexdocs.pm/crawly/Crawly.SimpleStorage

Initializes the storage mechanism used to store spider information. This function is crucial for setting up the storage before other operations can be performed. It returns an {:ok, any()} tuple on successful initialization or an error tuple.

```elixir
Crawly.SimpleStorage.init()
```

--------------------------------

### Run Crawly.Pipelines.Validate for Item Validation

Source: https://hexdocs.pm/crawly/Crawly.Pipelines.Validate

This example shows how to use the Validate pipeline to check if a scraped item contains the required fields. It returns a boolean indicating success and the item itself. If the item lacks required fields, it's dropped.

```elixir
# Drops the scraped item that does not have the required fields
iex> Validate.run(%{my: "field"}, %{}, fields: [:id])
{false, %{}}
```

--------------------------------

### Get DataWorker Stats

Source: https://hexdocs.pm/crawly/Crawly.DataStorage.Worker

Retrieves statistics from the data worker process. It takes the process ID (pid) as input and returns a tuple containing the number of stored items. This function is useful for monitoring the worker's activity and storage usage.

```elixir
stats(pid()) :: {:stored_items, non_neg_integer()}
```

--------------------------------

### Request and Pipeline Utilities

Source: https://hexdocs.pm/crawly/Crawly.Utils

Utilities for creating request structures and pipeline operations.

```APIDOC
## request_from_url(url)

### Description
A helper function which returns a Request structure for the given URL.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `Crawly.Request.t()` - A Crawly Request struct.

#### Response Example
```json
{
  "url": "http://example.com",
  "method": "GET"
}
```

## requests_from_urls(urls)

### Description
A helper function which converts a list of URLs into a requests list.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
- `list(Crawly.Request.t())` - A list of Crawly Request structs.

#### Response Example
```json
[
  {
    "url": "http://example.com/page1",
    "method": "GET"
  },
  {
    "url": "http://example.com/page2",
    "method": "GET"
  }
]
```

## pipe(arg1, item, state)

### Description
Pipeline/Middleware helper.

### Method
N/A (Function)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
#### Success Response (200)
N/A

#### Response Example
N/A
```

--------------------------------

### Unwrap Setting Module and Options

Source: https://hexdocs.pm/crawly/Crawly.Utils

Parses a setting term to extract the associated module and any associated options. This function is used internally to manage and apply spider configurations.

```elixir
unwrap_module_and_options(setting) :: {atom(), maybe_improper_list()}
```

--------------------------------

### Override Global Settings in a Spider

Source: https://hexdocs.pm/crawly/configuration

This example demonstrates how to override global Crawly settings on a per-spider basis. By defining the `override_settings/0` callback within a spider module, specific parameters like `concurrent_requests_per_domain` and `closespider_timeout` can be customized for that spider's execution.

```elixir
def override_settings() do
   [
    concurrent_requests_per_domain: 5,
    closespider_timeout: 6
   ]
end
```

--------------------------------

### Custom Item Pipeline: Struct-Based Pattern Matching

Source: https://hexdocs.pm/crawly/basic_concepts

Demonstrates using struct-based pattern matching in an item pipeline to selectively process items that match a specific struct, like `%MyItem{}`. Non-matching items are passed through unchanged.

```Elixir
defmodule MyApp.MyCustomPipeline do
  @impl Crawly.Pipeline
  def run(%MyItem{} = item, state, _opts \ []) do
    # do something
  end
  # do nothing if it does not match
  def run(item, state, _opts), do: {item, state}
end
```

--------------------------------

### Load Spider Configurations - Elixir

Source: https://hexdocs.pm/crawly/Crawly.Models.YMLSpider

This function iterates through a list of spider configurations and loads them. It returns :ok upon successful iteration and loading.

```Elixir
Crawly.Models.YMLSpider.load()
```

--------------------------------

### List All Spider Configurations - Elixir

Source: https://hexdocs.pm/crawly/Crawly.Models.YMLSpider

Lists all available spider configurations stored in SimpleStorage. This function takes no arguments and returns a list of terms, where each term represents a spider configuration.

```Elixir
Crawly.Models.YMLSpider.list()
```

--------------------------------

### Create New Job

Source: https://hexdocs.pm/crawly/Crawly.Models.Job

Creates and stores a new web scraping job with a unique crawl ID and spider name.

```APIDOC
## POST /jobs

### Description
Creates a new job with the given `crawl_id` and `spider_name`, and stores it in the SimpleStorage instance.

### Method
POST

### Endpoint
`/jobs`

### Parameters
#### Request Body
- **crawl_id** (binary) - Required - The unique ID for the new job.
- **spider_name** (binary) - Required - The name of the spider to be used for this job.

### Request Example
```json
{
  "crawl_id": "my-new-crawl-id",
  "spider_name": "new-spider"
}
```

### Response
#### Success Response (201)
- **message** (string) - Indicates successful job creation.

#### Error Response (400)
- **error** (string) - A message indicating why the job could not be created (e.g., missing parameters, ID already exists).

### Response Example (Success)
```json
{
  "message": "Job created successfully"
}
```

### Response Example (Error)
```json
{
  "error": "Invalid parameters provided"
}
```
```

--------------------------------

### Configure CrawlyRenderServer Fetcher

Source: https://hexdocs.pm/crawly/basic_concepts

Configuration snippet for Crawly to use the `CrawlyRenderServer` fetcher for browser rendering. This involves setting the `:fetcher` configuration in `config.exs` to point to the running CrawlyRenderServer instance.

```elixir
import Config

config :crawly,
  fetcher: {Crawly.Fetchers.CrawlyRenderServer, [base_url: "http://localhost:3000/render"]},

```

--------------------------------

### Preview Spider Results from YML

Source: https://hexdocs.pm/crawly/Crawly.Utils

Generates a preview of spider results based on a provided YML configuration. The output can include URLs, extracted items, and generated requests, or detailed error information.

```elixir
preview(yml) :: [result]
when yml: binary(),
     result:
       %{url: binary(), items: [map()], requests: [binary()]}
       | %{error: term()}
       | %{error: term(), url: binary()}
```

--------------------------------

### Handle Continue Callback for Crawly.Engine

Source: https://hexdocs.pm/crawly/Crawly.Engine

Callback implementation for `GenServer.handle_continue/2`. This function is part of the GenServer behavior and handles continuation messages.

```elixir
handle_continue(atom, state)
```

--------------------------------

### Create New Crawly Job

Source: https://hexdocs.pm/crawly/Crawly.Models.Job

Creates a new job entry with a unique crawl ID and spider name. This function initializes a job record and stores it in the SimpleStorage instance. It returns :ok on success or an error tuple if creation fails.

```elixir
iex> Crawly.Models.Job.new("my-crawl-id", "my-spider")
:ok
```

--------------------------------

### Declare SameDomainFilter Middleware

Source: https://hexdocs.pm/crawly/Crawly.Middlewares.SameDomainFilter

This snippet demonstrates how to declare the SameDomainFilter middleware in the Crawly configuration. It is added to the 'middlewares' list in the project's configuration file.

```elixir
middlewares: [
  Crawly.Middlewares.SameDomainFilter
]
```

--------------------------------

### Load Spiders from Directory

Source: https://hexdocs.pm/crawly/Crawly.Utils

Loads spiders from a specified directory and stores them persistently. This allows for the use of spiders not part of the main Crawly application. Returns an error if the spider directory is not found.

```elixir
load_spiders() :: {:ok, [module()]} | {:error, :no_spiders_dir}
```

--------------------------------

### Configure erlang-node-discovery Hosts and Ports

Source: https://hexdocs.pm/crawly/experimental_ui

Configures the erlang-node-discovery application to specify the hosts and ports for node discovery. This allows Crawly nodes to locate the CrawlyUI node within the Erlang cluster.

```elixir
config :erlang_node_discovery, hosts: ["127.0.0.1", "crawlyui.com"], node_ports: [ {:ui, 0} ]
```

--------------------------------

### Process Item Through Pipelines

Source: https://hexdocs.pm/crawly/Crawly.Utils

Executes a list of pipelines (or middleware) on a given item and state. Mimics filtermap behavior by passing the item through each pipeline's `run/3` function. An item can be dropped by returning `false`, and state can be updated across pipelines.

```elixir
pipe(pipelines, item, state) :: result
when pipelines: [Crawly.Pipeline.t()],
     item: map(),
     state: map(),
     result: {new_item | false, new_state},
     new_item: map(),
     new_state: map()
```

```elixir
item = %{my: "item"}
state = %{}
pipelines = [ MyCustomPipelineOrMiddleware ]
{new_item, new_state} = Crawly.Utils.pipe(pipelines, item, state)
```

--------------------------------

### Custom Blog Post Pipeline in Elixir

Source: https://hexdocs.pm/crawly/basic_concepts

An Elixir module `MyApp.BlogPostPipeline` implementing a custom Crawly pipeline. It uses pattern matching to identify and process items with a `:blog_post` key, updating the blog post data. Items not matching this pattern are passed through unchanged.

```elixir
defmodule MyApp.BlogPostPipeline do
  @impl Crawly.Pipeline
  def run(%{blog_post: old_blog_post} = item, state, _opts \ []) do
    # process the blog post
    updated_item = Map.put(item, :blog_post, %{my: "data"})
    {updated_item, state}
  end
  # do nothing if it does not match
  def run(item, state, _opts), do: {item, state}
end

```

--------------------------------

### Configure Splash Fetcher

Source: https://hexdocs.pm/crawly/basic_concepts

Configuration snippet for Crawly to use the `Splash` fetcher for browser rendering. This involves setting the `:fetcher` configuration to the base URL of a running Splash instance.

```elixir
fetcher: {Crawly.Fetchers.Splash, [base_url: "http://localhost:8050/render.html"]}

```