### Install Scrapyd via pip

Source: https://github.com/scrapy/scrapyd/blob/master/docs/index.rst

This command installs the Scrapyd server package using pip, the Python package installer. It's the first step to setting up a Scrapyd instance on your system.

```shell
pip install scrapyd
```

--------------------------------

### Start the Scrapyd server

Source: https://github.com/scrapy/scrapyd/blob/master/docs/index.rst

Execute this command to launch the Scrapyd server. Once running, it listens for incoming requests to deploy projects and schedule spider crawls, making it ready for operation.

```shell
scrapyd
```

--------------------------------

### Install Scrapyd for Development

Source: https://github.com/scrapy/scrapyd/blob/master/docs/contributing/index.rst

Install Scrapyd in an editable mode, which is suitable for development. This command also includes extra dependencies required for running tests and building documentation, allowing local changes to be immediately reflected.

```shell
pip install -e .[test,docs]
```

--------------------------------

### Run Scrapyd Integration Tests

Source: https://github.com/scrapy/scrapyd/blob/master/docs/contributing/index.rst

Perform integration tests for Scrapyd. This involves setting up a temporary configuration file with credentials, creating a log directory, starting the Scrapyd server in the background, and then running the dedicated integration test suite.

```shell
printf "[scrapyd]\nusername = hello12345\npassword = 67890world\n" > scrapyd.conf
mkdir logs
scrapyd &
pytest integration_tests
```

--------------------------------

### Scrapyd Launcher Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `launcher` configuration option in Scrapyd, which defines the class responsible for starting Scrapy processes.

```APIDOC
launcher:
  Description: The class that starts Scrapy processes.
  Default: scrapyd.launcher.Launcher
  Options: Any Twisted Service (https://docs.twisted.org/en/stable/api/twisted.application.service.Service.html)
```

--------------------------------

### Deploy a Scrapy project to Scrapyd

Source: https://github.com/scrapy/scrapyd/blob/master/docs/index.rst

This command utilizes `scrapyd-deploy` from the `scrapyd-client` package to build a Python egg from your Scrapy project and upload it to the running Scrapyd server. This makes your spiders available for scheduling and execution.

```shell
scrapyd-deploy
```

--------------------------------

### Scrapy Project Deployment Configuration (scrapy.cfg)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/deploy.rst

Example `scrapy.cfg` file used by `scrapyd-deploy` to define default project settings and the deployment URL for the Scrapyd service.

```ini
[settings]
default = myproject.settings

[deploy]
url = http://localhost:6800
project = myproject
```

--------------------------------

### Run Scrapyd Unit Tests

Source: https://github.com/scrapy/scrapyd/blob/master/docs/contributing/index.rst

Execute the unit test suite for the Scrapyd project using pytest to ensure core functionalities are working correctly.

```shell
pytest tests
```

--------------------------------

### Scrapyd Service Runtime Configuration (scrapyd.conf)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/deploy.rst

Example `scrapyd.conf` file for configuring the Scrapyd service's runtime behavior, including the bind address and directories for logs, scraped items, databases, and project eggs.

```ini
[scrapyd]
bind_address      = 0.0.0.0
logs_dir          = /var/lib/scrapyd/logs
items_dir         = /var/lib/scrapyd/items
dbs_dir           = /var/lib/scrapyd/dbs
eggs_dir          = /src/eggs
```

--------------------------------

### Execute Scrapy Crawl via Scrapyd

Source: https://github.com/scrapy/scrapyd/blob/master/docs/overview.rst

This snippet illustrates how Scrapyd initiates a crawl for a specific spider. When a crawl is scheduled via Scrapyd's API, it spawns a new process that essentially runs this command to start the Scrapy spider.

```shell
scrapy crawl myspider
```

--------------------------------

### Scrapyd API: List All Projects (listprojects.json)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Documents the `listprojects.json` endpoint, used to retrieve a list of all projects deployed on the Scrapyd server. It supports GET requests.

```APIDOC
listprojects.json:
  Description: Get the projects deployed on Scrapyd.
  Methods:
    - GET
  Response:
    Type: JSON Object
    Properties:
      node_name: string (e.g., "mynodename")
      status: string (e.g., "ok")
      projects: array of strings (List of project names, e.g., ["myproject", "anotherproject"])
```

--------------------------------

### Schedule a Scrapy spider crawl via HTTP API

Source: https://github.com/scrapy/scrapyd/blob/master/docs/index.rst

This `curl` command demonstrates how to schedule a specific spider within a deployed project on Scrapyd using its HTTP API. It sends a POST request to the `/schedule.json` endpoint, specifying the project and spider name, initiating a crawl.

```shell
curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2
```

--------------------------------

### Scrapyd Environment Variables for Scrapy Processes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/contributing/index.rst

Documentation of environment variables that Scrapyd uses to communicate with and configure the Scrapy processes it launches. These variables provide crucial context and configuration details to the spawned Scrapy instances.

```APIDOC
SCRAPY_PROJECT:
  Description: The project to use. See scrapyd/runner.py.
SCRAPYD_EGG_VERSION:
  Description: The version of the project, to be retrieved as an egg from eggstorage and activated.
SCRAPY_SETTINGS_MODULE:
  Description: The Python path to the settings module of the project. This is usually the module from the entry points of the egg, but can be the module from the [settings] section of a scrapy.cfg file. See scrapyd/environ.py.
```

--------------------------------

### Scrapyd Job Management Interfaces and Workflow

Source: https://github.com/scrapy/scrapyd/blob/master/docs/contributing/index.rst

Detailed documentation of the core interfaces and their methods involved in the Scrapyd job lifecycle, including scheduling, polling for new jobs, and storing information about finished jobs. This section outlines the interaction between different components during job processing.

```APIDOC
ISpiderQueue:
  Description: Represents a queue for pending spider jobs.
  Methods:
    pop(): Retrieves and removes a pending job (dict object).
    list(): Lists pending jobs (dict objects).

ISpiderScheduler:
  Description: Handles the scheduling of spider jobs.
  Methods:
    schedule(message: dict): Adds a job message to the project's ISpiderQueue.

IPoller:
  Description: Polls for new jobs to be launched.
  Methods:
    poll(): Retrieves a short-lived job message.
  Attributes:
    queues: Implements __getitem__ to get a project's ISpiderQueue by project name.

IJobStorage:
  Description: Stores information about finished jobs.
  Methods:
    add(process_protocol: ScrapyProcessProtocol): Adds a finished ScrapyProcessProtocol object to storage.

Job Workflow Overview:
  - The schedule.json webservice calls ISpiderScheduler.schedule, which adds the job message to the project's ISpiderQueue.
  - A TimerService periodically calls IPoller.poll at a defined poll_interval.
  - The QueuePoller implementation calls ISpiderQueue.pop, modifies the retrieved message (adding '_project' and renaming 'name' to '_spider'), and then fires a callback.
  - The Launcher service, having added a callback to the Deferred returned by IPoller.next, adapts the message to instantiate a ScrapyProcessProtocol object, adds another callback, and spawns a new process for the job.
  - When the spawned process finishes, its callback fires, and the Launcher service calls IJobStorage.add, passing the ScrapyProcessProtocol object to store information about the completed job.
```

--------------------------------

### Add Custom Webservice to Scrapyd

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Example configuration to add a new custom webservice endpoint to Scrapyd by defining its module path in the `[services]` section of the configuration file.

```ini
[services]
mywebservice.json = amodule.anothermodule.MyWebService
```

--------------------------------

### Scrapyd Unix Socket Path Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Example configuration for setting the Unix socket path for Scrapyd's web UI and API. This allows Scrapyd to listen for connections on a filesystem path instead of a TCP port, enhancing security and local inter-process communication.

```ini
unix_socket_path = /var/run/scrapyd/web.socket
```

--------------------------------

### Scrapyd API: Get Daemon Status (daemonstatus.json)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Documents the `daemonstatus.json` endpoint, used to check the current load status of the Scrapyd service. It supports GET requests and returns information about pending, running, and finished jobs.

```APIDOC
daemonstatus.json:
  Description: Check the load status of a service.
  Methods:
    - GET
  Response:
    Type: JSON Object
    Properties:
      node_name: string (e.g., "mynodename")
      status: string (e.g., "ok")
      pending: integer (Number of pending jobs)
      running: integer (Number of running jobs)
      finished: integer (Number of finished jobs)
```

```shell-session
curl http://localhost:6800/daemonstatus.json
{"node_name": "mynodename", "status": "ok", "pending": 0, "running": 0, "finished": 0}
```

--------------------------------

### Scrapyd API: Get Job Status (status.json)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Documents the `status.json` endpoint, used to retrieve the current status of a specific job. It requires the job ID and optionally the project name, supporting GET requests.

```APIDOC
status.json:
  Description: Get the status of a job.
  Methods:
    - GET
  Parameters:
    job (required): string - The job ID.
    project (optional): string - The project name.
  Response:
    Type: JSON Object
    Properties:
      node_name: string (e.g., "mynodename")
      status: string (e.g., "ok")
      currstate: string (The current state of the job, e.g., "running", "pending", "finished")
```

```shell-session
curl http://localhost:6800/status.json?job=6487ec79947edab326d6db28a2d86511e8247444
{"node_name": "mynodename", "status": "ok", "currstate": "running"}
```

--------------------------------

### Remove Default Scrapyd Webservice

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Configuration example to disable a default Scrapyd webservice by setting its entry to empty within the `[services]` section.

```ini
[services]
daemonstatus.json =
```

--------------------------------

### Reconfigure Scrapy Logger for Remote Service

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Python code example demonstrating how to reconfigure Scrapy's logger to send logs to a remote Logstash service, typically used within a Scrapy project.

```python
import logging
import logstash

logger = logging.getLogger("scrapy")
logger.handlers.clear()
logger.addHandler(logstash.LogstashHandler("https://user:pass@id.us-east-1.aws.found.io", 5959, version=1))
```

--------------------------------

### Scrapyd Job Object Definition and Interface Attribute Mappings

Source: https://github.com/scrapy/scrapyd/blob/master/docs/contributing/index.rst

This API documentation defines the attributes of a 'finished job' object and illustrates how various job-related concepts are handled and named within different Scrapyd core interfaces. It specifies attribute names, indicates if a concept is not applicable (✗), or describes the format of the data.

```APIDOC
Finished Job Object Definition:
  Description: An object representing a completed job.
  Attributes:
    project: string
    spider: string
    job: string
    start_time: datetime
    end_time: datetime
  Access Methods (via IJobStorage):
    list()
    __iter__()

Scrapyd Interface Job Concept Mappings:

ISpiderQueue:
  Attributes for Job Concepts:
    Project: *not specified*
    Spider: name
    Job ID: _job
    Egg version: _version
    Scrapy settings: settings
    Spider arguments: *remaining keys*
    Environment variables: ✗
    Process ID: ✗
    Start time: ✗
    End time: ✗

IPoller:
  Attributes for Job Concepts:
    Project: _project
    Spider: _spider
    Job ID: _job
    Egg version: _version
    Scrapy settings: settings
    Spider arguments: *remaining keys*
    Environment variables: ✗
    Process ID: ✗
    Start time: ✗
    End time: ✗

ScrapyProcessProtocol:
  Attributes for Job Concepts:
    Project: project
    Spider: spider
    Job ID: job
    Egg version: ✗
    Scrapy settings: args ("-s k=v")
    Spider arguments: args ("-a k=v")
    Environment variables: env
    Process ID: pid
    Start time: start_time
    End time: end_time

IJobStorage:
  Attributes for Job Concepts:
    Project: project
    Spider: spider
    Job ID: job
    Egg version: ✗
    Scrapy settings: ✗
    Spider arguments: ✗
    Environment variables: ✗
    Process ID: ✗
    Start time: start_time
    End time: end_time
```

--------------------------------

### Disable Scrapy Log Storage

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Example INI configuration to disable log storage in Scrapyd by setting `logs_dir` to an empty value.

```ini
logs_dir =
```

--------------------------------

### Set Jobs to Keep to Arbitrarily Large Value

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

INI configuration example to effectively 'disable' the job retention limit by setting `jobs_to_keep` to a very large integer value.

```ini
jobs_to_keep = 9223372036854775807
```

--------------------------------

### Disable Scrapyd TCP Server Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Example configuration to disable the TCP server for Scrapyd by setting the bind_address to an empty value. This is useful when only a Unix socket server is desired, preventing Scrapyd from listening on a TCP port.

```ini
bind_address =
```

--------------------------------

### Scrapyd CLI Usage and Options Reference

Source: https://github.com/scrapy/scrapyd/blob/master/docs/cli.rst

This snippet provides a comprehensive list of command-line options for the `scrapyd` application. It covers flags for debugging, file and process management, user/group ID settings, logging configuration, profiling, and displaying version information.

```Shell
Usage: scrapyd [options]
Options:
  -b, --debug          Run the application in the Python Debugger (implies
                       nodaemon),         sending SIGUSR2 will drop into
                       debugger
      --chroot=        Chroot to a supplied directory before running
  -e, --encrypted      The specified tap/aos file is encrypted.
      --euid           Set only effective user-id rather than real user-id.
                       (This option has no effect unless the server is running
                       as root, in which case it means not to shed all
                       privileges after binding ports, retaining the option to
                       regain privileges in cases such as spawning processes.
                       Use with caution.)
  -f, --file=          read the given .tap file [default: twistd.tap]
  -g, --gid=           The gid to run as.  If not specified, the default gid
                       associated with the specified --uid is used.
      --help           Display this help and exit.
      --help-reactors  Display a list of possibly available reactor names.
  -l, --logfile=       log to a specified file, - for stdout
      --logger=        A fully-qualified name to a log observer factory to use
                       for the initial log observer.  Takes precedence over
                       --logfile and --syslog (when available).
  -n, --nodaemon       don't daemonize, don't use default umask of 0077
  -o, --no_save        do not save state on shutdown
      --originalname   Don't try to change the process name
  -p, --profile=       Run in profile mode, dumping results to specified file.
      --pidfile=       Name of the pidfile [default: twistd.pid]
      --prefix=        use the given prefix when syslogging [default: twisted]
      --profiler=      Name of the profiler to use (profile, cprofile).
                       [default: cprofile]
  -r, --reactor=       Which reactor to use (see --help-reactors for a list of
                       possibilities)
  -s, --source=        Read an application from a .tas file (AOT format).
      --savestats      save the Stats object rather than the text output of the
                       profiler.
      --spew           Print an insanely verbose log of everything that happens.
                       Useful when debugging freezes or locks in complex code.
      --syslog         Log to syslog, not to file
  -u, --uid=           The uid to run as.
      --umask=         The (octal) file creation mask to apply.
      --version        Print version information and exit.
```

--------------------------------

### Build Docker Image for Scrapy Project with Scrapyd

Source: https://github.com/scrapy/scrapyd/blob/master/docs/deploy.rst

This Dockerfile provides a multi-stage build process to create a Docker image for deploying Scrapy projects via Scrapyd. It builds the project egg in the first stage and then sets up the Scrapyd environment in the second stage, including dependencies and volume mounts.

```dockerfile
# Build an egg of your project.

FROM python as build-stage

RUN pip install --no-cache-dir scrapyd-client

WORKDIR /workdir

COPY . .

RUN scrapyd-deploy --build-egg=myproject.egg

# Build the image.

FROM python:alpine

# Install Scrapy dependencies - and any others for your project.

RUN apk --no-cache add --virtual build-dependencies \
   gcc \
   musl-dev \
   libffi-dev \
   libressl-dev \
   libxml2-dev \
   libxslt-dev \
 && pip install --no-cache-dir \
   scrapyd \
 && apk del build-dependencies \
 && apk add \
   libressl \
   libxml2 \
   libxslt

# Mount two volumes for configuration and runtime.

VOLUME /etc/scrapyd/ /var/lib/scrapyd/

COPY ./scrapyd.conf /etc/scrapyd/

RUN mkdir -p /src/eggs/myproject

COPY --from=build-stage /workdir/myproject.egg /src/eggs/myproject/1.egg

EXPOSE 6800

ENTRYPOINT ["scrapyd", "--pidfile="]
```

--------------------------------

### Scrapyd Application Configuration Parameters

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Detailed documentation for the core application-level configuration parameters within Scrapyd. These settings control fundamental aspects like the Twisted application instance, network binding, authentication, and the class responsible for managing spider queues.

```APIDOC
application:
  Description: The function that returns the Twisted Application to use. Override to fully control Scrapyd's behavior.
  Default: scrapyd.app.application
  Options: Any Twisted Application instance
bind_address:
  Description: The IP address on which the web UI and API listen for connections.
  Default: 127.0.0.1
  Options:
    - 127.0.0.1 (local IPv4 only)
    - 0.0.0.0 (all IPv4 connections)
    - ::0 (all IPv4 and IPv6 connections; IPv6 only if sysctl net.ipv6.bindv6only is true)
http_port:
  Description: The TCP port on which the web UI and API listen for connections.
  Default: 6800
  Options: Any integer
unix_socket_path:
  Description: The filesystem path of the Unix socket on which the web UI and API listen for connections. The file's mode is set to 660.
  Version Added: 1.5.0
  Note: If bind_address and http_port are also set, a TCP server will start in addition to the Unix server. Set bind_address = to disable the TCP server.
username:
  Description: Enables basic authentication when set to a non-empty value along with 'password'.
  Version Added: 1.3.0
  Default: "" (empty)
password:
  Description: Enables basic authentication when set to a non-empty value along with 'username'.
  Version Added: 1.3.0
  Default: "" (empty)
spiderqueue:
  Description: The class that stores pending jobs for projects.
  Version Added: 1.4.2
  Default: scrapyd.spiderqueue.SqliteSpiderQueue
  Options:
    - scrapyd.spiderqueue.SqliteSpiderQueue: Stores spider queues in SQLite databases named after each project, located in the 'dbs_dir' directory.
    - Custom implementation: Must use the scrapyd.interfaces.ISpiderQueue interface.
  Used by:
    - addversion.json webservice (to create a queue if project is new)
    - schedule.json webservice (to add a pending job)
    - cancel.json webservice (to remove a pending job)
    - listjobs.json webservice (to list pending jobs)
    - daemonstatus.json webservice (to count pending jobs)
    - webui (to list pending jobs and create queues for transient projects)
```

--------------------------------

### List Project Versions via Scrapyd API

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Fetches a JSON list of all deployed versions for a specified project. Versions are ordered, with the latest version appearing last. This endpoint is crucial for managing different deployments of a single project.

```APIDOC
Endpoint: /listversions.json
Method: GET
Parameters:
  project: string (required) - The name of the project.
Response:
  node_name: string (The name of the Scrapyd node)
  status: string (e.g., "ok")
  versions: array of string (List of project versions, e.g., "r99", "r156")
```

```shell-session
$ curl http://localhost:6800/listversions.json?project=myproject
{"node_name": "mynodename", "status": "ok", "versions": ["r99", "r156"]}
```

--------------------------------

### Scrapyd API: Add Project Version (addversion.json)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Documents the `addversion.json` endpoint for adding a new version of a project to Scrapyd's egg storage. It requires the project name, version, and a Python egg file. The egg must define a Scrapy settings entry point.

```APIDOC
addversion.json:
  Description: Add a version to a project in eggstorage, creating the project if needed.
  Methods:
    - POST
  Parameters:
    project (required): string - The project name.
    version (required): string - The project version. Scrapyd uses packaging.Version to interpret.
    egg (required): file - A Python egg containing the project's code. The egg must set an entry point to its Scrapy settings (e.g., {'scrapy': ['settings = projectname.settings']}).
  Response:
    Type: JSON Object
    Properties:
      node_name: string (e.g., "mynodename")
      status: string (e.g., "ok")
      spiders: integer (Number of spiders found in the uploaded egg)
```

```python
setup(
    name         = 'project',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = projectname.settings']},
)
```

```shell-session
curl http://localhost:6800/addversion.json -F project=myproject -F version=r23 -F egg=@myproject.egg
{"node_name": "mynodename", "status": "ok", "spiders": 3}
```

--------------------------------

### Scrapyd Web Root Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `webroot` configuration option in Scrapyd, defining the class that provides the web UI and API as a Twisted Resource.

```APIDOC
webroot:
  Version Added: 1.2.0
  Description: The class that defines the webui and api, as a Twisted Resource. Override to fully control how the web UI and API work.
  Default: scrapyd.website.Root
  Options: Any Twisted Resource (https://docs.twisted.org/en/stable/web/howto/using-twistedweb.html#resource-objects)
```

--------------------------------

### Default Scrapy Project Settings in scrapy.cfg

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Illustrates the default `[settings]` section found in a `scrapy.cfg` file, typically used to define the default project settings module for Scrapy projects.

```ini
[settings]
default = projectname.settings
```

--------------------------------

### Scrapyd Runner Script Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `runner` configuration option in Scrapyd, which specifies the Python script used to run Scrapy's command-line interface.

```APIDOC
runner:
  Description: The Python script to run Scrapy's CLI. Override to fully control how the Scrapy CLI is called.
  Default: scrapyd.runner
  Options: Any Python script
  Also used by: listspiders.json webservice (to run Scrapy's list command)
```

--------------------------------

### List Deployed Projects via Scrapyd API

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Retrieves a JSON list of all projects currently deployed on the Scrapyd server. This endpoint provides an overview of available projects.

```APIDOC
Endpoint: /listprojects.json
Method: GET
Description: Get a list of all deployed projects.
Response:
  node_name: string (The name of the Scrapyd node)
  status: string (e.g., "ok")
  projects: array of string (List of project names)
```

```shell-session
$ curl http://localhost:6800/listprojects.json
{"node_name": "mynodename", "status": "ok", "projects": ["myproject", "otherproject"]}
```

--------------------------------

### Scrapyd Platform Compatibility Updates

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Outlines improvements and fixes related to Scrapyd's compatibility and behavior on different operating systems, particularly Windows, including testing and specific issue resolutions.

```Python
Platform Support:
- Scrapyd is now tested on macOS and Windows, in addition to Linux.
- The `cancel.json` webservice now works on Windows, by using SIGBREAK instead of SIGINT or SIGTERM.
- The `dbs_dir` setting no longer causes an error if it contains a drive letter on Windows.
- The `items_dir` setting is considered a local path if it contains a drive letter on Windows.
- The `jobs_to_keep` setting no longer causes an error if a file to delete can't be deleted (for example, if the file is open on Windows).
```

--------------------------------

### Scrapyd Maximum Processes Per CPU Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `max_proc_per_cpu` configuration option in Scrapyd, used in conjunction with `max_proc` when `max_proc` is set to 0.

```APIDOC
max_proc_per_cpu:
  Description: See max_proc.
  Default: 4
```

--------------------------------

### Scrapyd Prefix Header Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `prefix_header` configuration option in Scrapyd, used when Scrapyd is behind a reverse proxy and the public URL includes a base path.

```APIDOC
prefix_header:
  Version Added: 1.4.2
  Description: The header for the base path of the original request. Relevant only if Scrapyd is running behind a reverse proxy, and if the public URL contains a base path (e.g., /base/path).
  Default: x-forwarded-prefix
```

--------------------------------

### Scrapyd Poller Configuration Parameters

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for configuration parameters related to Scrapyd's job poller. These settings define how Scrapyd tracks job capacity, manages concurrent processes, and schedules the execution of new jobs.

```APIDOC
poller:
  Description: The class that tracks capacity for new jobs and starts jobs when ready.
  Version Added: 1.5.0
  Default: scrapyd.poller.QueuePoller
  Options:
    - scrapyd.poller.QueuePoller: When using default application and launcher values, the launcher adds max_proc capacity at startup and one capacity each time a Scrapy process ends. The application starts a timer to start jobs every 'poll_interval' seconds if capacity is available.
    - Custom implementation: Must use the scrapyd.interfaces.IPoller interface.
poll_interval:
  Description: The number of seconds between capacity checks performed by the poller.
  Default: (Not explicitly stated)
  Options: Any integer
```

--------------------------------

### Scrapyd 1.1.0 Release Notes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

This entry outlines the changes in Scrapyd version 1.1.0, released on June 29, 2015. It adds node_name and start_time to webservice responses for better job tracking and relocates the scrapyd-deploy command to a separate package. Key fixes include ensuring spiders exist before scheduling, sanitizing version names, and generating correct feed URIs.

```APIDOC
Added:
- Add node_name (hostname) to webservice responses.
- Add start_time to the running jobs in the response from the listjobs.json webservice.

Changed:
- Move scrapyd-deploy command to scrapyd-client package.
- Allow the items_dir setting to be a URL.
- Look for a ~/.scrapyd.conf file in the user's home directory.

Fixed:
- Check if a spider exists before scheduling it.
- Sanitize version names when creating egg paths.
- Generate correct feed URIs, using w3lib.
- Fix git versioning for projects without annotated tags.
- Use valid HTML markup on website pages.
```

--------------------------------

### Scrapyd Core Component Behavior Changes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Documents various behavioral changes in Scrapyd's core components, including settings interpretation, job storage ordering, and method return types.

```Python
- If the `items_dir` setting is a URL and the path component ends with `/`, the `FEEDS` setting no longer contains double slashes.
- The `MemoryJobStorage` class returns finished jobs in reverse chronological order, like the `SqliteJobStorage` class.
- The `list_projects` method of the `SpiderScheduler` class returns a `list`, instead of `dict_keys`.
- Log errors to Scrapyd's log, even when `debug` mode is enabled.
- List the closest `scrapy.cfg` file as a configuration source.
```

--------------------------------

### List Project Jobs (Pending, Running, Finished) via Scrapyd API

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Provides a comprehensive JSON overview of all jobs associated with a project, categorized into pending, running, and finished states. This endpoint is essential for monitoring the execution status of Scrapy crawls. Note that job storage defaults to in-memory, and log/item URLs might be null if disabled or files don't exist.

```APIDOC
Endpoint: /listjobs.json
Method: GET
Parameters:
  project: string (optional) - Filter results by project name.
Response:
  node_name: string (The name of the Scrapyd node)
  status: string (e.g., "ok")
  pending: array of object (List of pending jobs)
    id: string (Job ID)
    project: string (Project name)
    spider: string (Spider name)
    version: string (Project version)
    settings: object (Job-specific settings)
    args: object (Job arguments)
  running: array of object (List of running jobs)
    id: string (Job ID)
    project: string (Project name)
    spider: string (Spider name)
    pid: integer (Process ID)
    start_time: string (Job start timestamp)
    log_url: string or null (URL to job log)
    items_url: string or null (URL to job items)
  finished: array of object (List of finished jobs)
    id: string (Job ID)
    project: string (Project name)
    spider: string (Spider name)
    start_time: string (Job start timestamp)
    end_time: string (Job end timestamp)
    log_url: string or null (URL to job log)
    items_url: string or null (URL to job items)
```

```shell-session
$ curl http://localhost:6800/listjobs.json?project=myproject | python -m json.tool
{
    "node_name": "mynodename",
    "status": "ok",
    "pending": [
        {
            "id": "78391cc0fcaf11e1b0090800272a6d06",
            "project": "myproject",
            "spider": "spider1",
            "version": "0.1",
            "settings": {"DOWNLOAD_DELAY=2"},
            "args": {"arg1": "val1"}
        }
    ],
    "running": [
        {
            "id": "422e608f9f28cef127b3d5ef93fe9399",
            "project": "myproject",
            "spider": "spider2",
            "pid": 93956,
            "start_time": "2012-09-12 10:14:03.594664",
            "log_url": "/logs/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.log",
            "items_url": "/items/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.jl"
        }
    ],
    "finished": [
        {
            "id": "2f16646cfcaf11e1b0090800272a6d06",
            "project": "myproject",
            "spider": "spider3",
            "start_time": "2012-09-12 10:14:03.594664",
            "end_time": "2012-09-12 10:24:03.594664",
            "log_url": "/logs/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.log",
            "items_url": "/items/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.jl"
        }
    ]
}
```

--------------------------------

### List Spiders in a Project Version via Scrapyd API

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Retrieves a JSON list of all spiders available within a specific version of a project. If the project was configured via `scrapy.cfg` and not uploaded, the `version` parameter should be omitted. This helps in identifying runnable spiders for a given deployment.

```APIDOC
Endpoint: /listspiders.json
Method: GET
Parameters:
  project: string (required) - The project name.
  _version: string (optional) - The project version (defaults to the latest).
Response:
  node_name: string (The name of the Scrapyd node)
  status: string (e.g., "ok")
  spiders: array of string (List of spider names, e.g., "spider1", "spider2")
```

```shell-session
$ curl http://localhost:6800/listspiders.json?project=myproject
{"node_name": "mynodename", "status": "ok", "spiders": ["spider1", "spider2", "spider3"]}
```

--------------------------------

### Configure Scrapy Feed Exports for Items Directory

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

JSON configuration for Scrapy's `FEEDS` setting, showing how items can be written to a specified `items_dir` path using the `jsonlines` format. This is applied when `items_dir` is non-empty in Scrapyd.

```json
{"file:///path/to/items_dir/project/spider/job.jl": {"format": "jsonlines"}}
```

--------------------------------

### Scrapyd Debug Mode Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `debug` configuration option in Scrapyd, which, when enabled, returns Python tracebacks for API errors.

```APIDOC
debug:
  Description: Whether debug mode is enabled. If enabled, a Python traceback is returned (as a plain-text response) when the api errors.
  Default: off
```

--------------------------------

### Scrapyd Egg Storage Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `eggstorage` configuration option in Scrapyd, which specifies the class responsible for storing project eggs.

```APIDOC
eggstorage:
  Version Added: 1.3.0
  Description: The class that stores project eggs.
  Default: scrapyd.eggstorage.FilesystemEggStorage
  Options:
```

--------------------------------

### Scrapyd 1.1.1 Release Notes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

This section details the updates in Scrapyd version 1.1.1, released on November 3, 2016. It focuses on documentation improvements for default settings and spider queue priority, along with fixes for SQLite type adapter issues and FEED_URI overriding. The release also addresses requirement incompatibilities and marks the package as zip-unsafe due to Twisted requirements.

```APIDOC
Added:
- Document and include missing settings in default_scrapyd.conf.
- Document the spider queue's priority argument.
- Enable some missing tests for the SQLite queues.

Removed:
- Disable bdist_wheel command in setup to define dynamic requirements, despite pip-7 wheel caching bug.

Fixed:
- Use correct type adapter for sqlite3 blobs. In some systems, a wrong type adapter leads to incorrect buffer reads/writes.
- FEED_URI was always overridden by Scrapyd.
- Specify maximum versions for requirements that became incompatible.
- Mark package as zip-unsafe because Twistd requires a plain txapp.py.
```

--------------------------------

### Scrapyd API: Schedule a Scrapy Job (schedule.json)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Documents the `schedule.json` endpoint, used to schedule a Scrapy crawl job. It allows specifying the project, spider, version, job ID, priority, and custom spider arguments or Scrapy settings. Spiders should accept arbitrary keyword arguments in their `__init__` method.

```APIDOC
schedule.json:
  Description: Schedule a job (a Scrapy crawl). Log files are written to {logs_dir}/{project}/{spider}/{jobid}.log if logs_dir is set. Spiders should allow arbitrary keyword arguments in their __init__ method.
  Methods:
    - POST
  Parameters:
    project (required): string - The project name.
    spider (required): string - The spider name.
    _version (optional): string - The project version (latest by default).
    jobid (optional): string - The job's ID (a hexadecimal UUID v1 by default). Configures the basename of the log file.
    priority (optional): integer - The job's priority in the project's spider queue (0 by default, higher number = higher priority).
    setting (optional): string - A Scrapy setting (e.g., DOWNLOAD_DELAY=2).
    Any other parameter: string - A spider argument (e.g., arg1=val1). Note: When such parameters are set multiple times, only the first value is sent to the spider.
  Response:
    Type: JSON Object
    Properties:
      node_name: string (e.g., "mynodename")
      status: string (e.g., "ok")
      jobid: string (The ID of the scheduled job, e.g., "6487ec79947edab326d6db28a2d86511e8247444")
```

```shell
curl http://localhost:6800/schedule.json -d setting=DOWNLOAD_DELAY=2 -d project=myproject -d spider=somespider
```

```shell
curl http://localhost:6800/schedule.json -d arg1=val1 -d project=myproject -d spider=somespider
```

```shell-session
curl http://localhost:6800/schedule.json -d project=myproject -d spider=somespider
{"node_name": "mynodename", "status": "ok", "jobid": "6487ec79947edab326d6db28a2d86511e8247444"}
```

--------------------------------

### Delete Entire Project via Scrapyd API

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Deletes an entire project and all its associated versions from the Scrapyd server's egg storage. This endpoint provides a way to completely remove a project from the deployment environment.

```APIDOC
Endpoint: /delproject.json
Method: POST
Parameters:
  project: string (required) - The project name to delete.
Response:
  node_name: string (The name of the Scrapyd node)
  status: string (e.g., "ok")
```

```shell-session
$ curl http://localhost:6800/delproject.json -d project=myproject
{"node_name": "mynodename", "status": "ok"}
```

--------------------------------

### Scrapyd 1.3.0 Release Notes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

This entry details the significant updates in Scrapyd version 1.3.0, released on January 12, 2022. Key additions include new settings for HTTP authentication (username, password), job storage, and egg storage, alongside enhancements to webservices like schedule.json and listjobs.json. The release also expanded Python support to versions 3.7 through 3.10 while removing support for older Python 2.x and 3.x versions.

```APIDOC
Added:
- Add username and password settings, for HTTP authentication.
- Add jobstorage and eggstorage settings.
- Add a priority argument to the schedule.json webservice.
- Add project to all jobs in the response from the listjobs.json webservice.
- Add shortcut to jobs page to cancel a job using the cancel.json webservice.
- Python 3.7, 3.8, 3.9, 3.10 support.

Changed:
- Make optional the project argument to the listjobs.json webservice, to easily query for all jobs.
- Improve HTTP headers across webservices.

Removed:
- Python 2, 3.3, 3.4, 3.5 support.
- PyPy 2 support.
- Documentation for Ubuntu installs (Zyte no longer maintains the Ubuntu package).

Fixed:
- Respect Scrapy's TWISTED_REACTOR setting.
- Replace deprecated SafeConfigParser with ConfigParser.
```

--------------------------------

### Authenticate Scrapyd API Requests with cURL

Source: https://github.com/scrapy/scrapyd/blob/master/docs/api.rst

Demonstrates how to include basic authentication credentials when making cURL requests to the Scrapyd API. This is necessary if basic authentication is enabled on the Scrapyd server.

```shell
curl -u yourusername:yourpassword http://localhost:6800/daemonstatus.json
```

--------------------------------

### Scrapyd Maximum Concurrent Processes Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `max_proc` configuration option in Scrapyd, controlling the maximum number of Scrapy processes that can run concurrently.

```APIDOC
max_proc:
  Description: The maximum number of Scrapy processes to run concurrently.
  Default: 0
  Options: Any non-negative integer, including 0 (to use max_proc_per_cpu multiplied by the number of CPUs)
```

--------------------------------

### Scrapyd 1.4.2 Release Notes (2023-05-01)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Summary of additions and changes introduced in Scrapyd version 1.4.2, including new settings and configuration handling.

```Python
Added:
- Add a `spiderqueue` setting.
- Add support for the X-Forwarded-Prefix HTTP header. Rename this header using the `prefix_header` setting.

Changed:
- `scrapyd.spiderqueue.SqliteSpiderQueue` is initialized with a `scrapyd.config.Config` object and a project name, rather than a SQLite connection string.
- If `dbs_dir` is set to `:memory:` or to a URL, it is passed through without modification and without creating a directory to `scrapyd.jobstorage.SqliteJobStorage` and `scrapyd.spiderqueue.SqliteSpiderQueue`.
- `scrapyd.utils.get_spider_queues` defers the creation of the `dbs_dir` directory to the spider queue implementation.
```

--------------------------------

### Scrapyd Items Directory Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `items_dir` configuration option in Scrapyd, specifying the directory for Scrapy item feeds. It explains how this setting influences Scrapy's `FEEDS` and recommends alternative methods for item storage.

```APIDOC
items_dir:
  Description: The directory in which to write Scrapy items. An item feed is written to {items_dir}/{project}/{spider}/{job}.jl. If non-empty, the FEEDS Scrapy setting is set accordingly. Recommended to use Feed exports or Item pipeline instead.
  Default: "" (empty)
  Also used by: webui (to link to item feeds)
  Attention: Each *_dir setting must point to a different directory.
```

--------------------------------

### Scrapyd 1.2.1 Release Notes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

This section outlines the fixes and removals introduced in Scrapyd version 1.2.1, released on June 17, 2019. It addresses issues such as HTTP header compatibility with newer Twisted versions, DeferredQueue behavior, and ensures the addversion.json webservice functions correctly on Windows. Additionally, deprecated SQLite utilities were removed in this version.

```APIDOC
Fixed:
- Fix HTTP header types for newer Twisted versions.
- DeferredQueue no longer hides a pending job when reaching max_proc.
- The addversion.json webservice now works on Windows.
- test: Update binary eggs to be compatible with Scrapy 1.x.

Removed:
- Remove deprecated SQLite utilities.
```

--------------------------------

### Scrapyd Logs Directory Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `logs_dir` configuration option in Scrapyd, specifying the directory for Scrapy log files. It also notes how to disable log storage and its use by the web UI.

```APIDOC
logs_dir:
  Description: The directory in which to write Scrapy logs. A log file is written to {logs_dir}/{project}/{spider}/{job}.log. To disable log storage, set this option to empty.
  Default: logs
  Also used by: webui (to link to log files)
  Attention: Each *_dir setting must point to a different directory.
```

--------------------------------

### Scrapyd Security Enhancements

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Describes security improvements in Scrapyd, focusing on preventing directory traversal and cross-site scripting (XSS) vulnerabilities through proper escaping and error handling in various classes and webservices.

```Python
Security Enhancements:
- `FilesystemEggStorage` class (used by `listversions.json`): Escapes project names (used in glob patterns) before globbing, to disallow listing arbitrary directories.
- `FilesystemEggStorage` class (used by `runner`, `addversion.json`, `listversions.json`, `delversion.json`, `delproject.json` webservices): Raises a `DirectoryTraversalError` error if the project parameter (used in file paths) would traverse directories.
- `Environment` class (used by `launcher`): Raises a `DirectoryTraversalError` error if the project, spider or job parameters (used in file paths) would traverse directories.
- `webui`: Escapes user input (project names, spider names, and job IDs) to prevent cross-site scripting (XSS).
```

--------------------------------

### Scrapyd Web Service API Changes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Details changes and updates to Scrapyd's web service API endpoints, including header handling, field values, and error responses.

```APIDOC
API Changes:
- `Content-Length` header: Counts the number of bytes, instead of the number of characters.
- `Access-Control-Allow-Methods` response header: Contains only the HTTP methods to which webservices respond.
- `listjobs.json` webservice: Sets the `log_url` and `items_url` fields to `null` if the files don't exist.
- `schedule.json` webservice: Sets the `node_name` field in error responses.
- `daemonstatus.json` and `listjobs.json` webservices: Previously unreported next pending job for all but one project; now correctly reported.
- `cancel.json` webservice: Previously could not cancel the unreported job; now cancellable.
```

--------------------------------

### Scrapyd 1.2.0 Release Notes

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

This entry summarizes the major changes in Scrapyd version 1.2.0, released on April 12, 2017. It introduces new webservices like daemonstatus.json and adds arguments to existing ones, improving job and spider management. Significant updates include Python 3 support, Twisted 16 compatibility, and a change to the default bind_address for enhanced security. This version also deprecated and removed several older components and Scrapy 0.x support.

```APIDOC
Added:
- Webservice
  - Add the daemonstatus.json webservice.
  - Add a _version argument to the schedule.json and listspiders.json webservices.
  - Add a jobid argument to the schedule.json webservice.
  - Add pid to the running jobs in the response from the listjobs.json webservice.
  - Include full tracebacks from Scrapy when failing to get spider list.
    This makes debugging deployment problems easier, but webservice output noisier.
- Website
  - Add a webroot setting for website root class.
  - Add start and finish times to jobs page.
- Make console script executable.
- Add contributing documentation.
- Twisted 16 support.
- Python 3 support.

Changed:
- Change bind_address default to 127.0.0.1, instead of 0.0.0.0, to listen only for connections from localhost.

Removed:
- Deprecate unused SQLite utilities in the scrapyd.sqlite module.
  - SqliteDict
  - SqlitePickleDict
  - SqlitePriorityQueue
  - PickleSqlitePriorityQueue
- Scrapy 0.x support.
- Python 2.6 support.

Fixed:
- Poller race condition for concurrently accessed queues.
```

--------------------------------

### Scrapyd Deprecated and Removed Features

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Lists features, classes, functions, and environment variables that have been removed or deprecated in Scrapyd, along with dropped Python version support.

```Python
Removed Features:
- Support for parsing URLs in `dbs_dir` (since SQLite writes only to paths or `:memory:`).
- The `JsonSqliteDict` and `UtilsCache` classes.
- The `native_stringify_dict` function.
- Undocumented and unused internal environment variables:
  - `SCRAPYD_FEED_URI`
  - `SCRAPYD_JOB`
  - `SCRAPYD_LOG_FILE`
  - `SCRAPYD_SLOT`
  - `SCRAPYD_SPIDER`
- Support for end-of-life Python version 3.7.
```

--------------------------------

### Scrapyd Jobs to Keep Configuration

Source: https://github.com/scrapy/scrapyd/blob/master/docs/config.rst

Documentation for the `jobs_to_keep` configuration option in Scrapyd, which controls the number of recent log files and item feeds to retain per spider.

```APIDOC
jobs_to_keep:
  Description: The number of finished jobs per spider, for which to keep the most recent log files in the logs_dir directory and item feeds in the items_dir directory. To "disable" this feature, set this to an arbitrarily large value.
  Default: 5
  Warning: Scrapyd deletes old files in these directories, regardless of origin.
```

--------------------------------

### Scrapyd 1.4.1 Release Notes (2023-02-10)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Summary of fixes introduced in Scrapyd version 1.4.1.

```Python
Fixed:
- Encode the `FEEDS` command-line argument as JSON.
```

--------------------------------

### Scrapyd 1.4.0 Release Notes (2023-02-07)

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

Summary of additions and changes introduced in Scrapyd version 1.4.0, including new API fields and Python version support.

```Python
Added:
- Add `log_url` and `items_url` to the finished jobs in the response from the `listjobs.json` webservice. (@mxdev88)
- Scrapy 2.8 support. Scrapyd sets `LOG_FILE` and `FEEDS` command-line arguments, instead of `SCRAPY_LOG_FILE` and `SCRAPY_FEED_URI` environment variables.
- Python 3.11 support.
- Python 3.12 support. Use `packaging.version.Version` instead of `distutils.LooseVersion`. (@pawelmhm)

Changed:
- Rename environment variables to avoid spurious Scrapy deprecation warnings:
  - `SCRAPY_EGG_VERSION` to `SCRAPYD_EGG_VERSION`
  - `SCRAPY_FEED_URI` to `SCRAPYD_FEED_URI`
  - `SCRAPY_JOB` to `SCRAPYD_JOB`
  - `SCRAPY_LOG_FILE` to `SCRAPYD_LOG_FILE`
```

--------------------------------

### Scrapyd Environment Variable Renames

Source: https://github.com/scrapy/scrapyd/blob/master/docs/news.rst

This section documents the renaming of two internal environment variables, SCRAPY_SLOT and SCRAPY_SPIDER, to their SCRAPYD_ prefixed counterparts. It highlights that these variables are largely undocumented and unused, with a warning that they may be removed in future versions. Users are encouraged to report their usage if they rely on them.

```APIDOC
- SCRAPY_SLOT to SCRAPYD_SLOT
- SCRAPY_SPIDER to SCRAPYD_SPIDER
```