Heritrix

https://heritrix.readthedocs.io/en/latest/
Admin
Heritrix is a scalable, open-source web crawler designed for large-scale, high-performance web...
Tokens:89,938
Snippets:842
Trust Score:9.6
Update:6 months ago
Show doc for...
Tokens:
Raw
### Schedule a URL via Action Directory

Source: https://heritrix.readthedocs.io/en/latest/_sources/operating.rst.txt

Example content for an .schedule file to enqueue a specific URL.

```text
F+ http://example.com
```

--------------------------------

### POST /engine/job/{jobname} (Launch)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Launches a crawl job, optionally starting from a specific checkpoint.

```APIDOC
## POST /engine/job/{jobname}

### Description
Launches a crawl job. The job can be launched in the "paused" state or the "unpaused" state.

### Method
POST

### Endpoint
https://{heritrixhost}:8443/engine/job/{jobname}

### Parameters
#### Path Parameters
- **jobname** (string) - Required - The name of the job to launch.

#### Request Body
- **action** (string) - Required - Must be "launch".
- **checkpoint** (string) - Optional - The name of a checkpoint or "latest" to resume from.
```

--------------------------------

### Sample Processors Report Output

Source: https://heritrix.readthedocs.io/en/latest/_sources/operating.rst.txt

Example output from the processors-report.txt file showing the activity and configuration order of processors in the crawl chains.

```text
CandidateChain - Processors report - 200910300032
  Number of Processors: 2

Processor: org.archive.crawler.prefetch.CandidateScoper

Processor: org.archive.crawler.prefetch.FrontierPreparer

FetchChain - Processors report - 200910300032
  Number of Processors: 9

Processor: org.archive.crawler.prefetch.Preselector

Processor: org.archive.crawler.prefetch.PreconditionEnforcer

Processor: org.archive.modules.fetcher.FetchDNS

Processor: org.archive.modules.fetcher.FetchHTTP
  Function:          Fetch HTTP URIs
  CrawlURIs handled: 1337
  Recovery retries:   0

Processor: org.archive.modules.extractor.ExtractorHTTP
  Function:          Extracts URIs from HTTP response headers
  CrawlURIs handled: 1337  Links extracted:   0

Processor: org.archive.modules.extractor.ExtractorHTML
  Function:          Link extraction on HTML documents
  CrawlURIs handled: 449
  Links extracted:   6894
...
```

--------------------------------

### View crawl.log entries

Source: https://heritrix.readthedocs.io/en/latest/_sources/operating.rst.txt

Example of two lines from the crawl.log file showing URI fetch details.

```text
2011-06-23T17:12:08.802Z   200       1299 http://content-5.powells.com/robots.txt LREP http://content-5.powells.com/cgi-bin/imageDB.cgi?isbn=9780385518635 text/plain #014 20110623171208574+225 sha1:YIUOKDGOLGI5JYHDTXRFFQ5FF4N2EJRV - -
2011-06-23T17:12:09.591Z   200      15829 http://www.identitytheory.com/etexts/poetics.html L http://www.identitytheory.com/ text/html #025 20110623171208546+922 sha1:7AJUMSDTOMT4FN7MBFGGNJU3Z56MLCMW - -
```

--------------------------------

### Configure JerichoExtractorHTML Bean

Source: https://heritrix.readthedocs.io/en/latest/bean-reference.html

Example XML configuration for the JerichoExtractorHTML bean, including commented-out properties for customization.

```xml
<bean id="jerichoExtractorHTML" class="org.archive.modules.extractor.JerichoExtractorHTML">
  <!-- <property name="extractJavascript" value="true" /> -->
  <!-- <property name="extractOnlyFormGets" value="true" /> -->
  <!-- <property name="extractValueAttributes" value="true" /> -->
  <!-- <property name="extractorJS" value="" /> -->
  <!-- <property name="extractorParameters" value="" /> -->
  <!-- <property name="ignoreFormActionUrls" value="false" /> -->
  <!-- <property name="ignoreUnexpectedHtml" value="true" /> -->
  <!-- <property name="loggerModule" value="" /> -->
  <!-- <property name="maxAttributeNameLength" value="64" /> -->
  <!-- <property name="maxAttributeValLength" value="2048" /> -->
  <!-- <property name="maxElementLength" value="64" /> -->
  <!-- <property name="metadata" value="" /> -->
  <!-- <property name="obeyRelNofollow" value="false" /> -->
  <!-- <property name="treatFramesAsEmbedLinks" value="true" /> -->
</bean>
```

--------------------------------

### Launch Heritrix with Web UI

Source: https://heritrix.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Start Heritrix with the Web UI enabled, providing admin credentials. The username and password default to 'admin' and 'admin' if not specified.

```bash
$HERITRIX_HOME/bin/heritrix -a admin:admin
```

--------------------------------

### Sample Mimetypes Report Output

Source: https://heritrix.readthedocs.io/en/latest/_sources/operating.rst.txt

Example output from the mimetype-report.txt file showing document counts and data volume per mime type.

```text
624 13248443 image/jpeg
450 8385573 text/html
261 2160104 image/gif
1 74708 application/x-javascript
1 59 text/dns
1 8488 text/plain
```

--------------------------------

### Configure Heritrix Memory Allocation

Source: https://heritrix.readthedocs.io/en/latest/getting-started.html

Set the JAVA_OPTS environment variable to configure the Java heap size for Heritrix. This example allocates 1GB of memory.

```bash
export JAVA_OPTS=-Xmx1024M

```

--------------------------------

### Sample ResponseCode Report Output

Source: https://heritrix.readthedocs.io/en/latest/_sources/operating.rst.txt

Example output from the responsecode-report.txt file displaying the number of successful documents downloaded per status code.

```text
[#urls] [rescode]
1306 200
31 404
1 1
```

--------------------------------

### POST /engine/job/{jobname} [action=launch]

Source: https://heritrix.readthedocs.io/en/latest/api.html

Launches a crawl job. The job can be started in a paused state or immediately begin crawling if unpaused. Supports launching from the latest checkpoint or a specified checkpoint.

```APIDOC
## Launch Job

POST https://(_heritrixhost_):8443/engine/job/(_jobname_) [action=launch]

### Description
Launches a crawl job. The job can be launched in the “paused” state or the “unpaused” state. If launched in the “unpaused” state the job will immediately begin crawling.

### Method
POST

### Endpoint
https://(_heritrixhost_):8443/engine/job/(_jobname_)

### Parameters
#### Query Parameters
- **action** (string) - Required - must be `launch`
- **checkpoint** (string) - Optional - If supplied, Heritrix will attempt to launch from a checkpoint. Should be the name of a checkpoint (e.g. `cp00001-20180102121229`) or (since version 3.3) the special value `latest`, which will automatically select the most recent checkpoint. If no `checkpoint` is specified (or if the `latest` checkpoint is requested and there are no valid checkpoints) a new crawl will be launched.

### Request Example
```bash
curl -v -d "action=launch" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
```

### Response
#### Success Response (200)
- **(Response details not specified in source)**

#### Response Example
```json
(JSON response example not specified in source)
```
```

--------------------------------

### Run Heritrix in Docker

Source: https://heritrix.readthedocs.io/en/latest/_sources/operating.rst.txt

Starts a detached Heritrix container with port mapping and volume mounting for job persistence.

```bash
mkdir jobs
docker run --init --rm -d -p 8443:8443 -e "USERNAME=admin" -e "PASSWORD=admin" -v $(pwd)/jobs:/opt/heritrix/jobs iipc/heritrix
```

--------------------------------

### Netscape Cookie File Format Example

Source: https://heritrix.readthedocs.io/en/latest/configuring-jobs.html

Represents a single entry in the 7-field tab-separated Netscape cookie file format.

```text
www.example.org FALSE / FALSE 1311699995 sessionid 60ddb868550a
```

--------------------------------

### Set JAVA_HOME Environment Variable

Source: https://heritrix.readthedocs.io/en/latest/getting-started.html

Set the JAVA_HOME environment variable to point to your Java installation directory. This is a prerequisite for running Heritrix.

```bash
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk

```

--------------------------------

### Set JAVA_HOME Environment Variable

Source: https://heritrix.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Configure the JAVA_HOME environment variable to point to your Java installation directory. This is essential for Heritrix to locate the Java runtime.

```bash
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk
```

--------------------------------

### Get Heritrix Engine Status (JSON)

Source: https://heritrix.readthedocs.io/en/latest/api.html

Retrieves engine status information in JSON format. Use this to get version, memory usage, and job lists. Requires authentication.

```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine

```

--------------------------------

### Get Heritrix Engine Status (XML)

Source: https://heritrix.readthedocs.io/en/latest/api.html

Retrieves engine status information in XML format. Use this to get version, memory usage, and job lists. Requires authentication.

```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine

```

--------------------------------

### Configure AggressiveExtractorHTML Bean

Source: https://heritrix.readthedocs.io/en/latest/bean-reference.html

Example Spring bean configuration for the AggressiveExtractorHTML module.

```xml
<bean id="aggressiveExtractorHTML" class="org.archive.modules.extractor.AggressiveExtractorHTML">
  <!-- <property name="extractJavascript" value="true" /> -->
  <!-- <property name="extractOnlyFormGets" value="true" /> -->
  <!-- <property name="extractValueAttributes" value="true" /> -->
  <!-- <property name="extractorJS" value="" /> -->
  <!-- <property name="extractorParameters" value="" /> -->
  <!-- <property name="ignoreFormActionUrls" value="false" /> -->
  <!-- <property name="ignoreUnexpectedHtml" value="true" /> -->
  <!-- <property name="loggerModule" value="" /> -->
  <!-- <property name="maxAttributeNameLength" value="64" /> -->
  <!-- <property name="maxAttributeValLength" value="2048" /> -->
  <!-- <property name="maxElementLength" value="64" /> -->
  <!-- <property name="metadata" value="" /> -->
  <!-- <property name="obeyRelNofollow" value="false" /> -->
  <!-- <property name="treatFramesAsEmbedLinks" value="true" /> -->
</bean>
```

--------------------------------

### Make Heritrix Executable

Source: https://heritrix.readthedocs.io/en/latest/getting-started.html

Grant execute permissions to the Heritrix startup script. This allows you to run Heritrix from the command line.

```bash
chmod u+x $HERITRIX_HOME/bin/heritrix

```

--------------------------------

### Enable HTTP/3 in FetchHTTP2

Source: https://heritrix.readthedocs.io/en/latest/_sources/configuring-jobs.rst.txt

To enable experimental HTTP/3 support, replace FetchHTTP with FetchHTTP2 and set the 'useHTTP3' property to 'true'. An appropriate jetty-quiche-native jar must also be present in the lib directory. HTTP/3 requests are logged with 'h3'.

```xml
<bean id="fetchHttp2" class="org.archive.modules.fetcher.FetchHTTP2">
   <property name="useHTTP3" value="true">
</bean>
```

--------------------------------

### Enable Authentication Logging

Source: https://heritrix.readthedocs.io/en/latest/_sources/configuring-jobs.rst.txt

Sets log levels to FINE for tracking authentication interactions in the console.

```properties
org.archive.crawler.fetcher.FetchHTTP.level = FINE
org.archive.crawler.prefetch.PreconditionEnforcer.level = FINE
```

--------------------------------

### GET /engine/job/(jobname)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Retrieves status information and statistics for a specified job. Supports both XML and JSON responses.

```APIDOC
## GET /engine/job/(jobname)

### Description
Returns status information and statistics about the chosen job.

### Method
GET

### Endpoint
https://(heritrixhost):8443/engine/job/(jobname)

### Parameters
#### Path Parameters
- **jobname** (string) - Required - The name of the job to retrieve status for.

### Request Example
#### XML Request
```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
```

#### JSON Request
```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
```

### Response
#### Success Response (200)
- **shortName** (string) - The short name of the job.
- **crawlControllerState** (string) - The current state of the crawl controller.
- **crawlExitStatus** (string) - The exit status of the crawl.
- **statusDescription** (string) - A human-readable description of the job status.
- **availableActions** (array) - A list of available actions for the job.
- **launchCount** (integer) - The number of times the job has been launched.
- **lastLaunch** (string) - The timestamp of the last job launch (ISO 8601 format).
- **isProfile** (boolean) - Indicates if the job is a profile.
- **primaryConfig** (string) - The path to the primary configuration file.
- **primaryConfigUrl** (string) - The URL of the primary configuration file.
- **url** (string) - The URL for the job.
- **jobLogTail** (array) - An array of strings representing the tail of the job log.
- **uriTotalsReport** (object) - Contains URI statistics.
  - **downloadedUriCount** (integer) - Number of URIs downloaded.
  - **queuedUriCount** (integer) - Number of URIs in the queue.
  - **totalUriCount** (integer) - Total number of URIs.
  - **futureUriCount** (integer) - Number of future URIs.
- **sizeTotalsReport** (object) - Contains size statistics.
  - **dupByHash** (integer) - Size of duplicate content by hash.
  - **dupByHashCount** (integer) - Count of duplicate content by hash.
  - **notModified** (integer) - Size of not modified content.
  - **notModifiedCount** (integer) - Count of not modified content.
  - **novel** (integer) - Size of novel content.
  - **novelCount** (integer) - Count of novel content.
  - **total** (integer) - Total size.
  - **totalCount** (integer) - Total count.
  - **warcNovelContentBytes** (integer) - Size of novel WARC content bytes.
  - **warcNovelUrls** (integer) - Count of novel WARC URLs.
- **rateReport** (object) - Contains rate statistics.
  - **currentDocsPerSecond** (number) - Current documents per second.
  - **averageDocsPerSecond** (number) - Average documents per second.
  - **currentKiBPerSec** (integer) - Current KiB per second.
  - **averageKiBPerSec** (integer) - Average KiB per second.
- **loadReport** (object) - Contains load statistics.
  - **busyThreads** (integer) - Number of busy threads.
  - **totalThreads** (integer) - Total number of threads.
  - **congestionRatio** (number) - Congestion ratio.
  - **averageQueueDepth** (integer) - Average queue depth.
  - **deepestQueueDepth** (integer) - Deepest queue depth.
- **elapsedReport** (object) - Contains elapsed time statistics.
  - **elapsedMilliseconds** (integer) - Elapsed time in milliseconds.
  - **elapsedPretty** (string) - Elapsed time in a human-readable format.
- **threadReport** (object) - Thread-related information (content omitted in example).
- **frontierReport** (object) - Frontier statistics.
  - **totalQueues** (integer) - Total number of queues.
  - **inProcessQueues** (integer) - Number of queues in process.
  - **readyQueues** (integer) - Number of ready queues.
  - **snoozedQueues** (integer) - Number of snoozed queues.
  - **activeQueues** (integer) - Number of active queues.
  - **inactiveQueues** (integer) - Number of inactive queues.
  - **ineligibleQueues** (integer) - Number of ineligible queues.
  - **retiredQueues** (integer) - Number of retired queues.
  - **exhaustedQueues** (integer) - Number of exhausted queues.
  - **lastReachedState** (string) - The last state reached by the frontier.
- **crawlLogTail** (array) - An array of strings representing the tail of the crawl log.
- **configFiles** (array) - Information about configuration files.
- **isLaunchInfoPartial** (boolean) - Indicates if launch information is partial.
- **isRunning** (boolean) - Indicates if the job is currently running.
- **isLaunchable** (boolean) - Indicates if the job is launchable.
- **hasApplicationContext** (boolean) - Indicates if the job has an application context.
- **alertCount** (integer) - The number of alerts.
- **checkpointFiles** (array) - Information about checkpoint files.
- **alertLogFilePath** (string) - The file path for the alert log.
- **crawlLogFilePath** (string) - The file path for the crawl log.
- **reports** (array) - A list of reports associated with the job.
  - **className** (string) - The class name of the report.
  - **shortName** (string) - The short name of the report.
- **heapReport** (object) - Heap memory statistics.
  - **usedBytes** (integer) - Used heap memory in bytes.
  - **totalBytes** (integer) - Total heap memory in bytes.
  - **maxBytes** (integer) - Maximum heap memory in bytes.

#### Response Example
#### XML Response Example
```xml
<job>
  <shortName>myjob</shortName>
  <crawlControllerState>FINISHED</crawlControllerState>
  <crawlExitStatus>FINISHED</crawlExitStatus>
  <statusDescription>Finished: FINISHED</statusDescription>
  <availableActions>
    <value>teardown</value>
  </availableActions>
  <launchCount>1</launchCount>
  <lastLaunch>2020-04-01T02:07:42.531Z</lastLaunch>
  <isProfile>false</isProfile>
  <primaryConfig>/heritrix/jobs/myjob/crawler-beans.cxml</primaryConfig>
  <primaryConfigUrl>https://localhost:8443/engine/job/myjob/jobdir/crawler-beans.cxml</primaryConfigUrl>
  <url>https://localhost:8443/engine/job/myjob/job/myjob</url>
  <jobLogTail>
    <value>2020-04-01T03:50:44.708Z INFO FINISHED 20200401020744</value>
    <value>2020-04-01T03:50:42.670Z INFO EMPTY 20200401020744</value>
    <value>2020-04-01T03:50:42.669Z INFO STOPPING 20200401020744</value>
  </jobLogTail>
  <uriTotalsReport>
    <downloadedUriCount>3920</downloadedUriCount>
    <queuedUriCount>0</queuedUriCount>
    <totalUriCount>3920</totalUriCount>
    <futureUriCount>0</futureUriCount>
  </uriTotalsReport>
  <sizeTotalsReport>
    <dupByHash>0</dupByHash>
    <dupByHashCount>0</dupByHashCount>
    <notModified>0</notModified>
    <notModifiedCount>0</notModifiedCount>
    <novel>2177235508</novel>
    <novelCount>3920</novelCount>
    <total>2177235508</total>
    <totalCount>3920</totalCount>
    <warcNovelContentBytes>2177235508</warcNovelContentBytes>
    <warcNovelUrls>3920</warcNovelUrls>
  </sizeTotalsReport>
  <rateReport>
    <currentDocsPerSecond>0.0</currentDocsPerSecond>
    <averageDocsPerSecond>0.6354171124312226</averageDocsPerSecond>
    <currentKiBPerSec>0</currentKiBPerSec>
    <averageKiBPerSec>344</averageKiBPerSec>
  </rateReport>
  <loadReport>
    <busyThreads>0</busyThreads>
    <totalThreads>0</totalThreads>
    <congestionRatio>NaN</congestionRatio>
    <averageQueueDepth>0</averageQueueDepth>
    <deepestQueueDepth>0</deepestQueueDepth>
  </loadReport>
  <elapsedReport>
    <elapsedMilliseconds>6169176</elapsedMilliseconds>
    <elapsedPretty>1h42m49s176ms</elapsedPretty>
  </elapsedReport>
  <threadReport/>
  <frontierReport>
    <totalQueues>1</totalQueues>
    <inProcessQueues>0</inProcessQueues>
    <readyQueues>0</readyQueues>
    <snoozedQueues>0</snoozedQueues>
    <activeQueues>0</activeQueues>
    <inactiveQueues>0</inactiveQueues>
    <ineligibleQueues>0</ineligibleQueues>
    <retiredQueues>0</retiredQueues>
    <exhaustedQueues>1</exhaustedQueues>
    <lastReachedState>FINISH</lastReachedState>
  </frontierReport>
  <crawlLogTail>
    ...
  </crawlLogTail>
  <configFiles>
    ...
  </configFiles>
  <isLaunchInfoPartial>false</isLaunchInfoPartial>
  <isRunning>false</isRunning>
  <isLaunchable>false</isLaunchable>
  <hasApplicationContext>true</hasApplicationContext>
  <alertCount>549</alertCount>
  <checkpointFiles></checkpointFiles>
  <alertLogFilePath>/heritrix/jobs/myjob/20200401020744/logs/alerts.log</alertLogFilePath>
  <crawlLogFilePath>/heritrix/jobs/myjob/20200401020744/logs/crawl.log</crawlLogFilePath>
  <reports>
    <value>
      <className>CrawlSummaryReport</className>
      <shortName>CrawlSummary</shortName>
    </value>
    ...
  </reports>
  <heapReport>
    <usedBytes>66893400</usedBytes>
    <totalBytes>589824000</totalBytes>
    <maxBytes>2885681152</maxBytes>
  </heapReport>
</job>
```

#### JSON Response Example
```json
{
  "uriTotalsReport": {
    "downloadedUriCount": 3920,
    "queuedUriCount": 2,
    "futureUriCount": 0,
    "totalUriCount": 3920
  },
  "reports": [
    {
      "shortName": "CrawlSummary",
      "className": "CrawlSummaryReport"
    },
    // ...
  ],
  "rateReport": {
    "currentDocsPerSecond": 0.6354171124312226,
    "averageKiBPerSec": 344,
    "currentKiBPerSec": 0,
    "averageDocsPerSecond": 0
  }
}
```
```

--------------------------------

### GET /engine

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Retrieves information about the Heritrix instance, including version, memory usage, and a list of crawl jobs.

```APIDOC
## GET /engine

### Description
Returns information about this instance of Heritrix such as version number, memory usage and the list of crawl jobs.

### Method
GET

### Endpoint
https://(heritrixhost):8443/engine

### Response
#### Success Response (200)
- **heritrixVersion** (string) - The version of Heritrix.
- **heapReport** (object) - Memory usage statistics.
- **jobs** (array) - List of available crawl jobs.
```

--------------------------------

### GET /engine

Source: https://heritrix.readthedocs.io/en/latest/api.html

Retrieves information about the Heritrix instance, including version, memory usage, and a list of crawl jobs.

```APIDOC
## GET /engine

### Description
Returns information about this instance of Heritrix such as version number, memory usage and the list of crawl jobs.

### Method
GET

### Endpoint
https://{heritrixhost}:8443/engine

### Response
#### Success Response (200)
- **heritrixVersion** (string) - The version of the Heritrix instance.
- **heapReport** (object) - Memory usage statistics.
- **jobs** (array) - List of available crawl jobs.

#### Response Example
{
  "heritrixVersion": "3.3.0-SNAPSHOT-2017-07-12T04:17:56Z",
  "heapReport": {
    "usedBytes": 69529904,
    "totalBytes": 589824000,
    "maxBytes": 2885681152
  },
  "jobs": [{
    "shortName": "myjob",
    "statusDescription": "Unbuilt"
  }]
}
```

--------------------------------

### Define Robots Meta Tag

Source: https://heritrix.readthedocs.io/en/latest/_sources/configuring-jobs.rst.txt

Example of a robots meta tag used to instruct crawlers to ignore links.

```html
<meta name="robots" content="nofollow"/>
```

--------------------------------

### Configure PrerequisiteAcceptDecideRule

Source: https://heritrix.readthedocs.io/en/latest/bean-reference.html

Accepts all prerequisite URIs, typically used to ensure necessary resources are not blocked.

```xml
<bean id="prerequisiteAcceptDecideRule" class="org.archive.modules.deciderules.PrerequisiteAcceptDecideRule">
</bean>
```

--------------------------------

### GET /engine/job/{jobname}

Source: https://heritrix.readthedocs.io/en/latest/api.html

Retrieves status information and statistics for a specified Heritrix job. Supports both XML and JSON responses.

```APIDOC
## Get Job Status

### Description
Returns status information and statistics about the chosen job.

### Method
GET

### Endpoint
https://(_heritrixhost_):8443/engine/job/(_jobname_)

### Parameters
#### Path Parameters
- **jobname** (string) - Required - The name of the job to retrieve status for.

### Request Example (XML)
```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
```

### Request Example (JSON)
```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
```

### Response
#### Success Response (200)
- **XML Response**: Contains detailed job status and statistics in XML format.
- **JSON Response**: Contains detailed job status and statistics in JSON format.

#### Response Example (XML)
```xml
<job>
  <shortName>myjob</shortName>
  <crawlControllerState>FINISHED</crawlControllerState>
  <crawlExitStatus>FINISHED</crawlExitStatus>
  <statusDescription>Finished: FINISHED</statusDescription>
  <availableActions>
    <value>teardown</value>
  </availableActions>
  <launchCount>1</launchCount>
  <lastLaunch>2020-04-01T02:07:42.531Z</lastLaunch>
  <isProfile>false</isProfile>
  <primaryConfig>/heritrix/jobs/myjob/crawler-beans.cxml</primaryConfig>
  <primaryConfigUrl>https://localhost:8443/engine/job/myjob/jobdir/crawler-beans.cxml</primaryConfigUrl>
  <url>https://localhost:8443/engine/job/myjob/job/myjob</url>
  <jobLogTail>
    <value>2020-04-01T03:50:44.708Z INFO FINISHED 20200401020744</value>
    <value>2020-04-01T03:50:42.670Z INFO EMPTY 20200401020744</value>
    <value>2020-04-01T03:50:42.669Z INFO STOPPING 20200401020744</value>
  </jobLogTail>
  <uriTotalsReport>
    <downloadedUriCount>3920</downloadedUriCount>
    <queuedUriCount>0</queuedUriCount>
    <totalUriCount>3920</totalUriCount>
    <futureUriCount>0</futureUriCount>
  </uriTotalsReport>
  <sizeTotalsReport>
    <dupByHash>0</dupByHash>
    <dupByHashCount>0</dupByHashCount>
    <notModified>0</notModified>
    <notModifiedCount>0</notModifiedCount>
    <novel>2177235508</novel>
    <novelCount>3920</novelCount>
    <total>2177235508</total>
    <totalCount>3920</totalCount>
    <warcNovelContentBytes>2177235508</warcNovelContentBytes>
    <warcNovelUrls>3920</warcNovelUrls>
  </sizeTotalsReport>
  <rateReport>
    <currentDocsPerSecond>0.0</currentDocsPerSecond>
    <averageDocsPerSecond>0.6354171124312226</averageDocsPerSecond>
    <currentKiBPerSec>0</currentKiBPerSec>
    <averageKiBPerSec>344</averageKiBPerSec>
  </rateReport>
  <loadReport>
    <busyThreads>0</busyThreads>
    <totalThreads>0</totalThreads>
    <congestionRatio>NaN</congestionRatio>
    <averageQueueDepth>0</averageQueueDepth>
    <deepestQueueDepth>0</deepestQueueDepth>
  </loadReport>
  <elapsedReport>
    <elapsedMilliseconds>6169176</elapsedMilliseconds>
    <elapsedPretty>1h42m49s176ms</elapsedPretty>
  </elapsedReport>
  <threadReport/>
  <frontierReport>
    <totalQueues>1</totalQueues>
    <inProcessQueues>0</inProcessQueues>
    <readyQueues>0</readyQueues>
    <snoozedQueues>0</snoozedQueues>
    <activeQueues>0</activeQueues>
    <inactiveQueues>0</inactiveQueues>
    <ineligibleQueues>0</ineligibleQueues>
    <retiredQueues>0</retiredQueues>
    <exhaustedQueues>1</exhaustedQueues>
    <lastReachedState>FINISH</lastReachedState>
  </frontierReport>
  <crawlLogTail>
    <value>...</value>
  </crawlLogTail>
  <configFiles>
    <value>...</value>
  </configFiles>
  <isLaunchInfoPartial>false</isLaunchInfoPartial>
  <isRunning>false</isRunning>
  <isLaunchable>false</isLaunchable>
  <hasApplicationContext>true</hasApplicationContext>
  <alertCount>549</alertCount>
  <checkpointFiles></checkpointFiles>
  <alertLogFilePath>/heritrix/jobs/myjob/20200401020744/logs/alerts.log</alertLogFilePath>
  <crawlLogFilePath>/heritrix/jobs/myjob/20200401020744/logs/crawl.log</crawlLogFilePath>
  <reports>
    <value>
      <className>CrawlSummaryReport</className>
      <shortName>CrawlSummary</shortName>
    </value>
    <value>...</value>
  </reports>
  <heapReport>
    <usedBytes>66893400</usedBytes>
    <totalBytes>589824000</totalBytes>
    <maxBytes>2885681152</maxBytes>
  </heapReport>
</job>
```

#### Response Example (JSON)
```json
{
  "uriTotalsReport": {
    "downloadedUriCount": 3920,
    "queuedUriCount": 2,
    "futureUriCount": 0,
    "totalUriCount": 3920
  },
  "reports": [
    {
      "shortName": "CrawlSummary",
      "className": "CrawlSummaryReport"
    },
    // ...
  ],
  "rateReport": {
    "currentDocsPerSecond": 0.6354171124312226,
    "averageKiBPerSec": 344,
    "currentKiBPerSec": 0,
    "averageDocsPerSecond": 0
  },
  "lastLaunch": "2020-04-01T02:07:42.531Z",
  "frontierReport": {
    "lastReachedState": "FINISH",
    "totalQueues": 1,
    "queueReadiedCount": 0,
    "inProcessQueues": 0,
    "readyQueues": 0,
    "activeQueues": 0,
    "inactiveQueues": 0,
    "ineligibleQueues": 0,
    "retiredQueues": 0,
    "exhaustedQueues": 1,
    "snoozedQueues": 0
  },
  "sizeTotalsReport": {
    "notModifiedCount": 0,
    "total": 2177235508,
    "notModified": 0,
    "dupByHashCount": 0,
    "novelCount": 3920,
    "sizeOnDisk": 0,
    "totalCount": 3920,
    "dupByHash": 0,
    "novel": 2177235508,
    "warcNovelContentBytes": 2177235508,
    "warcNovelUrls": 3920
  },
  "checkpointFiles": [],
  "crawlLogTail": [
    // ...
  ]
}
```
```

--------------------------------

### Add FetchFTP to FetchChain

Source: https://heritrix.readthedocs.io/en/latest/_sources/configuring-jobs.rst.txt

Define the FetchFTP bean and add it to the FetchChain processors to enable FTP crawling. Seeds for FTP should be in the format 'ftp://sftp.example.org/directory'.

```xml
<bean id="fetchProcessors" class="org.archive.modules.FetchChain">
  <property name="processors">
    <list>...<ref bean="fetchFTP"/>...</list>
  </property>
</bean>
```

--------------------------------

### GET /status-codes

Source: https://heritrix.readthedocs.io/en/latest/_sources/glossary.rst.txt

Retrieves the list of custom status codes used by Heritrix to report the result of a URI fetch operation.

```APIDOC
## GET /status-codes

### Description
Returns the mapping of custom status codes used by Heritrix to indicate the result of a URI fetch. These codes supplement standard HTTP status codes.

### Method
GET

### Endpoint
/status-codes

### Response
#### Success Response (200)
- **1** (integer) - Successful DNS lookup
- **0** (integer) - Fetch never tried (protocol unsupported or illegal URI)
- **-1** (integer) - DNS lookup failed
- **-2** (integer) - HTTP connect failed
- **-3** (integer) - HTTP connect broken
- **-4** (integer) - HTTP timeout
- **-5** (integer) - Unexpected runtime exception
- **-6** (integer) - Prerequisite domain-lookup failed
- **-7** (integer) - URI recognized as unsupported or illegal
- **-8** (integer) - Multiple retries failed, retry limit reached
- **-50** (integer) - Temporary status assigned to URIs awaiting preconditions
- **-60** (integer) - URIs assigned a failure status, could not be queued
- **-61** (integer) - Prerequisite robots.txt fetch failed
- **-62** (integer) - Other prerequisite failed
- **-63** (integer) - Prerequisite could not be scheduled
- **-404** (integer) - Empty HTTP response interpreted as a 404
- **-3000** (integer) - Severe Java Error condition
- **-4000** (integer) - Chaff detection applied
- **-4001** (integer) - URI too many link hops away from seed
- **-4002** (integer) - URI too many embed/transitive hops away
- **-5000** (integer) - URI out of scope upon reexamination
- **-5001** (integer) - Blocked from fetch by user setting
```

--------------------------------

### Configure Politeness Settings

Source: https://heritrix.readthedocs.io/en/latest/configuring-jobs.html

Defines delay factors and limits for URI fetching from the same host using the DispositionProcessor bean.

```xml
<bean id="disposition" class="org.archive.crawler.postprocessor.DispositionProcessor">
  <property name="delayFactor" value="5.0" />
  <property name="maxDelayMs" value="30000" />
  <property name="minDelayMs" value="3000" />
</bean>
```

--------------------------------

### Create New Heritrix Job (HTML)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Creates a new crawl job using the default configuration. This endpoint requires the 'createpath' and 'action=create' parameters. Authentication is necessary.

```bash
curl -v -d "createpath=myjob&action=create" -k -u admin:admin --anyauth --location https://localhost:8443/engine
```

--------------------------------

### Add FetchSFTP to FetchChain

Source: https://heritrix.readthedocs.io/en/latest/_sources/configuring-jobs.rst.txt

Define the FetchSFTP bean and add it to the FetchChain processors to enable SFTP crawling. Seeds for SFTP should be in the format 'sftp://sftp.example.org/directory'.

```xml
<bean id="fetchProcessors" class="org.archive.modules.FetchChain">
  <property name="processors">
    <list>
      ...
      <ref bean="fetchSFTP"/>
      ...
    </list>
  </property>
</bean>
```

--------------------------------

### Create New Job (HTML)

Source: https://heritrix.readthedocs.io/en/latest/api.html

Creates a new crawl job using the default profile configuration. Specify the job name with 'createpath'. Requires authentication.

```bash
curl -v -d "createpath=myjob&action=create" -k -u admin:admin --anyauth --location \
  https://localhost:8443/engine

```

--------------------------------

### Set Execute Permission for Heritrix Startup Script

Source: https://heritrix.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Grant execute permissions to the Heritrix startup script. This allows you to run the script to launch Heritrix.

```bash
chmod u+x $HERITRIX_HOME/bin/heritrix
```

--------------------------------

### POST /engine/job/{jobname} [action=build]

Source: https://heritrix.readthedocs.io/en/latest/api.html

Builds the job configuration for a specified job. This step is necessary before a crawl can be launched, as it processes the XML descriptor and prepares the Java objects for execution.

```APIDOC
## Build Job Configuration

POST https://(_heritrixhost_):8443/engine/job/(_jobname_) [action=build]

### Description
Builds the job configuration for the chosen job. It reads an XML descriptor file and uses Spring to build the Java objects that are necessary for running the crawl. Before a crawl can be run it must be built.

### Method
POST

### Endpoint
https://(_heritrixhost_):8443/engine/job/(_jobname_)

### Parameters
#### Query Parameters
- **action** (string) - Required - must be `build`

### Request Example
```bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
```

### Response
#### Success Response (200)
- **(Response details not specified in source)**

#### Response Example
```json
(JSON response example not specified in source)
```
```

--------------------------------

### Get Heritrix Engine Status (JSON)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Retrieves information about the Heritrix instance in JSON format. Use this for programmatic access to crawler status. Requires authentication.

```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine
```

--------------------------------

### Submit CXML Configuration File

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Uploads a crawler-beans.cxml configuration file to a specific job directory.

```bash
curl -v -T my-crawler-beans.cxml -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob/jobdir/crawler-beans.cxml
```

--------------------------------

### Build Job Configuration with cURL

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Use this command to build a job configuration. It reads an XML descriptor and uses Spring to build necessary Java objects for the crawl. Ensure the action parameter is set to 'build'.

```bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location https://localhost:8443/engine/job/myjob
```

```bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine/job/myjob
```

```bash
curl -v -d "action=build" -k -u admin:admin --anyauth --location -H "Accept: application/json" https://localhost:8443/engine/job/myjob
```

--------------------------------

### Configure Heritrix Java Heap Size

Source: https://heritrix.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Adjust the amount of memory allocated to Heritrix by setting the JAVA_OPTS environment variable. This example allocates 1GB of memory.

```bash
export JAVA_OPTS=-Xmx1024M
```

--------------------------------

### Create New Heritrix Job (XML)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Creates a new crawl job and requests the response in XML format. Ensure the 'createpath' parameter is set to the desired job name and 'action' is 'create'. Requires authentication.

```bash
curl -v -d "createpath=myjob&action=create" -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine
```

--------------------------------

### Set HERITRIX_HOME Environment Variable

Source: https://heritrix.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Set the HERITRIX_HOME environment variable to the path where Heritrix is installed. This helps in locating Heritrix's executable files and configurations.

```bash
export HERITRIX_HOME=/home/user/heritrix3.1
```

--------------------------------

### BrowserProcessor Configuration

Source: https://heritrix.readthedocs.io/en/latest/bean-reference.html

Configures the BrowserProcessor to interact with web pages using WebDriver BiDi.

```APIDOC
## BrowserProcessor

### Description
Opens web pages in a local browser via WebDriver BiDi to run behaviors and record subresources.

### Parameters
- **behaviors** (java.util.List<org.archive.modules.behaviors.Behavior>) - Optional - List of behaviors to execute on each page.
- **concurrency** (int) - Optional - Maximum number of concurrent browser pages. Default is 20.
- **executable** (java.lang.String) - Optional - Path to the WebDriver executable.
- **options** (java.util.List<java.lang.String>) - Optional - Command-line options for the WebDriver executable.
```

--------------------------------

### Heritrix Job Status Response (JSON)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

This is an example of the JSON response you can expect when querying job status. It includes summary reports, rate information, and other job statistics.

```json
{
  "uriTotalsReport": {
    "downloadedUriCount": 3920,
    "queuedUriCount": 2,
    "futureUriCount": 0,
    "totalUriCount": 3920
  },
  "reports": [
    {
      "shortName": "CrawlSummary",
      "className": "CrawlSummaryReport"
    },
    // ...
  ],
  "rateReport": {
    "currentDocsPerSecond": 0.6354171124312226,
    "averageKiBPerSec": 344,
    "currentKiBPerSec": 0,
    "averageDocsPerSecond": 0

```

--------------------------------

### POST /engine (Create Job)

Source: https://heritrix.readthedocs.io/en/latest/api.html

Creates a new crawl job using the default configuration provided by the profile-defaults profile.

```APIDOC
## POST /engine

### Description
Creates a new crawl job. It uses the default configuration provided by the profile-defaults profile.

### Method
POST

### Endpoint
https://{heritrixhost}:8443/engine

### Parameters
#### Request Body
- **action** (string) - Required - Must be 'create'.
- **createpath** (string) - Required - The name of the new job.

### Request Example
curl -v -d "createpath=myjob&action=create" -k -u admin:admin https://localhost:8443/engine
```

--------------------------------

### Configure NotMatchesStatusCodeDecideRule

Source: https://heritrix.readthedocs.io/en/latest/bean-reference.html

Use this rule to reject URIs with fetch status codes outside a specified inclusive range. For example, to reject client errors, set the range from 400 to 499.

```xml
<bean id="notMatchesStatusCodeDecideRule" class="org.archive.modules.deciderules.NotMatchesStatusCodeDecideRule">
  <!-- <property name="decision" value="" /> -->
  <!-- <property name="lowerBound" value="" /> -->
  <!-- <property name="upperBound" value="" /> -->
  <!-- <property name="upperBound" value="" /> -->
</bean>
```

--------------------------------

### POST /engine (Create Job)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Creates a new crawl job using the default configuration provided by the profile-defaults profile.

```APIDOC
## POST /engine

### Description
Creates a new crawl job. It uses the default configuration provided by the profile-defaults profile.

### Method
POST

### Endpoint
https://(heritrixhost):8443/engine

### Parameters
#### Request Body
- **action** (string) - Required - Must be "create".
- **createpath** (string) - Required - The name of the new job.
```

--------------------------------

### Configure BrowserProcessor

Source: https://heritrix.readthedocs.io/en/latest/bean-reference.html

Configuration for the BrowserProcessor, which uses WebDriver BiDi to interact with pages. Must be used with FetchHTTP2.

```xml
<bean id="browserProcessor" class="org.archive.crawler.processor.BrowserProcessor">
  <!-- <property name="behaviors" value="" /> -->
  <!-- <property name="concurrency" value="20" /> -->
  <!-- <property name="executable" value="" /> -->
  <!-- <property name="options" value="" /> -->
</bean>
```

--------------------------------

### Get Heritrix Engine Status (XML)

Source: https://heritrix.readthedocs.io/en/latest/_sources/api.rst.txt

Retrieves information about the Heritrix instance, including version, memory usage, and job details. Use this to check the overall state of the crawler. Requires authentication.

```bash
curl -v -k -u admin:admin --anyauth --location -H "Accept: application/xml" https://localhost:8443/engine
```

--------------------------------

### SourceTags Report Output

Source: https://heritrix.readthedocs.io/en/latest/operating.html

Sample output from the source-report.txt file, showing the relationship between seeds and accessed hosts.

```text
[source] [host] [#urls]
http://www.fizzandpop.com/ dns: 1
http://www.fizzandpop.com/ www.fizzandpop.com 1

```
Tokens:
Raw