### Start Web Crawl

Source: https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/README.md

Initiate a web crawl after completing the installation and configuration steps.

```bash
Start your web crawl (Ensure that you followed the above steps and have started your xvfb display as shown above)
```

--------------------------------

### Example: Route documents by host field

Source: https://github.com/apache/nutch/blob/master/src/plugin/exchange-jexl/README.md

This example demonstrates routing documents to 'indexer_solr_1' and 'indexer_rabbit_1' if their 'host' field matches 'example.org'. Other documents are routed to the default writer.

```xml
<exchanges xmlns="http://lucene.apache.org/nutch"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://lucene.apache.org/nutch exchanges.xsd">
  <exchange id="exchange_jexl_1" class="org.apache.nutch.exchange.jexl.JexlExchange">
    <writers>
      <writer id="indexer_solr_1" />
      <writer id="indexer_rabbit_1" />
    </writers>
    <params>
      <param name="expr" value="doc.getFieldValue('host')=='example.org'" />
    </params>
  </exchange>
  <exchange id="default" class="default">
    <writers>
      <writer id="indexer_dummy_1" />
    </writers>
    <params />
  </exchange>
</exchanges>
```

--------------------------------

### IndexReplace Configuration Example

Source: https://github.com/apache/nutch/blob/master/src/plugin/index-replace/README.txt

This example shows the format for configuring the index.replace.regexp property, defining replacements for specific fields.

```xml
<property>
      <name>index.replace.regexp</name>
      <value>
        id=/file\:/http\:my.site.com/
        url=/file\:/http\:my.site.com/2
      </value>
    </property>
```

--------------------------------

### Start Docker and Initialize Environment

Source: https://github.com/apache/nutch/blob/master/docker/README.md

Starts the Docker daemon and initializes the environment variables required for Docker commands. This is typically needed before running Docker containers.

```bash
boot2docker up
$(boot2docker shellinit | grep export)
```

--------------------------------

### Install Firefox (Ubuntu)

Source: https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/README.md

Installs the Firefox browser on Ubuntu systems. This is a prerequisite for using Selenium in local mode with Firefox.

```bash
sudo apt-get install firefox
```

--------------------------------

### List and Format Dependencies

Source: https://github.com/apache/nutch/blob/master/src/plugin/lib-selenium/howto_upgrade_selenium.md

Execute these commands to list current dependencies and format them for inclusion in `plugin.xml`. Note that MacOSX's Sed may require GNU Sed installation.

```bash
$ ant -f ./build-ivy.xml
$ ls lib | sed 's/^/     <library name="/g' | sed 's/$/">\n       <export name="*"/>\n     <\/library>/g'
```

```bash
$ brew install gnu-sed --with-default-names
```

--------------------------------

### Install Xvfb and Associates (Ubuntu)

Source: https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/README.md

Installs Xvfb and related packages required for running graphical applications like Firefox in a headless environment on Ubuntu. This is often necessary for local Selenium execution.

```bash
sudo apt-get install xorg synaptic xvfb gtk2-engines-pixbuf xfonts-cyrillic xfonts-100dpi \
    xfonts-75dpi xfonts-base xfonts-scalable freeglut3-dev dbus-x11 openbox x11-xserver-utils \
    libxrender1 cabextract
```

--------------------------------

### Build Nutch Docker Image

Source: https://github.com/apache/nutch/blob/master/docker/README.md

Builds the Nutch Docker image from the files in the current directory. Ensure Docker is installed and accessible.

```bash
$(boot2docker shellinit | grep export)
docker build -t apache/nutch .
```

--------------------------------

### Configure CSV Index Writer

Source: https://github.com/apache/nutch/blob/master/src/plugin/indexer-csv/README.md

Example configuration for the CSVIndexWriter in conf/index-writers.xml. Customize writer ID, mapping, and parameters as needed.

```xml
<writer id="<writer_id>" class="org.apache.nutch.indexwriter.csv.CSVIndexWriter">
  <mapping>
    ...
  </mapping>
  <parameters>
    ...
  </parameters>   
</writer>
```

--------------------------------

### Configure and run a crawl within a Nutch Docker container

Source: https://context7.com/apache/nutch/llms.txt

Set up seed URLs, configure the agent name, and run a crawl within an active Nutch Docker container. This example performs a 3-round crawl.

```bash
echo "https://example.com" > /tmp/seeds.txt
mkdir -p $NUTCH_HOME/urls
cp /tmp/seeds.txt $NUTCH_HOME/urls/seeds.txt

cat > $NUTCH_HOME/conf/nutch-site.xml <<'EOF'
<configuration>
  <property><name>http.agent.name</name><value>MyNutchBot</value></property>
</configuration>
EOF

crawl -i -s $NUTCH_HOME/urls/ --num-threads 10 --size-fetchlist 1000 $NUTCH_HOME/crawl/ 3
```

--------------------------------

### Get Server Status

Source: https://context7.com/apache/nutch/llms.txt

Retrieves the current status of the Nutch server, including start time, configurations, and job information.

```APIDOC
## GET /admin/ — Get Server Status

### Description
Returns current server state including start time, known configurations, all jobs, and currently running jobs.

### Method
GET

### Endpoint
`http://localhost:8081/admin/`

### Authentication
HTTP Basic Authentication (`admin:admin`)

### Response Example (200 OK)
```json
{
  "startDate": "2026-02-13T10:00:00Z",
  "configuration": ["default", "my-config"],
  "jobs": [],
  "runningJobs": []
}
```
```

--------------------------------

### Get Nutch Server Status

Source: https://context7.com/apache/nutch/llms.txt

Use `curl` to retrieve the current server status, including start time, configurations, and job information. HTTP Basic Authentication is required.

```bash
# Get server status
curl -u admin:admin http://localhost:8081/admin/

# Expected response
# {
#   "startDate": "2026-02-13T10:00:00Z",
#   "configuration": ["default", "my-config"],
#   "jobs": [],
#   "runningJobs": []
# }
```

--------------------------------

### Build Nutch and Run Tests

Source: https://github.com/apache/nutch/blob/master/src/plugin/indexer-elastic/howto_upgrade_es.md

After updating dependencies, navigate to the Nutch root directory and execute this command to clean, build the runtime, and run all unit tests.

```bash
cd ../../../
ant clean runtime test
```

--------------------------------

### Generate OpenSearch Dependencies

Source: https://github.com/apache/nutch/blob/master/src/plugin/indexer-opensearch-1x/howto_upgrade_opensearch.md

Execute this command to list OpenSearch dependencies for inclusion in plugin.xml. Ensure you are in the correct directory.

```bash
cd src/plugin/indexer-opensearch-1x/
ant -f ./build-ivy.xml
ls lib | sed 's/^/    <library name="/g' | sed 's/$/"\]\/>/g'
```

--------------------------------

### Get status of a specific job

Source: https://context7.com/apache/nutch/llms.txt

Retrieves the status of a specific job within a crawl.

```APIDOC
## GET /job/{jobId}

### Description
Gets the status of a specific job within a crawl.

### Method
GET

### Endpoint
/job/{jobId}

### Parameters
#### Query Parameters
- **crawlId** (string) - Required - The identifier for the crawl.

### Response
#### Success Response (200)
- **id** (string) - The ID of the job.
- **type** (string) - The type of the job.
- **state** (string) - The current state of the job.
- **crawlId** (string) - The crawl ID associated with the job.
```

--------------------------------

### Show available `nutch` commands

Source: https://context7.com/apache/nutch/llms.txt

Displays a list of all available commands for the `nutch` CLI. This is useful for understanding the tool's capabilities.

```bash
bin/nutch
```

--------------------------------

### Get Configuration Properties

Source: https://context7.com/apache/nutch/llms.txt

Retrieves all properties for a specific configuration or a single property value.

```APIDOC
## GET /config/{configId} — Get Configuration Properties

### Description
Retrieves all properties of a given configuration ID.

### Method
GET

### Endpoint
`http://localhost:8081/config/{configId}`

### Parameters
#### Path Parameters
- **configId** (string) - Required - The ID of the configuration to retrieve.

### Authentication
HTTP Basic Authentication (`admin:admin`)

### Response Example (200 OK)
```json
{
  "http.agent.name": "NutchBot",
  "http.robots.agents": "NutchBot,*",
  ...
}
```
```

```APIDOC
## GET /config/{configId}/{propertyName} — Get Single Property Value

### Description
Retrieves the value of a single property within a specific configuration.

### Method
GET

### Endpoint
`http://localhost:8081/config/{configId}/{propertyName}`

### Parameters
#### Path Parameters
- **configId** (string) - Required - The ID of the configuration.
- **propertyName** (string) - Required - The name of the property to retrieve.

### Authentication
HTTP Basic Authentication (`admin:admin`)

### Response Example (200 OK)
```
NutchBot
```
```

--------------------------------

### List OkHttp JARs for plugin.xml

Source: https://github.com/apache/nutch/blob/master/src/plugin/protocol-okhttp/howto_upgrade_okhttp.md

Execute this command in the Nutch root directory to list all OkHttp related JAR files, excluding the main protocol-okhttp.jar. This output is used to update the dependencies in plugin.xml.

```bash
ant clean runtime
ls build/plugins/protocol-okhttp/ \
     | grep '\.jar$' \
     | grep -vF protocol-okhttp.jar \
     | sed 's/^/      <library name="/g' | sed 's/$/"\/>/g'
```

--------------------------------

### Remove Local Dependencies

Source: https://github.com/apache/nutch/blob/master/src/plugin/lib-selenium/howto_upgrade_selenium.md

Remove the locally installed dependencies from the `lib/` directory before proceeding with the upgrade.

```bash
$ rm -rf lib/
```

--------------------------------

### List Configurations

Source: https://context7.com/apache/nutch/llms.txt

Lists all available configuration IDs on the server.

```APIDOC
## GET /config/ — List Configurations

### Description
Lists all available configuration IDs.

### Method
GET

### Endpoint
`http://localhost:8081/config/`

### Authentication
HTTP Basic Authentication (`admin:admin`)

### Response Example (200 OK)
```json
["default", "my-custom-config"]
```
```

--------------------------------

### Get Webgraph Node Reader Schema

Source: https://context7.com/apache/nutch/llms.txt

Retrieves the schema definition for the webgraph node reader.

```APIDOC
## GET /reader/node

### Description
Gets the schema for the webgraph node reader, detailing the fields and their types.

### Method
GET

### Endpoint
http://localhost:8081/reader/node

### Response
#### Success Response (200)
- **key_url** (string) - The URL of the node.
- **num_inlinks** (int) - The number of inlinks to the node.
- **num_outlinks** (int) - The number of outlinks from the node.
- **inlink_score** (float) - The score of inlinks.
- **outlink_score** (float) - The score of outlinks.
- **metadata** (string) - Metadata associated with the node.
```

--------------------------------

### Get Webgraph Link Reader Schema

Source: https://context7.com/apache/nutch/llms.txt

Retrieves the schema definition for the webgraph link reader.

```APIDOC
## GET /reader/link

### Description
Gets the schema for the webgraph link reader, detailing the fields and their types.

### Method
GET

### Endpoint
http://localhost:8081/reader/link

### Response
#### Success Response (200)
- **key_url** (string) - The URL of the link.
- **timestamp** (int) - The timestamp associated with the link.
- **score** (float) - The score of the link.
- **anchor** (string) - The anchor text of the link.
- **linktype** (string) - The type of the link.
- **url** (string) - The target URL of the link.
```

--------------------------------

### Build and Test Nutch

Source: https://github.com/apache/nutch/blob/master/src/plugin/lib-selenium/howto_upgrade_selenium.md

After updating dependencies, navigate to the project root and execute these commands to clean, build, and run all unit tests.

```bash
$ cd ../../../
$ ant clean runtime test
```

--------------------------------

### Show resolved Nutch+Hadoop configuration with `nutch`

Source: https://context7.com/apache/nutch/llms.txt

Prints all resolved configuration properties for Nutch and its underlying Hadoop environment. Useful for diagnosing configuration problems.

```bash
bin/nutch showproperties
```

--------------------------------

### Configure Selenium Grid Binary Path

Source: https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/README.md

Provide the path to the browser binary for each node in the Selenium Grid.

```xml
<property>
  <name>selenium.grid.binary</name>
  <value></value>
  <description>A String value representing the path to the browser binary
    location for each node
 </description>
</property>
```

--------------------------------

### Remove Local Elasticsearch Dependencies

Source: https://github.com/apache/nutch/blob/master/src/plugin/indexer-elastic/howto_upgrade_es.md

Remove locally installed Elasticsearch dependencies from the lib/ directory within the indexer-elastic plugin.

```bash
rm -rf lib/
```

--------------------------------

### Get FetchDB node info

Source: https://context7.com/apache/nutch/llms.txt

Retrieves node information from the FetchDB, which represents the fetch graph. Allows specifying a range of entries to retrieve.

```APIDOC
## GET /db/fetchdb

### Description
Gets FetchDB node info (fetch graph), allowing retrieval of entries within a specified range.

### Method
GET

### Endpoint
/db/fetchdb

### Parameters
#### Query Parameters
- **from** (integer) - Optional - The starting index of the entries to retrieve.
- **to** (integer) - Optional - The ending index of the entries to retrieve.

### Response
#### Success Response (200)
- Returns a JSON array of objects, where each object represents a node in the FetchDB with details like URL, status, and children.
```

--------------------------------

### Set up Xvfb Display

Source: https://github.com/apache/nutch/blob/master/src/plugin/protocol-selenium/README.md

Configures Xvfb to create a virtual display server and sets the DISPLAY environment variable. This allows applications like Firefox to run without a physical display.

```bash
sudo /usr/bin/Xvfb :11 -screen 0 1024x768x24 &
sudo export DISPLAY=:11
```

--------------------------------

### Exempt URLs Ending with Image Extensions

Source: https://github.com/apache/nutch/blob/master/src/plugin/urlfilter-ignoreexempt/README.md

Define exemption rules in 'conf/db-ignore-external-exemptions.txt'. This example exempts URLs ending with common image file extensions.

```regex
+(?i)\.(jpg|png|gif)$
```

--------------------------------

### List Nutch Configurations

Source: https://context7.com/apache/nutch/llms.txt

Use `curl` to list all available configuration IDs on the Nutch server. HTTP Basic Authentication is required.

```bash
# List all configuration IDs
curl -u admin:admin http://localhost:8081/config/
# ["default", "my-custom-config"]
```

--------------------------------

### Get Nutch Configuration Properties

Source: https://context7.com/apache/nutch/llms.txt

Retrieve all properties for a specific Nutch configuration or a single property value using `curl`. HTTP Basic Authentication is required.

```bash
# Get all properties of a configuration
curl -u admin:admin http://localhost:8081/config/default
# {
#   "http.agent.name": "NutchBot",
#   "http.robots.agents": "NutchBot,*",
```

```bash
# Get a single property value
curl -u admin:admin http://localhost:8081/config/default/http.agent.name
# NutchBot
```

--------------------------------

### URL Filter Rule Syntax

Source: https://github.com/apache/nutch/blob/master/src/plugin/urlfilter-fast/README.md

Defines the syntax for rules in the urlfilter.fast.file. Rules can specify host or domain matching, followed by path or path-and-query exclusions. Comments start with '#'.

```plaintext
Host www.example.org
  DenyPath /path/to/be/excluded
  DenyPath /some/other/path/excluded

# Deny everything from *.example.com and example.com
Domain example.com
  DenyPath .*

Domain example.org
  DenyPathQuery /resource/.*?action=exclude
```