### Start Docker Compose Services

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Command to start the Docker Compose services in detached mode.

```bash
docker compose up -d
```

--------------------------------

### Configure HTTP Protocol Versions

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Control the HTTP protocol versions used by the crawler. This example forces the crawler to use only HTTP/1.1, disabling HTTP/2.

```yaml
http.protocol.versions:
- "http/1.1"
```

--------------------------------

### RegexURLFilter Configuration

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

This JSON configures the RegexURLFilter with a list of rules. Rules starting with '-' deny a URL, while those starting with '+' accept it. The most specific rule should be placed first.

```json
{
    "urlFilters": [
        "-^(file|ftp|mailto):",
        "+."
    ]
}

```

--------------------------------

### CollectionTagger ParseFilter Configuration

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Example JSON configuration for the CollectionTagger ParseFilter. It allows defining collections with include and exclude patterns to tag documents based on their URLs.

```json
{
  "collections": [
    {
      "name": "stormcrawler",
      "includePatterns": ["https://stormcrawler.net/.+"]
    },
    {
      "name": "crawler",
      "includePatterns": [".+crawler.+", ".+nutch.+"],
      "excludePatterns": [".+baby.+", ".+spider.+"]
    }
  ]
}
```

--------------------------------

### Configure Proxy File for MultiProxyManager

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Specify the path to a TXT file containing connection strings for multiple proxies. This file is used by `MultiProxyManager` for load balancing.

```properties
http.proxy.file
```

--------------------------------

### Configure Verbatim User Agent String

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Specify the complete User-Agent string directly using the 'http.agent' configuration. Note that 'http.agent.name' is still required for robots.txt parsing.

```properties
http.agent=MyCrawler/1.0 (Description; http://example.com; contact@example.com)
```

--------------------------------

### Configure Single Proxy Settings

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Set up a single proxy server for StormCrawler to use. This includes host, port, type, and optional authentication credentials.

```properties
http.proxy.host
http.proxy.port
http.proxy.type
http.proxy.user (optional)
http.proxy.pass (optional)
```

--------------------------------

### Docker Compose for StormCrawler Services

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

A sample docker-compose.yml configuration to set up Zookeeper, Storm Nimbus, Supervisor, UI, and URLFrontier. Ensure ports are adjusted for your environment and network connectivity between services is maintained.

```yaml
services:
  zookeeper:
    image: zookeeper:3.9.3
    container_name: zookeeper
    restart: always

  nimbus:
    image: storm:latest
    container_name: nimbus
    hostname: nimbus
    command: storm nimbus
    depends_on:
      - zookeeper
    restart: always

  supervisor:
    image: storm:latest
    container_name: supervisor
    command: storm supervisor -c worker.childopts=-Xmx%HEAP-MEM%m
    depends_on:
      - nimbus
      - zookeeper
    restart: always

  ui:
    image: storm:latest
    container_name: ui
    command: storm ui
    depends_on:
      - nimbus
    restart: always
    ports:
      - "127.0.0.1:8080:8080"

  urlfrontier:
    image: crawlercommons/url-frontier:latest
    container_name: urlfrontier
    restart: always
    ports:
      - "127.0.0.1:7071:7071"
```

--------------------------------

### Configure Custom Proxy Manager

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Specify a custom Java class to implement the `ProxyManager` interface for advanced proxy management and load balancing logic.

```properties
http.proxy.manager: "org.apache.stormcrawler.proxy.MultiProxyManager"
```

--------------------------------

### Configure HTTP/S Protocol Implementation

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Specify custom implementations for HTTP and HTTPS protocols in your crawler configuration. This is useful for integrating alternative HTTP clients like OKHttp.

```yaml
http.protocol.implementation: "org.apache.stormcrawler.protocol.okhttp.HttpProtocol"
https.protocol.implementation: "org.apache.stormcrawler.protocol.okhttp.HttpProtocol"
```

--------------------------------

### View Docker Compose Logs

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Command to follow the logs of all running Docker Compose services.

```bash
docker compose logs -f
```

--------------------------------

### Configure Proxy Load Balancing Scheme

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Set the load balancing scheme for `MultiProxyManager`. Options include ROUND_ROBIN, RANDOM, and LEAST_USED.

```properties
http.proxy.rotation
```

--------------------------------

### FastURLFilter Configuration

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

This JSON configures the FastURLFilter using scopes and patterns. Scopes can be GLOBAL, domain, metadata, or host-specific. Patterns determine whether to allow or deny URLs based on their path and query elements.

```json
[
    {
        "scope": "GLOBAL",
        "patterns": [
            "DenyPathQuery \.jpg"
        ]
      },
      {
        "scope": "domain:stormcrawler.net",
        "patterns": [
            "AllowPath /digitalpebble/",
            "DenyPath .+"
        ]
      },
      {
        "scope": "metadata:key=value",
        "patterns": [
           "DenyPath .+"
        ]
    }]

```

--------------------------------

### Compile StormCrawler Project

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Build the StormCrawler project into an uberjar using the 'mvn package' command. The resulting JAR file will be located in the 'target' directory.

```bash
mvn package
```

--------------------------------

### Configure Metadata Serialization via YAML

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Alternatively, register Metadata for Kryo serialization by specifying it in the Storm configuration file under 'topology.kryo.register'. This is a declarative way to handle serialization registration.

```yaml
topology.kryo.register:
  - org.apache.storm.crawler.Metadata
```

--------------------------------

### List Docker Networks

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Command to list available Docker networks, useful for identifying the network name to use with the `docker run` command.

```bash
docker network ls
```

--------------------------------

### Inject Seed URLs into URLFrontier

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Use the URLFrontier client to inject seed URLs from a file into the running URLFrontier service. Ensure the JAR path and seed file name are correct.

```bash
java -cp target/${artifactId}-${version}.jar crawlercommons.urlfrontier.client.Client PutURLs -f seeds.txt
```

--------------------------------

### Instantiate MetadataTransfer

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Obtain an instance of MetadataTransfer using the static getInstance method, passing the Storm configuration map. This class is central to managing metadata transfer and filtering within the crawler pipeline.

```java
public static MetadataTransfer getInstance(Map<String, Object> conf)
```

--------------------------------

### Run First StormCrawler Topology

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Command to run a StormCrawler topology in distributed mode on a Storm Cluster using Docker. Ensure the crawler-conf.yaml, crawler.flux, and the JAR file are in the current directory.

```bash
docker run --network ${NETWORK} -it \
--rm \
-v "$(pwd)/crawler-conf.yaml:/apache-storm/crawler-conf.yaml" \
-v "$(pwd)/crawler.flux:/apache-storm/crawler.flux" \
-v "$(pwd)/${artifactId}-${version}.jar:/apache-storm/${artifactId}-${version}.jar" \
storm:latest \
storm jar ${artifactId}-${version}.jar org.apache.storm.flux.Flux --remote crawler.flux
```

--------------------------------

### Generate StormCrawler Project with Maven Archetype

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Use this Maven command to generate a new StormCrawler project structure. Replace `<CURRENT_VERSION>` with the latest release. You will be prompted for project details like groupId, artifactId, and version.

```bash
mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler \
                       -DarchetypeArtifactId=stormcrawler-archetype \
                       -DarchetypeVersion=<CURRENT_VERSION>
```

--------------------------------

### Persist Metadata for HTTP Requests

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Configure which metadata keys should be persisted across requests. This is essential for headers like 'last-modified' and 'protocol.etag'.

```yaml
metadata.persist:
  - last-modified
  - protocol.etag
  - protocol.set-cookie
  - ...
```

--------------------------------

### Set Custom HTTP Headers via Metadata

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Use the 'protocol.set-header' metadata key to add custom headers to HTTP requests. Values are specified as key-value pairs.

```json
"protocol%2Eset-header": [
  "header1=value1",
  "header2=value2"
]
```

--------------------------------

### Register Metadata for Kryo Serialization

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Manually register StormCrawler's Metadata class for Kryo serialization if not extending ConfigurableTopology. This ensures Metadata objects can be serialized and deserialized correctly by Storm.

```java
Config.registerSerialization(conf, Metadata.class);
```

--------------------------------

### ParseFilter Interface Methods

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Defines the three core methods required for implementing a custom ParseFilter: filter, configure, and needsDOM. The filter method handles data extraction, configure is for initialization, and needsDOM indicates DOM dependency.

```java
public void filter(String URL, byte[] content, DocumentFragment doc, ParseResult parse);

public void configure(Map stormConf, JsonNode filterParams);

public boolean needsDOM();
```

--------------------------------

### Configure Robots Exclusion Protocol Rules

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Define rules for the robots.txt protocol to specify which parts of a website are allowed or disallowed for crawling. The `User-Agent` directive matches crawler identities, and `Disallow`/`Allow` directives specify URL patterns.

```properties
User-Agent: *
Disallow: *.gif$
Disallow: /example/
Allow: /publications/
```

--------------------------------

### Enable Remote Debugging for StormCrawler Topology

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Use this command to enable remote debugging for a StormCrawler topology. The topology will suspend execution until a debugger attaches.

```bash
export STORM_JAR_JVM_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=localhost:8000"
```

--------------------------------

### RegexURLNormalizer Configuration

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

This JSON configures the RegexURLNormalizer with patterns and their corresponding substitutions for normalizing URLs. It can be used to remove or modify parts of a URL.

```json
{
    "urlNormalizers": [
        {
            "pattern": "#.*?(\?|&amp;|$)",
            "substitution": "$1"
        },
        {
            "pattern": "\?&amp;",
            "substitution": "\?"
        }
    ]
}

```

--------------------------------

### Declare Output Fields for Fetcher Bolts

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Defines the output fields for the default stream and the status stream used by Fetcher Bolts. The status stream handles redirections and errors, while the default stream provides fetched content and metadata.

```java
declarer.declare(new Fields("url", "content", "metadata"));
declarer.declareStream(
        org.apache.storm.crawler.Constants.StatusStreamName,
        new Fields("url", "metadata", "status"));
```

--------------------------------

### Declare Output Fields for Indexer Bolts

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Specifies the output fields for the status stream, which is used by Indexer Bolts to report the processing status of a URL, including metadata and the final status.

```java
public void declareOutputFields(OutputFieldsDeclarer declarer) {
    declarer.declareStream(
            org.apache.stormcrawler.Constants.StatusStreamName,
            new Fields("url", "metadata", "status"));
}
```

--------------------------------

### Transfer Cookies to Outlinks

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

Configure metadata transfer to ensure cookies are passed to outlinks. This requires setting 'metadata.transfer' to include 'set-cookie'.

```yaml
metadata.transfer:
  - set-cookie
```

--------------------------------

### Declare Status Stream in StormCrawler

Source: https://stormcrawler.apache.org/docs/3.5.1/index.html

This code snippet demonstrates how a bolt declares its output stream for status updates in Apache StormCrawler. It specifies the stream name and the fields within the tuples.

```java
declarer.declareStream(
    org.apache.storm.crawler.Constants.StatusStreamName,
    new Fields("url", "metadata", "status"));
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.