### SimpleFetcherBolt Topology Setup with URLPartitionerBolt

Source: https://context7.com/apache/stormcrawler/llms.txt

Example of setting up SimpleFetcherBolt with URLPartitionerBolt for explicit partitioning. This configuration ensures that URLs are processed by specific instances based on a key.

```java
builder.setBolt("partitioner", new URLPartitionerBolt(), 1)
       .shuffleGrouping("spout");
builder.setBolt("fetcher", new SimpleFetcherBolt(), 4)  // 4 instances
       .fieldsGrouping("partitioner", new Fields("key"));
```

--------------------------------

### FetcherBolt Topology Setup

Source: https://context7.com/apache/stormcrawler/llms.txt

Example of setting up FetcherBolt in a StormCrawler topology using TopologyBuilder. It demonstrates how to configure parallel instances and group by key for partitioning.

```java
builder.setBolt("fetcher", new FetcherBolt(), 2)        // 2 parallel instances
       .fieldsGrouping("partitioner", new Fields("key")); // same key → same instance
```

--------------------------------

### Complete Storm Topology Example with Metadata Serialization

Source: https://github.com/apache/stormcrawler/wiki/Start/Registering-Metadata-for-Serialization

A full example of a Storm topology demonstrating how to register the Metadata class for serialization, based on storm-starter's ExclamationTopology.

```java
package storm.starter;

import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.testing.TestWordSpout;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseRichBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;

import org.apache.storm.crawler.Metadata;

import java.util.Map;

/**
 * This is a basic example of a Storm topology, with serialization for storm-crawler's Metadata.
 */
public class ExclamationTopology {

  public static class ExclamationBolt extends BaseRichBolt {
    OutputCollector _collector;

    @Override
    public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
      _collector = collector;
    }

    @Override
    public void execute(Tuple tuple) {
      _collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
      _collector.ack(tuple);
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
      declarer.declare(new Fields("word"));
    }


  }

  public static void main(String[] args) throws Exception {
    TopologyBuilder builder = new TopologyBuilder();

    builder.setSpout("word", new TestWordSpout(), 10);
    builder.setBolt("exclaim1", new ExclamationBolt(), 3).shuffleGrouping("word");
    builder.setBolt("exclaim2", new ExclamationBolt(), 2).shuffleGrouping("exclaim1");

    Config conf = new Config();
    conf.setDebug(true);

    // Register Metadata for serialization
    Config.registerSerialization(conf, Metadata.class);

    if (args != null && args.length > 0) {
      conf.setNumWorkers(3);

      StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());
    }
    else {

      LocalCluster cluster = new LocalCluster();
      cluster.submitTopology("test", conf, builder.createTopology());
      Utils.sleep(10000);
      cluster.killTopology("test");
      cluster.shutdown();
    }
  }
}

```

--------------------------------

### Build Core and OpenSearch Modules with Dependencies

Source: https://context7.com/apache/stormcrawler/llms.txt

Compile and install the 'core' and 'external/opensearch' modules, including their transitive dependencies.

```bash
mvn clean install -pl core,external/opensearch -am
```

--------------------------------

### CollectionTagger Configuration Example

Source: https://github.com/apache/stormcrawler/wiki/Components/Filters/ParseFilters

Example JSON configuration for CollectionTagger, specifying collection names with include and exclude URL patterns.

```json
{
    "collections": [{
             "name": "stormcrawler",
             "includePatterns": ["http://stormcrawler.net/.+"]
         },
         {
             "name": "crawler",
             "includePatterns": [".+crawler.+", ".+nutch.+"],
             "excludePatterns": [".+baby.+", ".+spider.+"]
         }
     ]
  }
```

--------------------------------

### Example Robots.txt

Source: https://context7.com/apache/stormcrawler/llms.txt

Illustrates robots.txt rules for specific user-agents and general crawlers. Includes Disallow directives and Crawl-delay.

```text
# Example robots.txt the crawler would respect:
User-Agent: ResearchCrawler
Disallow: /private/
Crawl-delay: 2

User-Agent: *
Disallow: /admin/
```

--------------------------------

### Install Chromium Browser with Maven

Source: https://github.com/apache/stormcrawler/blob/main/external/playwright/README.md

Use Maven to execute the Playwright CLI command for installing the Chromium browser. This is an alternative to Playwright's automatic installation.

```bash
mvn exec:java -e -Dexec.mainClass=com.microsoft.playwright.CLI -Dexec.args="install chromium"

```

--------------------------------

### Topology Wiring with OpenSearch Components

Source: https://context7.com/apache/stormcrawler/llms.txt

Example Java code demonstrating how to wire the AggregationSpout, IndexerBolt, and StatusUpdaterBolt in a Storm topology for OpenSearch integration.

```java
import org.apache.stormcrawler.opensearch.persistence.AggregationSpout;
import org.apache.stormcrawler.opensearch.persistence.StatusUpdaterBolt;
import org.apache.stormcrawler.opensearch.bolt.IndexerBolt;

builder.setSpout("spout", new AggregationSpout(), 1);
// ... parser bolts ...
builder.setBolt("index", new IndexerBolt(), 1)
       .localOrShuffleGrouping("parse");
builder.setBolt("status", new StatusUpdaterBolt(), 1)
       .fieldsGrouping("index", "status", new Fields("url"))
       .fieldsGrouping("parse", "status", new Fields("url'))
       .fieldsGrouping("fetcher", "status", new Fields("url"));
```

--------------------------------

### Define Proxies in proxies.txt

Source: https://context7.com/apache/stormcrawler/llms.txt

List proxies with their protocol, host, and port. User credentials can be included. Comments start with # or //.

```text
# proxies.txt — one proxy per line (comments with # or //)
# format: protocol://[user:pass@]host:port
http://proxy1.example.com:8080
http://user:pass@proxy2.example.com:3128
https://proxy3.example.example.com:8443
# http://disabled-proxy.example.com:8080
```

--------------------------------

### Start OpenSearch with Docker

Source: https://context7.com/apache/stormcrawler/llms.txt

Launches a single-node OpenSearch instance using Docker. Exposes ports 9200 and 9600 for communication.

```shell
docker run -d -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  opensearchproject/opensearch:latest
```

--------------------------------

### Example OpenSearch Dashboards Import Response

Source: https://github.com/apache/stormcrawler/blob/main/external/opensearch/archetype/src/main/resources/archetype-resources/README.md

This is an example of the JSON response received after successfully importing dashboards into OpenSearch Dashboards.

```json
{"successCount":4,"success":true,"successResults":[{"type":"index-pattern","id":"7445c390-7339-11e9-9289-ffa3ee6775e4","meta":{"title":"status","icon":"indexPatternApp"}},{"type":"visualization","id":"status-count","meta":{"title":"status count","icon":"visualizeApp"}},{"type":"visualization","id":"Top-Hosts","meta":{"title":"Top Hosts","icon":"visualizeApp"}},{"type":"dashboard","id":"Crawl-status","meta":{"title":"Crawl status","icon":"dashboardApp"}}]}
```

--------------------------------

### Enable Pre-commit Format Hooks

Source: https://context7.com/apache/stormcrawler/llms.txt

Run a full Maven build and install, ensuring code formatting checks are active.

```bash
mvn clean install -Dskip.format.code=false
```

--------------------------------

### FastURLFilter Configuration Example

Source: https://github.com/apache/stormcrawler/wiki/Components/Filters/URLFilters

A JSON configuration file for FastURLFilter, demonstrating how to define rules based on scope (GLOBAL, domain, metadata) and patterns for allowing or denying URLs.

```json
[
    {
            "scope": "GLOBAL",
            "patterns": [
                "DenyPathQuery \.jpg"
            ]
        },
        {
            "scope": "domain:stormcrawler.net",
            "patterns": [
                "AllowPath /digitalpebble/",
                "DenyPath .+"
            ]
        },
        {
            "scope": "metadata:key=value",
            "patterns": [
               "DenyPath .+"
            ]
        }
    ]

```

--------------------------------

### Start Solr in Cloud Mode and Upload Configsets

Source: https://github.com/apache/stormcrawler/blob/main/external/solr/archetype/src/main/resources/archetype-resources/README.md

Use this script to initialize Solr in cloud mode, upload necessary configuration sets, and create the required collections for StormCrawler.

```sh
./setup-solr.sh
```

--------------------------------

### Build Only the Core Module

Source: https://context7.com/apache/stormcrawler/llms.txt

Compile and install only the 'core' module of the Apache StormCrawler project.

```bash
mvn clean install -pl core
```

--------------------------------

### Example Solr Record with Metadata

Source: https://github.com/apache/stormcrawler/blob/main/external/solr/archetype/src/main/resources/archetype-resources/README.md

An example JSON record demonstrating how URL metadata is stored in Solr. Metadata fields are prefixed, and the prefix can be configured.

```json
{
  "url": "http://test.com",
  "host": "test.com",
  "status": "DISCOVERED",
  "key": "test.com",
  "metadata.url.path": "http://test.com",
  "metadata.depth": "1",
  "nextFetchDate": "2015-10-30T17:26:34.386Z"
}
```

--------------------------------

### HTTP/S Protocol Implementation

Source: https://github.com/apache/stormcrawler/blob/main/docs/src/main/asciidoc/internals.adoc

Specify the HTTP and HTTPS protocol implementations for StormCrawler. This example shows how to configure the OkHttp protocol for both.

```yaml
http.protocol.implementation: "org.apache.stormcrawler.protocol.okhttp.HttpProtocol"
https.protocol.implementation: "org.apache.stormcrawler.protocol.okhttp.HttpProtocol"

```

--------------------------------

### Start Docker Compose Services

Source: https://github.com/apache/stormcrawler/blob/main/docs/src/main/asciidoc/quick-start.adoc

Starts the services defined in your Docker Compose file in detached mode. Ensure network connectivity between services.

```shell
docker compose up -d
```

--------------------------------

### Robots.txt Directives Example

Source: https://github.com/apache/stormcrawler/wiki/User-Agent-Configuration

Illustrates common directives in a robots.txt file, including User-Agent, Disallow, and Allow rules. The '*' in User-Agent matches any agent not explicitly specified.

```robots.txt
User-Agent: *
Disallow: *.gif$
Disallow: /example/
Allow: /publications/
```

--------------------------------

### Flux Topology Definition Example

Source: https://context7.com/apache/stormcrawler/llms.txt

An example of a Flux topology definition in YAML format. This allows for declarative configuration of StormCrawler topologies without recompiling Java code.

```yaml

```

--------------------------------

### AdaptiveScheduler Configuration

Source: https://github.com/apache/stormcrawler/blob/main/docs/src/main/asciidoc/internals.adoc

Configuration example for the AdaptiveScheduler. Adjusts fetch intervals based on page content changes.

```yaml
scheduler.class: "org.apache.stormcrawler.persistence.AdaptiveScheduler"
scheduler.adaptive.setLastModified: true
scheduler.adaptive.fetchInterval.min: 60
scheduler.adaptive.fetchInterval.max: 20160
```

--------------------------------

### Add PGP Key to KEYS File

Source: https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Examples of how to add your PGP key to the KEYS file in the Apache distribution repository. Ensure your PGP fingerprint is registered on id.apache.org.

```bash
pgp -kxa <your name> and append it to this file.
```

```bash
(pgpk -ll <your name> && pgpk -xa <your name>) >> this file.
```

```bash
(gpg --list-sigs <your name>
&& gpg --armor --export <your name>) >> this file.
```

--------------------------------

### Verify Source Artifact Signature (Bash)

Source: https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Use this bash script to download release artifacts and verify their signatures using GPG. Ensure you have GPG installed and the KEYS file is available.

```bash
#!/bin/bash

mkdir /tmp/test
cd /tmp/test
curl -s -O https://dist.apache.org/repos/dist/release/stormcrawler/KEYS
curl -s -O https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-x.y.z-RC1/apache-stormcrawler-x.y.z-source-release.tar.gz
curl -s -O https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-x.y.z-RC1/apache-stormcrawler-x.y.z-source-release.tar.gz.asc

echo "
list keys
"
gpg --homedir . --list-keys

echo "
import KEYS file
"
gpg --homedir . --import KEYS

echo "
verify signature
"
gpg --homedir . --output  apache-stormcrawler-x.y.z-source-release.tar.gz --decrypt apache-stormcrawler-x.y.z-source-release.tar.gz.asc

```

--------------------------------

### Build StormCrawler from Source

Source: https://github.com/apache/stormcrawler/blob/main/README.md

Build the StormCrawler project from source using Maven. Ensure you have JDK 17+ and Apache Maven 3 installed.

```shell
mvn clean install

```

--------------------------------

### Submit StormCrawler Topology

Source: https://github.com/apache/stormcrawler/wiki/Start/Configuration

Submit your StormCrawler topology to Storm using the 'storm jar' command. This example includes passing a custom configuration file and running in local mode.

```bash
storm jar path/to/allmycode.jar org.me.MyCrawlTopology -conf my-crawler-conf.yaml -local
```

--------------------------------

### Start Standalone Chrome for Debugging

Source: https://github.com/apache/stormcrawler/blob/main/external/playwright/README.md

Launch Chrome with remote debugging enabled. This allows Playwright to connect to an existing browser instance.

```bash
./chrome --remote-debugging-port=9222 --user-data-dir=remote-profile --headless &

```

```bash
chrome --allow-pre-commit-input --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-gpu --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-software-rasterizer --disable-sync --dns-prefetch-disable --enable-automation --enable-logging --headless --log-level=0 --mute-audio --no-first-run --no-sandbox --no-service-autorun --password-store=basic --remote-debugging-port=9222 --use-mock-keychain --user-data-dir=/tmp/.org.chromium.Chromium.CPTbw3 data:,

```

--------------------------------

### Start Crawl Topology in Local Mode

Source: https://github.com/apache/stormcrawler/blob/main/external/opensearch/archetype/src/main/resources/archetype-resources/README.md

Initiate the crawl topology in local mode using URLs from seeds.txt. The --local-ttl flag sets the topology's runtime.

```sh
storm local target/${artifactId}-${version}.jar  org.apache.storm.flux.Flux injection.flux --local-ttl 3600
```

--------------------------------

### RegexURLFilter Configuration (JSON)

Source: https://github.com/apache/stormcrawler/wiki/Components/Filters/URLFilters

Configure the RegexURLFilter using a JSON array of rules. The first matching rule determines if a URL is kept or discarded. Rules starting with '-' discard, and those starting with '+' keep.

```json
{
    "urlFilters": [
        "-^(file|ftp|mailto):",
        "+."
    ]
}
```

--------------------------------

### Perform Trial Maven Release Build

Source: https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Conduct a trial build of the release artifacts using Maven with the 'apache-release' and 'apache-gpg' profiles.

```bash
mvn package -Papache-release,apache-gpg
```

--------------------------------

### Instantiate WARCSpout in TopologyBuilder

Source: https://github.com/apache/stormcrawler/blob/main/external/warc/README.md

Set up the WARCSpout using TopologyBuilder, specifying the input directory and a pattern for WARC files. This is for programmatic topology creation.

```java
TopologyBuilder builder = new TopologyBuilder();

builder.setSpout("spout", new WARCSpout("input/", "*.{paths,txt}"));
```

--------------------------------

### Run URLFrontier Docker Container

Source: https://github.com/apache/stormcrawler/blob/main/archetype/src/main/resources/archetype-resources/README.md

Launches a Docker container for URLFrontier. Ensure URLFrontier is accessible.

```bash
docker pull crawlercommons/url-frontier
docker run --rm --name frontier -p 7071:7071  crawlercommons/url-frontier
```

--------------------------------

### Using the Metadata Class in Java

Source: https://context7.com/apache/stormcrawler/llms.txt

Demonstrates the creation, population, and retrieval of data using StormCrawler's Metadata class. It shows how to set single or multiple values, check for key existence, and merge metadata with a prefix.

```java
import org.apache.stormcrawler.Metadata;

// Create and populate
Metadata meta = new Metadata();
meta.setValue("custom.tag", "news");
meta.setValues("parse.keywords", new String[]{"crawler", "storm", "apache"});

// Retrieve values
String tag = meta.getFirstValue("custom.tag");           // "news"
String[] keywords = meta.getValues("parse.keywords");    // ["crawler", "storm", "apache"]

// Check presence
boolean hasTag = meta.containsKey("custom.tag");         // true
boolean isNews = meta.containsKeyWithValue("custom.tag", "news"); // true

// Merge with a prefix (avoids key collisions when combining metadata sources)
Metadata httpHeaders = new Metadata();
httpHeaders.setValue("Content-Type", "text/html");
meta.putAll(httpHeaders, "http.response.");
// Now meta has: "http.response.Content-Type" = "text/html"

// Useful well-known keys set by StormCrawler internally:
//   fetch.statusCode       — HTTP status code (e.g., "200")
//   fetch.error.count      — cumulative fetch error count
//   error.cause            — exception message on errors
//   error.source           — bolt that raised the error
//   isSitemap              — "true" if document is a sitemap
//   isFeed                 — "true" if document is an RSS/Atom feed
//   _redirTo               — target URL on HTTP redirect
//   depth                  — crawl depth from seed URL
//   url.path               — list of ancestor URLs (breadcrumb trail)

```

--------------------------------

### Create Seed URLs File

Source: https://github.com/apache/stormcrawler/blob/main/external/opensearch/archetype/src/main/resources/archetype-resources/README.md

Create a seeds.txt file and populate it with the initial URLs for the crawl.

```sh
echo "http://stormcrawler.net/" > seeds.txt
```

--------------------------------

### Configure RawLocalFileSystem for WARCHdfsBolt

Source: https://github.com/apache/stormcrawler/blob/main/external/warc/README.md

Configure the WARCHdfsBolt to use RawLocalFileSystem for proper content syncing with the local file system.

```java
  warcbolt.withConfigKey("warc");
  Map<String, Object> hdfsConf = new HashMap<>();
  hdfsConf.put("fs.file.impl", "org.apache.hadoop.fs.RawLocalFileSystem");
  getConf().put("warc", hdfsConf);
```

--------------------------------

### Run URLFrontier Service with Docker

Source: https://github.com/apache/stormcrawler/blob/main/external/urlfrontier/README.md

Use these Docker commands to pull the image and run the URLFrontier service locally. Ensure the service is accessible on port 7071.

```bash
docker pull crawlercommons/url-frontier:1.0
```

```bash
docker run --rm --name frontier -p 7071:7071 crawlercommons/url-frontier:1.0
```

--------------------------------

### Add WARC Module Dependency

Source: https://github.com/apache/stormcrawler/blob/main/external/warc/README.md

Include this dependency in your project's pom.xml to use the WARC module. Ensure the version matches your StormCrawler installation.

```xml
<dependency>
			<groupId>org.apache.stormcrawler</groupId>
			<artifactId>stormcrawler-warc</artifactId>
			<version>${stormcrawler.version}</version>
		</dependency>
```

--------------------------------

### Configure Built-in ParseFilters

Source: https://context7.com/apache/stormcrawler/llms.txt

Example configuration for common built-in ParseFilters in StormCrawler. These filters are applied after document parsing to extract or modify metadata and outlinks.

```json
// parsefilters.json — example with common built-in filters
{
  "org.apache.stormcrawler.parse.ParseFilters": [
    {
      "class": "org.apache.stormcrawler.parse.filter.DomainParseFilter",
      "name": "DomainParseFilter",
      "params": { "key": "domain", "byHost": false }
    },
    {
      "class": "org.apache.stormcrawler.parse.filter.MimeTypeNormalization",
      "name": "MimeTypeNormalization"
      // Converts MIME types into human-readable values: pdf, html, image, etc.
    },
    {
      "class": "org.apache.stormcrawler.parse.filter.CommaSeparatedToMultivaluedMetadata",
      "name": "CommaSeparatedToMultivaluedMetadata",
      "params": { "keys": ["parse.keywords"] }
    },
    {
      "class": "org.apache.stormcrawler.parse.filter.CollectionTagger",
      "name": "CollectionTagger",
      "params": { "file": "collections.json" }
      // Tags documents with collection names based on URL pattern matching
    },
    {
      "class": "org.apache.stormcrawler.parse.filter.MD5SignatureParseFilter",
      "name": "MD5SignatureParseFilter",
      "params": { "useText": true }
      // Generates MD5 signature from text for change detection
    }
  ]
}
```

--------------------------------

### Prepare Maven Release

Source: https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Prepare the release using Maven's release plugin. This command creates and pushes version update commits, tags the release, and pushes the tag. Answer the prompts carefully, using 'stormcrawler-x.y.z' for the tag name.

```bash
mvn release:prepare -Papache-release,apache-gpg
```

--------------------------------

### CollectionTagger ParseFilter Configuration

Source: https://github.com/apache/stormcrawler/blob/main/docs/src/main/asciidoc/internals.adoc

Example JSON configuration for the CollectionTagger ParseFilter. It allows defining collections with include and exclude URL patterns for tagging documents.

```json
{
  "collections": [
    {
      "name": "stormcrawler",
      "includePatterns": ["https://stormcrawler.net/.+"]
    },
    {
      "name": "crawler",
      "includePatterns": [".+crawler.+", ".+nutch.+"],
      "excludePatterns": [".+baby.+", ".+spider.+"]
    }
  ]
}
```

--------------------------------

### Emit on Status Stream from Bolt

Source: https://context7.com/apache/stormcrawler/llms.txt

Example of emitting a tuple on the status stream with a specific URL, metadata, and status (e.g., Status.ERROR). Acknowledges the input tuple.

```java
// Emitting on the status stream from a bolt's execute():
collector.emit(
    org.apache.stormcrawler.Constants.StatusStreamName,
    tuple,
    new Values(url, metadata, Status.ERROR)
);
collector.ack(tuple);
```

--------------------------------

### Add Hadoop AWS Dependency

Source: https://github.com/apache/stormcrawler/blob/main/external/warc/README.md

Include the hadoop-aws dependency in your project to enable access to WARC files on AWS S3. Match the version with your Apache Storm installation.

```xml
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-aws</artifactId>
  <version>2.10.1</version>
</dependency>
```

--------------------------------

### Run Apache Rat License Check

Source: https://context7.com/apache/stormcrawler/llms.txt

Execute the Apache Rat ( a license checker ) verification for the project.

```bash
mvn verify -Prat
```

--------------------------------

### MultiProxyManager Configuration

Source: https://github.com/apache/stormcrawler/wiki/Proxies

Configure multiple proxies by providing a file path via http.proxy.file. The TXT file should contain connection strings for all proxies and must be accessible to all topology nodes.

```properties
http.proxy.file
```

--------------------------------

### Setting Custom HTTP Headers via Metadata

Source: https://github.com/apache/stormcrawler/wiki/Components/Protocol/HTTPProtocol

Demonstrates how to set custom HTTP headers for a request using metadata. The 'protocol.set-header' key expects a list of 'header=value' strings.

```text
"protocol%2Eset-header": [
"header1=value1",
"header2=value2"
]
```

--------------------------------

### Build StormCrawler with Tests

Source: https://context7.com/apache/stormcrawler/llms.txt

Command to build all StormCrawler modules, including running integration tests. Requires JDK 17+, Maven 3, and Docker.

```shell
# Requirements: JDK 17+, Apache Maven 3, Docker (for integration tests)

# Build all modules including tests
mvn clean install
```

--------------------------------

### Docker Compose for StormCrawler Services

Source: https://github.com/apache/stormcrawler/blob/main/docs/src/main/asciidoc/quick-start.adoc

This Docker Compose configuration sets up Zookeeper, Storm Nimbus, Supervisor, UI, and URLFrontier. Ensure Docker is installed and running. Adjust ports if necessary.

```yaml
services:
  zookeeper:
    image: zookeeper:3.9.3
    container_name: zookeeper
    restart: always

  nimbus:
    image: storm:latest
    container_name: nimbus
    hostname: nimbus
    command: storm nimbus
    depends_on:
      - zookeeper
    restart: always

  supervisor:
    image: storm:latest
    container_name: supervisor
    command: storm supervisor -c worker.childopts=-Xmx2g
    depends_on:
      - nimbus
      - zookeeper
    restart: always

  ui:
    image: storm:latest
    container_name: ui
    command: storm ui
    depends_on:
      - nimbus
    restart: always
    ports:
      - "127.0.0.1:8080:8080"

  urlfrontier:
    image: crawlercommons/url-frontier:latest
    container_name: urlfrontier
    restart: always
    ports:
      - "127.0.0.1:7071:7071"
```

--------------------------------

### Perform Maven Release

Source: https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Execute the actual release using Maven. This will create a staged repository on repository.apache.org. After verification, close the staging repository.

```bash
mvn release:perform -Papache-gpg
```

--------------------------------

### Solr Configuration for URL Diversity

Source: https://github.com/apache/stormcrawler/blob/main/external/solr/archetype/src/main/resources/archetype-resources/README.md

Configure SolrSpout to fetch URLs with diversity using collapse and expand features. This balances the amount of URLs processed per bucket, for example, per domain.

```yaml
solr.status.bucket.field: host
solr.status.bucket.maxsize: 100
```

--------------------------------

### Implement Custom Protocol in Java

Source: https://github.com/apache/stormcrawler/blob/main/docs/src/main/asciidoc/extending.adoc

Implement the Protocol interface to define custom content fetching logic. Implement getProtocolOutput and getRobotRules. Register in configuration YAML.

```java
import org.apache.stormcrawler.protocol.Protocol;
import org.apache.stormcrawler.protocol.ProtocolResponse;
import org.apache.stormcrawler.Metadata;
import crawlercommons.robots.BaseRobotRules;
import org.apache.storm.Config;

public class MyProtocol implements Protocol {

    @Override
    public void configure(Config conf) {
        // Initialize from Storm configuration
    }

    @Override
    public ProtocolResponse getProtocolOutput(String url, Metadata metadata)
            throws Exception {
        byte[] content = /* fetch content */;
        int statusCode = 200;
        Metadata responseMeta = new Metadata();
        return new ProtocolResponse(content, statusCode, responseMeta);
    }

    @Override
    public BaseRobotRules getRobotRules(String url) {
        return RobotRulesParser.EMPTY_RULES;
    }

    @Override
    public void cleanup() {}
}
```

```yaml
http.protocol.implementation: "com.example.MyProtocol"
https.protocol.implementation: "com.example.MyProtocol"
```

--------------------------------

### Run Maven Tests

Source: https://github.com/apache/stormcrawler/blob/main/RELEASING.md

Execute all tests using Maven. Ensure a working Docker environment is available, as it's required for coverage computation and can cause build failures.

```bash
mvn test
```

--------------------------------

### Implement a Custom ParseFilter in Java

Source: https://context7.com/apache/stormcrawler/llms.txt

Example of creating a custom ParseFilter in Java. This filter demonstrates how to configure parameters and access the DOM to extract specific information, such as an author's name from meta tags.

```java
package com.example.crawler.filter;

import com.fasterxml.jackson.databind.JsonNode;
import org.apache.stormcrawler.Metadata;
import org.apache.stormcrawler.parse.ParseFilter;
import org.apache.stormcrawler.parse.ParseResult;
import org.w3c.dom.DocumentFragment;
import java.util.Map;

public class AuthorExtractFilter extends ParseFilter {

    private String metaKey;

    @Override
    public void configure(Map<String, Object> stormConf, JsonNode filterParams) {
        // Read the metadata key name from JSON config
        metaKey = filterParams.path("key").asText("author");
    }

    @Override
    public boolean needsDOM() {
        return true; // request DOM structure to be built
    }

    @Override
    public void filter(String url, byte[] content, DocumentFragment doc, ParseResult parse) {
        Metadata metadata = parse.get(url).getMetadata();
        // Extract <meta name="author" content="..."/> using the DOM
        // (real implementation would walk the DOM)
        String author = extractAuthor(doc);
        if (author != null) {
            metadata.setValue(metaKey, author);
        }
    }

    private String extractAuthor(DocumentFragment doc) {
        // DOM traversal logic...
        return null;
    }
}
```

```json
// Register in parsefilters.json:
{
  "class": "com.example.crawler.filter.AuthorExtractFilter",
  "name": "AuthorExtractFilter",
  "params": { "key": "parse.author" }
}
```

--------------------------------

### Enable Pre-commit Format Hooks

Source: https://github.com/apache/stormcrawler/blob/main/README.md

Enable pre-commit format hooks for your project by running this Maven command. This ensures code is formatted automatically before commits.

```shell
mvn clean install -Dskip.format.code=false

```

--------------------------------

### Configure WARCHdfsBolt in Java

Source: https://github.com/apache/stormcrawler/blob/main/external/warc/README.md

Configure the WARCHdfsBolt for WARC file generation, including path, metadata, request records, filesystem URL, and rotation policy.

```java
        String warcFilePath = "/warc";

        FileNameFormat fileNameFormat = new WARCFileNameFormat()
                .withPath(warcFilePath);

        Map<String,String> fields = new HashMap<>();
        fields.put("software:", "Apache StormCrawler 3.4.0 http://stormcrawler.net/");
        fields.put("format", "WARC File Format 1.0");
        fields.put("conformsTo:",
                "https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/");
        
        WARCHdfsBolt warcbolt = (WARCHdfsBolt) new WARCHdfsBolt ()
                .withFileNameFormat(fileNameFormat);
        warcbolt.withHeader(fields);

        // enable WARC request records
        warcbolt.withRequestRecords();

        // can specify the filesystem - will use the local FS by default
        String fsURL = "hdfs://localhost:9000";
        warcbolt.withFsUrl(fsURL);
        
        // a custom max length can be specified - 1 GB will be used as a default
        FileSizeRotationPolicy rotpol = new FileSizeRotationPolicy(50.0f,
                Units.MB);
        warcbolt.withRotationPolicy(rotpol);

        builder.setBolt("warc", warcbolt).localOrShuffleGrouping("fetch");
```