crawler4j

https://github.com/yasserg/crawler4j
Admin
crawler4j is an open source web crawler for Java that provides a simple interface for setting up...
Tokens:14,380
Snippets:122
Trust Score:7
License:Apache-2.0
Update:6 months ago
Show doc for...
Tokens:
Raw
### Initialize and Start CrawlController

Source: https://context7.com/yasserg/crawler4j/llms.txt

Demonstrates how to initialize CrawlController with basic configurations, add seed URLs, and start the crawling process with a specified number of threads. Ensure the crawl storage folder exists.

```java
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.CrawlController;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtServer;

public class CrawlControllerExample {
    public static void main(String[] args) throws Exception {
        // Configure the crawler
        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder("/tmp/crawler4j/");
        config.setPolitenessDelay(1000);  // 1 second between requests
        config.setMaxDepthOfCrawling(2);
        config.setMaxPagesToFetch(1000);

        // Initialize page fetcher and robots.txt handler
        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);

        // Create the controller
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        // Add seed URLs (starting points for crawling)
        controller.addSeed("https://www.example.com/");
        controller.addSeed("https://www.example.com/blog/");
        controller.addSeed("https://www.example.com/products/");

        // Start crawling with 8 concurrent threads (blocking)
        int numberOfCrawlers = 8;
        controller.start(MyCrawler.class, numberOfCrawlers);

        System.out.println("Crawling completed!");
    }
}
```

--------------------------------

### Configure and start the CrawlController

Source: https://github.com/yasserg/crawler4j/wiki/Quickstart

Set up storage, thread count, and seed URLs, then initiate the crawling process.

```java
    public class Controller {
        public static void main(String[] args) throws Exception {
            String crawlStorageFolder = "/data/crawl/root";
            int numberOfCrawlers = 7;

            CrawlConfig config = new CrawlConfig();
            config.setCrawlStorageFolder(crawlStorageFolder);

            /*
             * Instantiate the controller for this crawl.
             */
            PageFetcher pageFetcher = new PageFetcher(config);
            RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
            RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
            CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

            /*
             * For each crawl, you need to add some seed urls. These are the first
             * URLs that are fetched and then the crawler starts following links
             * which are found in these pages
             */
            controller.addSeed("http://www.ics.uci.edu/~lopes/");
            controller.addSeed("http://www.ics.uci.edu/~welling/");
            controller.addSeed("http://www.ics.uci.edu/");

            /*
             * Start the crawl. This is a blocking operation, meaning that your code
             * will reach the line after this only when crawling is finished.
             */
            controller.start(MyCrawler.class, numberOfCrawlers);
        }
    }
```

--------------------------------

### ImageCrawlerExample: Crawling and Saving Binary Content

Source: https://context7.com/yasserg/crawler4j/llms.txt

This example shows how to set up a crawler to specifically visit and save image files larger than 10KB. It configures the crawler to include binary content and defines custom logic for visiting and processing image URLs.

```java
import com.google.common.io.Files;
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.CrawlController;
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
import edu.uci.ics.crawler4j.parser.BinaryParseData;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtServer;
import edu.uci.ics.crawler4j.url.WebURL;

import java.io.File;
import java.io.IOException;
import java.util.UUID;
import java.util.regex.Pattern;

public class ImageCrawlerExample {

    public static class ImageCrawler extends WebCrawler {
        private static final Pattern FILTERS = Pattern.compile(
            ".*(\.(css|js|mp3|mp4|zip|gz|pdf))$);
        private static final Pattern IMG_PATTERNS = Pattern.compile(
            ".*(\.(bmp|gif|jpe?g|png|tiff?))$);

        private final File storageFolder;
        private final String targetDomain;

        public ImageCrawler(File storageFolder, String targetDomain) {
            this.storageFolder = storageFolder;
            this.targetDomain = targetDomain;
        }

        @Override
        public boolean shouldVisit(Page referringPage, WebURL url) {
            String href = url.getURL().toLowerCase();

            // Skip non-useful files
            if (FILTERS.matcher(href).matches()) {
                return false;
            }

            // Always visit images
            if (IMG_PATTERNS.matcher(href).matches()) {
                return true;
            }

            // Follow links within target domain
            return href.startsWith(targetDomain);
        }

        @Override
        public void visit(Page page) {
            String url = page.getWebURL().getURL();

            // Only process images larger than 10KB
            if (!IMG_PATTERNS.matcher(url.toLowerCase()).matches()) {
                return;
            }

            if (!(page.getParseData() instanceof BinaryParseData)) {
                return;
            }

            if (page.getContentData().length < 10 * 1024) {
                return;  // Skip small images
            }

            // Generate unique filename
            String extension = url.substring(url.lastIndexOf('.'));
            String filename = UUID.randomUUID() + extension;
            File outputFile = new File(storageFolder, filename);

            try {
                Files.write(page.getContentData(), outputFile);
                System.out.println("Saved: " + url + " -> " + filename +
                    " (" + page.getContentData().length + " bytes)");
            } catch (IOException e) {
                System.err.println("Failed to save: " + url + " - " + e.getMessage());
            }
        }
    }

    public static void main(String[] args) throws Exception {
        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder("/tmp/crawler4j/");
        config.setIncludeBinaryContentInCrawling(true);  // Enable binary content
        config.setMaxPagesToFetch(500);
        config.setPolitenessDelay(500);

        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        controller.addSeed("https://www.example.com/gallery/");

        File imageFolder = new File("/tmp/downloaded_images/");
        imageFolder.mkdirs();
        String domain = "https://www.example.com";

        CrawlController.WebCrawlerFactory<ImageCrawler> factory =
            () -> new ImageCrawler(imageFolder, domain);

        controller.start(factory, 4);

        System.out.println("Image crawling completed!");
    }
}
```

--------------------------------

### Install crawler4j via Build Tools

Source: https://context7.com/yasserg/crawler4j/llms.txt

Add the crawler4j dependency to your project using Maven or Gradle.

```xml
<!-- Maven -->
<dependency>
    <groupId>edu.uci.ics</groupId>
    <artifactId>crawler4j</artifactId>
    <version>4.4.0</version>
</dependency>
```

```groovy
// Gradle
compile group: 'edu.uci.ics', name: 'crawler4j', version: '4.4.0'
```

--------------------------------

### Accessing Page and HTML Data in Crawler4j

Source: https://context7.com/yasserg/crawler4j/llms.txt

This example shows how to access various properties of a fetched page, including URL information, status codes, headers, raw content, and parsed HTML data like text, title, meta tags, and outgoing URLs. It requires extending the WebCrawler class and overriding the visit method.

```java
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.parser.HtmlParseData;
import edu.uci.ics.crawler4j.url.WebURL;
import org.apache.http.Header;

import java.util.Map;
import java.util.Set;

public class PageDataExample extends WebCrawler {

    @Override
    public boolean shouldVisit(Page referringPage, WebURL url) {
        return url.getURL().startsWith("https://www.example.com/");
    }

    @Override
    public void visit(Page page) {
        // === WebURL Information ===
        WebURL webUrl = page.getWebURL();
        System.out.println("URL: " + webUrl.getURL());
        System.out.println("Document ID: " + webUrl.getDocid());
        System.out.println("Domain: " + webUrl.getDomain());
        System.out.println("Subdomain: " + webUrl.getSubDomain());
        System.out.println("Path: " + webUrl.getPath());
        System.out.println("Depth: " + webUrl.getDepth());
        System.out.println("Parent URL: " + webUrl.getParentUrl());
        System.out.println("Anchor Text: " + webUrl.getAnchor());

        // === Page Status and Headers ===
        System.out.println("Status Code: " + page.getStatusCode());
        System.out.println("Content Type: " + page.getContentType());
        System.out.println("Content Charset: " + page.getContentCharset());
        System.out.println("Content Encoding: " + page.getContentEncoding());
        System.out.println("Is Redirect: " + page.isRedirect());
        System.out.println("Redirected To: " + page.getRedirectedToUrl());

        // Response headers
        Header[] headers = page.getFetchResponseHeaders();
        System.out.println("Response Headers:");
        for (Header header : headers) {
            System.out.println("  " + header.getName() + ": " + header.getValue());
        }

        // === Raw Content ===
        byte[] contentData = page.getContentData();
        System.out.println("Content Size: " + contentData.length + " bytes");
        System.out.println("Content Truncated: " + page.isTruncated());

        // === Parsed HTML Data ===
        if (page.getParseData() instanceof HtmlParseData) {
            HtmlParseData htmlData = (HtmlParseData) page.getParseData();

            // Text content (no HTML tags)
            String text = htmlData.getText();
            System.out.println("Text Length: " + text.length());

            // Full HTML
            String html = htmlData.getHtml();
            System.out.println("HTML Length: " + html.length());

            // Page title
            String title = htmlData.getTitle();
            System.out.println("Title: " + title);

            // Meta tags
            Map<String, String> metaTags = htmlData.getMetaTags();
            System.out.println("Meta Tags:");
            for (Map.Entry<String, String> entry : metaTags.entrySet()) {
                System.out.println("  " + entry.getKey() + ": " + entry.getValue());
            }

            // Get specific meta tag value
            String robots = htmlData.getMetaTagValue("robots");
            String description = htmlData.getMetaTagValue("description");

            // Outgoing links
            Set<WebURL> outgoingUrls = htmlData.getOutgoingUrls();
            System.out.println("Outgoing Links: " + outgoingUrls.size());
            for (WebURL outUrl : outgoingUrls) {
                System.out.println("  -> " + outUrl.getURL());
                System.out.println("     Tag: " + outUrl.getTag());
                System.out.println("     Anchor: " + outUrl.getAnchor());
            }
        }
    }
}
```

--------------------------------

### Initialization Function

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Initializes the page by fetching names and page data, then calling the show function when both are ready.

```javascript
function init() { var json function ready () { names && json && show(json) } fetchNames(function(data) { names = data.split(/\r?\n/); ready()}) fetchPage(function(data) { json = JSON.parse(data); ready()})
```

--------------------------------

### Configure RobotstxtConfig in crawler4j

Source: https://context7.com/yasserg/crawler4j/llms.txt

Demonstrates how to instantiate and configure RobotstxtConfig, including setting the user-agent, cache size, and compliance settings before initializing the CrawlController.

```java
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.CrawlController;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtServer;

public class RobotstxtExample {
    public static void main(String[] args) throws Exception {
        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder("/tmp/crawler4j/");

        PageFetcher pageFetcher = new PageFetcher(config);

        // Configure robots.txt handling
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();

        // Enable/disable robots.txt compliance (default: true)
        robotstxtConfig.setEnabled(true);

        // User-agent name for robots.txt rule matching
        robotstxtConfig.setUserAgentName("MyCrawler");

        // Cache size for robots.txt files (default: 500)
        robotstxtConfig.setCacheSize(1000);

        // Ignore user-agent discrimination (use Allow rules for all bots)
        robotstxtConfig.setIgnoreUADiscrimination(false);

        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        // The controller will now respect robots.txt rules
        controller.addSeed("https://www.example.com/");

        // If robots.txt disallows a seed URL, you'll see a warning:
        // "Robots.txt does not allow this seed: ..."
    }
}
```

--------------------------------

### Implement Controller class in Java

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Configure the crawl environment and initiate the crawling process using CrawlController.

```java
public class Controller {
    public static void main(String[] args) throws Exception {
        String crawlStorageFolder = "/data/crawl/root";
        int numberOfCrawlers = 7;

        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder(crawlStorageFolder);

        // Instantiate the controller for this crawl.
        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        // For each crawl, you need to add some seed urls. These are the first
        // URLs that are fetched and then the crawler starts following links
        // which are found in these pages
        controller.addSeed("https://www.ics.uci.edu/~lopes/");
        controller.addSeed("https://www.ics.uci.edu/~welling/");
    	controller.addSeed("https://www.ics.uci.edu/");
    	
    	// The factory which creates instances of crawlers.
        CrawlController.WebCrawlerFactory<BasicCrawler> factory = MyCrawler::new;
        
        // Start the crawl. This is a blocking operation, meaning that your code
        // will reach the line after this only when crawling is finished.
        controller.start(factory, numberOfCrawlers);
    }
}
```

--------------------------------

### Create Advanced CrawlConfig

Source: https://context7.com/yasserg/crawler4j/llms.txt

Illustrates how to create a CrawlConfig object with advanced settings, including storage, politeness, content processing, proxy, authentication, and error handling. This configuration is useful for fine-tuning crawler behavior.

```java
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.authentication.BasicAuthInfo;
import org.apache.http.message.BasicHeader;

import java.util.Arrays;

public class CrawlConfigExample {
    public static CrawlConfig createAdvancedConfig() throws Exception {
        CrawlConfig config = new CrawlConfig();

        // Storage and resumability
        config.setCrawlStorageFolder("/data/crawl/storage");
        config.setResumableCrawling(true);  // Resume after crash/stop

        // Crawl limits
        config.setMaxDepthOfCrawling(5);      // Max link depth (-1 for unlimited)
        config.setMaxPagesToFetch(10000);     // Max pages (-1 for unlimited)
        config.setMaxOutgoingLinksToFollow(5000);  // Links per page

        // Politeness and timing
        config.setPolitenessDelay(500);       // 500ms between requests to same host
        config.setSocketTimeout(20000);       // 20 second socket timeout
        config.setConnectionTimeout(30000);   // 30 second connection timeout

        // Content settings
        config.setIncludeHttpsPages(true);
        config.setIncludeBinaryContentInCrawling(true);   // Fetch images, PDFs, etc.
        config.setProcessBinaryContentInCrawling(true);   // Parse binary with Tika
        config.setMaxDownloadSize(1048576);   // 1MB max page size

        // Proxy configuration
        config.setProxyHost("proxy.example.com");
        config.setProxyPort(8080);
        config.setProxyUsername("user");
        config.setProxyPassword("pass");

        // Custom user agent
        config.setUserAgentString("MyBot/1.0 (https://example.com/bot)");

        // Custom headers
        config.setDefaultHeaders(Arrays.asList(
            new BasicHeader("Accept-Language", "en-US"),
            new BasicHeader("Accept", "text/html,application/xhtml+xml")
        ));

        // Respect nofollow/noindex meta tags
        config.setRespectNoFollow(true);
        config.setRespectNoIndex(true);

        // Basic authentication for protected pages
        config.addAuthInfo(new BasicAuthInfo("username", "password",
            "https://protected.example.com/"));

        // Error handling
        config.setHaltOnError(false);  // Continue on errors

        return config;
    }
}
```

--------------------------------

### Configure Docker Location for Maven Surefire Tests

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j-examples/crawler4j-examples-postgres/README.md

If Maven Surefire tests fail on Windows due to 'docker.exe' or 'docker-compose.exe' not being found, create a '.mvn/maven.config' file at the project root. Ensure the paths specified in 'docker.location' and 'docker.compose.location' are correct for your system.

```properties
"-Ddocker.location=C:\\Program Files\\Docker Toolbox\\docker.exe "-Ddocker.compose.location=C:\\Program Files\\Docker Toolbox\\docker-compose.exe
```

--------------------------------

### Configure proxy settings

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Sets proxy host and port for the crawler.

```java
crawlConfig.setProxyHost("proxyserver.example.com");
crawlConfig.setProxyPort(8080);
```

--------------------------------

### Enable binary content crawling

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Enables the fetching of binary files like images and audio.

```java
crawlConfig.setIncludeBinaryContentInCrawling(true);
```

--------------------------------

### Urchin Tracker Initialization

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Initializes the Urchin (Google Analytics) tracker with account ID and domain.

```javascript
_uacct = "UA-2377314-2";_udn="c2.com"; urchinTracker();
```

--------------------------------

### Fetch Page Names

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Fetches the list of page names by calling loadDoc with the 'names.txt' URL and a callback function.

```javascript
function fetchNames (done) { loadDoc('names.txt', done) }
```

--------------------------------

### Implement Non-Blocking Crawl and Graceful Shutdown

Source: https://context7.com/yasserg/crawler4j/llms.txt

Utilize `startNonBlocking()` for asynchronous crawling and `shutdown()` for a controlled termination. This is ideal for time-sensitive operations or interactive applications where the main thread needs to remain responsive.

```java
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.CrawlController;
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtServer;
import edu.uci.ics.crawler4j.url.WebURL;

public class NonBlockingCrawlExample {

    public static class SimpleCrawler extends WebCrawler {
        @Override
        public boolean shouldVisit(Page referringPage, WebURL url) {
            return url.getURL().startsWith("https://www.example.com/");
        }

        @Override
        public void visit(Page page) {
            System.out.println("Crawled: " + page.getWebURL().getURL());
        }
    }

    public static void main(String[] args) throws Exception {
        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder("/tmp/crawler4j/");
        config.setPolitenessDelay(1000);
        config.setMaxPagesToFetch(-1);  // Unlimited

        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        controller.addSeed("https://www.example.com/");

        // Start non-blocking - returns immediately
        controller.startNonBlocking(SimpleCrawler.class, 4);

        System.out.println("Crawling started in background...");

        // Let it run for 60 seconds
        Thread.sleep(60 * 1000);

        // Gracefully shutdown
        System.out.println("Initiating shutdown...");
        controller.shutdown();

        // Wait for all threads to finish
        controller.waitUntilFinish();

        System.out.println("Crawling finished!");
    }
}
```

--------------------------------

### Configure proxy authentication

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Sets credentials for authenticated proxy access.

```java
crawlConfig.setProxyUsername(username);
crawlConfig.setProxyPassword(password);
```

--------------------------------

### Enable SSL crawling

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Configures the crawler to include HTTPS pages.

```java
CrawlConfig config = new CrawlConfig();

config.setIncludeHttpsPages(true);
```

--------------------------------

### Set maximum pages to fetch

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Sets a limit on the total number of pages to crawl.

```java
crawlConfig.setMaxPagesToFetch(maxPagesToFetch);
```

--------------------------------

### Markup Text Rendering

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Splits text into lines, renders each line using the 'render' function, and joins them back with newlines. It also appends a completed empty string, likely for consistent formatting.

```javascript
function markup (text) { var lines = text.replace(/\\\\n/,' ').split(/\\n/) var expand = lines.map(render).join("\n") return expand + complete('') }
```

--------------------------------

### Origin Polyfill for HTTPS

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Provides a polyfill for window.location.origin for environments that might not support it, particularly for HTTPS URLs.

```javascript
if (!window.location.origin) { // https://github.com/WardCunningham/remodeling/issues/20 window.location.origin = window.location.protocol + '//' + window.location.hostname + (window.location.port ? (':' + window.location.port) : ''); }
```

--------------------------------

### Fetch Page Content

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Determines the page title from the URL and fetches the corresponding page content from the database using loadDoc.

```javascript
function fetchPage (done) { title = (location.search.match(/\w+/)||["WelcomeVisitors"])[0] var database = 'http://c2.com/wiki/remodel/pages/' loadDoc(database + title, done) }
```

--------------------------------

### Link and Media Embedding

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Functions to handle link stashing, YouTube embedding, and ISBN link generation.

```javascript
function links (text, sanitize) { // link conversion happens in four phases: // unexpected markers are adulterated // links are found, converted, and stashed away properly escaped // remaining text is processed and escaped // unique markers are replaced with unstashed links var stashed = [] function stash (text) { var here = stashed.length stashed.push(text) return "〖" + here + "〗" } function unstash (match, digits) { return stashed[+digits] } function internal (title) { if (names && names.indexOf(title)!=-1) { var url = location.origin + location.pathname + "?" + title return stash("<a href=\"" + url + "\">" + title + "</a>") } else { return title } } function external (url) { if (url.match(/\.(gif|jpg|jpeg|png)$/)) { return stash("<img src=\"" + url.replace(/^http:\/wiki\//,'http://c2.com/wiki/') + "\">") } else { return stash("<a href=\"" + url + "\" rel=\"nofollow\" target=\"_blank\">" + url + "</a>") } } function youtube (match, p1, p2) { var embed = "<object width=\"425\" height=\"344\">" + "<param name=\"movie\" value=\"http://www.youtube.com/v/$2&hl=en&fs=1&\"></param>" + "<param name=\"allowFullScreen\" value=\"true\"></param>" + "<param name=\"allowscriptaccess\" value=\"always\"></param>" + "<embed src=\"http://www.youtube.com/v/" + p2 + "&hl=en&fs=1&\" type=\"application/x-shockwave-flash\" allowscriptaccess=\"always\" allowfullscreen=\"true\" width=\"425\" height=\"344\"></embed>" + "</object>" return stash(embed) } function isbn (match, isbn) { var code = isbn.replace(/[- ]/g, "") if (code.match(/^\d{9}.$/)) { return stash("<a href=\"http://www.amazon.com/exec/obidos/ISBN=" + code + "/portlandpatternrA/\" rel=\"nofollow\">ISBN " + isbn + "</a>") } else { return "ISBN " + isbn } } function titlesearch () { function link(text) { return "<a href=?" + text + ">" + text + "</a>" } get = function() { var want = search.value if (want.length > 1) { var found = names.filter(function (e) { return e.includes(want) }) if (found.length) { return window.searchresult.innerHTML = found.slice(0,500).map(link).join('<br>') } } window.searchresult.innerHTML = '' } got = function (e) { if (!e) e = window.event; if ((e.keyCode || e.which) == '13') { search.value = '' window.searchresult.innerHTML = '' } } return stash("<input type=text id=search onInput='get()' onKeyPress='got()'><br><div id=searchresult></div>") } function fullsearch () { return stash("<form action=\"http://c2.com/cgi/fullSearch\"><input type=text name=search></form>") } var prepass = text .replace(/〖(\d+)〗/g, '〖 $1 〗') .replace(/^http:\/\/(www.)?youtube.com\/watch\?v=([-\w]+)/, youtube) .replace(/\[Search\]/, titlesearch) .replace(/\[Fullsearch\]/, fullsearch) .replace(/\[?ISBN:? *([0-9- xX]{10,})\]?/i, isbn) }
```

--------------------------------

### Implement a custom WebCrawler

Source: https://github.com/yasserg/crawler4j/wiki/Quickstart

Define crawling logic by overriding shouldVisit to filter URLs and visit to process fetched page content.

```java
     public class MyCrawler extends WebCrawler {

                private final static Pattern FILTERS = Pattern.compile(".*(\\.(css|js|gif|jpg|png|mp3|mp3|zip|gz))$");

                /**
                * This method receives two parameters. The first parameter is the page
                * in which we have discovered this new url and the second parameter is
                * the new url. You should implement this function to specify whether
                * the given url should be crawled or not (based on your crawling logic).
                * In this example, we are instructing the crawler to ignore urls that
                * have css, js, git, ... extensions and to only accept urls that start
                * with "http://www.ics.uci.edu/". In this case, we didn't need the
                * referringPage parameter to make the decision.
                */
                @Override
                public boolean shouldVisit(Page referringPage, WebURL url) {
                        String href = url.getURL().toLowerCase();
                        return !FILTERS.matcher(href).matches() && href.startsWith("http://www.ics.uci.edu/");
                }

                /**
                * This function is called when a page is fetched and ready
                * to be processed by your program.
                */
                @Override
                public void visit(Page page) {
                        String url = page.getWebURL().getURL();
                        System.out.println("URL: " + url);
        
                        if (page.getParseData() instanceof HtmlParseData) {
                                HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
                                String text = htmlParseData.getText();
                                String html = htmlParseData.getHtml();
                                Set<WebURL> links = htmlParseData.getOutgoingUrls();

                                System.out.println("Text length: " + text.length());
                                System.out.println("Html length: " + html.length());
                                System.out.println("Number of outgoing links: " + links.size());
                        }
                }
        }
```

--------------------------------

### Implement WebCrawler class in Java

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Extend WebCrawler to define URL filtering logic in shouldVisit and page processing logic in visit.

```java
public class MyCrawler extends WebCrawler {

    private final static Pattern FILTERS = Pattern.compile(".*(\\.(css|js|gif|jpg"
                                                           + "|png|mp3|mp4|zip|gz))$");

    /**
     * This method receives two parameters. The first parameter is the page
     * in which we have discovered this new url and the second parameter is
     * the new url. You should implement this function to specify whether
     * the given url should be crawled or not (based on your crawling logic).
     * In this example, we are instructing the crawler to ignore urls that
     * have css, js, git, ... extensions and to only accept urls that start
     * with "https://www.ics.uci.edu/". In this case, we didn't need the
     * referringPage parameter to make the decision.
     */
     @Override
     public boolean shouldVisit(Page referringPage, WebURL url) {
         String href = url.getURL().toLowerCase();
         return !FILTERS.matcher(href).matches()
                && href.startsWith("https://www.ics.uci.edu/");
     }

     /**
      * This function is called when a page is fetched and ready
      * to be processed by your program.
      */
     @Override
     public void visit(Page page) {
         String url = page.getWebURL().getURL();
         System.out.println("URL: " + url);

         if (page.getParseData() instanceof HtmlParseData) {
             HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
             String text = htmlParseData.getText();
             String html = htmlParseData.getHtml();
             Set<WebURL> links = htmlParseData.getOutgoingUrls();

             System.out.println("Text length: " + text.length());
             System.out.println("Html length: " + html.length());
             System.out.println("Number of outgoing links: " + links.size());
         }
    }
}
```

--------------------------------

### Add crawler4j Dependency with Gradle

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Add this line to your build.gradle file to integrate crawler4j into your project.

```gradle
    compile group: 'edu.uci.ics', name: 'crawler4j', version: '4.4.0'
```

--------------------------------

### Page Display Logic

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Handles the display of page content. If the page is 'RandomPages', it modifies the text. Otherwise, it constructs the page using the header, marked-up content, and footer.

```javascript
function show (json) { if (Object.keys(json).length == 0) { trouble('Page unavailable until conversion completed.') } else { if (title == 'RandomPages') { json.text = json.text.replace(/\*\w+/g, anypage) } page.innerHTML = head() + markup(json.text) + foot(json) } }
```

--------------------------------

### Create Custom Crawlers with WebCrawlerFactory

Source: https://context7.com/yasserg/crawler4j/llms.txt

Use WebCrawlerFactory to create crawler instances with injected dependencies like shared state or configuration. This allows for more flexible and testable crawler implementations.

```java
import edu.uci.ics.crawler4j.crawler.CrawlConfig;
import edu.uci.ics.crawler4j.crawler.CrawlController;
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.fetcher.PageFetcher;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtConfig;
import edu.uci.ics.crawler4j.robotstxt.RobotstxtServer;
import edu.uci.ics.crawler4j.url.WebURL;

import java.util.concurrent.atomic.AtomicInteger;

public class FactoryExample {

    // Crawler that receives dependencies via constructor
    public static class CountingCrawler extends WebCrawler {
        private final AtomicInteger pageCount;
        private final String targetDomain;

        public CountingCrawler(AtomicInteger pageCount, String targetDomain) {
            this.pageCount = pageCount;
            this.targetDomain = targetDomain;
        }

        @Override
        public boolean shouldVisit(Page referringPage, WebURL url) {
            return url.getURL().startsWith(targetDomain);
        }

        @Override
        public void visit(Page page) {
            int count = pageCount.incrementAndGet();
            System.out.println("Page #" + count + ": " + page.getWebURL().getURL());
        }
    }

    public static void main(String[] args) throws Exception {
        CrawlConfig config = new CrawlConfig();
        config.setCrawlStorageFolder("/tmp/crawler4j/");
        config.setMaxPagesToFetch(100);

        PageFetcher pageFetcher = new PageFetcher(config);
        RobotstxtConfig robotstxtConfig = new RobotstxtConfig();
        RobotstxtServer robotstxtServer = new RobotstxtServer(robotstxtConfig, pageFetcher);
        CrawlController controller = new CrawlController(config, pageFetcher, robotstxtServer);

        controller.addSeed("https://www.example.com/");

        // Shared state across all crawler instances
        AtomicInteger totalPages = new AtomicInteger(0);
        String domain = "https://www.example.com";

        // Factory creates crawler instances with injected dependencies
        CrawlController.WebCrawlerFactory<CountingCrawler> factory =
            () -> new CountingCrawler(totalPages, domain);

        controller.start(factory, 4);

        System.out.println("Total pages crawled: " + totalPages.get());
    }
}
```

--------------------------------

### List and Structure Parsing

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Utilities for converting structured text patterns into HTML lists and blocks.

```javascript
function enter(key, replacement) { // $code = $key; // while (@code > $depth) {local($_) = pop @code; print "</$_>\n"} // while (@code < $depth) {push (@code, $key); print "<$key>\n"} // if ($code[$#code] ne $key) { // print "</$code[$#code]><$key>\n"; // $code[$#code] = $key; // } return function (match, p1, p2) { var depth = p1.length || 0 var adjust = '' code = key if (code != '...') { while (codes.length > depth) { adjust += "</" + codes.pop() + ">" } while (codes.length < depth) { adjust += "<" + key + ">"; codes.push(key) } if (codes.length && codes[codes.length-1] != key) { adjust += "</" + codes[codes.length-1] + "><" + key + ">" codes[codes.length-1] = key } } return adjust + replacement.replace(/{p1}/, p1).replace(/{p2}/, p2) } }
```

```javascript
function bullets (text) { // $code = ""; // s/^\s*$/<p><\/p>/ && ($code = '...'); // s/^(\t+)(.+):\t/<dt>$2<dd>/ && &enter('DL', length $1); // s/^(\t+)\*/<li>/ && &enter('UL', length $1); // s/^(\*+)/<li>/ && &enter('UL', length $1); // s/^(\t+)\d+\.?/<li>/ && &enter('OL', length $1); // /^\s/ && &enter('PRE', 1); code = '' result = text .replace(/^\s*$/, enter('...', '<p></p>' ) ) .replace(/^(\t+)(.+):\t/, enter('DL', '<dt>{p2}<dd>') ) .replace(/^(\t+)\*/, enter('UL', '<li>') ) .replace(/^(\*+)/, enter('UL', '<li>') ) .replace(/^(\t+)\d+\.?/, enter('OL', '<li>') ) .replace(/^(\s)/, enter('PRE', '{p1}') ) return result }
```

--------------------------------

### Default user agent string

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

The default user agent string used by crawler4j.

```text
"crawler4j (https://github.com/yasserg/crawler4j/)"
```

--------------------------------

### Implement Custom Crawling Logic with WebCrawler

Source: https://context7.com/yasserg/crawler4j/llms.txt

Extend the WebCrawler class to define URL filtering in shouldVisit and page processing logic in visit.

```java
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.parser.HtmlParseData;
import edu.uci.ics.crawler4j.url.WebURL;

import java.util.Set;
import java.util.regex.Pattern;

public class MyCrawler extends WebCrawler {

    private static final Pattern FILTERS = Pattern.compile(
        ".*(\\.(css|js|gif|jpg|png|mp3|mp4|zip|gz))$");

    /**
     * Determines whether a URL should be crawled.
     * @param referringPage The page where this URL was found
     * @param url The URL to evaluate
     * @return true if the URL should be crawled
     */
    @Override
    public boolean shouldVisit(Page referringPage, WebURL url) {
        String href = url.getURL().toLowerCase();
        // Skip media files and only crawl specific domain
        return !FILTERS.matcher(href).matches()
               && href.startsWith("https://www.example.com/");
    }

    /**
     * Called when a page is successfully fetched and parsed.
     * @param page The fetched page with parsed data
     */
    @Override
    public void visit(Page page) {
        String url = page.getWebURL().getURL();
        int docId = page.getWebURL().getDocid();
        String domain = page.getWebURL().getDomain();
        String path = page.getWebURL().getPath();
        short depth = page.getWebURL().getDepth();

        System.out.println("URL: " + url);
        System.out.println("DocID: " + docId + ", Depth: " + depth);

        if (page.getParseData() instanceof HtmlParseData) {
            HtmlParseData htmlParseData = (HtmlParseData) page.getParseData();
            String text = htmlParseData.getText();
            String html = htmlParseData.getHtml();
            String title = htmlParseData.getTitle();
            Set<WebURL> links = htmlParseData.getOutgoingUrls();

            System.out.println("Title: " + title);
            System.out.println("Text length: " + text.length());
            System.out.println("Html length: " + html.length());
            System.out.println("Outgoing links: " + links.size());
        }
    }
}
```

--------------------------------

### Page Header Generation

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Generates the HTML for the page header, including the title formatted for display and a link to a search page.

```javascript
function head() { var words = document.title = title.replace(/(\[a-z\])(\[A-Z\])/g, '$1 $2'); return [ '<h1><img src=wiki.gif><div><span>', '<a href="http://c2.com/cgi/fullSearch?search=' + title + '" rel=nofollow>' + words + '</a>', '</span></div></h1>' ].join('') }
```

--------------------------------

### Page Footer Generation

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Generates the HTML for the page footer, displaying the last edit date and a link to the project's GitHub repository.

```javascript
function foot(json) { return [ '<hr>Last edit ', json.date, ', See <a href=https://github.com/WardCunningham/remodeling target="_blank">github</a> about remodling.' ].join('') }
```

--------------------------------

### Enable resumable crawling

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Allows the crawler to resume after an unexpected termination, though it may impact performance.

```java
crawlConfig.setResumableCrawling(true);
```

--------------------------------

### Implement WebCrawler Lifecycle Hooks

Source: https://context7.com/yasserg/crawler4j/llms.txt

Override lifecycle methods to execute custom code at specific points during crawling. This includes initialization, page processing hooks, error handling, and cleanup.

```java
import edu.uci.ics.crawler4j.crawler.Page;
import edu.uci.ics.crawler4j.crawler.WebCrawler;
import edu.uci.ics.crawler4j.crawler.exceptions.ParseException;
import edu.uci.ics.crawler4j.url.WebURL;

public class LifecycleHooksExample extends WebCrawler {

    /**
     * Called once when the crawler thread starts.
     * Use for initialization, opening database connections, etc.
     */
    @Override
    public void onStart() {
        System.out.println("Crawler " + getMyId() + " starting...");
        // Initialize resources
    }

    /**
     * Called before processing each URL. Can modify the URL before processing.
     * @param curURL The URL about to be processed
     * @return The (possibly modified) URL to process
     */
    @Override
    protected WebURL handleUrlBeforeProcess(WebURL curURL) {
        // Example: strip tracking parameters
        String url = curURL.getURL();
        if (url.contains("?utm_")) {
            url = url.substring(0, url.indexOf("?utm_"));
            curURL.setURL(url);
        }
        return curURL;
    }

    /**
     * Called after fetching the page header, before downloading content.
     * @param webUrl The URL being fetched
     * @param statusCode HTTP status code
     * @param statusDescription HTTP status description
     */
    @Override
    protected void handlePageStatusCode(WebURL webUrl, int statusCode, String statusDescription) {
        if (statusCode >= 400) {
            System.out.println("Error " + statusCode + " for: " + webUrl.getURL());
        }
    }

    /**
     * Called when a 3xx redirect is encountered.
     */
    @Override
    protected void onRedirectedStatusCode(Page page) {
        System.out.println("Redirect: " + page.getWebURL().getURL() +
            " -> " + page.getRedirectedToUrl());
    }

    /**
     * Called when page exceeds max download size.
     */
    @Override
    protected void onPageBiggerThanMaxSize(String urlStr, long pageSize) {
        System.out.println("Page too large (" + pageSize + " bytes): " + urlStr);
    }

    /**
     * Called when content fetch fails (network error, timeout, etc).
     */
    @Override
    protected void onContentFetchError(Page page) {
        System.out.println("Fetch failed: " + page.getWebURL().getURL());
    }

    /**
     * Called when HTML parsing fails.
     */
    @Override
    protected void onParseError(WebURL webUrl, ParseException e) throws ParseException {
        System.out.println("Parse error: " + webUrl.getURL() + " - " + e.getMessage());
    }

    /**
     * Called for unexpected HTTP status codes (not 2xx or 3xx).
     */
    @Override
    protected void onUnexpectedStatusCode(String urlStr, int statusCode,
                                          String contentType, String description) {
        System.out.println("Unexpected status " + statusCode + ": " + urlStr);
    }

    /**
     * Called for any unhandled exception during page processing.
     */
    @Override
    protected void onUnhandledException(WebURL webUrl, Throwable e) {
        System.err.println("Unhandled exception for " + webUrl.getURL() + ": " + e.getMessage());
    }

    @Override
    public boolean shouldVisit(Page referringPage, WebURL url) {
        return url.getURL().startsWith("https://www.example.com/");
    }

    @Override
    public void visit(Page page) {
        System.out.println("Visited: " + page.getWebURL().getURL());
    }

    /**
     * Called just before the crawler thread exits.
     * Use for cleanup, closing connections, final logging, etc.
     */
    @Override
    public void onBeforeExit() {
        System.out.println("Crawler " + getMyId() + " shutting down...");
        // Cleanup resources
    }
}
```

--------------------------------

### Overwrite user agent string

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Sets a custom user agent string for the crawler.

```java
crawlConfig.setUserAgentString(userAgentString);
```

--------------------------------

### Set politeness delay

Source: https://github.com/yasserg/crawler4j/blob/master/README.md

Adjusts the wait time between requests to avoid overloading servers.

```java
crawlConfig.setPolitenessDelay(politenessDelay);
```

--------------------------------

### Document Loading Function

Source: https://github.com/yasserg/crawler4j/blob/master/crawler4j/src/test/resources/html/wiki.c2.com.html

Loads a document from a given URL using XMLHttpRequest. Handles success by calling a callback function and errors by invoking the 'trouble' function.

```javascript
function loadDoc(url, done) { var xhttp = new XMLHttpRequest(); xhttp.onreadystatechange = function() { if (this.readyState == 4) { if (this.status == 200) { done(this.responseText) } else { trouble(url + "<br>can't fetch document<br>" + this.statusText) } } } xhttp.open("GET", url, true); xhttp.send(); }
```
Tokens:
Raw