### Start Docs Server Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/develop/docs.md Starts the local documentation server. ```bash mkdocs serve ``` -------------------------------- ### Install Mkdocs Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/develop/docs.md Installs the necessary Mkdocs package for building the documentation. ```bash pip install mkdocs-material ``` -------------------------------- ### SOCKS5 Proxy Example Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of running the crawler with a SOCKS5 proxy. ```sh docker run -v $PWD/crawls/:/crawls/ -e PROXY_SERVER=socks5://path-to-proxy-host.example.com:9001 webrecorder/browsertrix-crawler crawl --url https://example.com/ ``` -------------------------------- ### SSH Proxy Example (Private Key) Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of running the crawler through an SSH proxy, providing the private key file. ```sh docker run -v $PWD/crawls/:/crawls/ -v $PWD/my-proxy-private-key:/tmp/private-key webrecorder/browsertrix-crawler crawl --url https://httpbin.org/ip --proxyServer ssh://user@path-to-ssh-host.example.com --sshProxyPrivateKeyFile /tmp/private-key ``` -------------------------------- ### HTTP Proxy Example (CLI Flag) Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of running the crawler with an HTTP proxy specified via the --proxyServer CLI flag. ```sh docker run -v $PWD/crawls/:/crawls/ webrecorder/browsertrix-crawler crawl --url https://example.com/ --proxyServer http://path-to-proxy-host.example.com:9000 ``` -------------------------------- ### HTTP Proxy Example (Environment Variable) Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of running the crawler with an HTTP proxy specified via the PROXY_SERVER environment variable. ```sh docker run -v $PWD/crawls/:/crawls/ -e PROXY_SERVER=http://path-to-proxy-host.example.com:9000 webrecorder/browsertrix-crawler crawl --url https://example.com/ ``` -------------------------------- ### Docker Run with Config File Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/yaml-config.md Example of starting a crawl with a configuration file passed via the --config option. ```shell docker run -v $PWD/crawl-config.yaml:/app/crawl-config.yaml -v $PWD/crawls:/crawls/ webrecorder/browsertrix-crawler crawl --config /app/crawl-config.yaml ``` -------------------------------- ### Docker command for screencasting Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/common-options.md Example command to run the Browsertrix Crawler with screencasting enabled, mapping the port and mounting a volume. ```sh docker run -p 9037:9037 -v $PWD/crawls:/crawls/ webrecorder/browsertrix-crawler crawl --url https://www.example.com --screencastPort 9037 ``` -------------------------------- ### Example YAML Config File Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/yaml-config.md An example of a basic YAML configuration file for the browsertrix-crawler. ```yaml seeds: - https://example.com/ - https://www.iana.org/ combineWARC: true ``` -------------------------------- ### SOCKS5 Proxy with Authentication Example Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of running the crawler with a SOCKS5 proxy that requires username and password authentication. ```sh docker run-v $PWD/crawls/:/crawls/ -e PROXY_SERVER=socks5://user:password@path-to-proxy-host.example.com:9001 webrecorder/browsertrix-crawler crawl --url https://example.com/ ``` -------------------------------- ### SSH Proxy Example (Private Key and Known Hosts) Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of running the crawler through an SSH proxy, providing both the private key and the known hosts file for verification. ```sh docker run -v $PWD/crawls/:/crawls/ -v $PWD/my-proxy-private-key:/tmp/private-key -v $PWD/known_hosts:/tmp/known_hosts webrecorder/browsertrix-crawler crawl --url https://httpbin.org/ip --proxyServer ssh://user@path-to-ssh-host.example.com --sshProxyPrivateKeyFile /tmp/private-key --sshProxyKnownHostsFile /tmp/known_hosts ``` -------------------------------- ### Basic run() loop example Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md An example of a basic async iterator run() loop that iterates over elements and clicks them, logging the number of clicks. ```javascript async* run(ctx) { let click = 0; let dblClick = 0; for await (const elem of document.querySelectorAll(".my-selector")) { if (elem.clickTwice) { elem.click(); elem.click(); dblClick++; } else { elem.click(); click++; } ctx.log({msg: "Clicked on elem", click, dblClick}); } } ``` -------------------------------- ### Proxy Configuration YAML Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example YAML file for configuring host-specific proxies, including matching hosts and proxy definitions. ```yaml matchHosts: # load all URLs from example.com through 'example-1-proxy' example.com/.*: example-1-proxy # load all URLS from https://my-social.example.com/.*/posts/ through # a different proxy https://my-social.example.com/.*/posts/: social-proxy # optional default proxy "": default-proxy proxies: # SOCKS5 proxy just needs a URL example-1-proxy: socks5://username:password@my-socks-5-proxy.example.com # SSH proxy also should have at least a 'privateKeyFile' social-proxy: url: ssh://user@my-social-proxy.example.com privateKeyFile: /proxies/social-proxy-private-key # optional publicHostsFile: /proxies/social-proxy-public-hosts default-proxy: url: ssh://user@my-social-proxy.example.com privateKeyFile: /proxies/default-proxy-private-key ``` -------------------------------- ### awaitPageLoad example with assertContentValid Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md An example of an awaitPageLoad method that uses assertContentValid to check for the absence of 'Log In' text, failing the crawl if found. ```javascript async awaitPageLoad(ctx: any) { const { sleep, assertContentValid } = ctx.Lib; await sleep(5); assertContentValid(() => !document.documentElement.outerHTML.match(/Log In/i), "not_logged_in"); } ``` -------------------------------- ### Example datapackage.json with deduplication dependencies Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/develop/dedupe.md An example of how the 'relation.requires' entry in datapackage.json might look when a WACZ file has dependencies on other WACZs from different crawls. ```json { "resources": { //resources in this WACZ file }, ... "relation": { "requires": [ { "filename": "", "hash": ", "crawlId": "" }, ... ] } } ``` -------------------------------- ### Run a Crawl with Docker Compose Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/develop/index.md Starts a crawl with 2 workers, generates CDX, and specifies the URL. ```sh docker-compose run crawler crawl --url https://webrecorder.net/ --generateCDX --collection wr-net --workers 2 ``` -------------------------------- ### Browser Profile Creation with SSH Proxy Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example of creating a browser profile using an SSH proxy, including private key and known hosts file. ```sh docker run -p 6080:6080 -p 9223:9223 -v $PWD/crawls/profiles:/crawls/profiles -v $PWD/my-proxy-private-key:/tmp/private-key -v $PWD/known_hosts:/tmp/known_hosts webrecorder/browsertrix-crawler create-login-profile --url https://example.com/ --proxyServer ssh://user@path-to-ssh-host.example.com --sshProxyPrivateKeyFile /tmp/private-key --sshProxyKnownHostsFile /tmp/known_hosts ``` -------------------------------- ### Example crawl command Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/qa.md This command demonstrates how to perform an initial crawl, ensuring WACZ generation, text extraction, and screenshot capture, which are prerequisites for QA analysis. ```sh docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --url https://webrecorder.net/ --collection example-crawl --text to-warc --screenshot view --generateWACZ ``` -------------------------------- ### Docker Run Command with Proxy Config Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/proxies.md Example command to run the Browsertrix Crawler Docker container with a specified proxy configuration file. ```sh docker run -v $PWD/crawls:/crawls -v $PWD/proxies:/proxies -it webrecorder/browsertrix-crawler --url https://example.com/ --proxyServerConfig /proxies/proxyConfig.yaml ``` -------------------------------- ### Custom WARCinfo Fields via Command-Line Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/common-options.md Example of how to add custom 'operator' and 'hostname' fields to the warcinfo record using command-line flags. ```sh --warcinfo.operator my-org --warcinfo.hostname hostname.my-org ``` -------------------------------- ### Example QA analysis command Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/qa.md This command shows how to run the QA analysis on an existing crawl WACZ, specifying the source WACZ and a new collection for the QA results. ```sh docker run -v $PWD/crawls/:/crawls/ -it webrecorder/browsertrix-crawler qa --qaSource /crawls/collections/example-crawl/example-crawl.wacz --collection example-qa --generateWACZ ``` -------------------------------- ### Create deduplication index from existing crawls Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/dedupe.md Example of using the 'indexer' entrypoint to import data from an existing WACZ file into the deduplication index. ```sh docker run -it webrecorder/browsertrix-crawler index --sourceUrl my-crawl.wacz --redisDedupeUrl redis://localhost:10000/0 ``` -------------------------------- ### Running crawls against an imported deduplication index Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/dedupe.md Example of running a crawl after the deduplication index has been populated from existing WACZ files. ```sh docker run -it webrecorder/browsertrix-crawler crawl --redisDedupeUrl redis://localhost:10000/0 ... ``` -------------------------------- ### Running crawls with deduplication Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/dedupe.md Example of running two crawls against the same Redis deduplication index. The second crawl should be significantly smaller due to deduplication. ```sh docker run -it webrecorder/browsertrix-crawler crawl --redisDedupeUrl redis://localhost:10000/0 --url https://old.webrecorder.net/ --generateWACZ --collection firstCrawl --limit 5 ``` ```sh docker run -it webrecorder/browsertrix-crawler crawl --redisDedupeUrl redis://localhost:10000/0 --url https://old.webrecorder.net/ --generateWACZ --collection secondCrawl --limit 5 ``` -------------------------------- ### Crawl with URL and Custom Behaviors Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md Example of running browsertrix-crawler with a specific URL and two custom behavior URLs. ```sh docker run -v $PWD/test-crawls:/crawls webrecorder/browsertrix-crawler crawl --url https://specs.webrecorder.net --customBehaviors https://example.com/custom-behavior-1 --customBehaviors https://example.org/custom-behavior-2 ``` -------------------------------- ### Custom Behaviors - Local filepath (directory) Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md Example of loading custom behaviors from a local directory using Docker. ```shell docker run -v $PWD/test-crawls:/crawls -v $PWD/tests/custom-behaviors/:/custom-behaviors/ webrecorder/browsertrix-crawler crawl --url https://specs.webrecorder.net --customBehaviors /custom-behaviors/ ``` -------------------------------- ### Docker Run with Config from Stdin Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/yaml-config.md Example of passing configuration via stdin, requiring the -i flag for docker run and --config stdin. ```shell cat ./crawl-config.yaml | docker run -i -v $PWD/crawls:/crawls/ webrecorder/browsertrix-crawler crawl --config stdin ``` -------------------------------- ### Custom WARCinfo Fields via YAML Config Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/common-options.md Example of how to add custom 'operator' and 'hostname' fields to the warcinfo record using a YAML configuration file. ```yaml warcinfo: operator: my-org hostname: hostname.my-org ``` -------------------------------- ### Custom Behaviors - Local filepath (file) Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md Example of loading a single custom behavior file from a local path using Docker. ```shell docker run -v $PWD/test-crawls:/crawls -v $PWD/tests/custom-behaviors/:/custom-behaviors/ webrecorder/browsertrix-crawler crawl --url https://specs.webrecorder.net --customBehaviors /custom-behaviors/custom.js ``` -------------------------------- ### run() loop using getState() utility function Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md A condensed example of the run() loop using the getState() utility function to simplify state management. ```javascript async* run(ctx) { const { Lib } = ctx; for await (const elem of document.querySelectorAll(".my-selector")) { if (elem.clickTwice) { elem.click(); elem.click(); yield Lib.getState(ctx, "Double-Clicked on elem", "dblClick"); } else { elem.click(); yield Lib.getState(ctx, "Single-Clicked on elem", "click"); } } } ``` -------------------------------- ### Crawl from Git Repository with Custom Behaviors Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md Example of running browsertrix-crawler with a URL and a custom behavior specified via a Git repository URL. ```sh docker run -v $PWD/test-crawls:/crawls webrecorder/browsertrix-crawler crawl --url https://example.com/ --customBehaviors "git+https://git.example.com/custom-behaviors?branch=dev&path=path/to/behaviors" ``` -------------------------------- ### Docker Run with Seed File Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/yaml-config.md Example of passing a seed file to the crawler by mounting it as a volume and specifying its path with --seedFile. ```shell docker run -v $PWD/seedFile.txt:/app/seedFile.txt -v $PWD/crawls:/crawls/ webrecorder/browsertrix-crawler crawl --seedFile /app/seedFile.txt ``` -------------------------------- ### Regular expression exclude rules Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/crawl-scope.md Example configuration for excluding specific URLs based on regular expressions. The crawl starts at a seed URL and avoids pages matching the exclude patterns. ```yaml seeds: - url: https://example.com/startpage.html scopeType: "host" exclude: - example.com/skip.* - example.com/search.* - postfeed ``` -------------------------------- ### create-login-profile CLI Options Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/cli-options.md Command-line options for creating a login profile with the Browsertrix Crawler. ```bash Options: --help Show help [boolean] --version Show version number [boolean] --url The URL of the login page [string] [required] --user The username for the login. If not specified, will b e prompted [string] --password The password for the login. If not specified, will b e prompted (recommended) [string] --filename The filename for the profile tarball, stored within /crawls/profiles if absolute path not provided [string] [default: "/crawls/profiles/profile.tar.gz"] --debugScreenshot If specified, take a screenshot after login and save as this filename [boolean] [default: false] --headless Run in headless mode, otherwise start xvfb [boolean] [default: false] --automated Start in automated mode, no interactive browser [boolean] [default: false] --interactive Deprecated. Now the default option! [boolean] [default: false] --shutdownWait Shutdown browser in interactive after this many seco nds, if no pings received [number] [default: 0] --profile Path or HTTP(S) URL to tar.gz file which contains th e browser profile directory [string] [default: ""] --windowSize Browser window dimensions, specified as: width,heigh t [string] [default: "1360,1020"] --cookieDays If >0, set all cookies, including session cookies, t o have this duration in days before saving profile [number] [default: 7] --proxyServer if set, will use specified proxy server. Takes prece dence over any env var proxy settings [string] --proxyServerConfig if set, path to yaml/json file that configures multi ple path servers per URL regex [string] --sshProxyPrivateKeyFile path to SSH private key for SOCKS5 over SSH proxy co nnection [string] --sshProxyKnownHostsFile path to SSH known hosts file for SOCKS5 over SSH pro xy connection [string] ``` -------------------------------- ### CLI Options Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/cli-options.md A list of available command-line options for the Browsertrix Crawler, including their types and default values. ```bash --qaSource Required for QA mode. Source (WACZ o r multi WACZ) for QA [string] --qaDebugImageDiff if specified, will write crawl.png, replay.png and diff.png for each pag e where they're different [boolean] --sshProxyPrivateKeyFile path to SSH private key for SOCKS5 o ver SSH proxy connection [string] --sshProxyKnownHostsFile path to SSH known hosts file for SOC KS5 over SSH proxy connection [string] --extraChromeArgs Extra arguments to pass directly to the Chrome instance (space-separated or multiple --extraChromeArgs) [array] [default: []] --useRobots, --robots If set, fetch and respect page disal lows specified in per-host robots.tx t [boolean] [default: false] --robotsAgent Agent to check in addition to '*' fo r robots rules [string] [default: "Browsertrix/1.x"] --reportSkipped If set, write information about URLs encountered but not queued to repor ts/skippedPages.jsonl [boolean] [default: false] --config Path to YAML config file ``` -------------------------------- ### Browsertrix Crawler CLI Options Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/cli-options.md Lists the available command-line options for the browsertrix-crawler, including their descriptions, types, and default values. ```bash Options: --help Show help [boolean] --version Show version number [boolean] --seeds, --url The URL to start crawling from [array] [default: []] --seedFile, --urlFile If set, read a list of seed urls, on e per line, from the specified [string] -w, --workers The number of workers to run in para llel [number] [default: 1] --crawlId, --id A user provided ID for this crawl or crawl configuration (can also be se t via CRAWL_ID env var), defaults to combination of Docker container hos tname and collection [string] --waitUntil Puppeteer page.goto() condition to w ait for before continuing, can be mu ltiple separated by ',' [array] [choices: "load", "domcontentloaded", "networkidle0", "networkidle2"] [default: ["load","networkidle2"]] --depth The depth of the crawl for all seeds [number] [default: -1] --extraHops Number of extra 'hops' to follow, be yond the current scope [number] [default: 0] --pageLimit, --limit Limit crawl to this number of pages [number] [default: 0] --maxPageLimit Maximum pages to crawl, overriding pageLimit if both are set [number] [default: 0] --pageLoadTimeout, --timeout Timeout for each page to load (in se conds) [number] [default: 90] --scopeType A predefined scope of the crawl. For more customization, use 'custom' and set scopeIncludeRx regexes [string] [choices: "page", "page-spa", "prefix", "host", "domain", "any", "cus tom"] --scopeIncludeRx, --include Regex of page URLs that should be in cluded in the crawl (defaults to the immediate directory of URL)[string] --scopeExcludeRx, --exclude Regex of page URLs that should be ex cluded from the crawl. [string] --allowHashUrls Allow Hashtag URLs, useful for singl e-page-application crawling or when different hashtags load dynamic cont ent --selectLinks, --linkSelector One or more selectors for extracting links, in the format [css selector] ->[property to use],[css selector]-> @[attribute to use] [array] [default: ["a[href]->href"]] --clickSelector Selector for elements to click when using the autoclick behavior [string] [default: "a"] --blockRules Additional rules for blocking certai n URLs from being loaded, by URL reg ex and optionally via text match in an iframe [array] [default: []] --blockMessage If specified, when a URL is blocked, a record with this error message is added instead[string] [default: ""] --blockAds, --blockads If set, block advertisements from be ing loaded (based on Stephen Black's blocklist) ``` -------------------------------- ### Automated Profile Creation Example Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/browser-profiles.md This command automatically creates a logged-in browser profile by launching a Docker container. It mounts a local directory for profiles and initiates the profile creation process for a specified URL. The script will then prompt for login credentials. ```bash docker run -v $PWD/crawls/profiles:/crawls/profiles -it webrecorder/browsertrix-crawler create-login-profile --url "https://digipres.club/" ``` -------------------------------- ### Enable Built-In Behaviors Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md Examples of enabling built-in behaviors using the --behaviors option. ```shell --behaviors autoscroll,autoplay,autofetch,siteSpecific ``` ```shell --behaviors autoscroll ``` ```shell --behaviors autoclick,autoplay,autofetch,siteSpecific ``` ```shell --behaviors "" ``` -------------------------------- ### Per-Seed Settings in YAML Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/yaml-config.md Example of configuring per-seed settings like depth and scopeType directly in the YAML configuration. ```yaml seeds: - url: https://webrecorder.net/ depth: 1 scopeType: "prefix" ``` -------------------------------- ### indexer CLI Options Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/cli-options.md Command-line options for the indexer functionality of the Browsertrix Crawler. ```bash Options: --help Show help [boolean] --version Show version number [boolean] --redisDedupeUrl URL for remote redis instance to index into [string] [required] --sourceUrl Source WACZ or Multi WACZ or Multi WACZ JSON to index [string] --sourceCrawlId If single WACZ, use this id as source id [string] --removing If set, also remove unused crawls/hashes from index [boolean] [default: false] --commitCrawlId If provided, commit single uncommitted crawl to merged index and exit [string] --cancelCrawlId If provided, delete data for uncommitted crawl and exit [string] ``` -------------------------------- ### Indexer Entrypoint - Index Importing Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/develop/dedupe.md Command to populate an index from existing WACZ files. ```bash webrecorder/browsertrix-crawler indexer --sourceUrl [--remove] ``` -------------------------------- ### run() loop with yield for pausing Source: https://github.com/webrecorder/browsertrix-crawler/blob/main/docs/docs/user-guide/behaviors.md An example of a run() loop that uses yield to allow the behavior to be paused after each major step. ```javascript async* run(ctx) { let click = 0; let dblClick = 0; for await (const elem of document.querySelectorAll(".my-selector")) { if (elem.clickTwice) { elem.click(); elem.click(); dblClick++; // allows behavior to be paused here yield {msg: "Double-clicked on elem", click, dblClick}; } else { elem.click(); click++; // allows behavior to be paused here yield {msg: "Single-clicked on elem", click, dblClick}; } } } ```