### Install Go

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/slyrz_warc/README.md

Installs the Go programming language, a prerequisite for building the benchmark.

```bash
apt install golang
```

--------------------------------

### Install Go Dependencies

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/gowarc/README.md

Installs the Go programming language, which is a prerequisite for building and running the gowarc benchmarks.

```bash
sudo apt install golang
```

--------------------------------

### Build FastWARC Python Bindings from Source

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/fastwarc-py/README.md

Steps to build FastWARC Python bindings from the GitHub repository, including cloning, Rust toolchain installation, virtual environment setup, and installation options.

```bash
# Clone repository

git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
cd chatnoir-resiliparse

# Install Rust toolchain (if not already installed)

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Create a fresh venv first (recommended)

python3 -m venv venv && source venv/bin/activate

# Option 1: Build and install in editable mode (best for development)

python3 -m pip install -e ./fastwarc-py

# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)

python3 -m pip wheel -w build ./fastwarc-py
ls ./build/*.whl | xargs python3 -m pip install
```

--------------------------------

### Install Rust

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/rust_warc/README.md

Installs the Rust programming language toolchain using the official installation script.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

--------------------------------

### Install npm

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcpp/README.md

Installs the Node Package Manager (npm) using apt. This is a prerequisite for building and running the benchmarks.

```bash
sudo apt install npm
```

--------------------------------

### Install Dependencies for jwarc Benchmark

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/jwarc/README.md

Installs necessary Java Development Kit and Maven for building the benchmark.

```bash
sudo apt install openjdk-25-jdk maven
```

--------------------------------

### Install vcpkg Dependencies

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/resiliparse-py/README.md

Install build-time dependencies using vcpkg. Ensure to use a triplet suitable for your platform.

```bash
git clone https://github.com/Microsoft/vcpkg
./vcpkg/bootstrap-vcpkg.sh

# Install dependencies to vcpkg_installed (must be run from sources root)
./vcpkg/vcpkg install --triplet=x64-linux
```

--------------------------------

### Install Build Dependencies

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcio.js/README.md

Installs the necessary build tools for compiling the benchmark. This command is typically run on Debian-based systems.

```bash
sudo apt install build-essential
```

--------------------------------

### Install vcpkg and Dependencies

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/installation.md

Steps to install vcpkg and its required build-time dependencies for compiling Resiliparse from source. Ensure to use a triplet suitable for your platform.

```bash
git clone https://github.com/Microsoft/vcpkg
./vcpkg/bootstrap-vcpkg.sh
./vcpkg/vcpkg install --triplet=x64-linux
```

--------------------------------

### Truncate Examples by Length

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/cli/resiliparse.md

Truncate examples to a specified length for processing.

```shell
-t, --truncate <truncate>
```

--------------------------------

### Build and Install Resiliparse (Wheel Distribution)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/installation.md

Build wheels for Resiliparse and FastWARC in separate steps, then install them. This method is best for redistribution.

```bash
python3 -m pip wheel ./fastwarc ./resiliparse
ls ./FastWARC-*.whl ./Resiliparse-*.whl | xargs python3 -m pip install
```

--------------------------------

### Build FastWARC from source

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/fastwarc.md

Clone the repository, set up a virtual environment, and install FastWARC in editable mode for development.

```bash
git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
cd chatnoir-resiliparse
python3 -m venv venv && source venv/bin/activate
python3 -m pip install -e ./fastwarc-py
```

--------------------------------

### Resiliparse CLI Installation

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/cli/resiliparse.md

Instructions on how to install the Resiliparse CLI tool using pip, including optional flags for benchmarking dependencies.

```APIDOC
## Resiliparse CLI Installation

The `resiliparse` command line utility provides tools for maintaining and benchmarking Resiliparse. At the moment, these tools are aimed primarily at developers of Resiliparse. General-purpose tools geared towards users of the library may be added later.

To install the Resiliparse CLI tool, specify the `cli` flag (and optionally the `cli-benchmark` flag for any third-party benchmarking dependencies) in your `pip install` command:

```console
$ pip install 'resiliparse[cli]'
```

Once installed, run `resiliparse [COMMAND] --help` for detailed help listings.
```

--------------------------------

### Install Python 3

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/fastwarc-py-legacy/README.md

Installs Python 3 using apt package manager. This is a prerequisite for building and running the benchmark.

```bash
sudo apt install python3
```

--------------------------------

### Build and Install Resiliparse (Editable Mode)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/installation.md

Build and install Resiliparse and FastWARC in editable mode, which is ideal for development. A virtual environment is recommended.

```bash
python3 -m venv venv && source venv/bin/activate
python3 -m pip install -e ./fastwarc ./resiliparse
```

--------------------------------

### Install Resiliparse CLI via Pip

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/installation.md

Install the Resiliparse CLI and its dependencies by installing the package with the 'cli' flag.

```bash
pip install 'resiliparse[cli]'
```

--------------------------------

### Install Resiliparse with Beam Dependencies

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/beam.md

Install Resiliparse including the necessary dependencies for Apache Beam transforms.

```bash
pip install 'resiliparse[beam]'
```

--------------------------------

### Build Resiliparse from Source (Editable Mode)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/resiliparse-py/README.md

Build and install Resiliparse in editable mode, which is recommended for development. Ensure a virtual environment is activated.

```bash
# Create a fresh venv first (recommended)
python3 -m venv venv && source venv/bin/activate

# Option 1: Build and install in editable mode (best for development)
python3 -m pip install -e ./resiliparse-py
```

--------------------------------

### Build Resiliparse from Source (Wheel Distribution)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/resiliparse-py/README.md

Build Resiliparse wheels in separate steps, suitable for redistribution. This involves building the wheels and then installing them.

```bash
# Create a fresh venv first (recommended)
python3 -m venv venv && source venv/bin/activate

# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
python3 -m pip wheel -w build ./resiliparse-py
ls ./build/*.whl | xargs python3 -m pip install
```

--------------------------------

### TimeGuard as a Context Manager

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/process-guard.md

This example demonstrates using `TimeGuard` as a context manager with a `with` statement to guard arbitrary program sections. Progress reporting requires calling the `progress()` method on the guard instance.

```python
from time import sleep
from resiliparse.process_guard import time_guard, ExecutionTimeout

with time_guard(timeout=10):
    for _ in range(1000):
        try:
            sleep(0.1)
        except ExecutionTimeout:
            break
```

```python
from time import sleep
from resiliparse.process_guard import time_guard, ExecutionTimeout

with time_guard(timeout=10) as guard:
    for _ in range(1000):
        try:
            sleep(0.1)
            guard.progress()
        except ExecutionTimeout:
            break
```

--------------------------------

### CMake Minimum Version and Project Setup

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcpp/CMakeLists.txt

Sets the minimum required CMake version and defines the project name and languages.

```cmake
cmake_minimum_required(VERSION 3.16)
project(warcpp_profile LANGUAGES CXX)
```

--------------------------------

### Build Python Bindings

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/fastwarc-rs/README.md

Build the FastWARC Python bindings from source. This includes installing Rust and using pip to install the bindings in editable mode or build wheels.

```bash
# Install Rust toolchain (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Create a fresh venv first (recommended)
python3 -m venv venv && source venv/bin/activate

# Option 1: Build and install in editable mode (best for development)
python3 -m pip install -e ./fastwarc-py

# Option 2 (alternative): Build and install wheels in separate steps (best for redistribution)
python3 -m pip wheel -w build ./fastwarc-py
ls ./build/*.whl | xargs python3 -m pip install
```

--------------------------------

### Install Resiliparse Python Package

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/README.md

Use pip to install the Resiliparse Python package. Pre-built wheels are available for Linux, macOS, and Windows.

```bash
pip install resiliparse
```

--------------------------------

### Install FastWARC Python Package

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/README.md

Use pip to install the FastWARC Python package. Pre-built wheels are available for Linux, macOS, and Windows.

```bash
pip install fastwarc
```

--------------------------------

### Print Start and End Tags During DOM Traversal

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/parse/html.md

Implement both start and end element callbacks with `traverse_dom` to print the start and end tags of elements, excluding their content. This helps visualize the HTML structure without inner details.

```python
def start_cb(ctx: DOMContext):
    if ctx.node.type == NodeType.ELEMENT:
        print(f'<{ctx.node.tag}>', end='')

def end_cb(ctx: DOMContext):
    if ctx.node.type == NodeType.ELEMENT:
        print(f'</{ctx.node.tag}>', end='')

traverse_dom(tree.body, start_cb, end_cb)
```

--------------------------------

### Build Benchmark

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcpp/README.md

Builds the benchmark executable. This command compiles the necessary components to run performance tests.

```bash
make
```

--------------------------------

### Specify Input Directory for Training

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/cli/resiliparse.md

Required argument for training. Provide the path to the directory containing language-coded subdirectories (e.g., 'en/', 'fr/'). Each subdirectory should contain 'test.txt', 'train.txt', and 'val.txt' files.

```shell
INDIR
```

--------------------------------

### Get Multiple Language Detection Results

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/parse/lang.md

To analyze and weight potential language matches, use the n_results parameter to get a sorted list of possible language detections.

```python
print(d('''To Sherlock Holmes she is always the woman. I have seldom heard him
           mention her under any other name. In his eyes she eclipses and
           predominates the whole of her sex. It was not that he felt any emotion
           akin to love for Irene Adler.''', n_results=5))
# >>> [('en', 430), ('da', 463), ('no', 469), ('af', 481), ('fr', 491)]
```

--------------------------------

### Create and Serialize a WARC Response Record

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/fastwarc.md

Demonstrates initializing a WARC record, setting headers and payload, and writing it to a file. Use this to generate new WARC records from scratch.

```python
from fastwarc import WarcRecord, WarcRecordType

record = WarcRecord()

# Initialize mandatory headers.
record.init_headers(WarcRecordType.response, b'uuid:record-uuid')

# Alternative: initialize mandatory headers with an auto-generated random ID.
record.init_headers(WarcRecordType.response, None)

# Set the target ID header.
record.headers.append_bytes(WarcHeader.WARC_TARGET_URI, b'https://example.com/index.html')

# Set the payload bytes (automatically adjusts the Content-Length header).
payload = (b'HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n' +
           b'<!DOCTYPE html><html><body>Hello, world!</body></html>')
record.set_bytes_payload(payload)

# Mark the record as an HTTP record by setting the correct Content-Type.
record.is_http = True

# Write out the record.
with open('out.warc', 'wb') as out_file:
    record.write(out_file)
```

--------------------------------

### Accessing Text and HTML Serialization

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/parse/html.md

Use the `.text` property to get the plaintext content and the `.html` property to get the HTML serialization of a DOM node. Casting a DOMNode to `str` is equivalent to accessing its `.html` property.

```python
print(tree.body.get_element_by_id('a').text)
# >>> Hello world!
```

```python
print(tree.body.get_element_by_id('a').html)
# >>> <p id="a">Hello <span class="bar">world</span>!</p>
```

```python
print(tree.body.get_element_by_id('a'))
# >>> <p id="a">Hello <span class="bar">world</span>!</p>
```

--------------------------------

### Benchmark Results (Gzipped WARC)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcio.js/README.md

Sample output from running the benchmark on a gzipped WARC file. It displays performance metrics including records per second and MiB per second.

```console
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
$ ./profile CC-MAIN-20231005012006-20231005042006-00899.warc.gz
Reading WARC file: CC-MAIN-20231005012006-20231005042006-00899.warc.gz
2203 records/s, 49.7 MiB/s, 23.1 KiB/rec (1104 total, 24.9 MiB)
3990 records/s, 84.0 MiB/s, 21.6 KiB/rec (3099 total, 66.9 MiB)
4301 records/s, 86.5 MiB/s, 20.6 KiB/rec (5250 total, 110.2 MiB)
4204 records/s, 102.3 MiB/s, 24.9 KiB/rec (7374 total, 161.8 MiB)
4361 records/s, 106.8 MiB/s, 25.1 KiB/rec (9555 total, 215.3 MiB)
4353 records/s, 100.8 MiB/s, 23.7 KiB/rec (11733 total, 265.7 MiB)
4532 records/s, 106.3 MiB/s, 24.0 KiB/rec (14007 total, 319.0 MiB)
2664 records/s, 122.0 MiB/s, 46.9 KiB/rec (15339 total, 380.1 MiB)
2798 records/s, 117.5 MiB/s, 43.0 KiB/rec (16740 total, 438.9 MiB)
3016 records/s, 120.4 MiB/s, 40.9 KiB/rec (18255 total, 499.4 MiB)
2677 records/s, 122.6 MiB/s, 46.9 KiB/rec (19602 total, 561.1 MiB)
2676 records/s, 109.3 MiB/s, 41.8 KiB/rec (20940 total, 615.7 MiB)
2887 records/s, 118.9 MiB/s, 42.2 KiB/rec (22386 total, 675.3 MiB)
2603 records/s, 118.2 MiB/s, 46.5 KiB/rec (23691 total, 734.6 MiB)
2760 records/s, 122.3 MiB/s, 45.4 KiB/rec (25071 total, 795.7 MiB)
2861 records/s, 123.9 MiB/s, 44.4 KiB/rec (26502 total, 857.7 MiB)
2849 records/s, 122.1 MiB/s, 43.9 KiB/rec (27939 total, 919.3 MiB)
3071 records/s, 117.4 MiB/s, 39.1 KiB/rec (29478 total, 978.1 MiB)
2717 records/s, 124.5 MiB/s, 46.9 KiB/rec (30837 total, 1040.4 MiB)
2359 records/s, 114.1 MiB/s, 49.5 KiB/rec (32019 total, 1097.6 MiB)
2707 records/s, 118.2 MiB/s, 44.7 KiB/rec (33375 total, 1156.8 MiB)
2977 records/s, 118.1 MiB/s, 40.6 KiB/rec (34866 total, 1215.9 MiB)
2769 records/s, 119.2 MiB/s, 44.1 KiB/rec (36255 total, 1275.8 MiB)
2788 records/s, 118.9 MiB/s, 43.7 KiB/rec (37650 total, 1335.3 MiB)
2761 records/s, 117.0 MiB/s, 43.4 KiB/rec (39045 total, 1394.3 MiB)
2760 records/s, 115.9 MiB/s, 43.0 KiB/rec (40425 total, 1452.3 MiB)
2462 records/s, 122.0 MiB/s, 50.7 KiB/rec (41656 total, 1513.3 MiB)
2718 records/s, 119.6 MiB/s, 45.1 KiB/rec (43015 total, 1573.1 MiB)
2810 records/s, 120.8 MiB/s, 44.0 KiB/rec (44421 total, 1633.6 MiB)
2936 records/s, 121.2 MiB/s, 42.3 KiB/rec (45891 total, 1694.2 MiB)
2855 records/s, 121.1 MiB/s, 43.5 KiB/rec (47319 total, 1754.8 MiB)
2596 records/s, 126.5 MiB/s, 49.9 KiB/rec (48621 total, 1818.3 MiB)
2806 records/s, 120.7 MiB/s, 44.1 KiB/rec (50025 total, 1878.7 MiB)
2459 records/s, 120.7 MiB/s, 50.2 KiB/rec (51255 total, 1939.0 MiB)
2638 records/s, 122.0 MiB/s, 47.4 KiB/rec (52574 total, 2000.1 MiB)
2624 records/s, 115.3 MiB/s, 45.0 KiB/rec (53886 total, 2057.7 MiB)
2669 records/s, 107.3 MiB/s, 41.2 KiB/rec (55221 total, 2111.4 MiB)
2600 records/s, 113.3 MiB/s, 44.6 KiB/rec (56521 total, 2168.0 MiB)
2456 records/s, 108.7 MiB/s, 45.3 KiB/rec (57753 total, 2222.6 MiB)
2895 records/s, 114.2 MiB/s, 40.4 KiB/rec (59202 total, 2279.7 MiB)
3013 records/s, 121.1 MiB/s, 41.2 KiB/rec (60711 total, 2340.4 MiB)
2433 records/s, 120.2 MiB/s, 50.6 KiB/rec (61932 total, 2400.7 MiB)
2668 records/s, 120.0 MiB/s, 46.1 KiB/rec (63266 total, 2460.7 MiB)
2888 records/s, 119.1 MiB/s, 42.2 KiB/rec (64713 total, 2520.3 MiB)
2712 records/s, 117.7 MiB/s, 44.4 KiB/rec (66081 total, 2579.7 MiB)
2547 records/s, 122.6 MiB/s, 49.3 KiB/rec (67356 total, 2641.1 MiB)
2893 records/s, 116.3 MiB/s, 41.2 KiB/rec (68820 total, 2700.0 MiB)
2561 records/s, 122.5 MiB/s, 49.0 KiB/rec (70101 total, 2761.2 MiB)
2697 records/s, 121.4 MiB/s, 46.1 KiB/rec (71457 total, 2822.2 MiB)
```

--------------------------------

### traverse_dom

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/api/parse/html.md

Traverses the DOM tree starting at a base node, executing callbacks for each node encountered.

```APIDOC
## traverse_dom

### Description
DOM traversal helper. Traverses the DOM tree starting at `base_node` in pre-order and calls `start_callback` at each child node. If `end_callback` is not `None`, it will be called each time a DOM element’s end tag is encountered.

The callbacks are expected to take exactly one [`DOMContext`](#resiliparse.parse.html.DOMContext) context parameter, which keeps track of the current node and traversal depth. The context object will be the same throughout the whole traversal process, so it can be mutated with custom data.

### Method
`traverse_dom(base_node, start_callback, end_callback=None, context=None, elements_only=False)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Build a Benchmark

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/README.md

Builds the FastWARC benchmark executable. This command is run within the specific benchmark directory (e.g., `fastwarc`).

```console
$ cd fastwarc
$ make
cargo build --release
   Compiling libc v0.2.186
   Compiling shlex v2.0.1
   Compiling find-msvc-tools v0.1.9
...

```

--------------------------------

### index_action

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/api/beam/elasticsearch.md

Build an index bulk action.

```APIDOC
### resiliparse.beam.elasticsearch.index_action(doc_id, index, doc)

#### Description
Build an index bulk action.

#### Parameters
* **doc_id** (*str*) – document ID (`None` for using autogenerated IDs)
* **index** (*Union* *[* *<class 'NoneType'>* *,*  *<class 'str'>* *]*) – index name
* **doc** (*Dict* *[* *<class 'str'>* *,* *Any* *]*) – document as dict
#### Returns
index action dict
```

--------------------------------

### Benchmark Resiliparse Language Detection

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/cli/resiliparse.md

Benchmark Resiliparse against FastText and Langid. Requires one of the packages to be installed.

```bash
resiliparse lang benchmark [OPTIONS] INFILE
```

--------------------------------

### Gzip WARC Benchmark Results

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/jwarc/README.md

Sample output from running the jwarc benchmark on a gzipped WARC file, showing performance metrics over time.

```console
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
$ ./profile CC-MAIN-20231005012006-20231005042006-00899.warc.gz
Reading WARC file: CC-MAIN-20231005012006-20231005042006-00899.warc.gz
9655 records/s, 205.0 MiB/s, 21.7 KiB/rec (4828 total, 102.5 MiB)
12702 records/s, 304.4 MiB/s, 24.5 KiB/rec (11182 total, 254.8 MiB)
10494 records/s, 349.7 MiB/s, 34.1 KiB/rec (16429 total, 429.6 MiB)
8752 records/s, 375.6 MiB/s, 44.0 KiB/rec (20806 total, 617.5 MiB)
8319 records/s, 360.4 MiB/s, 44.4 KiB/rec (24967 total, 797.7 MiB)
9012 records/s, 380.4 MiB/s, 43.2 KiB/rec (29473 total, 987.9 MiB)
8337 records/s, 384.3 MiB/s, 47.2 KiB/rec (33651 total, 1180.5 MiB)
9021 records/s, 379.2 MiB/s, 43.0 KiB/rec (38164 total, 1370.2 MiB)
8605 records/s, 393.9 MiB/s, 46.9 KiB/rec (42469 total, 1567.3 MiB)
8972 records/s, 386.3 MiB/s, 44.1 KiB/rec (46955 total, 1760.4 MiB)
8509 records/s, 392.8 MiB/s, 47.3 KiB/rec (51211 total, 1956.9 MiB)
8980 records/s, 393.7 MiB/s, 44.9 KiB/rec (55702 total, 2153.8 MiB)
9157 records/s, 384.6 MiB/s, 43.0 KiB/rec (60283 total, 2346.2 MiB)
8225 records/s, 375.4 MiB/s, 46.7 KiB/rec (64396 total, 2533.9 MiB)
8526 records/s, 374.4 MiB/s, 45.0 KiB/rec (68662 total, 2721.2 MiB)
8414 records/s, 373.9 MiB/s, 45.5 KiB/rec (72869 total, 2908.2 MiB)
7110 records/s, 417.4 MiB/s, 60.1 KiB/rec (76426 total, 3117.0 MiB)
7156 records/s, 402.1 MiB/s, 57.5 KiB/rec (80005 total, 3318.1 MiB)
6881 records/s, 410.0 MiB/s, 61.0 KiB/rec (83449 total, 3523.3 MiB)
7165 records/s, 404.4 MiB/s, 57.8 KiB/rec (87037 total, 3725.8 MiB)
6966 records/s, 401.1 MiB/s, 59.0 KiB/rec (90520 total, 3926.4 MiB)
6251 records/s, 390.1 MiB/s, 63.9 KiB/rec (93646 total, 4121.4 MiB)
6545 records/s, 401.3 MiB/s, 62.8 KiB/rec (96919 total, 4322.2 MiB)
7298 records/s, 393.5 MiB/s, 55.2 KiB/rec (100570 total, 4519.0 MiB)
6912 records/s, 382.0 MiB/s, 56.6 KiB/rec (104026 total, 4710.0 MiB)
6914 records/s, 389.3 MiB/s, 57.7 KiB/rec (107483 total, 4904.7 MiB)
6762 records/s, 419.6 MiB/s, 63.5 KiB/rec (110864 total, 5114.5 MiB)
Summary: 14.0s, 8189 records/s, 379.7 MiB/s, 47.5 KiB/rec (114274 total, 5298.4 MiB)
```

--------------------------------

### resiliparse lang benchmark

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/cli/resiliparse.md

Benchmarks Resiliparse language detection against FastText and Langid. Requires one of the packages to be installed.

```APIDOC
## resiliparse lang benchmark

### Description

Benchmark Resiliparse against FastText and Langid. Either package must be installed for this comparison. Install Resiliparse with the `cli-benchmark` flag to install all optional third-party dependencies automatically.

### Usage

```shell
resiliparse lang benchmark [OPTIONS] INFILE
```

### Options

#### -r, --rounds <rounds>

Number of rounds to benchmark

* **Default:**
  `10000`

#### -f, --fasttext-model <fasttext_model>

FastText model to benchmark

### Arguments

#### INFILE

Required argument
```

--------------------------------

### Run Benchmark from RAM (SSD)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcpp/README.md

Executes the benchmark by reading the WARC file directly from RAM. This bypasses disk I/O, providing insights into the library's processing speed without storage bottlenecks.

```bash
./profile tmpfs/CC-MAIN-20231005012006-20231005042006-00899.warc
```

--------------------------------

### Run a Benchmark with Custom Buffer Size

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/README.md

Runs the benchmark profile with a specified input buffer size to optimize performance based on storage and CPU. The buffer size is set using the `BUFFER_SIZE` environment variable.

```console
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
$ BUFFER_SIZE=$((768 << 10)) ./profile CC-MAIN-20231005012006-20231005042006-00899.warc  # 768 KiB
Reading WARC file: CC-MAIN-20231005012006-20231005042006-00899.warc
40202 records/s, 1175.2 MiB/s, 29.9 KiB/rec (20143 total, 588.8 MiB)
27629 records/s, 1200.7 MiB/s, 44.5 KiB/rec (33982 total, 1190.2 MiB)
28660 records/s, 1262.2 MiB/s, 45.1 KiB/rec (48312 total, 1821.3 MiB)
33065 records/s, 1460.5 MiB/s, 45.2 KiB/rec (64846 total, 2551.6 MiB)
30555 records/s, 1549.1 MiB/s, 51.9 KiB/rec (80176 total, 3328.8 MiB)
26573 records/s, 1563.3 MiB/s, 60.2 KiB/rec (93463 total, 4110.5 MiB)
29442 records/s, 1679.8 MiB/s, 58.4 KiB/rec (108202 total, 4951.5 MiB)
Summary: 3.7s, 30552 records/s, 1416.5 MiB/s, 47.5 KiB/rec (114274 total, 5298.4 MiB)

```

--------------------------------

### Benchmark Results (Uncompressed WARC)

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/warcio.js/README.md

Sample output from running the benchmark on an uncompressed WARC file. It shows records per second, MiB per second, and KiB per record.

```console
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
$ ./profile CC-MAIN-20231005012006-20231005042006-00899.warc
27350 records/s, 630.2 MiB/s, 23.6 KiB/rec (13675 total, 315.1 MiB)
20214 records/s, 862.7 MiB/s, 23.6 KiB/rec (13675 total, 315.1 MiB)
20296 records/s, 885.2 MiB/s, 44.7 KiB/rec (33930 total, 1189.0 MiB)
20940 records/s, 918.6 MiB/s, 44.9 KiB/rec (44412 total, 1648.9 MiB)
21804 records/s, 974.9 MiB/s, 45.8 KiB/rec (55314 total, 2136.3 MiB)
22307 records/s, 972.8 MiB/s, 44.7 KiB/rec (66480 total, 2623.3 MiB)
20104 records/s, 1001.6 MiB/s, 51.0 KiB/rec (76560 total, 3125.5 MiB)
18260 records/s, 1051.1 MiB/s, 58.9 KiB/rec (85740 total, 3653.9 MiB)
18173 records/s, 1084.5 MiB/s, 61.1 KiB/rec (94827 total, 4196.2 MiB)
18524 records/s, 1035.5 MiB/s, 57.2 KiB/rec (104089 total, 4714.0 MiB)
17401 records/s, 1036.8 MiB/s, 61.0 KiB/rec (112800 total, 5233.0 MiB)
Summary: 5.6s, 20532 records/s, 952.0 MiB/s, 47.5 KiB/rec (114274 total, 5298.4 MiB)
```

--------------------------------

### Iterate remote WARC files with fsspec

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/fastwarc.md

Use ArchiveIterator with remote URLs (e.g., S3) when fsspec is installed. Pass credentials via fsspec_args.

```python
from fastwarc import ArchiveIterator

# Read remote S3 object (with optional credentials)
creds = {'key': '...',
         'secret': '...',
         'endpoint_url': '...'}
for record in ArchiveIterator('s3://mybucket/warcfile.warc.gz', fsspec_args=creds):
    print(record.record_id)

# Or without stream auto-detection:
for record in ArchiveIterator(GzipReader('s3://mybucket/warcfile.warc.gz', fsspec_args=creds)):
    print(record.record_id)
```

--------------------------------

### Create Language Detection Dataset

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/cli/resiliparse.md

Create a language detection dataset from Wikipedia article dumps. Requires INDIR containing language-specific subdirectories with plaintext files and specifies the output directory OUTDIR.

```bash
resiliparse lang create-dataset [OPTIONS] INDIR OUTDIR
```

--------------------------------

### Language Detection Benchmark

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/parse/lang.md

Benchmark Resiliparse's language detector against FastText and Langid. This command-line example shows performance differences on a large input file.

```console
$ resiliparse lang benchmark -f lid.176.bin -r 50000 INFILE.txt

Benchmarking language detectors (10,000 rounds):
Resiliparse: 1.7s
FastText:    6.8s
Langid:      101.7s
```

--------------------------------

### Write Zstandard WARC with Custom Dictionary

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/fastwarc.md

Illustrates creating a Zstandard-compressed WARC file that includes a custom compression dictionary. This can improve compression ratios for repetitive data. The dictionary is trained from sample records.

```python
from fastwarc import WarcRecord
from fastwarc.stream_io import ZstdWriter, zstd_train_dictionary_from_samples

# record_bytes = [b'WARC/1.1\r\n...', ...];

# Train dict on the first few records (raises OSError if size is too small).
MAX_DICT_SIZE = 128 << 10  # 128 KiB
zstd_dict = zstd_train_dictionary_from_samples(record_bytes[:10], MAX_DICT_SIZE)

# Write a Zstandard WARC with a dictionary frame.
with ZstdWriter('out.warc.zst', dictionary=zstd_dict) as zstd_writer:
    for b in record_bytes:
        r = WarcRecord.from_bytes(b)
        r.write(zstd_writer)
        zstd_writer.finish()
```

--------------------------------

### Fix Up Wheels with auditwheel on Linux

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/installation.md

Example of using 'auditwheel repair' on Linux to fix up build wheels, embedding linked shared libraries. This is necessary if the wheels are not suitable for general redistribution.

```bash
LD_LIBRARY_PATH=$(pwd)/vcpkg_installed/x64-linux/lib     auditwheel repair --plat linux_x86_64 build/Resiliparse*.whl build/FastWARC*.whl
```

--------------------------------

### Run a Benchmark with Default Settings

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/benchmarks/warc/README.md

Executes the benchmark profile on a WARC file after clearing the system's page cache. It outputs statistics on records per second, MiB per second, and KiB per record.

```console
$ sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
$ ./profile CC-MAIN-20231005012006-20231005042006-00899.warc
Reading WARC file: CC-MAIN-20231005012006-20231005042006-00899.warc
40216 records/s, 1175.6 MiB/s, 29.9 KiB/rec (20143 total, 588.8 MiB)
37786 records/s, 1638.5 MiB/s, 44.4 KiB/rec (39046 total, 1408.5 MiB)
34980 records/s, 1562.6 MiB/s, 45.7 KiB/rec (56587 total, 2192.0 MiB)
34982 records/s, 1587.0 MiB/s, 46.5 KiB/rec (74122 total, 2987.5 MiB)
27994 records/s, 1596.1 MiB/s, 58.4 KiB/rec (88183 total, 3789.2 MiB)
26794 records/s, 1570.6 MiB/s, 60.0 KiB/rec (101584 total, 4574.8 MiB)
Summary: 3.4s, 33208 records/s, 1539.7 MiB/s, 47.5 KiB/rec (114274 total, 5298.4 MiB)

```

--------------------------------

### Reporting Progress with TimeGuard

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/process-guard.md

This example shows how to manually report progress within a timed function using the `progress()` function. This is useful for loops where individual iterations should not exceed a certain duration.

```python
from time import sleep
from resiliparse.process_guard import progress, time_guard, ExecutionTimeout

@time_guard(timeout=10)
def foo():
    for _ in range(1000):
        try:
            sleep(0.1)
            progress()
        except ExecutionTimeout:
            print('Time out!')
            break

foo()
```

--------------------------------

### Print Text Nodes During DOM Traversal

Source: https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/docs/man/parse/html.md

Use `traverse_dom` with a start callback to process and print the values of all non-empty text nodes encountered during traversal. Ensure the `resiliparse.parse.html` module is imported.

```python
from resiliparse.parse.html import *

def start_cb(ctx: DOMContext):
    if ctx.node.type == NodeType.TEXT and ctx.node.value.strip():
        print(ctx.node.value.strip(), end=' ')

traverse_dom(tree.body, start_cb)
```