### YAML Configuration File Example

Source: https://context7.com/allenai/duplodocus/llms.txt

Example structure of a YAML configuration file for MinHash parameters. Use this for complex setups and to ensure reproducibility.

```yaml

```

--------------------------------

### YAML Configuration for MinHash

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Example YAML configuration file for fine-tuning MinHash parameters. This allows for more complex setups than command-line arguments.

```yaml
# minhash_config.yaml
minhash_params:
  num_buckets: 26
  bucket_size: 11
  ngram_size: 5
  permutation_seed: 42
  tokenizer: "cl100k_base"
eng_params:
  num_docs: 1000000
  max_lines_per_path: 100000
  num_sig_chunks: 8
output_params:
  annotate: false
  annotate_key: metadata.minhash # minhash output data location
  remove_duplicates: true # just annotate, don't remove
  delete_while_cleaning: false
```

--------------------------------

### Build Duplodocus from Source

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Install the Rust toolchain and compile the project from the repository.

```bash
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.bashrc

# Clone and build
git clone git@github.com:allenai/duplodocus.git
cd dedup-tool
cargo build --release

# Binary will be at: ./target/release/dedup-tool
```

--------------------------------

### Configure AWS EC2 for Large-Scale Processing

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Setup a RAID0 array on NVMe drives and install necessary build dependencies for high-performance processing.

```bash
# Configure RAID0 array from NVMe drives
sudo yum install mdadm -y
sudo mdadm --create /dev/md0 --level=0 --raid-devices=8 \
  /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 \
  /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1
sudo mkfs.xfs /dev/md0
sudo mkdir /mnt/raid0
sudo mount /dev/md0 /mnt/raid0
sudo chown -R $USER /mnt/raid0

# Install build dependencies
sudo yum install gcc cmake openssl-devel g++ htop git -y

# Install s5cmd for fast S3 transfers
wget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz
tar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz
sudo mv s5cmd /usr/local/bin
```

--------------------------------

### Build Union-Find Structure (with Config)

Source: https://context7.com/allenai/duplodocus/llms.txt

Builds the Union-Find structure using parameters specified in a YAML configuration file. This allows for reproducible and complex setups.

```bash
cargo run --release -- mh-build-uf \
  --storage-dir /shared/work \
  --num-path-chunks 10 \
  --max-lines-per-path 100000 \
  --config /shared/work/config.yaml
```

--------------------------------

### Compute MinHash Signatures (with Config File)

Source: https://context7.com/allenai/duplodocus/llms.txt

Uses a YAML configuration file to specify parameters for MinHash signature computation. This is useful for complex setups and reproducibility.

```bash
cargo run --release -- mh-hash-docs \
  --local-input /data/documents \
  --storage-dir /shared/work \
  --text-key "text" \
  --path-chunk 0 \
  --num-path-chunks 10 \
  --config /shared/work/config.yaml
```

--------------------------------

### Hash Documents in Parallel (Disk-Based)

Source: https://github.com/allenai/duplodocus/blob/main/README.md

This command hashes documents in parallel across multiple workers for disk-based deduplication. Adjust `--path-chunk` and `--num-path-chunks` based on your worker setup.

```bash
# Worker 0
cargo run --release -- mh-hash-docs \
  --local-input /data/docs \
  --storage-dir /shared/work \
  --text-key "text" \
  --path-chunk 0 \
  --num-path-chunks 10 \
  --num-buckets 20 \
  --bucket-size 5

# Worker 1
cargo run --release -- mh-hash-docs \
  --local-input /data/docs \
  --storage-dir /shared/work \
  --text-key "text" \
  --path-chunk 1 \
  --num-path-chunks 10 \
  --num-buckets 20 \
  --bucket-size 5

# ... repeat for workers 2-9
```

--------------------------------

### Build Filemap

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Create a filemap object to map file indices to actual files using either Rust or Python.

```bash
cargo run --release -- mh-build-file-map --input-dir test_data_inputs/fuzzy_multi --storage-dir test_data_outputs/s3_storage/
```

```bash
python python/file_map_builder.py --remote-dir test_data_inputs/fuzzy_multi --storage-dir test_data_outputs/s3_storage/
```

--------------------------------

### Download Data from S3

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Configure AWS credentials and download JSONL files using s5cmd.

```bash
# Configure AWS credentials
aws configure

# Download JSONL files
s5cmd cp -sp s3://your-bucket/path/to/data/* /mnt/raid0/input_data/
```

--------------------------------

### Run MinHash Deduplication with Config File

Source: https://context7.com/allenai/duplodocus/llms.txt

Execute the MinHash deduplication process using a configuration file via the CLI. Input, storage, and output directories must be specified.

```bash
# Use config file with CLI
cargo run --release -- minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --config minhash_config.yaml
```

--------------------------------

### Build File Map for Disk-Based Deduplication

Source: https://github.com/allenai/duplodocus/blob/main/README.md

This is the first step in disk-based deduplication, run once to create a file map. It prepares the data for distributed processing.

```bash
cargo run --release -- mh-build-file-map \
  --input-dir /data/docs \
  --storage-dir /shared/work
```

--------------------------------

### Build File Map with Python Script

Source: https://context7.com/allenai/duplodocus/llms.txt

Use this Python script to build a file map for the MinHash process. Specify the remote directory containing documents and the local storage directory.

```bash
python python/file_map_builder.py \
  --remote-dir /data/documents \
  --storage-dir /shared/work
```

--------------------------------

### Hash and group data

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_multi/README.md

Simulate a multi-node environment by hashing and grouping data in subdirectories.

```bash
for i in {0..3}; do
	cargo run --release -- exact-dedup-disk-group \
		--input-dir "test_data_inputs/exact_multi/subdir_0${i}" \
		--storage-dir test_data_outputs/exact_storage/preshuffle \
		--hash-key metadata.text_hash \
		--num-bins 4
done
```

--------------------------------

### Verify output data with Python

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Use this snippet to load and validate the content of JSONL files in the test_data_outputs directory.

```python
#! python
import glob, json
all_data = [json.loads(_) for f in glob.glob('test_data_outputs/fuzzy_multi/*.jsonl') for _ in open(f).read().splitlines()]
assert len(all_data) == 25 # check only 25 surviving docs
assert set(_['text'].split(' ')[-1] for _ in all_data) == set('LOREM_%02d' % i for i in range(25)) # Check the content of the surviving docs
```

--------------------------------

### Override MinHash Config with CLI Arguments

Source: https://context7.com/allenai/duplodocus/llms.txt

Demonstrates how command-line arguments can override values specified in the MinHash configuration file, such as num_buckets.

```bash
# CLI arguments override config file values
cargo run --release -- minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --config minhash_config.yaml \
  --num-buckets 30  # Overrides config file value
```

--------------------------------

### MinHash Disk-Based - Build File Map

Source: https://context7.com/allenai/duplodocus/llms.txt

Required first step for disk-based MinHash operations. Creates an index mapping file paths to integer IDs.

```bash
cargo run --release -- mh-build-file-map \
  --input-dir /data/documents \
  --storage-dir /shared/work
```

--------------------------------

### Gather Edges (Explicit Parameters)

Source: https://context7.com/allenai/duplodocus/llms.txt

Configures edge gathering with explicit parameters for large datasets. Adjust num-docs and max-lines-per-path for performance tuning.

```bash
cargo run --release -- mh-gather-edges \
  --storage-dir /shared/work \
  --num-docs 1000000000 \
  --max-lines-per-path 100000
```

--------------------------------

### MinHash Configuration

Source: https://context7.com/allenai/duplodocus/llms.txt

Configure MinHash parameters for deduplication. Adjust num_buckets, bucket_size, and ngram_size for matching strictness. Ensure tokenizer is set appropriately.

```yaml
minhash_params:
  num_buckets: 26        # Number of LSH bands (more = stricter matching)
  bucket_size: 11        # Hash values per band (more = stricter matching)
  ngram_size: 5          # N-gram size for document shingling
  permutation_seed: 42   # Random seed for reproducibility
  tokenizer: "cl100k_base"  # Tokenizer: cl100k, p50k, uniseg, or character-level

eng_params:
  num_docs: 1000000000      # Expected total documents (overestimate!)
  max_lines_per_path: 100000  # Max lines per file (overestimate!)
  num_sig_chunks: 256       # Signature partitions (~100 per TB)

output_params:
  annotate: true                    # Add duplicate metadata
  annotate_key: metadata.minhash    # JSON key for annotations
  remove_duplicates: true           # Remove duplicates vs annotate only
  delete_while_cleaning: false      # Delete input files after processing
```

--------------------------------

### Define Minhash Configuration

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Configuration file defining hyperparameters for Minhash and deduplication settings.

```yaml
# Minhash Configuration file
minhash_params:
  num_buckets: 26
  bucket_size: 11
  ngram_size: 5
  permutation_seed: 42
  tokenizer: "cl100k_base"
eng_params:
  num_docs: 1000000
  max_lines_per_path: 100000
  num_sig_chunks: 8
output_params:
  annotate: false
  annotate_key: metadata.minhash # minhash output data location
  remove_duplicates: true # just annotate, don't remove
  delete_while_cleaning: false
```

--------------------------------

### Run exact deduplication

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_simple/README.md

Executes the Rust-based deduplication tool to clean or annotate data.

```bash
cargo run --release exact-dedup-memory --input-dir test_data_inputs/exact --output-dir test_data_outputs/exact_memory
```

```bash
cargo run --release exact-dedup-memory --input-dir test_data_inputs/exact --output-dir test_data_outputs/exact_memory_anno --annotate-key metadata.exact_dedup
```

--------------------------------

### Build Union-Find Structure

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Merge edges to identify connected components for duplicate identification.

```bash
cargo run --release -- mh-build-uf \
	--storage-dir test_data_outputs/s3_storage \
	--num-path-chunks 4
```

--------------------------------

### Generate Multi-Node Test Data

Source: https://context7.com/allenai/duplodocus/llms.txt

Create test data split across multiple subdirectories for simulating multi-node or distributed processing. Use the `--multi` flag.

```bash
# Generate multi-node test data (split into 4 subdirectories)
python examples/make_example_pools.py \
  --target-dir test_data_inputs/exact_multi \
  --multi
```

```bash
python examples/make_example_pools.py \
  --target-dir test_data_inputs/fuzzy_multi \
  --fuzzy \
  --multi
```

--------------------------------

### Generate Hashes

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Generate and save hashes for data slices. Increase ulimit if encountering 'Too many open files' errors.

```bash
for i in {0..3}; do 
	cargo run --release -- mh-hash-docs \
		--local-input test_data_inputs/fuzzy_multi \
		--storage-dir test_data_outputs/s3_storage \
		--num-buckets 26 \
		--bucket-size 11 \
		--ngram-size 5 \
		--permutation-seed 42 \
		--path-chunk $i \
		--num-path-chunks 4
done
```

--------------------------------

### Gather Edges (Distributed)

Source: https://context7.com/allenai/duplodocus/llms.txt

Performs distributed edge gathering by processing each signature band separately. This involves copying data to a local directory, running the command, and then copying results back.

```bash
for band in /shared/work/sig_storage/*; do
  mkdir -p /local/work/sig_storage/
  cp /shared/work/filemap.json.gz /local/work/
  cp -r $band /local/work/sig_storage/
  cargo run --release -- mh-gather-edges \
    --storage-dir /local/work
  cp -r /local/work/edges/* /shared/work/edges/
done
```

--------------------------------

### Generate test data for deduplication

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_simple/README.md

Creates the necessary input files for the deduplication process.

```bash
python examples/make_example_pools.py --target-dir test_data_inputs/exact
```

--------------------------------

### Read compressed JSONL output files

Source: https://context7.com/allenai/duplodocus/llms.txt

Use this snippet to aggregate documents from all .jsonl.zst files in the output directory.

```python
all_data = [
    doc
    for f in glob.glob('output/*.jsonl.zst')
    for doc in read_zst_jsonl(f)
]
print(f"Total documents: {len(all_data)}")
```

--------------------------------

### Run Memory-Based Deduplication

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Use this command for smaller datasets requiring all-in-one fuzzy deduplication. Ensure input and storage directories are correctly specified.

```bash
cargo run --release -- minhash-memory \
  --input-dir /data/docs \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --num-buckets 20 \
  --bucket-size 5 \
  --ngram-size 5 \
  --remove-duplicates true \
  --cleanup-storage
```

--------------------------------

### Generate Edges from Hashes

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Generate edges by operating on hash space slices in a simulated distributed environment.

```bash
mkdir -p test_data_outputs/s3_storage/edges/

for band in test_data_outputs/s3_storage/sig_storage/*; do 
	rm -rf test_data_outputs/local_storage
	mkdir -p test_data_outputs/local_storage/sig_storage/
	cp test_data_outputs/s3_storage/filemap.json.gz test_data_outputs/local_storage
	cp -r $band test_data_outputs/local_storage/sig_storage/
	cargo run --release -- mh-gather-edges \
		--storage-dir test_data_outputs/local_storage/ 
	cp -r test_data_outputs/local_storage/edges/* test_data_outputs/s3_storage/edges/
	rm -rf test_data_outputs/local_storage
done
```

--------------------------------

### Run fuzzy deduplication

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_simple/README.md

Executes the MinHash memory deduplication tool. Use the annotate flag to keep all documents with metadata instead of filtering.

```bash
cargo run --release -- minhash-memory \
	--input-dir test_data_inputs/fuzzy \
	--storage-dir test_data_outputs/fuzzy_mem_storage \
	--output-dir test_data_outputs/fuzzy_mem \
	--num-buckets 26 \
	--bucket-size 11 \
	--ngram-size 5 \
	--permutation-seed 42 \
	--cleanup-storage
```

```bash
cargo run --release -- minhash-memory \
	--input-dir test_data_inputs/fuzzy \
	--storage-dir test_data_outputs/fuzzy_mem_storage \
	--output-dir test_data_outputs/fuzzy_mem_anno \
	--num-buckets 26 \
	--bucket-size 11 \
	--ngram-size 5 \
	--permutation-seed 42 \
	--cleanup-storage \
	--remove-duplicates false \
	--annotate true \
	--annotate-key metadata.minhash
```

--------------------------------

### Reorganize data by hash-slices

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_multi/README.md

Shuffle data chunks so that all chunks of the same hash slice reside in a single subdirectory.

```bash
for i in {0..3}; do
	mkdir -p "test_data_outputs/exact_storage/chunk_${i}"
	for f in  test_data_outputs/exact_storage/preshuffle/chunk_0000000${i}*;
	do
		mv $f "test_data_outputs/exact_storage/chunk_${i}/"
	done
done
```

--------------------------------

### Verify outputs

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_multi/README.md

Validate the deduplicated and annotated data using Python scripts.

```python
import zstandard, glob, json
reader = lambda x : [json.loads(_) for _ in zstandard.ZstdDecompressor().stream_reader(open(x, 'rb').read()).read().splitlines()]
all_data = [_ for f in glob.glob('test_data_outputs/exact_multi/*.jsonl.zst') for _ in reader(f)]
assert len(all_data) == 25
assert set(_['text'] for _ in all_data) == set('this doc has content: %02d' % i for i in range(25)) # Check the content of the surviving docs


annotated_data = [_ for f in glob.glob('test_data_outputs/exact_multi_anno/*.jsonl.zst') for _ in reader(f)]
assert len(annotated_data) == 325 # check all data survived
assert all(_['metadata']['exact_dedup']['num_dups'] == 25 - int(_['text'].split(' ')[-1]) for _ in annotated_data) # check data is as expected
```

--------------------------------

### Perform Memory-Based Exact Deduplication

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Execute exact deduplication for datasets under 100GB in a single pass.

```bash
cargo run --release -- exact-dedup-memory \
  --input-dir /data/docs \
  --output-dir /data/unique \
  --text-key "content" \
  --annotate-key "duplicate_info"  # Optional: annotate instead of remove
```

--------------------------------

### Perform Exact Deduplication (Small Dataset)

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Remove documents with identical content using the memory-based exact deduplication method.

```bash
cargo run --release -- exact-dedup-memory \
  --input-dir /data/documents \
  --output-dir /data/unique \
  --text-key "content"
```

--------------------------------

### Run Memory-Based Deduplication with YAML Config

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Execute memory-based deduplication using parameters defined in a YAML configuration file. This is useful for managing numerous settings.

```bash
cargo run --release -- minhash-memory \
  --input-dir /data/docs \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --config minhash_config.yaml
```

--------------------------------

### Perform Disk-Based Exact Deduplication

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Execute exact deduplication for large datasets using a two-step grouping and pruning process.

```bash
cargo run --release -- exact-dedup-disk-group \
  --input-dir /data/docs \
  --storage-dir /scratch/work \
  --hash-key "doc_hash" \
  --num-bins 100
```

```bash
cargo run --release -- exact-dedup-disk-prune \
  --storage-dir /scratch/work \
  --output-dir /data/unique \
  --hash-key "doc_hash"
```

--------------------------------

### Exact Deduplication (Disk-Based) - Grouping

Source: https://context7.com/allenai/duplodocus/llms.txt

First stage for large datasets, hashing documents and grouping them into bins for distributed processing. Supports multi-node processing.

```bash
cargo run --release -- exact-dedup-disk-group \
  --input-dir /data/documents \
  --storage-dir /scratch/work \
  --text-key "content" \
  --hash-key "doc_hash" \
  --hash-bits 128 \
  --num-bins 100
```

```bash
for i in {0..3}; do
  cargo run --release -- exact-dedup-disk-group \
    --input-dir "/data/documents/shard_${i}" \
    --storage-dir /shared/storage/preshuffle \
    --hash-key "metadata.text_hash" \
    --num-bins 100
done
```

--------------------------------

### Perform Fuzzy Deduplication (Small Dataset)

Source: https://github.com/allenai/duplodocus/blob/main/README.md

Identify and remove near-duplicates using the MinHash memory-based method.

```bash
cargo run --release -- minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --num-buckets 20 \
  --bucket-size 5 \
  --remove-duplicates true \
  --cleanup-storage
```

--------------------------------

### Generate test data

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_simple/README.md

Creates the necessary input data for the deduplication process.

```bash
python examples/make_example_pools.py --target-dir test_data_inputs/fuzzy --fuzzy
```

--------------------------------

### Build Union-Find for Disk-Based Deduplication

Source: https://github.com/allenai/duplodocus/blob/main/README.md

This step builds the Union-Find data structure on a single machine after edges have been gathered. It's essential for identifying connected components.

```bash
cargo run --release -- mh-build-uf \
  --storage-dir /shared/work \
  --num-path-chunks 10
```

--------------------------------

### Compute MinHash Signatures (Single Worker)

Source: https://context7.com/allenai/duplodocus/llms.txt

Computes MinHash signatures for documents using a single worker. Ensure the text key and storage directory are correctly specified. Adjust parameters like ngram size and permutation seed as needed.

```bash
cargo run --release -- mh-hash-docs \
  --local-input /data/documents \
  --storage-dir /shared/work \
  --text-key "text" \
  --path-chunk 0 \
  --num-path-chunks 1 \
  --num-buckets 20 \
  --bucket-size 5 \
  --ngram-size 5 \
  --permutation-seed 42
```

--------------------------------

### Clean Files (Annotate Duplicates)

Source: https://context7.com/allenai/duplodocus/llms.txt

Annotates duplicate files instead of removing them. Use the 'annotate' flag and specify the annotation key. Ensure 'remove-duplicates' is set to false.

```bash
cargo run --release -- mh-clean-files \
  --input-dir /data/documents \
  --storage-dir /shared/work \
  --output-dir /data/annotated \
  --path-chunk 0 \
  --num-path-chunks 1 \
  --annotate true \
  --annotate-key "metadata.minhash" \
  --remove-duplicates false
```

--------------------------------

### Gather Edges for Disk-Based Deduplication

Source: https://github.com/allenai/duplodocus/blob/main/README.md

This command gathers edges after all documents have been hashed. It requires all signatures to be available and is run once.

```bash
cargo run --release -- mh-gather-edges \
  --storage-dir /shared/work
```

--------------------------------

### True Jaccard Verification (Parallel & Hotnode)

Source: https://context7.com/allenai/duplodocus/llms.txt

Performs true Jaccard verification with parallel processing and hotnode handling for large-scale verification. Configure parallel-nest and hotnode parameters as needed.

```bash
cargo run --release -- true-jaccard \
  --input-dir /data/annotated \
  --output-dir /data/verified \
  --jaccard-threshold 0.8 \
  --annotate-key "metadata.true_jaccard" \
  --parallel-nest 8 \
  --hotnode-size 1000 \
  --hotnode-dir /data/hotnodes
```

--------------------------------

### Generate Exact Deduplication Test Data

Source: https://context7.com/allenai/duplodocus/llms.txt

Create synthetic test data for exact deduplication using the Python utility. Specify the target directory for the generated files.

```bash
# Generate exact deduplication test data
python examples/make_example_pools.py \
  --target-dir test_data_inputs/exact
```

--------------------------------

### Verify Exact Deduplication Results

Source: https://context7.com/allenai/duplodocus/llms.txt

Python script to verify exact deduplication by reading JSONL output files, counting unique documents, and checking for expected annotations.

```python
import glob
import json

# Verify exact deduplication results
all_data = [
    json.loads(line)
    for f in glob.glob('output/*.jsonl')
    for line in open(f).read().splitlines()
]
# Check unique documents remain
unique_texts = set(doc['text'] for doc in all_data)
print(f"Unique documents: {len(unique_texts)}")

# Verify annotated data
annotated_data = [
    json.loads(line)
    for f in glob.glob('annotated/*.jsonl')
    for line in open(f).read().splitlines()
]
# Check annotations exist
for doc in annotated_data:
    if 'metadata' in doc and 'exact_dedup' in doc['metadata']:
        print(f"Doc {doc['id']}: {doc['metadata']['exact_dedup']['num_dups']} duplicates")

# Verify fuzzy deduplication with MinHash annotations
for doc in annotated_data:
    if 'metadata' in doc and 'minhash' in doc['metadata']:
        mh = doc['metadata']['minhash']
        print(f"Doc {doc['id']}: CC={mh['cc_id']}, size={mh['cc_size']}")
```

--------------------------------

### Generate Test Data

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Programmatically generate test data for the fuzzy deduplication process.

```bash
python examples/make_example_pools.py --target-dir test_data_inputs/fuzzy_multi --fuzzy
```

--------------------------------

### Compute MinHash Signatures (Distributed)

Source: https://context7.com/allenai/duplodocus/llms.txt

Distributes MinHash signature computation across multiple workers. Each worker processes a specific path chunk. Adjust the number of path chunks and bucket parameters for optimal performance.

```bash
for i in {0..9}; do
  cargo run --release -- mh-hash-docs \
    --local-input /data/documents \
    --storage-dir /shared/work \
    --text-key "text" \
    --path-chunk $i \
    --num-path-chunks 10 \
    --num-buckets 26 \
    --bucket-size 11 \
    --ngram-size 5 \
    --permutation-seed 42
done
```

--------------------------------

### Verify deduplication outputs

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_simple/README.md

Validates the resulting JSONL files to ensure the expected number of documents and metadata annotations are present.

```python
import glob, json
all_data = [json.loads(_) for f in glob.glob('test_data_outputs/fuzzy_mem/*.jsonl') for _ in open(f).read().splitlines()]
assert len(all_data) == 25 # check only 25 surviving docs
assert set(_['text'].split(' ')[-1] for _ in all_data) == set('LOREM_%02d' % i for i in range(25)) # Check the content of the surviving docs


annotated_data = [json.loads(_) for f in glob.glob('test_data_outputs/fuzzy_mem_anno/*.jsonl') for _ in open(f).read().splitlines()]
assert len(annotated_data) == 325 # check all data survived
missing_minhash_count = 0
for doc in annotated_data:
	if 'metadata' not in doc:
		missing_minhash_count += 1
	else:
		assert doc['metadata']['minhash']['cc_size'] == 25 - int(doc['text'].split('_')[-1])

assert missing_minhash_count == 1
```

--------------------------------

### Verify deduplication outputs

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_simple/README.md

Validates the integrity and content of the generated output files using Python.

```python
import glob, json
all_data = [json.loads(_) for f in glob.glob('test_data_outputs/exact_memory/*.jsonl') for _ in open(f).read().splitlines()]
assert len(all_data) == 25 # check only 25 surviving docs
assert set(_['text'] for _ in all_data) == set('this doc has content: %02d' % i for i in range(25)) # Check the content of the surviving docs


annotated_data = [json.loads(_) for f in glob.glob('test_data_outputs/exact_memory_anno/*.jsonl') for _ in open(f).read().splitlines()]
assert len(annotated_data) == 325 # check all data survived
assert all(_['metadata']['exact_dedup']['num_dups'] == 25 - int(_['text'].split(' ')[-1]) for _ in annotated_data) # check data is as expected
```

--------------------------------

### Clean Files (Remove Duplicates)

Source: https://context7.com/allenai/duplodocus/llms.txt

Removes duplicate files based on the computed MinHash signatures. Specify input, storage, and output directories. Set 'remove-duplicates' to true.

```bash
cargo run --release -- mh-clean-files \
  --input-dir /data/documents \
  --storage-dir /shared/work \
  --output-dir /data/deduped \
  --path-chunk 0 \
  --num-path-chunks 1 \
  --remove-duplicates true
```

--------------------------------

### Clean Files (Distributed)

Source: https://context7.com/allenai/duplodocus/llms.txt

Distributes the file cleaning process across multiple workers. Each worker handles a specific path chunk. This is suitable for large datasets.

```bash
for i in {0..9}; do
  cargo run --release -- mh-clean-files \
    --input-dir /data/documents \
    --storage-dir /shared/work \
    --output-dir /data/deduped \
    --path-chunk $i \
    --num-path-chunks 10 \
    --remove-duplicates true
done
```

--------------------------------

### Scrub Output Data

Source: https://github.com/allenai/duplodocus/blob/main/examples/fuzzy_multi/README.md

Apply deduplication results to the data by processing individual clean files.

```bash
for i in {0..3}; do 
	rm -rf test_data_outputs/local_storage
	mkdir -p test_data_outputs/local_storage/clean
	cp "test_data_outputs/s3_storage/clean/chunk_0000000${i}.00000000.clean.bin" test_data_outputs/local_storage/clean/
	cp test_data_outputs/s3_storage/filemap.json.gz test_data_outputs/local_storage/
	cargo run --release -- mh-clean-files \
	--input-dir test_data_inputs/fuzzy_multi/ \
	--storage-dir test_data_outputs/local_storage \
	--output-dir test_data_outputs/fuzzy_multi \
	--path-chunk $i \
	--num-path-chunks 4 \
	--annotate false --remove-duplicates true
	rm -rf test_data_outputs/local_storage
done
```

--------------------------------

### Generate test data

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_multi/README.md

Programmatically generate the test data pool for the deduplication process.

```python
python examples/make_example_pools.py --target-dir test_data_inputs/exact_multi --multi
```

--------------------------------

### Generate Fuzzy Deduplication Test Data

Source: https://context7.com/allenai/duplodocus/llms.txt

Generate synthetic test data for fuzzy deduplication, enabling the `--fuzzy` flag. This prepares data for MinHash-based comparisons.

```bash
# Generate fuzzy deduplication test data
python examples/make_example_pools.py \
  --target-dir test_data_inputs/fuzzy \
  --fuzzy
```

--------------------------------

### Configure Global Thread Count

Source: https://context7.com/allenai/duplodocus/llms.txt

Set the number of threads for parallel processing across all Duplodocus commands. Use 0 to default to all available threads.

```bash
# Limit to 8 threads
cargo run --release -- --threads 8 minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text"
```

```bash
# Use all available threads (default, threads=0)
cargo run --release -- exact-dedup-memory \
  --input-dir /data/documents \
  --output-dir /data/unique \
  --text-key "content"
```

--------------------------------

### Verify MinHash with True Jaccard

Source: https://context7.com/allenai/duplodocus/llms.txt

Computes actual pairwise Jaccard similarity for documents identified as duplicates by MinHash. This is for verification purposes. Specify input/output directories and thresholds.

```bash
cargo run --release -- true-jaccard \
  --input-dir /data/annotated \
  --output-dir /data/verified \
  --minhash-cc-id "metadata.minhash.cc_id" \
  --jaccard-threshold 0.8 \
  --ngram-size 5 \
  --tokenizer "cl100k" \
  --annotate-key "metadata.true_jaccard"
```

--------------------------------

### MinHash Fuzzy Deduplication (Memory-Based)

Source: https://context7.com/allenai/duplodocus/llms.txt

Performs fuzzy deduplication using MinHash LSH in memory. Identifies near-duplicates based on Jaccard similarity. Supports annotation and cleanup.

```bash
cargo run --release -- minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --num-buckets 20 \
  --bucket-size 5 \
  --remove-duplicates true \
  --cleanup-storage
```

```bash
cargo run --release -- minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/annotated \
  --text-key "text" \
  --num-buckets 26 \
  --bucket-size 11 \
  --ngram-size 5 \
  --permutation-seed 42 \
  --remove-duplicates false \
  --annotate true \
  --annotate-key "metadata.minhash" \
  --cleanup-storage
```

```bash
cargo run --release -- minhash-memory \
  --input-dir /data/documents \
  --storage-dir /tmp/work \
  --output-dir /data/deduped \
  --text-key "text" \
  --tokenizer "cl100k" \
  --num-buckets 20 \
  --bucket-size 5
```

--------------------------------

### Prune duplicates

Source: https://github.com/allenai/duplodocus/blob/main/examples/exact_multi/README.md

Run the pruning operation on each hash-slice to remove duplicates, with an optional annotated output.

```bash
for i in {0..3}; do 
	cargo run --release -- exact-dedup-disk-prune \
		--storage-dir "test_data_outputs/exact_storage/chunk_${i}" \
		--output-dir test_data_outputs/exact_multi \
		--hash-key metadata.text_hash
done
```

```bash
for i in {0..3}; do 
	cargo run --release -- exact-dedup-disk-prune \
		--storage-dir "test_data_outputs/exact_storage/chunk_${i}" \
		--output-dir test_data_outputs/exact_multi_anno \
		--hash-key metadata.text_hash \
		--annotate-key metadata.exact_dedup
done
```

--------------------------------

### Clean Files in Parallel (Disk-Based)

Source: https://github.com/allenai/duplodocus/blob/main/README.md

This command cleans up files in parallel across workers after the Union-Find structure is built. Specify the correct `--path-chunk` for each worker.

```bash
# Worker 0
cargo run --release -- mh-clean-files \
  --input-dir /data/docs \
  --storage-dir /shared/work \
  --output-dir /data/deduped \
  --path-chunk 0 \
  --num-path-chunks 10 \
  --remove-duplicates true

# Repeat for other workers...
```

--------------------------------

### Exact Deduplication (Memory-Based)

Source: https://context7.com/allenai/duplodocus/llms.txt

Removes or annotates exact duplicates in memory. Best for datasets under 10GB. Can use a pre-computed hash field.

```bash
cargo run --release -- exact-dedup-memory \
  --input-dir /data/documents \
  --output-dir /data/unique \
  --text-key "content"
```

```bash
cargo run --release -- exact-dedup-memory \
  --input-dir /data/documents \
  --output-dir /data/annotated \
  --text-key "content" \
  --annotate-key "metadata.exact_dedup"
```

```bash
cargo run --release -- exact-dedup-memory \
  --input-dir /data/documents \
  --output-dir /data/unique \
  --text-key "content" \
  --hash-key "doc_hash" \
  --hash-bits 128
```

--------------------------------

### Read Zstd-Compressed JSONL Files

Source: https://context7.com/allenai/duplodocus/llms.txt

Python function to read and parse data from zstd-compressed JSONL files, commonly used for disk-based output.

```python
import zstandard
import glob
import json

def read_zst_jsonl(filepath):
    """Read zstd-compressed JSONL file."""
    with open(filepath, 'rb') as f:
        dctx = zstandard.ZstdDecompressor()
        content = dctx.stream_reader(f).read()
        return [json.loads(line) for line in content.splitlines()]

```

--------------------------------

### Exact Deduplication (Disk-Based) - Pruning

Source: https://context7.com/allenai/duplodocus/llms.txt

Second stage for large datasets, processing grouped documents to remove or annotate duplicates. Supports multi-node processing per hash bin.

```bash
cargo run --release -- exact-dedup-disk-prune \
  --storage-dir /scratch/work \
  --output-dir /data/unique \
  --hash-key "doc_hash"
```

```bash
cargo run --release -- exact-dedup-disk-prune \
  --storage-dir /scratch/work \
  --output-dir /data/annotated \
  --hash-key "doc_hash" \
  --annotate-key "metadata.exact_dedup"
```

```bash
for i in {0..3}; do
  cargo run --release -- exact-dedup-disk-prune \
    --storage-dir "/shared/storage/chunk_${i}" \
    --output-dir /data/deduped \
    --hash-key "metadata.text_hash"
done
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.