### Install dedups via Bash Script

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

This snippet downloads and executes an installation script for dedups, automating the process of detecting the OS/architecture, downloading the correct binary, and installing it to a user's PATH.

```bash
curl -sSL https://raw.githubusercontent.com/AtlasPilotPuppy/dedup/main/install.sh | bash
```

```bash
curl -sSL https://raw.githubusercontent.com/AtlasPilotPuppy/dedup/main/install.sh > install.sh && chmod +x install.sh && ./install.sh
```

--------------------------------

### Media Deduplication CLI Usage Examples

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Provides command-line examples for using the `dedups` tool for media deduplication. Covers enabling media mode, setting resolution and format preferences, adjusting similarity thresholds, and defining workflows for professional photography and web optimization.

```bash
# Enable media mode with default settings
dedups /photos --media-mode

# Prefer highest resolution images
dedups /photos --media-mode --media-resolution highest

# Prefer specific formats (highest priority first)
dedups /photos --media-mode --media-formats raw,png,jpg

# Adjust similarity threshold (0-100)
dedups /photos --media-mode --media-similarity 85

# Professional photography workflow
dedups /photo_archive \
  --media-mode \
  --media-resolution highest \
  --media-formats raw,tiff,png,jpg \
  --media-similarity 95 \
  --move-to /duplicates

# Web optimization workflow (keep medium resolution)
dedups /web_images \
  --media-mode \
  --media-resolution 1920x1080 \
  --media-formats webp,jpg,png \
  --delete \
  --mode newest_modified
```

--------------------------------

### Algorithm Selection Guide (Rust)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Illustrates the characteristics of different hashing algorithms available in the dedups project, such as speed, output size, and collision risk. It includes a conceptual benchmark example.

```rust
// Algorithm characteristics (from implementation)
//
// Algorithm | Output Size | Speed      | Collision Risk | Use Case
// ----------|-------------|------------|----------------|------------------
// xxHash    | 16 hex      | Fastest    | Low            | Default (recommended)
// Blake3    | 64 hex      | Very Fast  | Very Low       | High security needs
// SHA256    | 64 hex      | Fast       | Very Low       | Cryptographic verification
// CRC32     | 8 hex       | Ultra Fast | Medium         | Quick checks only
// GxHash    | 16 hex      | Fastest*   | Low            | Linux-only optimization
// FNV-1a    | 16 hex      | Fast       | Low            | Simple applications
// MD5       | 32 hex      | Fast       | Medium         | Legacy compatibility
// SHA1      | 40 hex      | Fast       | Medium         | Legacy compatibility

// Example: Performance benchmark (conceptual)
fn benchmark_algorithms() {
    let test_file = "/data/large_file.bin"; // 1GB file

    // xxHash:  ~2-3 seconds
    // Blake3:  ~3-4 seconds
    // SHA256:  ~5-7 seconds
    // MD5:     ~4-5 seconds
    // CRC32:   ~1-2 seconds (but not recommended for deduplication)
}

```

--------------------------------

### Bash Examples for DEDUPS CLI Usage

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Provides practical command-line examples for using the DEDUPS tool. Demonstrates basic duplicate finding, specifying algorithms and threads, deleting duplicates, moving duplicates, and outputting results in JSON format.

```bash
# Find duplicates in current directory
dedups .

# Find duplicates with specific algorithm and thread count
dedups /path/to/directory --algorithm blake3 --parallel 8

# Find and delete duplicates, keeping newest files
dedups /path/to/directory --delete --mode newest_modified

# Move duplicates to separate folder
dedups /path/to/directory --move-to /path/to/duplicates --mode shortest_path

# Output results to JSON file
dedups /path/to/directory --output duplicates.json --format json

# Expected output format in JSON:
# [
#   {
#     "files": [
#       {"path": "/path/file1.jpg", "size": 2048576, "hash": "abc123", ...},
#       {"path": "/path/file2.jpg", "size": 2048576, "hash": "abc123", ...}
#     ],
#     "size": 2048576,
#     "hash": "abc123"
#   }
# ]
```

--------------------------------

### TUI Key Bindings Guide (Bash)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Lists the essential keyboard shortcuts and controls for navigating and operating within the dedups interactive TUI, covering file operations, actions, and settings.

```bash
# TUI controls:
# Navigation:
#   ↑/↓, j/k     - Move selection up/down
#   Tab          - Cycle between panels (Sets → Files → Jobs)
#   h/l, ←/→     - Switch between sets and files
#   Ctrl+G       - Toggle log focus
#
# File Operations:
#   s            - Mark to keep (others in set marked for deletion)
#   d            - Mark for deletion
#   c            - Copy file (prompts for destination)
#   a            - Toggle all files in set
#   i            - Ignore file
#
# Actions:
#   Ctrl+E       - Execute pending jobs
#   x/Del/Bksp   - Remove selected job
#   Ctrl+D       - Toggle dry run mode
#   Ctrl+R       - Rescan directory
#
# Settings:
#   Ctrl+S       - Open settings menu
#   Ctrl+L       - Clear log
#   h            - Show help screen
#   q/Ctrl+C     - Quit

```

--------------------------------

### Media Deduplication Workflow Example

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

A specialized workflow for deduplicating media files. It includes options for specifying media formats, similarity thresholds, and moving lower-quality duplicates to a separate directory.

```bash
#!/bin/bash
# Professional photo archive deduplication

PHOTO_DIR="/archive/photos"
DUPLICATES_DIR="/archive/duplicates"

# Step 1: Media scan with highest quality preference
dedups "$PHOTO_DIR" \
  --media-mode \
  --media-resolution highest \
  --media-formats raw,tiff,png,jpg \
  --media-similarity 95 \
  --output media_scan.json

# Step 2: Interactive review to verify similar images
dedups -i "$PHOTO_DIR" --media-mode

# Step 3: Move lower quality duplicates
dedups "$PHOTO_DIR" \
  --media-mode \
  --media-resolution highest \
  --media-formats raw,tiff,png,jpg \
  --move-to "$DUPLICATES_DIR" \
  --log --log-file media_cleanup.log

# Step 4: Review moved files
ls -lh "$DUPLICATES_DIR"
```

--------------------------------

### Install dedups using Cargo

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Installs the dedups binary using the Rust package manager, Cargo. This is a straightforward method for users who have Rust and Cargo set up.

```rust
cargo install dedup
```

--------------------------------

### Basic Cleanup Workflow Example

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

A comprehensive bash script illustrating a common cleanup workflow. It includes scanning, interactive review, automated deletion based on modification time, and log file generation for verification.

```bash
#!/bin/bash
# Complete workflow for cleaning up a photo directory

# Step 1: Scan and export report
dedups /photos --output scan_results.json --algorithm xxhash

# Step 2: Review results (open scan_results.json)
# Identify duplicate patterns

# Step 3: Interactive review
dedups -i /photos

# Step 4: Automated cleanup with newest files kept
dedups /photos \
  --delete \
  --mode newest_modified \
  --include "*.jpg" "*.png" "*.raw" \
  --exclude "*backup*" "*tmp*" \
  --log \
  --log-file cleanup.log

# Step 5: Verify results
grep "Deleted" cleanup.log
```

--------------------------------

### Backup Synchronization Workflow Example

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

This bash script outlines a workflow for synchronizing files to a backup location. It leverages caching for performance and deduplicates both the source and the backup directories.

```bash
#!/bin/bash
# Synchronize files from source to backup

SOURCE="/data/photos"
BACKUP="/backup/photos"
CACHE_DIR="/tmp/dedup_cache"

# Step 1: Cache hashes for source files (first run only)
dedups "$SOURCE" \
  --cache-location "$CACHE_DIR" \
  --output source_scan.json

# Step 2: Clean duplicates in source
dedups "$SOURCE" \
  --delete \
  --mode newest_modified \
  --cache-location "$CACHE_DIR" \
  --fast-mode \
  --log --log-file source_cleanup.log

# Step 3: Copy missing files to backup
dedups "$SOURCE" "$BACKUP" \
  --cache-location "$CACHE_DIR" \
  --fast-mode

# Step 4: Deduplicate across both directories
dedups "$SOURCE" "$BACKUP" \
  --deduplicate \
  --output sync_results.json

# Total time saved with caching: ~80% on subsequent runs
```

--------------------------------

### Filter File Format Example

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Illustrates the format for filter files used with the `dedups` tool, mimicking rclone's style. It shows how to define include patterns (prefixed with '+') and exclude patterns (prefixed with '-') within a text file, and how to use this file with the `--filter-from` CLI option.

```bash
# Create a filter file (rclone-style format)
cat > filters.txt << 'EOF'
# Include patterns (prefixed with +)
+ *.jpg
+ *.png
+ *.mp4

# Exclude patterns (prefixed with -)
- *tmp*
- *cache*
- *backup*
- *.partial

# Comments start with # or ;
; This is also a comment
EOF

# Use the filter file
dedups /data --filter-from filters.txt --delete --mode newest_modified
```

--------------------------------

### Parallel Hash Calculation in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

An example of parallel hash calculation using the Rayon library in Rust. This snippet shows how to configure the thread pool and process files concurrently for efficient hashing.

```rust
// Parallel hash calculation (simplified from actual implementation)
use rayon::prelude::*;
use std::sync::mpsc;

fn calculate_hashes_parallel(
    files: Vec<FileInfo>,
    algorithm: &str,
    thread_count: usize
) -> Vec<FileInfo> {
    // Configure Rayon thread pool
    rayon::ThreadPoolBuilder::new()
        .num_threads(thread_count)
        .build()
        .unwrap();

    // Process files in parallel
    files.par_iter()
        .map(|file_info| {
            let mut updated_file = file_info.clone();
            updated_file.hash = Some(calculate_hash(&file_info.path, algorithm));
            updated_file
        })
        .collect()
}
```

--------------------------------

### Sort Criteria and Order Enumerations (Rust)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Defines the `SortCriterion` and `SortOrder` enumerations used for sorting files within the dedups project. Includes an example of parsing these from strings.

```rust
use dedups::file_utils::{SortCriterion, SortOrder};
use std::str::FromStr;

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SortCriterion {
    FileName,
    FileSize,
    CreatedAt,
    ModifiedAt,
    PathLength,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SortOrder {
    Ascending,
    Descending,
}

// Example: Using sort options
fn sort_examples() -> anyhow::Result<()> {
    // Parse from string
    let criterion = SortCriterion::from_str("modified")?;
    let order = SortOrder::from_str("desc")?;

    println!("Sorting by: {}", criterion); // "modified"
    println!("Order: {}", order);           // "descending"

    Ok(())
}

```

--------------------------------

### Scheduled Deduplication Tasks (Cron)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Cron job entries for scheduling weekly and daily file deduplication tasks using the DEDUPS command-line tool. These examples demonstrate how to configure deduplication for general files and media workflows, including specifying target directories, deletion modes, cache locations, logging, and media-specific options.

```cron
0 2 * * 0 /usr/local/bin/dedups /data --delete --mode newest_modified --cache-location /var/cache/dedups --fast-mode --log --log-file /var/log/dedups-weekly.log 2>&1

# Daily media deduplication for photography workflow
0 1 * * * /usr/local/bin/dedups /photos --media-mode --media-resolution highest --move-to /photos/duplicates --log --log-file /var/log/photo-dedup.log 2>&1
```

--------------------------------

### Prepare and Run Sample Media Deduplication Script

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Steps to make the sample media script executable, run it to generate diverse media files, and then test deduplication in interactive and CLI modes.

```bash
# Make the script executable
chmod +x sample_media.sh

# Run the script to create sample media files
./sample_media.sh

# Test media deduplication on the sample files (interactive mode)
dedups -i demo --media-mode

# For CLI mode with specific options
dedups --dry-run demo --media-mode --media-resolution highest --media-formats png,jpg,mp4
```

--------------------------------

### Recommended Media Deduplication Settings for Audio

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Configuration for audio collections, prioritizing lossless formats like FLAC, followed by common compressed formats like MP3 and Ogg.

```bash
dedups /path/to/audio --media-mode --media-formats flac,mp3,ogg
```

--------------------------------

### Hash Algorithm Selection via CLI

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Lists command-line options for selecting different hash algorithms with the `dedups` tool. It covers legacy MD5, secure SHA256, fast Blake3, the default xxHash, Linux-specific GxHash, and FNV-1a.

```bash
# MD5 (legacy, fast but less secure)
dedups /data --algorithm md5

# SHA256 (cryptographically secure)
dedups /data --algorithm sha256

# Blake3 (modern, fast, secure)
dedups /data --algorithm blake3

# xxHash (fastest, default, good collision resistance)
dedups /data --algorithm xxhash

# GxHash (Linux-only, optimized for modern CPUs)
dedups /data --algorithm gxhash

# FNV-1a (fast, simple)
dedups /data --algorithm fnv1a
```

--------------------------------

### Recommended Media Deduplication Settings for Photography

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Optimized settings for professional photography, prioritizing the highest resolution and common professional formats like RAW, TIFF, PNG, and JPG.

```bash
dedups /path/to/photos --media-mode --media-resolution highest --media-formats raw,tiff,png,jpg
```

--------------------------------

### Build dedups from Source using Cargo

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Provides instructions to build the dedups project from its source code using Cargo. This involves cloning the repository, navigating to the directory, and executing the build command. The release binary will be located in `target/release/dedup`.

```rust
# Clone the repository
git clone https://github.com/AtlasPilotPuppy/dedup
cd dedup

# Build in release mode
cargo build --release

# The binary will be available at target/release/dedup
```

--------------------------------

### Recommended Media Deduplication Settings for Web/Mobile

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Settings tailored for web and mobile optimization, focusing on a common resolution (1920x1080) and web-friendly formats like WebP, JPG, and PNG.

```bash
dedups /path/to/images --media-mode --media-resolution 1920x1080 --media-formats webp,jpg,png
```

--------------------------------

### TUI Interactive Mode Launch (Bash)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Shows how to launch the dedups tool in interactive TUI mode for managing duplicate files manually.

```bash
# Launch interactive mode
dedups -i /path/to/directory

```

--------------------------------

### Load Dedup Configuration from TOML and Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Illustrates loading deduplication configuration from a TOML file and using it within a Rust application. It covers loading from default and custom paths, and ensuring a default configuration file exists.

```toml
# ~/.deduprc configuration file example
algorithm = "xxhash"
parallel = 4
mode = "newest_modified"
format = "json"
progress = true
sort_by = "modified"
sort_order = "descending"

include = ["*.jpg", "*.png", "*.mp4"]
exclude = ["*tmp*", "*cache*", "*.partial"]

cache_location = "/home/user/.dedup_cache"
fast_mode = true

[media_dedup]
enabled = false
similarity_threshold = 90
```

```rust
// Loading and using configuration
use dedups::config::DedupConfig;

fn load_config() -> anyhow::Result<()> {
    // Load from default location
    let config = DedupConfig::load()?;
    println!("Using algorithm: {}", config.algorithm);

    // Or load from custom path
    let custom_config = DedupConfig::load_from_path(
        &PathBuf::from("/path/to/custom-config.toml")
    )?;

    // Create default config file if it doesn't exist
    if DedupConfig::create_default_if_not_exists()? {
        println!("Created default configuration file");
    }

    Ok(())
}
```

--------------------------------

### Basic dedups Command-Line Usage

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Demonstrates fundamental ways to use the dedups command-line tool. Includes initiating the interactive TUI, scanning a specific directory, and performing a non-interactive delete operation with a specified mode.

```bash
# Find duplicates in the current directory using the TUI
dedups -i

# Find duplicates in a specific directory
dedups /path/to/directory

# Find and delete duplicates (non-interactive)
dedups /path/to/directory --delete --mode newest_modified

# Use a custom config file
dedups /path/to/directory --config-file /path/to/my-config.toml
```

--------------------------------

### Enable Media Deduplication Mode

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Activates the media deduplication feature, allowing the tool to detect similar media files across different formats, resolutions, and quality levels.

```bash
dedups /path/to/media --media-mode
```

--------------------------------

### Selection Strategies Usage (Bash)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Demonstrates command-line usage of the dedups tool for selecting and deleting duplicate files based on various strategies like newest/oldest modified, shortest/longest path, and sorting output.

```bash
# Keep newest files, delete older duplicates
dedups /data --delete --mode newest_modified

# Keep oldest files (original copies)
dedups /data --delete --mode oldest_modified

# Keep files with shortest paths (closest to root)
dedups /data --delete --mode shortest_path

# Keep files with longest paths (deeply nested)
dedups /data --delete --mode longest_path

# Sort results by different criteria
dedups /data --sort-by size --sort-order desc --output results.json
dedups /data --sort-by name --sort-order asc --output sorted.json
dedups /data --sort-by modified --sort-order desc

```

--------------------------------

### Execute Dry Run for Simulated Operations

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Demonstrates the `--dry-run` option, which allows users to preview the actions dedup would take without actually modifying any files. This is crucial for verifying cleanup or synchronization logic before execution.

```bash
# Simulate operations without making changes
dedups /data --delete --mode newest_modified --dry-run

# Output shows what would happen:
# [DRY RUN] Would delete: /data/duplicate1.jpg
# [DRY RUN] Would delete: /data/duplicate2.jpg
# [DRY RUN] Would keep: /data/original.jpg
# [DRY RUN] Total space that would be freed: 15.3 MB

# Use with interactive mode
dedups -i /data --dry-run
# (Toggle dry run in TUI with Ctrl+D)
```

--------------------------------

### Execute Multi-Directory Operations with Dedups CLI

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Demonstrates how to use the dedups command-line interface to copy, deduplicate, and manage files across multiple directories. Supports various options like dry-run and specifying a target directory.

```bash
# Copy missing files from source to target
dedups /source/photos /backup/photos

# Deduplicate across multiple directories
dedups /source/photos /backup/photos --deduplicate

# Copy from multiple sources to single target
dedups /photos/2020 /photos/2021 /photos/2022 --target /archive/all_photos

# Dry run to see what would happen
dedups /source /target --deduplicate --dry-run
```

--------------------------------

### Specify Custom Configuration File with dedups

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

This command demonstrates how to specify a custom configuration file for the dedups tool using the --config-file option. This is useful for managing different sets of configurations for various directories or projects. Ensure the specified path points to a valid TOML configuration file.

```bash
dedups --config-file /path/to/my-config.toml /path/to/directory
```

--------------------------------

### Configure Logging Output

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Shows how to enable and configure logging for dedup operations. Users can specify log file locations and control the verbosity of the output using different levels.

```bash
# Enable logging to default file (dedups.log)
dedups /data --log

# Specify custom log file
dedups /data --log --log-file /var/log/dedups.log

# Verbose output levels
dedups /data -v         # INFO level
dedups /data -vv        # DEBUG level
dedups /data -vvv       # TRACE level

# Combine logging options
dedups /data --log --log-file scan.log -vv --delete --mode newest_modified
```

--------------------------------

### Rust CLI Structure Definition with Clap

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Defines the command-line interface structure using the Clap library for parsing arguments. It includes options for directories, target, deduplication actions, output formats, hashing algorithms, and various filtering and mode settings.

```rust
use clap::Parser;
use std::path::PathBuf;

#[derive(Parser, Debug, Clone)]
#[clap(author, version, about, long_about = None)]
pub struct Cli {
    pub directories: Vec<PathBuf>,
    pub target: Option<PathBuf>,
    pub deduplicate: bool,
    pub delete: bool,
    pub move_to: Option<PathBuf>,
    pub log: bool,
    pub log_file: Option<PathBuf>,
    pub output: Option<PathBuf>,
    pub format: String,
    pub algorithm: String,
    pub parallel: Option<usize>,
    pub mode: String,
    pub interactive: bool,
    pub include: Vec<String>,
    pub exclude: Vec<String>,
    pub filter_from: Option<PathBuf>,
    pub progress: bool,
    pub cache_location: Option<PathBuf>,
    pub fast_mode: bool,
    pub media_mode: bool,
    pub media_resolution: String,
    pub media_formats: Vec<String>,
    pub media_similarity: u32,
    pub dry_run: bool,
}

// Example: Parsing CLI arguments with config file
use dedups::Cli;

fn main() -> anyhow::Result<()> {
    // Parse CLI args and apply config file defaults
    let cli = Cli::with_config()?;

    println!("Scanning directories: {:?}", cli.directories);
    println!("Algorithm: {}", cli.algorithm);
    println!("Parallel threads: {:?}", cli.parallel);

    Ok(())
}
```

--------------------------------

### Set Media Resolution Preference

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Configures the preferred resolution for media deduplication. Options include 'highest', 'lowest', or a custom resolution specified as 'WIDTHxHEIGHT'.

```bash
dedups /path/to/media --media-mode --media-resolution highest
```

```bash
dedups /path/to/media --media-mode --media-resolution lowest
```

```bash
dedups /path/to/media --media-mode --media-resolution 1280x720
```

--------------------------------

### Define and Create Dedup Configuration in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Shows how to define a `DedupConfig` struct in Rust using Serde for serialization/deserialization, and then create and save a default configuration with custom settings. This configuration manages deduplication parameters.

```rust
use serde::{Deserialize, Serialize};
use std::path::PathBuf;

#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct DedupConfig {
    pub algorithm: String,
    pub parallel: Option<usize>,
    pub mode: String,
    pub format: String,
    pub progress: bool,
    pub sort_by: String,
    pub sort_order: String,
    pub include: Vec<String>,
    pub exclude: Vec<String>,
    pub cache_location: Option<PathBuf>,
    pub fast_mode: bool,
    pub media_dedup: MediaDedupOptions,
}

// Example: Creating and saving a configuration
use dedups::config::DedupConfig;

fn setup_config() -> anyhow::Result<()> {
    let mut config = DedupConfig::default();
    config.algorithm = "blake3".to_string();
    config.parallel = Some(8);
    config.include = vec!["*.jpg".to_string(), "*.png".to_string()];
    config.exclude = vec!["*tmp*".to_string(), "*cache*".to_string()];
    config.cache_location = Some(PathBuf::from("/home/user/.dedup_cache"));
    config.fast_mode = true;

    // Save to ~/.deduprc (Linux/macOS) or %USERPROFILE%.deduprc (Windows)
    config.save()?;

    Ok(())
}
```

--------------------------------

### Apply File Filters Programmatically in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Shows how to programmatically create and apply `FilterRules` using the `dedups::file_utils` module in Rust. It demonstrates initializing `FilterRules` from `Cli` arguments, which include include and exclude patterns for file filtering during operations.

```rust
use dedups::file_utils::FilterRules;
use dedups::Cli;

// FilterRules are created from CLI arguments
#[derive(Debug, Default)]
pub struct FilterRules {
    includes: Vec<Pattern>,
    excludes: Vec<Pattern>,
}

// Example: Using filter rules programmatically
fn apply_filters() -> anyhow::Result<()> {
    let cli = Cli {
        directories: vec![PathBuf::from("/data")],
        include: vec!["*.jpg".to_string(), "*.png".to_string()],
        exclude: vec!["*tmp*".to_string(), "*cache*".to_string()],
        filter_from: None,
        ..Default::default()
    };

    let filter_rules = FilterRules::new(&cli)?;
    // Filter rules are now applied during file scanning

    Ok(())
}
```

--------------------------------

### Configure Media Deduplication Options in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Illustrates how to create and configure `MediaDedupOptions` in Rust for various deduplication scenarios. It covers preferences for resolution (highest, lowest, closest to specific dimensions) and formats, along with similarity thresholds.

```rust
use dedups::media_dedup::{
    MediaDedupOptions,
    ResolutionPreference,
    FormatPreference
};

// Example: Configuring media deduplication options
fn configure_media_dedup() -> MediaDedupOptions {
    let options = MediaDedupOptions {
        enabled: true,
        resolution_preference: ResolutionPreference::Highest,
        format_preference: FormatPreference {
            formats: vec![
                "raw".to_string(),
                "png".to_string(),
                "jpg".to_string(),
            ],
        },
        similarity_threshold: 90, // 90% similarity required
    };

    // For keeping lowest resolution instead
    let low_res_options = MediaDedupOptions {
        enabled: true,
        resolution_preference: ResolutionPreference::Lowest,
        format_preference: FormatPreference::default(),
        similarity_threshold: 85,
    };

    // For targeting specific resolution
    let target_res_options = MediaDedupOptions {
        enabled: true,
        resolution_preference: ResolutionPreference::ClosestTo(1920, 1080),
        format_preference: FormatPreference::default(),
        similarity_threshold: 95,
    };

    options
}
```

--------------------------------

### Detect Media File Types with Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Demonstrates how to use the `detect_media_type` function from the `dedups::media_dedup` module to identify if a given file path corresponds to an image, video, or audio file. It handles different media kinds and unknown types.

```rust
use dedups::media_dedup::{MediaKind, detect_media_type};
use std::path::Path;

// Example: Detecting media file types
fn detect_media_files() {
    let image_path = Path::new("/photos/vacation.jpg");
    let video_path = Path::new("/videos/movie.mp4");
    let audio_path = Path::new("/music/song.mp3");

    match detect_media_type(image_path) {
        MediaKind::Image => println!("Image file detected"),
        MediaKind::Video => println!("Video file detected"),
        MediaKind::Audio => println!("Audio file detected"),
        MediaKind::Unknown => println!("Unknown media type"),
    }
}
```

--------------------------------

### dedups Multi-Directory Operations

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Illustrates how to use dedups for operations involving multiple directories. This includes copying missing files between directories, explicitly defining a target directory, and performing deduplication with output to a JSON file.

```bash
# Copy missing files from source to target directory
dedups /source/directory /target/directory

# Explicitly specify a target directory (can be useful with multiple source directories)
dedups /source/dir1 /source/dir2 --target /target/directory

# Deduplicate between directories and copy missing files
dedups /source/directory /target/directory --deduplicate

# Find duplicates in both source and target (without copying)
# and save the results to a file
dedups /source/directory /target/directory --deduplicate -o duplicates.json
```

--------------------------------

### Set Media Format Preferences

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Specifies a comma-separated list of preferred media formats, ordered by priority, for deduplication. This helps in identifying or prioritizing certain file types.

```bash
dedups /path/to/media --media-mode --media-formats raw,png,jpg
```

--------------------------------

### Selection Strategy Implementation (Rust)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Provides a conceptual Rust implementation for selecting which file to keep from a set of duplicates based on different strategies, such as modification time or path length.

```rust
// Selection strategy implementation (conceptual)
fn select_files_to_keep(
    duplicate_set: &DuplicateSet,
    strategy: &str
) -> Vec<FileInfo> {
    match strategy {
        "newest_modified" => {
            // Keep file with most recent modification time
            let mut files = duplicate_set.files.clone();
            files.sort_by_key(|f| f.modified_at);
            vec![files.last().unwrap().clone()]
        }
        "shortest_path" => {
            // Keep file with shortest path
            let mut files = duplicate_set.files.clone();
            files.sort_by_key(|f| f.path.to_string_lossy().len());
            vec![files.first().unwrap().clone()]
        }
        // ... other strategies
        _ => vec![],
    }
}

```

--------------------------------

### One-Step Deduplication Between Directories

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Performs a deduplication operation directly between a source and target directory in a single command, synchronizing unique files while handling duplicates.

```bash
dedups /source/photos /target/backup --deduplicate
```

--------------------------------

### Utilize File Cache for Deduplication Scans in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Demonstrates how to use the `FileCache` system in Rust to speed up file scans by storing and retrieving file hashes. It shows initializing the cache, checking for existing hashes, calculating and storing new ones, and persisting the cache.

```rust
use dedups::file_cache::FileCache;
use dedups::file_utils::FileInfo;
use std::path::{Path, PathBuf};

// Example: Using file cache for faster scans
fn use_file_cache() -> anyhow::Result<()> {
    let cache_dir = PathBuf::from("/home/user/.dedup_cache");
    let algorithm = "xxhash";

    // Initialize cache (loads existing cache if available)
    let mut cache = FileCache::new(&cache_dir, algorithm)?;

    let file_path = Path::new("/home/user/photo.jpg");

    // Try to get hash from cache
    if let Some(cached_hash) = cache.get_hash(file_path) {
        println!("Cache hit! Hash: {}", cached_hash);
        // Skip recalculating hash
    } else {
        println!("Cache miss - calculating hash");

        // Calculate hash (pseudo-code)
        let file_info = FileInfo {
            path: file_path.to_path_buf(),
            size: 2048576,
            hash: Some("calculated_hash_value".to_string()),
            modified_at: Some(std::time::SystemTime::now()),
            created_at: None,
        };

        // Store in cache
        cache.store(&file_info, algorithm)?;
    }

    // Persist cache to disk
    cache.save()?;

    Ok(())
}
```

--------------------------------

### Perform Batch Operations on File Cache in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Illustrates how to efficiently store multiple file hashes in the cache using a batch operation in Rust. This method is more performant than storing files individually. It also shows how to clear the cache.

```rust
use dedups::file_cache::FileCache;
use dedups::file_utils::FileInfo;

// Example: Batch storing file hashes
fn batch_cache_operation() -> anyhow::Result<()> {
    let cache_dir = PathBuf::from("/tmp/cache");
    let mut cache = FileCache::new(&cache_dir, "blake3")?;

    let files = vec![
        FileInfo {
            path: PathBuf::from("/data/file1.dat"),
            size: 1024,
            hash: Some("hash1".to_string()),
            modified_at: Some(std::time::SystemTime::now()),
            created_at: None,
        },
        FileInfo {
            path: PathBuf::from("/data/file2.dat"),
            size: 2048,
            hash: Some("hash2".to_string()),
            modified_at: Some(std::time::SystemTime::now()),
            created_at: None,
        },
    ];

    // Batch store is more efficient than individual stores
    cache.store_batch(&files, "blake3")?;
    cache.save()?;

    // Clear cache if needed
    cache.clear();

    Ok(())
}
```

--------------------------------

### Integrate Dedup into Backup Scripts

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

A bash function demonstrating how to integrate dedup into a backup script. It performs deduplication before initiating a backup, ensuring that only unique files are transferred.

```bash
#!/bin/bash
# Integration with backup script

function dedup_before_backup() {
    local DIR="$1"
    local LOG="/var/log/dedup-$(date +%Y%m%d).log"

    echo "Deduplicating $DIR before backup..."

    dedups "$DIR" \
        --delete \
        --mode newest_modified \
        --algorithm blake3 \
        --parallel 8 \
        --cache-location /var/cache/dedups \
        --fast-mode \
        --log --log-file "$LOG" \
        --exclude "*tmp*" "*cache*" "*.partial"

    local EXIT_CODE=$?
    if [ $EXIT_CODE -eq 0 ]; then
        echo "Deduplication completed successfully"
        return 0
    else
        echo "Deduplication failed with exit code $EXIT_CODE"
        return $EXIT_CODE
    fi
}

# Use in backup pipeline
dedup_before_backup "/data/documents" && rsync -av /data/documents /backup/
```

--------------------------------

### Full Synchronization with Deduplication

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

A multi-step process to ensure synchronization between two directories. It first cleans duplicates in both source and target, then copies missing files from source to target.

```bash
# Step 1: Clean duplicates in the target directory
dedups /target/backup --delete --mode newest_modified
# Step 2: Clean duplicates in the source directory
dedups /source/photos --delete --mode newest_modified
# Step 3: Copy missing files from source to target
dedups /source/photos /target/backup
```

--------------------------------

### Adjust Media Similarity Threshold

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Sets the similarity threshold for media deduplication, ranging from 0 to 100. The default value is 90. Lower values detect more, potentially less similar, files.

```bash
dedups /path/to/media --media-mode --media-similarity 85
```

--------------------------------

### TUI Application State Structure (Rust)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Defines a conceptual Rust structure for the `AppState` in the dedups TUI, outlining the data it holds, such as grouped duplicate files, pending jobs, and operational modes like dry run.

```rust
use dedups::tui_app::AppState;

// Example: TUI state structure (conceptual)
pub struct AppState {
    pub grouped_data: Vec<ParentFolderGroup>,
    pub jobs: Vec<Job>,
    pub is_loading: bool,
    pub dry_run: bool,
    pub log_messages: Vec<String>,
    pub media_mode: bool,
    pub current_algorithm: String,
    pub parallel_threads: usize,
}

pub enum ActionType {
    Keep,
    Delete,
    Move(PathBuf),
    Copy(PathBuf),
    Ignore,
}

pub struct Job {
    pub action: ActionType,
    pub file_path: PathBuf,
}

```

--------------------------------

### Rust FileInfo Structure Definition

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Defines the FileInfo structure used to store metadata about files, including path, size, hash, and modification/creation times. It is serializable and deserializable using serde and supports comparison.

```rust
use std::path::PathBuf;
use std::time::SystemTime;

#[derive(Debug, Clone, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
pub struct FileInfo {
    pub path: PathBuf,
    pub size: u64,
    pub hash: Option<String>,
    pub modified_at: Option<SystemTime>,
    pub created_at: Option<SystemTime>,
}

// Example: Creating a FileInfo
let file = FileInfo {
    path: PathBuf::from("/home/user/documents/photo.jpg"),
    size: 2048576,
    hash: Some("a3c4e5f7b9d1".to_string()),
    modified_at: Some(SystemTime::now()),
    created_at: Some(SystemTime::now()),
};
```

--------------------------------

### Schedule Weekly Deduplication with Cron

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

A comment providing guidance on how to schedule weekly deduplication tasks using cron jobs. This ensures regular maintenance of storage space.

```bash
# Add to crontab for weekly deduplication

```

--------------------------------

### Control Parallel Processing with Thread Count

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Demonstrates how to specify the number of threads for dedup operations. This allows users to control resource utilization, with options ranging from auto-detection to explicit counts and using the maximum available CPU cores.

```bash
# Auto-detect thread count (default)
dedups /data

# Explicit thread count
dedups /data --parallel 8

# Single-threaded (useful for debugging)
dedups /data --parallel 1

# Maximum threads (number of CPU cores)
dedups /data --parallel $(nproc)
```

--------------------------------

### Copy Missing Files Between Directories

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Copies files present in source directories but missing in the target directory. This is useful for backing up or synchronizing data.

```bash
dedups /source/dir1 /source/dir2 /source/dir3 /target/directory
```

```bash
dedups /source/photos /target/backup
```

```bash
dedups /photos/2020 /photos/2021 /photos/2022 /backup/all_photos
```

--------------------------------

### Export Duplicate Report

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Scans a directory for duplicates and exports a report detailing the duplicates found in JSON format to a specified file.

```bash
dedups /path/to/photos -o duplicates.json
```

--------------------------------

### Deduplicate Target Directory and Copy Unique Files

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

First, it cleans duplicates within the target directory, keeping the newest modified files. Then, it copies unique files from the source to the target.

```bash
dedups /target/directory --delete --mode newest_modified
dedups /source/directory /target/directory
```

--------------------------------

### Use File Caching for Faster Scans

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Enables file caching using the '--cache-location' option and '--fast-mode' to speed up subsequent scans of the same directories by reusing previous scan results.

```bash
dedups /path/to/photos --cache-location ~/.dedup_cache --fast-mode
```

--------------------------------

### List Duplicates in a Single Directory

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Scans a specified directory to find and list all duplicate files without taking any action on them. Useful for initial review.

```bash
dedups /path/to/photos
```

--------------------------------

### Analyze Deduplication Results (Python)

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

A Python script to parse and analyze JSON output from the DEDUPS tool. It calculates the total number of duplicate files, estimates wasted disk space, and provides a breakdown of duplicates by file type. Requires the 'json' and 'sys' modules. The script takes a JSON file path as a command-line argument.

```python
#!/usr/bin/env python3
# Parse dedups JSON output

import json
import sys

def analyze_duplicates(json_file):
    with open(json_file, 'r') as f:
        duplicate_sets = json.load(f)

    total_duplicates = 0
    total_wasted_space = 0

    for dup_set in duplicate_sets:
        file_count = len(dup_set['files'])
        if file_count > 1:
            # Count all but one as duplicates
            total_duplicates += (file_count - 1)
            # Wasted space = size * (count - 1)
            total_wasted_space += dup_set['size'] * (file_count - 1)

    print(f"Total duplicate files: {total_duplicates}")
    print(f"Total wasted space: {total_wasted_space / (1024**3):.2f} GB")

    # Group by file extension
    extensions = {}
    for dup_set in duplicate_sets:
        for file_info in dup_set['files']:
            path = file_info['path']
            ext = path.split('.')[-1].lower() if '.' in path else 'none'
            extensions[ext] = extensions.get(ext, 0) + 1

    print("\nDuplicates by file type:")
    for ext, count in sorted(extensions.items(), key=lambda x: x[1], reverse=True):
        print(f"  .{ext}: {count} files")

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print("Usage: analyze_duplicates.py <duplicates.json>")
        sys.exit(1)

    analyze_duplicates(sys.argv[1])
```

--------------------------------

### Conceptual File Filtering Logic in Rust

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Presents a conceptual Rust function `should_include_file` that outlines the logic for filtering files based on `FilterRules`. It prioritizes checking exclude patterns first, then checks include patterns if they exist, and defaults to inclusion if no exclude matches.

```rust
// Filter matching logic (conceptual)
fn should_include_file(path: &Path, rules: &FilterRules) -> bool {
    // 1. Check excludes first - if any match, exclude
    for exclude_pattern in &rules.excludes {
        if exclude_pattern.matches_path(path) {
            return false; // Excluded
        }
    }

    // 2. Check includes - if includes exist, must match at least one
    if !rules.includes.is_empty() {
        let matches_include = rules.includes.iter()
            .any(|pattern| pattern.matches_path(path));
        return matches_include;
    }

    // 3. Default: include if not excluded
    true
}
```

--------------------------------

### Move Duplicates to a Separate Folder

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Identifies duplicate files and moves them to a specified 'duplicates' folder instead of deleting them. The 'shortest_path' mode determines which file to keep based on its path.

```bash
dedups /path/to/photos --move-to /path/to/duplicates --mode shortest_path
```

--------------------------------

### Rust DuplicateSet Structure Definition

Source: https://context7.com/atlaspilotpuppy/dedup/llms.txt

Defines the DuplicateSet structure, which groups multiple FileInfo objects that represent duplicate files. It stores a vector of FileInfo, the total size of duplicates, and their common hash. It is serializable.

```rust
#[derive(Debug, Clone, serde::Serialize)]
pub struct DuplicateSet {
    pub files: Vec<FileInfo>,
    pub size: u64,
    pub hash: String,
}

// Example: A duplicate set with 3 identical files
let duplicate_set = DuplicateSet {
    files: vec![
        FileInfo {
            path: PathBuf::from("/photos/vacation1.jpg"),
            size: 1048576,
            hash: Some("abc123".to_string()),
            modified_at: None,
            created_at: None,
        },
        FileInfo {
            path: PathBuf::from("/backup/vacation1.jpg"),
            size: 1048576,
            hash: Some("abc123".to_string()),
            modified_at: None,
            created_at: None,
        },
        FileInfo {
            path: PathBuf::from("/archive/photos/vacation1.jpg"),
            size: 1048576,
            hash: Some("abc123".to_string()),
            modified_at: None,
            created_at: None,
        },
    ],
    size: 1048576,
    hash: "abc123".to_string(),
};
```

--------------------------------

### Delete Duplicates in a Single Directory

Source: https://github.com/atlaspilotpuppy/dedup/blob/main/README.md

Finds duplicate files in a directory and deletes them, retaining the file that was most recently modified. Use with caution.

```bash
dedups /path/to/photos --delete --mode newest_modified
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.