FastCDC

https://docs.rs/fastcdc/3.2.1/
Admin
FastCDC is a Rust library that implements Fast Content-Defined Chunking, an algorithm for data...
Tokens:39,919
Snippets:295
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### AsyncStreamCDC Example Usage

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/async_stream_cdc.rs.html

Demonstrates how to create an AsyncStreamCDC instance, obtain an async stream of chunks, and collect them. This example requires the 'futures' or 'tokio' feature flag.

```rust
use std::fs::File;
use fastcdc::v2020::AsyncStreamCDC;
#[cfg(all(feature = "futures", not(feature = "tokio")))]
use futures::stream::StreamExt;
#[cfg(all(feature = "tokio", not(feature = "futures")))]
use tokio_stream::StreamExt;

async fn run() {
    let source = std::fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
    let mut chunker = AsyncStreamCDC::new(source.as_ref(), 4096, 16384, 65535);
    let stream = chunker.as_stream();

    let chunks = stream.collect::<Vec<_>>().await;

    for result in chunks {
        let chunk = result.unwrap();
        println!("offset={} length={}", chunk.offset, chunk.length);
    }
}
```

--------------------------------

### AsyncStreamCDC Usage Example

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.AsyncStreamCDC.html

Demonstrates how to use AsyncStreamCDC to read a file, generate chunks asynchronously, and collect them. This example requires a file named 'test/fixtures/SekienAkashita.jpg' to exist.

```rust
async fn run() {
    let source = std::fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
    let mut chunker = AsyncStreamCDC::new(source.as_ref(), 4096, 16384, 65535);
    let stream = chunker.as_stream();

    let chunks = stream.collect::<Vec<_>>().await;

    for result in chunks {
        let chunk = result.unwrap();
        println!("offset={} length={}", chunk.offset, chunk.length);
    }
}
```

--------------------------------

### Migration Example: Pre-3.0 vs. Post-3.0 FastCDC Initialization

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/lib.rs.html

Illustrates the change in how to instantiate the FastCDC chunker before and after version 3.0 of the crate. The 'ronomon' implementation remains consistent.

```rust
# use fastcdc::ronomon as fastcdc;
# use std::fs;
# let contents = fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = fastcdc::FastCDC::new(&contents, 8192, 16384, 32768);
```

```rust
# use std::fs;
# let contents = fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = fastcdc::ronomon::FastCDC::new(&contents, 8192, 16384, 32768);
```

--------------------------------

### Basic Usage of v2020 FastCDC

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/index.html

Example demonstrating how to use the v2020 FastCDC implementation to read a file and print chunk offsets and sizes.

```rust
use std::fs;
use fastcdc::v2020;
let contents = fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = v2020::FastCDC::new(&contents, 4096, 16384, 65535);
for entry in chunker {
    println!("offset={} size={}", entry.offset, entry.length);
}

```

--------------------------------

### Basic FastCDC Chunking with v2020 Implementation

Source: https://docs.rs/fastcdc/3.2.1/fastcdc

A simple example demonstrating how to use the v2020 FastCDC implementation to chunk a file. It reads file contents and iterates over the generated chunk entries, printing their offset and size.

```rust
use std::fs;
use fastcdc::v2020;
let contents = fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = v2020::FastCDC::new(&contents, 4096, 16384, 65535);
for entry in chunker {
    println!("offset={} size={}", entry.offset, entry.length);
}
```

--------------------------------

### StreamCDC Usage Example

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.StreamCDC.html

Demonstrates how to construct and use StreamCDC to process a file and print chunk information. Ensure the 'test/fixtures/SekienAkashita.jpg' file exists or replace it with a valid path. The chunker iterates over detected chunks, yielding their offset and length.

```rust
use fastcdc::v2020::StreamCDC;
use std::fs::File;

let source = File::open("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = StreamCDC::new(source, 4096, 16384, 65535);
for result in chunker {
    let chunk = result.unwrap();
    println!("offset={} length={}", chunk.offset, chunk.length);
}
```

--------------------------------

### Any Trait Implementation

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Provides a way to get the TypeId of a value, useful for dynamic trait objects.

```rust
impl<T> Any for T
where T: 'static + ?Sized,
```

```rust
fn type_id(&self) -> TypeId
```

--------------------------------

### Chunk

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Represents a chunk returned by the FastCDC iterator. Each chunk contains its hash, starting offset, and length.

```APIDOC
## Chunk

### Description
Represents a chunk returned from the [`FastCDC`] iterator.

### Fields

- **hash** (`u64`): The gear hash value as of the end of the chunk.
- **offset** (`usize`): Starting byte position within the source.
- **length** (`usize`): Length of the chunk in bytes.
```

--------------------------------

### CloneToUninit Implementation (Nightly)

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

An experimental nightly-only API for copying data to uninitialized memory.

```rust
impl<T> CloneToUninit for T
where T: Clone,
```

```rust
unsafe fn clone_to_uninit(&self, dest: *mut u8)
```

--------------------------------

### From and Into Implementations

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Enables conversion between types where one type can be created from another.

```rust
impl<T> From<T> for T
```

```rust
fn from(t: T) -> T
```

```rust
impl<T, U> Into<U> for T
where U: From<T>,
```

```rust
fn into(self) -> U
```

--------------------------------

### Migrate FastCDC Usage (Pre-3.0 to Post-3.0)

Source: https://docs.rs/fastcdc/3.2.1/fastcdc

Illustrates the change in how to instantiate the FastCDC chunker after version 3.0. The 'ronomon' implementation remains consistent.

```rust
let chunker = fastcdc::FastCDC::new(&contents, 8192, 16384, 32768);
```

```rust
let chunker = fastcdc::ronomon::FastCDC::new(&contents, 8192, 16384, 32768);
```

--------------------------------

### Construct FastCDC with Custom Normalization Level and Seed

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Initializes a `FastCDC` instance with a specific normalization level and a seed value for XORing with gear table values.

```rust
pub fn with_level_and_seed( source: &'a [u8], min_size: u32, avg_size: u32, max_size: u32, level: Normalization, seed: u64, ) -> Self
```

--------------------------------

### Using v2020 FastCDC with Normalization Level 3

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/index.html

Demonstrates using the `with_level` method for normalized chunking with the v2020 FastCDC implementation, specifying Normalization::Level3.

```rust
use std::fs;
use fastcdc::v2020::{FastCDC, Normalization};
let contents = fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = FastCDC::with_level(&contents, 8192, 16384, 32768, Normalization::Level3);
for entry in chunker {
    println!("offset={} size={}", entry.offset, entry.length);
}

```

--------------------------------

### Migrate from pre-3.0 fastcdc

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/index.html

Adjust the instantiation of FastCDC from the root to the `ronomon` module for pre-3.0 compatibility.

```rust
let chunker = fastcdc::FastCDC::new(&contents, 8192, 16384, 32768);

```

```rust
let chunker = fastcdc::ronomon::FastCDC::new(&contents, 8192, 16384, 32768);

```

--------------------------------

### StreamCDC struct documentation

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Provides documentation for the `StreamCDC` struct, highlighting its use for streaming chunk data and its buffer allocation strategy.

```rust
/// The FastCDC chunker implementation from 2020 with streaming support.
///
/// Use `new` to construct an instance, and then iterate over the [`ChunkData`]s
/// via the [`Iterator`] trait.
///
/// Note that this struct allocates a [`Vec<u8>`] of `max_size` bytes to act as a
/// buffer when reading from the source and finding chunk boundaries.
///
/// ```no_run
/// # use std::fs::File;
/// # use fastcdc::v2020::StreamCDC;
///

```

--------------------------------

### TryFrom and TryInto Implementations

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Enables fallible conversions between types, returning a Result.

```rust
impl<T, U> TryFrom<U> for T
where U: Into<T>,
```

```rust
type Error = Infallible
```

```rust
fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
```

```rust
impl<T, U> TryInto<U> for T
where U: TryFrom<T>,
```

```rust
type Error = <U as TryFrom<T>>::Error
```

```rust
fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
```

--------------------------------

### FastCDC Clone Implementation

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Provides methods for cloning and copy-assigning FastCDC instances.

```APIDOC
## impl<'a> Clone for FastCDC<'a>

### fn clone(&self) -> FastCDC<'a>

Returns a duplicate of the value.

### fn clone_from(&mut self, source: &Self)

Performs copy-assignment from `source`.
```

--------------------------------

### FastCDC struct initialization

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Initializes a new FastCDC chunker instance. It requires a source byte slice and configuration parameters for minimum, average, and maximum chunk sizes, along with a seed for the gear hash function.

```rust
pub fn new<'a, T: AsRef<[u8]> + ?Sized>(
        source: &'a T,
        min_size: u32,
        avg_size: u32,
        max_size: u32,
        seed: u64,
    ) -> Self {
        let (bits, normalization) = get_cdc_bits(avg_size);
        let mask_s = MASKS[(bits + normalization) as usize];
        let mask_l = MASKS[(bits - normalization) as usize];
        let (gear, gear_ls) = get_gear_with_seed(seed);
        Self {
            source,
            processed: 0,
            remaining: source.len(),
            min_size: min_size as usize,
            avg_size: avg_size as usize,
            max_size: max_size as usize,
            mask_s,
            mask_l,
            mask_s_ls: mask_s << 1,
            mask_l_ls: mask_l << 1,
            gear,
            gear_ls,
        }
    }

```

--------------------------------

### StructuralPartialEq Implementation for Chunk

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Indicates that Chunk implements StructuralPartialEq.

```rust
impl StructuralPartialEq for Chunk
```

--------------------------------

### AsyncStreamCDC Constructor with Level and Seed

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.AsyncStreamCDC.html

Initializes AsyncStreamCDC with a specific normalization level and a seed value. The seed is XORed with values in the gear tables, affecting the chunking algorithm's behavior.

```rust
pub fn with_level_and_seed( source: R, min_size: u32, avg_size: u32, max_size: u32, level: Normalization, seed: u64, ) -> Self
```

--------------------------------

### FastCDC Sekien Akashita 16k Chunks Test

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Tests chunking on the SekienAkashita.jpg fixture with 16k chunk size parameters. It iterates through expected chunk hashes and lengths, verifying the `cut` method's output.

```rust
let read_result = fs::read("test/fixtures/SekienAkashita.jpg");
assert!(read_result.is_ok());
let contents = read_result.unwrap();
let chunker = FastCDC::new(&contents, 4096, 16384, 65535);
let mut cursor: usize = 0;
let mut remaining: usize = contents.len();
let expected: Vec<(u64, usize)> = vec![
    (17968276318003433923, 21325),
    (8197189939299398838, 17140),
    (13019990849178155730, 28084),
    (4509236223063678303, 18217),
    (2504464741100432583, 24700),
];
for (e_hash, e_length) in expected.iter() {
    let (hash, pos) = chunker.cut(cursor, remaining);
    assert_eq!(hash, *e_hash);
    assert_eq!(pos, cursor + e_length);
    cursor = pos;
    remaining -= e_length;
}
assert_eq!(remaining, 0);
```

--------------------------------

### Verify FastCDC Chunking with Expected Data

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Tests the FastCDC chunker by comparing generated chunk hashes, offsets, and lengths against a predefined set of expected values. It also verifies the MD5 digest of each chunk's content.

```Rust
let expected_chunks = vec![
            ExpectedChunk {
                hash: 17968276318003433923,
                offset: 0,
                length: 21325,
                digest: "2bb52734718194617c957f5e07ee6054".into(),
            },
            ExpectedChunk {
                hash: 8197189939299398838,
                offset: 21325,
                length: 17140,
                digest: "badfb0757fe081c20336902e7131f768".into(),
            },
            ExpectedChunk {
                hash: 13019990849178155730,
                offset: 38465,
                length: 28084,
                digest: "18412d7414de6eb42f638351711f729d".into(),
            },
            ExpectedChunk {
                hash: 4509236223063678303,
                offset: 66549,
                length: 18217,
                digest: "04fe1405fc5f960363bfcd834c056407".into(),
            },
            ExpectedChunk {
                hash: 2504464741100432583,
                offset: 84766,
                length: 24700,
                digest: "1aa7ad95f274d6ba34a983946ebc5af3".into(),
            },
        ];
        let chunker = FastCDC::new(&contents, 4096, 16384, 65535);
        let mut index = 0;
        for chunk in chunker {
            assert_eq!(chunk.hash, expected_chunks[index].hash);
            assert_eq!(chunk.offset, expected_chunks[index].offset as usize);
            assert_eq!(chunk.length, expected_chunks[index].length);
            let mut hasher = Md5::new();
            hasher.update(&contents[chunk.offset..chunk.offset + chunk.length]);
            let table = hasher.finalize();
            let digest = format!("{:x}", table);
            assert_eq!(digest, expected_chunks[index].digest);
            index += 1;
        }
        assert_eq!(index, 5);
```

--------------------------------

### map_windows

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.StreamCDC.html

Calls a function for each contiguous window of a specified size over the iterator. It returns an iterator over the results of the function calls. This is an experimental API.

```APIDOC
## fn map_windows<F, R, const N: usize>(self, f: F) -> MapWindows<Self, F, N>

### Description
Calls the given function `f` for each contiguous window of size `N` over `self` and returns an iterator over the outputs of `f`. Like `slice::windows()`, the windows during mapping overlap as well.

### Parameters
- `f`: A closure that takes a slice representing the window and returns a value `R`.

### Returns
A `MapWindows` iterator adapter.

### Note
This is a nightly-only experimental API.
```

--------------------------------

### Logarithm2 Test Cases

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Unit tests for the `logarithm2` function, verifying its output for various powers of 2 and values in between.

```Rust
#[test]
    fn test_logarithm2() {
        assert_eq!(logarithm2(0), 0);
        assert_eq!(logarithm2(1), 0);
        assert_eq!(logarithm2(2), 1);
        assert_eq!(logarithm2(3), 2);
        assert_eq!(logarithm2(5), 2);
        assert_eq!(logarithm2(6), 3);
        assert_eq!(logarithm2(11), 3);
        assert_eq!(logarithm2(12), 4);
        assert_eq!(logarithm2(19), 4);
        assert_eq!(logarithm2(64), 6);
        assert_eq!(logarithm2(128), 7);
        assert_eq!(logarithm2(256), 8);
        assert_eq!(logarithm2(512), 9);
        assert_eq!(logarithm2(1024), 10);
        assert_eq!(logarithm2(16383), 14);
        assert_eq!(logarithm2(16384), 14);
        assert_eq!(logarithm2(16385), 14);
        assert_eq!(logarithm2(32767), 15);
        assert_eq!(logarithm2(32768), 15);
        assert_eq!(logarithm2(32769), 15);
        assert_eq!(logarithm2(65535), 16);
        assert_eq!(logarithm2(65536), 16);
        assert_eq!(logarithm2(65537), 16);
        assert_eq!(logarithm2(1_048_575), 20);
        assert_eq!(logarithm2(1_048_576), 20);
        assert_eq!(logarithm2(1_048_577), 20);
        assert_eq!(logarithm2(4_194_303), 22);
        assert_eq!(logarithm2(4_194_304), 22);
        assert_eq!(logarithm2(4_194_305), 22);
        assert_eq!(logarithm2(16_777_215), 24);
        assert_eq!(logarithm2(16_777_216), 24);
        assert_eq!(logarithm2(16_777_217), 24);
    }
```

--------------------------------

### FastCDC Sekien Akashita 32k Chunks Test

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Tests chunking on the SekienAkashita.jpg fixture with 32k chunk size parameters. It validates the chunk hashes and positions against a predefined expected list.

```rust
let read_result = fs::read("test/fixtures/SekienAkashita.jpg");
assert!(read_result.is_ok());
let contents = read_result.unwrap();
let chunker = FastCDC::new(&contents, 8192, 32768, 131072);
let mut cursor: usize = 0;
let mut remaining: usize = contents.len();
let expected: Vec<(u64, usize)> =
    vec![(15733367461443853673, 66549), (6321136627705800457, 42917)];
for (e_hash, e_length) in expected.iter() {
    let (hash, pos) = chunker.cut(cursor, remaining);
    assert_eq!(hash, *e_hash);
    assert_eq!(pos, cursor + e_length);
    cursor = pos;
    remaining -= e_length;
}
assert_eq!(remaining, 0);
```

--------------------------------

### FastCDC Sekien Akashita 16k Chunks with Seed Test

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Tests chunking on the SekienAkashita.jpg fixture with 16k chunk size and a specific seed (666). It compares the generated chunk hashes and lengths against expected values.

```rust
let read_result = fs::read("test/fixtures/SekienAkashita.jpg");
assert!(read_result.is_ok());
let contents = read_result.unwrap();
let chunker =
    FastCDC::with_level_and_seed(&contents, 4096, 16384, 65535, Normalization::Level1, 666);
let mut cursor: usize = 0;
let mut remaining: usize = contents.len();
let expected: Vec<(u64, usize)> = vec![
    (9312357714466240148, 10605),
    (226910853333574584, 55745),
    (12271755243986371352, 11346),
    (14153975939352546047, 5883),
    (5890158701071314778, 11586),
    (8981594897574481255, 14301),
];
for (e_hash, e_length) in expected.iter() {
    let (hash, pos) = chunker.cut(cursor, remaining);
    assert_eq!(hash, *e_hash);
    assert_eq!(pos, cursor + e_length);
    cursor = pos;
    remaining -= e_length;
}
assert_eq!(remaining, 0);
```

--------------------------------

### FastCDC Sekien Akashita 64k Chunks Test

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Tests chunking on the SekienAkashita.jpg fixture with 64k chunk size parameters. It asserts that the single expected chunk hash and length match the output of the `cut` method.

```rust
let read_result = fs::read("test/fixtures/SekienAkashita.jpg");
assert!(read_result.is_ok());
let contents = read_result.unwrap();
let chunker = FastCDC::new(&contents, 16384, 65536, 262144);
let mut cursor: usize = 0;
let mut remaining: usize = contents.len();
let expected: Vec<(u64, usize)> = vec![(2504464741100432583, 109466)];
for (e_hash, e_length) in expected.iter() {
    let (hash, pos) = chunker.cut(cursor, remaining);
    assert_eq!(hash, *e_hash);
    assert_eq!(pos, cursor + e_length);
    cursor = pos;
    remaining -= e_length;
}
assert_eq!(remaining, 0);
```

--------------------------------

### AsyncStreamCDC Constructor

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.AsyncStreamCDC.html

Constructs a new AsyncStreamCDC instance. It takes the source, minimum chunk size, average chunk size, and maximum chunk size as parameters. It uses chunk size normalization level 1 by default.

```rust
pub fn new(source: R, min_size: u32, avg_size: u32, max_size: u32) -> Self
```

--------------------------------

### StreamCDC::with_level_and_seed Constructor

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Initializes StreamCDC with custom normalization level and a hash seed. The seed allows for reproducible chunking, useful for testing or specific data processing scenarios.

```rust
pub fn with_level_and_seed(
    source: R,
    min_size: u32,
    avg_size: u32,
    max_size: u32,
    level: Normalization,
    seed: u64,
) -> Self {
    assert!(min_size >= MINIMUM_MIN);
    assert!(min_size <= MINIMUM_MAX);
    assert!(avg_size >= AVERAGE_MIN);
    assert!(avg_size <= AVERAGE_MAX);
    assert!(max_size >= MAXIMUM_MIN);
    assert!(max_size <= MAXIMUM_MAX);
    let bits = logarithm2(avg_size);
    let normalization = level.bits();
    let mask_s = MASKS[(bits + normalization) as usize];
    let mask_l = MASKS[(bits - normalization) as usize];
    let (gear, gear_ls) = get_gear_with_seed(seed);
    Self {
        buffer: vec![0_u8; max_size as usize],
        capacity: max_size as usize,
        length: 0,
        source,
        eof: false,
        processed: 0,
        min_size: min_size as usize,
        avg_size: avg_size as usize,
        max_size: max_size as usize,
        mask_s,
        mask_l,
        mask_s_ls: mask_s << 1,
        mask_l_ls: mask_l << 1,
        gear,
        gear_ls,
    }
}
```

--------------------------------

### AsyncStreamCDC Constructors

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.AsyncStreamCDC.html

Provides methods to construct an AsyncStreamCDC instance with different configuration options for chunk size normalization and seeding.

```APIDOC
## AsyncStreamCDC

An async-streamable version of the FastCDC chunker implementation from 2020 with streaming support.
Use `new` to construct an instance, and then `as_stream` to produce an async Stream of the chunks.
Both `futures` and `tokio`-based AsyncRead inputs are supported via feature flags. But, if necessary you can also use the `async_compat` crate to adapt your inputs as circumstances may require.
Note that this struct allocates a `Vec<u8>` of `max_size` bytes to act as a buffer when reading from the source and finding chunk boundaries.

### `new(source: R, min_size: u32, avg_size: u32, max_size: u32) -> Self`

Construct a `AsyncStreamCDC` that will process bytes from the given source.
Uses chunk size normalization level 1 by default.

### `with_level(source: R, min_size: u32, avg_size: u32, max_size: u32, level: Normalization) -> Self`

Create a new `AsyncStreamCDC` with the given normalization level.

### `with_level_and_seed(source: R, min_size: u32, avg_size: u32, max_size: u32, level: Normalization, seed: u64) -> Self`

Create a new `AsyncStreamCDC` with the given normalization level and seed to be XOR’d with the values in the gear tables.
```

--------------------------------

### FastCDC Iterator Implementation

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Details the iterator methods available for FastCDC, including basic iteration, chunking, and various adapter methods.

```APIDOC
## impl Iterator for FastCDC<'_>

### type Item = Chunk

The type of the elements being iterated over.

### fn next(&mut self) -> Option<Chunk>

Advances the iterator and returns the next value.

### fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the iterator.

### fn next_chunk<const N: usize>( &mut self, ) -> Result<[Self::Item; N], IntoIter<Self::Item, N>>

Advances the iterator and returns an array containing the next `N` values. (Nightly-only experimental API)

### fn count(self) -> usize

Consumes the iterator, counting the number of iterations and returning it.

### fn last(self) -> Option<Self::Item>

Consumes the iterator, returning the last element.

### fn advance_by(&mut self, n: usize) -> Result<(), NonZero<usize>>

Advances the iterator by `n` elements. (Nightly-only experimental API)

### fn nth(&mut self, n: usize) -> Option<Self::Item>

Returns the `n`th element of the iterator.

### fn step_by(self, step: usize) -> StepBy<Self>

Creates an iterator starting at the same point, but stepping by the given amount at each iteration.

### fn chain<U>(self, other: U) -> Chain<Self, <U as IntoIterator>::IntoIter>

Takes two iterators and creates a new iterator over both in sequence.

### fn zip<U>(self, other: U) -> Zip<Self, <U as IntoIterator>::IntoIter>

‘Zips up’ two iterators into a single iterator of pairs.

### fn intersperse(self, separator: Self::Item) -> Intersperse<Self>

Creates a new iterator which places a copy of `separator` between adjacent items of the original iterator. (Nightly-only experimental API)

### fn intersperse_with<G>(self, separator: G) -> IntersperseWith<Self, G>

Creates a new iterator which places an item generated by `separator` between adjacent items of the original iterator. (Nightly-only experimental API)

### fn map<B, F>(self, f: F) -> Map<Self, F>

Takes a closure and creates an iterator which calls that closure on each element.

### fn for_each<F>(self, f: F)

Calls a closure on each element of an iterator.

### fn filter<P>(self, predicate: P) -> Filter<Self, P>

Creates an iterator which uses a closure to determine if an element should be yielded.

### fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F>

Creates an iterator that both filters and maps.

### fn enumerate(self) -> Enumerate<Self>

Creates an iterator which gives the current iteration count as well as the next value.

### fn peekable(self) -> Peekable<Self>

Creates an iterator which can use the `peek` and `peek_mut` methods to look at the next element of the iterator without consuming it.

### fn skip_while<P>(self, predicate: P) -> SkipWhile<Self, P>

Creates an iterator that `skip`s elements based on a predicate.

### fn take_while<P>(self, predicate: P) -> TakeWhile<Self, P>

Creates an iterator that yields elements based on a predicate.

### fn map_while<B, P>(self, predicate: P) -> MapWhile<Self, P>

Creates an iterator that both yields elements based on a predicate and maps.

### fn skip(self, n: usize) -> Skip<Self>

Creates an iterator that skips the first `n` elements.
```

--------------------------------

### product

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Calculates the product of all elements in an iterator.

```APIDOC
## fn product<P>(self) -> P
where Self: Sized,
    P: Product<Self::Item>,

### Description
Iterates over the entire iterator, multiplying all the elements.

### Returns
The product of the elements in the iterator.
```

--------------------------------

### scan

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.StreamCDC.html

Creates an iterator adapter that holds internal state, similar to `fold`, but produces a new iterator. It applies a function to each element, allowing for stateful transformations.

```APIDOC
## fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F>

### Description
An iterator adapter which, like `fold`, holds internal state, but unlike `fold`, produces a new iterator.

### Parameters
- `initial_state`: The initial state for the scan operation.
- `f`: A closure that takes a mutable reference to the state and the current item, returning an `Option<B>`.

### Returns
A `Scan` iterator adapter.
```

--------------------------------

### Stream FastCDC Chunks from File

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Tests the streaming functionality of FastCDC by processing chunks from a file. The expected chunk results are asserted to match those of the non-streaming version.

```Rust
let file_result = File::open("test/fixtures/SekienAkashita.jpg");
        assert!(file_result.is_ok());
        let file = file_result.unwrap();
        // The set of expected results should match the non-streaming version.
        let expected_chunks = vec![
            ExpectedChunk {
                hash: 17968276318003433923,
                offset: 0,
                length: 21325,
                digest: "2bb52734718194617c957f5e07ee6054".into(),
            },
            ExpectedChunk {
                hash: 8197189939299398838,
                offset: 21325,
                length: 17140,
                digest: "badfb0757fe081c20336902e7131f768".into(),
            },
            ExpectedChunk {
                hash: 13019990849178155730,
                offset: 38465,
                length: 28084,
                digest: "18412d7414de6eb42f638351711f729d".into(),
            },
            ExpectedChunk {
                hash: 4509236223063678303,
                offset: 66549,
                length: 18217,
                digest: "04fe1405fc5f960363bfcd834c056407".into(),
            },
            ExpectedChunk {
                hash: 2504464741100432583,
                offset: 84766,
                length: 24700,
                digest: "1aa7ad95f274d6ba34a983946ebc5af3".into(),
            },
        ];
```

--------------------------------

### AsyncStreamCDC Stream Generation

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.AsyncStreamCDC.html

Demonstrates how to obtain an asynchronous stream of ChunkData from an AsyncStreamCDC instance.

```APIDOC
### `as_stream(&mut self) -> impl Stream<Item = Result<ChunkData, Error>> + '_`

Returns an asynchronous stream that yields `ChunkData` or `Error`.
```

--------------------------------

### get_gear_with_seed

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/fn.get_gear_with_seed.html

Generates the GEAR table and its left-shifted version, applying a bitwise XOR with a given seed.

```APIDOC
## Function get_gear_with_seed

### Description
Produces the GEAR table, and the left-shifted version, as heap-owned values. This will copy the original GEAR table, and its left-shifted twin, and perform a bitwise exclusive OR on the values using the given seed. If the seed is zero, no computation is performed.

### Signature
```rust
pub fn get_gear_with_seed(seed: u64) -> (Box<[u64; 256]>, Box<[u64; 256]>)
```

### Parameters
#### Path Parameters
- **seed** (u64) - Description: The seed value used for the bitwise XOR operation. If zero, no computation is performed.
```

--------------------------------

### Instantiate and Iterate FastCDC

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Reads a file into memory and splits it into chunks with specified minimum, average, and maximum sizes. The iterator yields chunk information including offset and length.

```rust
use std::fs;
use fastcdc::v2020;
let contents = fs::read("test/fixtures/SekienAkashita.jpg").unwrap();
let chunker = v2020::FastCDC::new(&contents, 8192, 16384, 65535);
for entry in chunker {
    println!("offset={} size={}", entry.offset, entry.length);
}
```

--------------------------------

### get_gear_with_seed Function Signature

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/fn.get_gear_with_seed.html

This is the function signature for `get_gear_with_seed`. It takes a `u64` seed and returns two `Box<[u64; 256]>` arrays representing the GEAR table and its shifted version.

```rust
pub fn get_gear_with_seed(seed: u64) -> (Box<[u64; 256]>, Box<[u64; 256]>)
```

--------------------------------

### StreamCDC::with_level Constructor

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Creates a StreamCDC instance with a specified normalization level. Choose the level based on desired chunk size distribution and deduplication efficiency.

```rust
pub fn with_level(
    source: R,
    min_size: u32,
    avg_size: u32,
    max_size: u32,
    level: Normalization,
) -> Self {
    StreamCDC::with_level_and_seed(source, min_size, avg_size, max_size, level, 0)
}
```

--------------------------------

### Read Next Chunk

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/async_stream_cdc.rs.html

Fills the buffer and then identifies the next data chunk using the configured gear and mask parameters. Returns the chunk data or an error if the source is empty or an issue occurs during processing.

```Rust
async fn read_chunk(&mut self) -> Result<ChunkData, Error> {
        self.fill_buffer().await?;
        if self.length == 0 {
            Err(Error::Empty)
        } else {
            let (hash, count) = cut_gear(
                &self.buffer[..self.length],
                self.min_size,
                self.avg_size,
                self.max_size,
                self.mask_s,
                self.mask_l,
                self.mask_s_ls,
                self.mask_l_ls,
                &self.gear,
                &self.gear_ls,
            );
            if count == 0 {
                Err(Error::Empty)
            } else {
                let offset = self.processed;
                self.processed += count as u64;
                let data = self.drain_bytes(count)?;
                Ok(ChunkData {
                    hash,
                    offset,
                    length: count,
                    data,
                })
            }
        }
    }
```

--------------------------------

### Test AsyncStreamCDC Initialization with Panicking Input

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/async_stream_cdc.rs.html

Tests that AsyncStreamCDC panics when the maximum buffer size is set too high, indicating an invalid configuration.

```rust
#[test]
    #[should_panic]
    fn test_maximum_too_high() {
        let array = [0u8; 1024];
        AsyncStreamCDC::new(array.as_slice(), 64, 256, 1_073_741_825);
    }
```

--------------------------------

### Blanket Implementations for StreamCDC

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.StreamCDC.html

This section details the blanket implementations available for types that implement StreamCDC, showcasing common Rust traits that can be utilized.

```APIDOC
## Blanket Implementations

### impl<T> Any for T
where T: 'static + ?Sized,
#### fn type_id(&self) -> TypeId
Gets the `TypeId` of `self`. Read more

### impl<T> Borrow<T> for T
where T: ?Sized,
#### fn borrow(&self) -> &T
Immutably borrows from an owned value. Read more

### impl<T> BorrowMut<T> for T
where T: ?Sized,
#### fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more

### impl<T> From<T> for T
#### fn from(t: T) -> T
Returns the argument unchanged.

### impl<T, U> Into<U> for T
where U: From<T>,
#### fn into(self) -> U
Calls `U::from(self)`.
That is, this conversion is whatever the implementation of `From<T> for U` chooses to do.

### impl<I> IntoIterator for I
where I: Iterator,
#### type Item = <I as Iterator>::Item
The type of the elements being iterated over.
#### type IntoIter = I
Which kind of iterator are we turning this into?
#### fn into_iter(self) -> I
Creates an iterator from a value. Read more

### impl<T, U> TryFrom<U> for T
where U: Into<T>,
#### type Error = Infallible
The type returned in the event of a conversion error.
#### fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
Performs the conversion.

### impl<T, U> TryInto<U> for T
where U: TryFrom<T>,
#### type Error = <U as TryFrom<T>>::Error
The type returned in the event of a conversion error.
#### fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
Performs the conversion.
```

--------------------------------

### FastCDC Debug Implementation

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Enables debugging capabilities for FastCDC instances.

```APIDOC
## impl<'a> Debug for FastCDC<'a>

### fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.
```

--------------------------------

### Construct FastCDC with Default Normalization

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.FastCDC.html

Constructs a `FastCDC` instance to process a given slice of bytes using default normalization level 1.

```rust
pub fn new( source: &'a [u8], min_size: u32, avg_size: u32, max_size: u32, ) -> Self
```

--------------------------------

### Test AsyncStreamCDC Streaming Chunk Iteration

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/async_stream_cdc.rs.html

Tests the asynchronous streaming of chunks from AsyncStreamCDC using a sample image file. It verifies chunk hashes, offsets, lengths, and MD5 digests against expected values.

```rust
use md5::{Digest, Md5};

    #[cfg(all(feature = "futures", not(feature = "tokio")))]
    use futures::stream::StreamExt;
    #[cfg(all(feature = "tokio", not(feature = "futures")))]
    use tokio_stream::StreamExt;

    #[cfg_attr(all(feature = "tokio", not(feature = "futures")), tokio::test)]
    #[cfg_attr(all(feature = "futures", not(feature = "tokio")), futures_test::test)]
    async fn test_iter_sekien_16k_chunks() {
        let read_result = std::fs::read("test/fixtures/SekienAkashita.jpg");
        assert!(read_result.is_ok());
        let contents = read_result.unwrap();
        // The digest values are not needed here, but they serve to validate
        // that the streaming version tested below is returning the correct
        // chunk data on each iteration.
        let expected_chunks = vec![
            ExpectedChunk {
                hash: 17968276318003433923,
                offset: 0,
                length: 21325,
                digest: "2bb52734718194617c957f5e07ee6054".into(),
            },
            ExpectedChunk {
                hash: 8197189939299398838,
                offset: 21325,
                length: 17140,
                digest: "badfb0757fe081c20336902e7131f768".into(),
            },
            ExpectedChunk {
                hash: 13019990849178155730,
                offset: 38465,
                length: 28084,
                digest: "18412d7414de6eb42f638351711f729d".into(),
            },
            ExpectedChunk {
                hash: 4509236223063678303,
                offset: 66549,
                length: 18217,
                digest: "04fe1405fc5f960363bfcd834c056407".into(),
            },
            ExpectedChunk {
                hash: 2504464741100432583,
                offset: 84766,
                length: 24700,
                digest: "1aa7ad95f274d6ba34a983946ebc5af3".into(),
            },
        ];
        let mut chunker = AsyncStreamCDC::new(contents.as_ref(), 4096, 16384, 65535);
        let stream = chunker.as_stream();

        let chunks = stream.collect::<Vec<_>>().await;

        let mut index = 0;

        for chunk in chunks {
            let chunk = chunk.unwrap();
            assert_eq!(chunk.hash, expected_chunks[index].hash);
            assert_eq!(chunk.offset, expected_chunks[index].offset);
            assert_eq!(chunk.length, expected_chunks[index].length);
            let mut hasher = Md5::new();
            hasher
                .update(&contents[(chunk.offset as usize)..(chunk.offset as usize) + chunk.length]);
            let table = hasher.finalize();
            let digest = format!("{:x}", table);
            assert_eq!(digest, expected_chunks[index].digest);
            index += 1;
        }
        assert_eq!(index, 5);
    }
```

--------------------------------

### Clone Implementation for Chunk

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Provides methods to duplicate and copy-assign Chunk values.

```rust
fn clone(&self) -> Chunk
```

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### PartialEq Implementation for Chunk

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Enables equality comparison between Chunk values.

```rust
fn eq(&self, other: &Chunk) -> bool
```

```rust
fn ne(&self, other: &Rhs) -> bool
```

--------------------------------

### Test AsyncStreamCDC Mask Calculations

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/async_stream_cdc.rs.html

Verifies the correct calculation of `mask_l` and `mask_s` for different buffer sizes in AsyncStreamCDC.

```rust
let source = [0u8; 1024];
        let chunker = AsyncStreamCDC::new(source.as_slice(), 64, 256, 1024);
        assert_eq!(chunker.mask_l, MASKS[7]);
        assert_eq!(chunker.mask_s, MASKS[9]);
        let chunker = AsyncStreamCDC::new(source.as_slice(), 8192, 16384, 32768);
        assert_eq!(chunker.mask_l, MASKS[13]);
        assert_eq!(chunker.mask_s, MASKS[15]);
        let chunker = AsyncStreamCDC::new(source.as_slice(), 1_048_576, 4_194_304, 16_777_216);
        assert_eq!(chunker.mask_l, MASKS[21]);
        assert_eq!(chunker.mask_s, MASKS[23]);
```

--------------------------------

### AsyncStreamCDC Constructor with Normalization Level

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.AsyncStreamCDC.html

Creates a new AsyncStreamCDC with a specified normalization level. This allows for finer control over chunk size distribution.

```rust
pub fn with_level( source: R, min_size: u32, avg_size: u32, max_size: u32, level: Normalization, ) -> Self
```

--------------------------------

### Hash Implementation for Chunk

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.Chunk.html

Provides methods for hashing Chunk values and slices of Chunks.

```rust
fn hash<__H: Hasher>(&self, state: &mut __H) -> ()
```

```rust
fn hash_slice<H>(data: &[Self], state: &mut H)
where H: Hasher,
      Self: Sized,
```

--------------------------------

### Async Stream Conversion (Tokio)

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/async_stream_cdc.rs.html

Converts the AsyncStreamCDC into a Tokio stream. Each item yielded by the stream is a Result containing either ChunkData or an Error. The stream terminates when `read_chunk` returns `Error::Empty`.

```Rust
pub fn as_stream(&mut self) -> impl Stream<Item = Result<ChunkData, Error>> + '_ {
        try_stream! {
            loop {
                match self.read_chunk().await {
                    Ok(chunk) => yield chunk,
                    Err(Error::Empty) => {
                        break;
                    }
                    error @ Err(_) => {
                        error?;
                    }
                }
            }
        }
    }
```

--------------------------------

### FastCDC cut method

Source: https://docs.rs/fastcdc/3.2.1/src/fastcdc/v2020/mod.rs.html

Finds the next cut point in the data. It returns a tuple containing the 64-bit hash and the byte offset of the end of the chunk. Handles the special case where remaining bytes are less than the minimum chunk size.

```rust
pub fn cut(&self, start: usize, remaining: usize) -> (u64, usize) {
        let end = start + remaining;
        let (hash, count) = cut_gear(
            &self.source[start..end],
            self.min_size,
            self.avg_size,
            self.max_size,
            self.mask_s,
            self.mask_l,
            self.mask_s_ls,
            self.mask_l_ls,
            &self.gear,
            &self.gear_ls,
        );
        (hash, start + count)
    }

```

--------------------------------

### StreamCDC::with_level_and_seed

Source: https://docs.rs/fastcdc/3.2.1/fastcdc/v2020/struct.StreamCDC.html

Creates a new StreamCDC instance with a specified normalization level and hash seed.

```APIDOC
## StreamCDC::with_level_and_seed

### Description
Create a new `StreamCDC` with the given normalization level and hash seed.

### Method
`with_level_and_seed`

### Parameters
- **source** (R: Read) - The byte source to read from.
- **min_size** (u32) - The minimum chunk size.
- **avg_size** (u32) - The average chunk size.
- **max_size** (u32) - The maximum chunk size.
- **level** (Normalization) - The normalization level to use.
- **seed** (u64) - The hash seed to use.

### Returns
- Self (StreamCDC<R>) - A new instance of StreamCDC.
```
Tokens:
Raw