### Install all dependencies with uv

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Install all project dependencies using uv sync.

```bash
uv sync
```

--------------------------------

### Install optional dependencies with uv

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Install specific optional dependencies for testing, Cassandra, or Redis support using uv sync with extra flags. Also shows how to install all extras.

```bash
# For testing
uv sync --extra test

# For Cassandra support
uv sync --extra cassandra

# For Redis support
uv sync --extra redis

# For all extras
uv sync --all-extras
```

--------------------------------

### Install datasketch with Cassandra dependency

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Install datasketch along with Cassandra support.

```bash
pip install datasketch[cassandra]
```

--------------------------------

### Install Optional Dependencies with uv

Source: https://github.com/ekzhu/datasketch/blob/master/README.rst

Installs specific optional dependencies for testing or database support.

```bash
uv sync --extra test
```

```bash
uv sync --extra cassandra
```

```bash
uv sync --extra redis
```

```bash
uv sync --all-extras
```

--------------------------------

### Asynchronous MinHash LSH Insertion Example

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates how to use AsyncMinHashLSH with an asynchronous insertion session. Ensure aiomongo is installed and a MongoDB instance is running.

```python
import asyncio
from datasketch.aio import AsyncMinHashLSH
from datasketch import MinHash

async def main():
    storage_config = {"type": "aiomongo", "mongo": {"host": "localhost", "port": 27017}}
    async with AsyncMinHashLSH(
        storage_config=storage_config, threshold=0.5, num_perm=16, prepickle=True
    ) as lsh:
        async with lsh.insertion_session(batch_size=1000) as session:
            m = MinHash(num_perm=16)
            m.update(b"data")
            await session.insert("key", m)

asyncio.run(main())
```

--------------------------------

### Async MinHash LSH Storage Configuration Example

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Example of a supported storage configuration for AsyncMinHashLSH using aiomongo.

```python
MONGO = {"type": "aiomongo", "basename": "base_name_1", "mongo": {"host": "localhost", "port": 27017}}
```

--------------------------------

### Install datasketch with Redis dependency

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Install datasketch along with Redis support.

```bash
pip install datasketch[redis]
```

--------------------------------

### Install datasketch with pip

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Install the base datasketch package. NumPy is installed as a dependency.

```bash
pip install datasketch
```

--------------------------------

### LSHBloom Example: Inserting and Querying Sets

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lshbloom.md

Demonstrates how to create an LSHBloom index, insert MinHash objects representing sets, and query for potential duplicates based on Jaccard similarity.

```python
from datasketch import MinHash, MinHashLSHBloom

set1 = set(['minhash', 'is', 'a', 'probabilistic', 'data', 'structure', 'for',
            'estimating', 'the', 'similarity', 'between', 'datasets'])
set2 = set(['minhash', 'is', 'a', 'probability', 'data', 'structure', 'for',
            'estimating', 'the', 'similarity', 'between', 'documents'])
set3 = set(['minhash', 'is', 'probability', 'data', 'structure', 'for',
            'estimating', 'the', 'similarity', 'between', 'documents'])

m1 = MinHash(num_perm=128)
m2 = MinHash(num_perm=128)
m3 = MinHash(num_perm=128)
for d in set1:
    m1.update(d.encode('utf8'))
for d in set2:
    m2.update(d.encode('utf8'))
for d in set3:
    m3.update(d.encode('utf8'))

# Create LSHBloom index
lsh = MinHashLSHBloom(threshold=0.5, num_perm=128, n=100, fp=0.001)
lsh.insert("m2", m2)
lsh.insert("m3", m3)
is_duplicate = lsh.query(m1)
print("Is Duplicate: ", is_duplicate)
```

--------------------------------

### Install datasketch with Bloom filter support

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lshbloom.md

Install the datasketch library with the necessary dependency for Bloom filter functionality using pip.

```bash
pip install datasketch[bloom]
```

--------------------------------

### Async MongoDB Storage Configuration with Additional Client Args

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Example of passing additional arguments to the PyMongo MongoClient constructor for advanced configurations like X509 authentication.

```python
_storage = {
    'type': 'aiomongo',
    'mongo':
        {
            ...
            'args': {
                'ssl': True,
                'ssl_ca_certs': 'root-ca.pem',
                'ssl_pem_passphrase': 'password',
                'ssl_certfile': 'certfile.pem',
                'authMechanism': "MONGODB-X509",
                'username': "username"
            }
        }
}
```

--------------------------------

### Asynchronous MinHash LSH Deletion Example

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Shows how to use AsyncMinHashLSH with an asynchronous deletion session. This example first inserts data and then removes it. Ensure aiomongo is installed and a MongoDB instance is running.

```python
import asyncio
from datasketch.aio import AsyncMinHashLSH
from datasketch import MinHash

async def main():
    storage_config = {"type": "aiomongo", "mongo": {"host": "localhost", "port": 27017}}
    async with AsyncMinHashLSH(
        storage_config=storage_config, threshold=0.5, num_perm=16, prepickle=True
    ) as lsh:
        # Insert some data first
        m = MinHash(num_perm=16)
        m.update(b"data")
        await lsh.insert("key1", m)

        # Delete using session
        async with lsh.delete_session(batch_size=100) as session:
            await session.remove("key1")

asyncio.run(main())
```

--------------------------------

### Async MongoDB Storage Configuration for Replica Set

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Configuration example for connecting AsyncMinHashLSH to a MongoDB replica set.

```python
_storage = {'type': 'aiomongo', 'mongo': {'replica_set': 'rs0', 'replica_set_nodes': 'node1:port1,node2:port2,node3:port3'}}
```

--------------------------------

### Create HNSW Index with Dictionary of Points

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates creating an HNSW index and batch inserting points from a dictionary. Ensure numpy is installed for random data generation.

```python
from datasketch.hnsw import HNSW
import numpy as np

data = np.random.random_sample((1000, 10))
index = HNSW(distance_func=lambda x, y: np.linalg.norm(x - y))

# Batch insert 1000 points.
index.update({i: d for i, d in enumerate(data)})
```

--------------------------------

### Run tests with uv

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Execute the project's tests using uv run and pytest to verify the installation.

```bash
uv run pytest
```

--------------------------------

### Create and Query LSHBloom Index

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates how to create a MinHashLSHBloom index, insert MinHash objects, and query for duplicates based on a Jaccard similarity threshold. Ensure the 'datasketch' library is installed and necessary MinHash objects are prepared.

```python
from datasketch import MinHash, MinHashLSH

set1 = set(
    [
        "minhash",
        "is",
        "a",
        "probabilistic",
        "data",
        "structure",
        "for",
        "estimating",
        "the",
        "similarity",
        "between",
        "datasets",
    ]
)
set2 = set(
    [
        "minhash",
        "is",
        "a",
        "probability",
        "data",
        "structure",
        "for",
        "estimating",
        "the",
        "similarity",
        "between",
        "documents",
    ]
)
set3 = set(
    [
        "minhash",
        "is",
        "probability",
        "data",
        "structure",
        "for",
        "estimating",
        "the",
        "similarity",
        "between",
        "documents",
    ]
)

m1 = MinHash(num_perm=128)
m2 = MinHash(num_perm=128)
m3 = MinHash(num_perm=128)
for d in set1:
    m1.update(d.encode("utf8"))
for d in set2:
    m2.update(d.encode("utf8"))
for d in set3:
    m3.update(d.encode("utf8"))

# Create LSHBloom index
lsh = MinHashLSHBloom(threshold=0.5, num_perm=128, n=100, fp=0.0001, save_dir="./index/")
lsh.insert(m2)
lsh.insert(m3)

# Query whether m1 is a duplicate according to the given threshold
is_duplicate = lsh.query(m1)
```

--------------------------------

### Create MinHashLSH Index with Redis Storage

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize a MinHashLSH index that uses Redis for storage. This example shows how to configure the Redis connection details and set a basename for the keys.

```python
lsh = MinHashLSH(
    threshold=0.9,
    num_perm=128,
    storage_config={
        "type": "redis",
        "basename": b"mylsh",  # optional, defaults to a random string.
        "redis": {"host": "localhost", "port": 6379},
    },
)
```

--------------------------------

### Async MongoDB Storage Configuration for Atlas Cluster

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Configuration example for connecting AsyncMinHashLSH to a MongoDB Atlas cluster using a connection URI.

```python
_storage = {'type': 'aiomongo', 'mongo': {'url': 'mongodb+srv://<username>:<password>@<cluster>.example.com/<db>'}}
```

--------------------------------

### Convert LeanMinHash to MinHash

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Create a MinHash object from a LeanMinHash. This allows for further updates to the MinHash if needed. The second example shows how to create a MinHash whose state is independent of the original LeanMinHash.

```python
minhash = MinHash(seed=lean_minhash.seed, hashvalues=lean_minhash.hashvalues)
```

```python
minhash = MinHash(seed=lean_minhash.seed, hashvalues=lean_minhash.digest())
```

--------------------------------

### Basic MinHash Initialization and Update

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates basic MinHash initialization with default settings and updating with string values that are encoded to bytes.

```python
from datasketch import MinHash

m = MinHash()
m.update_batch([s.encode("utf-8") for s in ["token1", "token2"]])
```

--------------------------------

### Create HNSW Index with Jaccard Distance

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

This example shows how to create an HNSW index for sets using Jaccard distance. The distance function calculates the Jaccard distance between two sets represented as numpy arrays. Ensure numpy is installed.

```python
from datasketch.hnsw import HNSW
import numpy as np

# Each set is represented as a 10-element vector of random integers
# between 0 and 100.
# Deduplication is handled by the distance function.
data = np.random.randint(0, 100, size=(1000, 10))
jaccard_distance = lambda x, y: (
    1.0 - float(len(np.intersect1d(x, y, assume_unique=False))) / float(len(np.union1d(x, y)))
)
index = HNSW(distance_func=jaccard_distance)
for i, d in enumerate(data):
    index[i] = d

# Query the index for the 10 nearest neighbors of the first set.
index.query(data[0], k=10)
```

--------------------------------

### Create and activate virtual environment with uv

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Create a Python virtual environment using uv and activate it. You can optionally specify a Python version.

```bash
# Create a virtual environment
# (Optional: specify Python version with --python 3.x)
uv venv
# Activate the virtual environment (optional, uv run commands work without it)
source .venv/bin/activate
```

--------------------------------

### Initialize HyperLogLog++

Source: https://github.com/ekzhu/datasketch/blob/master/docs/hyperloglog.md

Initialize a HyperLogLog++ object. HyperLogLog++ offers enhanced features over the standard HyperLogLog.

```python
from datasketch import HyperLogLogPlusPlus

# Initialize an HyperLogLog++ object.
hpp = HyperLogLogPlusPlus()
# Everything else is the same as HyperLogLog
```

--------------------------------

### MinHash Initialization with GPU Mode

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Shows how to initialize MinHash in GPU mode (if available) and update with byte literals.

```python
from datasketch import MinHash

m = MinHash(num_perm=256, gpu_mode="detect")
m.update_batch([b"token1", b"token2"])
```

--------------------------------

### bytesize

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Gets the size of the HyperLogLog sketch in bytes.

```APIDOC
### bytesize() → int

#### Description
Get the size of the HyperLogLog in bytes.

```

--------------------------------

### __len__

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Gets the size of the HyperLogLog sketch, which corresponds to the size of its internal state array.

```APIDOC
### __len__() → int

#### Returns
int: Get the size of the HyperLogLog as the size of

> reg.

```

--------------------------------

### Create and Query MinHash LSH Forest

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lshforest.md

Demonstrates creating a MinHash LSH Forest, adding MinHash objects with keys, indexing the forest, checking for membership, and performing a top-k query.

```python
from datasketch import MinHashLSHForest, MinHash

data1 = ['minhash', 'is', 'a', 'probabilistic', 'data', 'structure', 'for',
        'estimating', 'the', 'similarity', 'between', 'datasets']
data2 = ['minhash', 'is', 'a', 'probability', 'data', 'structure', 'for',
        'estimating', 'the', 'similarity', 'between', 'documents']
data3 = ['minhash', 'is', 'probability', 'data', 'structure', 'for',
        'estimating', 'the', 'similarity', 'between', 'documents']

# Create MinHash objects
m1 = MinHash(num_perm=128)
m2 = MinHash(num_perm=128)
m3 = MinHash(num_perm=128)
for d in data1:
    m1.update(d.encode('utf8'))
for d in data2:
    m2.update(d.encode('utf8'))
for d in data3:
    m3.update(d.encode('utf8'))

# Create a MinHash LSH Forest with the same num_perm parameter
forest = MinHashLSHForest(num_perm=128)

# Add m2 and m3 into the index
forest.add("m2", m2)
forest.add("m3", m3)

# IMPORTANT: must call index() otherwise the keys won't be searchable
forest.index()

# Check for membership using the key
print("m2" in forest)
print("m3" in forest)

# Using m1 as the query, retrieve top 2 keys that have the higest Jaccard
result = forest.query(m1, 2)
print("Top 2 candidates", result)
```

--------------------------------

### Indexing and Querying with MinHashLSHEnsemble

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lshensemble.md

Demonstrates how to create MinHash objects, build a MinHashLSHEnsemble index, and query for sets with a specified containment threshold. Ensure MinHash objects are created with the correct number of permutations.

```python
from datasketch import MinHashLSHEnsemble, MinHash

set1 = set(["cat", "dog", "fish", "cow"])
set2 = set(["cat", "dog", "fish", "cow", "pig", "elephant", "lion", "tiger",
             "wolf", "bird", "human"])
set3 = set(["cat", "dog", "car", "van", "train", "plane", "ship", "submarine",
             "rocket", "bike", "scooter", "motorcyle", "SUV", "jet", "horse"])

# Create MinHash objects
m1 = MinHash(num_perm=128)
m2 = MinHash(num_perm=128)
m3 = MinHash(num_perm=128)
for d in set1:
    m1.update(d.encode('utf8'))
for d in set2:
    m2.update(d.encode('utf8'))
for d in set3:
    m3.update(d.encode('utf8'))

# Create an LSH Ensemble index with threshold and number of partition
# settings.
lshensemble = MinHashLSHEnsemble(threshold=0.8, num_perm=128,
    num_part=32)

# Index takes an iterable of (key, minhash, size)
lshensemble.index([("m2", m2, len(set2)), ("m3", m3, len(set3))])

# Check for membership using the key
print("m2" in lshensemble)
print("m3" in lshensemble)

# Using m1 as the query, get an result iterator
print("Sets with containment > 0.8:")
for key in lshensemble.query(m1, len(set1)):
    print(key)
```

--------------------------------

### Run Top-K Benchmark for Flickr Dataset

Source: https://github.com/ekzhu/datasketch/blob/master/benchmark/indexes/jaccard/README.md

Execute the top-k benchmark for the Flickr dataset. This command generates a SQLite database containing the benchmark results.

```bash
python topk_benchmark.py --index-set-file FLICKR-london2y_dup_dr.inp --query-set-file FLICKR-london2y_dup_dr.inp --query-sample-ratio 0.01 --output flickr.sqlite
```

--------------------------------

### Create HNSW Index with Euclidean Distance

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Use this snippet to create an HNSW index for numerical vectors with Euclidean distance. Ensure numpy is installed for vector operations.

```python
from datasketch.hnsw import HNSW
import numpy as np

data = np.random.random_sample((1000, 10))
index = HNSW(distance_func=lambda x, y: np.linalg.norm(x - y))
for i, d in enumerate(data):
    index.insert(i, d)

# Query the index for the 10 nearest neighbors of the first vector.
index.query(data[0], k=10)
```

--------------------------------

### Initialize HyperLogLogPlusPlus

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize a HyperLogLogPlusPlus object with a specified precision parameter.

```python
from datasketch.hyperloglog import HyperLogLogPlusPlus

hll = HyperLogLogPlusPlus(p=14)
```

--------------------------------

### Initialize WeightedMinHashGenerator with default parameters

Source: https://github.com/ekzhu/datasketch/blob/master/docs/weightedminhash.md

Create a WeightedMinHashGenerator with the default sample_size (256) and seed (1). The dimension of the vectors must be provided.

```python
from datasketch import WeightedMinHashGenerator

# Using default sample_size 256 and seed 1
wmg = WeightedMinHashGenerator(1000)
```

--------------------------------

### MinHashLSHBloom Indexing and Querying

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates how to create a MinHashLSHBloom index, insert MinHash objects, and query for approximate duplicates. The threshold is approximate due to the nature of MinHash estimation.

```python
from datasketch import MinHash, MinHashLSHBloom
import numpy as np

# Generate 100 random MinHashes.
minhashes = MinHash.bulk(np.random.randint(low=0, high=30, size=(100, 10)), num_perm=128)

# Create LSHBloom index.
lsh = MinHashLSHBloom(threshold=0.5, num_perm=128, n=100, fp=0.0001, save_dir="./index/")
for i, m in enumerate(minhashes):
    lsh.insert(i, m)

# Get the duplication result from LSHBloom.
query = minhashes[0]
is_duplicate = lsh.query(query)
print(is_duplicate)
```

--------------------------------

### Initialize and Update HyperLogLog

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates how to initialize a HyperLogLog sketch and update it with a new string value. The default hash function (SHA1) requires byte input.

```python
hll = HyperLogLog()
hll.update("new value".encode("utf-8"))
```

--------------------------------

### Initialize HyperLogLog with Custom Hash Function

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Shows how to initialize a HyperLogLog sketch using a custom hash function, such as one from the 'farmhash' library. The update method can then accept string inputs directly.

```python
import farmhash

def _hash_32(b):
    return farmhash.hash32(b)

hll = HyperLogLog(hashfunc=_hash_32)
hll.update("new value")
```

--------------------------------

### Initialize WeightedMinHashGenerator with custom parameters

Source: https://github.com/ekzhu/datasketch/blob/master/docs/weightedminhash.md

Create a WeightedMinHashGenerator specifying the number of samples (sample_size) and a random seed.

```python
wmg = WeightedMinHashGenerator(1000, sample_size=512, seed=12)
```

--------------------------------

### HyperLogLog Initialization

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes a HyperLogLog sketch with specified precision, an optional internal state, and a hash function.

```APIDOC
## class datasketch.HyperLogLog

### Description
The HyperLogLog sketch for estimating cardinality of very large datasets in a single pass.

### Parameters
* **p** (*int*) – The precision parameter. It is ignored if the reg is given.
* **reg** (*Optional* *[**numpy.ndarray* *]*) – The internal state. This argument is for initializing the HyperLogLog from an existing one.
* **hashfunc** (*Callable*) – The hash function used by this MinHash. It takes the input passed to the update method and returns an integer that can be encoded with 32 bits. The default hash function is based on SHA1 from [hashlib](https://docs.python.org/3.5/library/hashlib.html).
* **hashobj** (**deprecated**) – This argument is deprecated since version 1.4.0. It is a no-op and has been replaced by hashfunc.
```

--------------------------------

### Configure MinHash LSH with Cassandra Storage

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Initialize MinHashLSH with Cassandra as the storage backend. Configure connection details, keyspace, and replication strategy. Options to drop existing keyspace/tables are available.

```python
from datasketch import MinHashLSH

lsh = MinHashLSH(
    threshold=0.5, num_perm=128, storage_config={
        'type': 'cassandra',
        'cassandra': {
            'seeds': ['127.0.0.1'],
            'keyspace': 'lsh_test',
            'replication': {
                'class': 'SimpleStrategy',
                'replication_factor': '1',
            },
            'drop_keyspace': False,
            'drop_tables': False,
        }
    }
)
```

--------------------------------

### Initialize and Update HyperLogLog

Source: https://github.com/ekzhu/datasketch/blob/master/docs/hyperloglog.md

Initialize a HyperLogLog object and update it with data. Use this to estimate the cardinality of a dataset.

```python
from datasketch import HyperLogLog

data1 = ['hyperloglog', 'is', 'a', 'probabilistic', 'data', 'structure', 'for',
'estimating', 'the', 'cardinality', 'of', 'dataset', 'dataset', 'a']

h = HyperLogLog()
for d in data1:
  h.update(d.encode('utf8'))
print("Estimated cardinality is", h.count())

s1 = set(data1)
print("Actual cardinality is", len(s1))
```

--------------------------------

### Create and Query MinHash LSH Index

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Demonstrates how to create MinHash objects for sets, insert them into a MinHashLSH index, and query for approximate neighbors with a Jaccard similarity above a specified threshold.

```python
from datasketch import MinHash, MinHashLSH

set1 = set(['minhash', 'is', 'a', 'probabilistic', 'data', 'structure', 'for',
            'estimating', 'the', 'similarity', 'between', 'datasets'])
set2 = set(['minhash', 'is', 'a', 'probability', 'data', 'structure', 'for',
            'estimating', 'the', 'similarity', 'between', 'documents'])
set3 = set(['minhash', 'is', 'probability', 'data', 'structure', 'for',
            'estimating', 'the', 'similarity', 'between', 'documents'])

m1 = MinHash(num_perm=128)
m2 = MinHash(num_perm=128)
m3 = MinHash(num_perm=128)
for d in set1:
    m1.update(d.encode('utf8'))
for d in set2:
    m2.update(d.encode('utf8'))
for d in set3:
    m3.update(d.encode('utf8'))

# Create LSH index
lsh = MinHashLSH(threshold=0.5, num_perm=128)
lsh.insert("m2", m2)
lsh.insert("m3", m3)
result = lsh.query(m1)
print("Approximate neighbours with Jaccard similarity > 0.5", result)
```

--------------------------------

### Create MinHashLSH Index with Custom Parameters

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize a MinHashLSH index with a specified number of bands and band size, bypassing the built-in optimization. This is useful when you want to manually control the LSH parameters.

```python
lsh = MinHashLSH(num_perm=128, params=(16, 8))
print(lsh.b, lsh.r)
# 16 8
```

--------------------------------

### Run Top-K Benchmark for Orkut Dataset

Source: https://github.com/ekzhu/datasketch/blob/master/benchmark/indexes/jaccard/README.md

Execute the top-k benchmark for the Orkut dataset. This command generates a SQLite database containing the benchmark results.

```bash
python topk_benchmark.py --index-set-file orkut_ge10.inp --query-set-file orkut_ge10.inp --query-sample-ratio 0.01 --output orkut.sqlite
```

--------------------------------

### AsyncMinHashLSH Constructor

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes an Asynchronous MinHashLSH index. It supports custom storage configurations, including aiomongo, and allows for fine-tuning of LSH parameters like threshold and number of permutations.

```APIDOC
## class datasketch.aio.AsyncMinHashLSH

### Description
Asynchronous MinHashLSH index.

### Parameters
* **threshold** (*float*) – Similarity threshold for LSH.
* **num_perm** (*int*) – The number of permutations for MinHash.
* **weights** (*tuple* *(**float* *,* *float* *)*) – Weights for MinHash.
* **params** (*tuple*) – Parameters for LSH.
* **storage_config** (*dict*) – Configuration for the storage service (e.g., aiomongo).
* **prepickle** (*bool*) – If True, keys are pickled to bytes before insertion.
```

--------------------------------

### Configure MinHash LSH with Redis Storage

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Initialize MinHashLSH with Redis as the storage backend. Ensure Redis is running on the specified host and port.

```python
from datasketch import MinHashLSH

lsh = MinHashLSH(
    threshold=0.5, num_perm=128, storage_config={
        'type': 'redis',
        'redis': {'host': 'localhost', 'port': 6379},
    }
)
```

--------------------------------

### Async MinHash LSH with Usual MongoDB Storage

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Demonstrates initializing AsyncMinHashLSH with a standard MongoDB storage configuration and performing insert and query operations.

```python
from datasketch.aio import AsyncMinHashLSH
from datasketch import MinHash

_storage = {'type': 'aiomongo', 'mongo': {'host': 'localhost', 'port': 27017, 'db': 'lsh_test'}}

async def func():
    lsh = await AsyncMinHashLSH(storage_config=_storage, threshold=0.5, num_perm=16)
    m1 = MinHash(16)
    m1.update('a'.encode('utf8'))
    m2 = MinHash(16)
    m2.update('b'.encode('utf8'))
    await lsh.insert('a', m1)
    await lsh.insert('b', m2)
    print(await lsh.query(m1))
    print(await lsh.query(m2))
    lsh.close()
```

--------------------------------

### Decompress Datasets

Source: https://github.com/ekzhu/datasketch/blob/master/benchmark/indexes/jaccard/README.md

Use gzip to decompress the downloaded datasets before running benchmarks.

```bash
gzip -d *.gz
```

--------------------------------

### Async MinHash LSH with Redis Configuration

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Shows the storage configuration for initializing AsyncMinHashLSH with a Redis backend. Custom Redis arguments can be passed via the 'redis' key.

```python
_storage = {'type': 'aioredis', 'redis': {'host': '127.0.0.1', 'port': '6379'}}
```

--------------------------------

### Async MinHash LSH with MongoDB

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Demonstrates how to initialize and use AsyncMinHashLSH with a MongoDB backend for insertion and deletion sessions. Ensure MongoDB is running on localhost:27017.

```python
from datasketch.aio import AsyncMinHashLSH
from datasketch import MinHash
import asyncio

def chunk(it, size):
    it = iter(it)
    return iter(lambda: tuple(islice(it, size)), ())

_chunked_str = chunk((random.choice(string.ascii_lowercase) for _ in range(10000)), 4)
seq = frozenset(chain((''.join(s) for s in _chunked_str), ('aahhb', 'aahh', 'aahhc', 'aac', 'kld', 'bhg', 'kkd', 'yow', 'ppi', 'eer')))
objs = [MinHash(16) for _ in range(len(seq))]
for e, obj in zip(seq, objs):
    for i in e:
        obj.update(i.encode('utf-8'))
data = [(e, m) for e, m in zip(seq, objs)]

_storage = {'type': 'aiomongo', 'mongo': {'host': 'localhost', 'port': 27017, 'db': 'lsh_test'}}
async def func():
    async with AsyncMinHashLSH(storage_config=_storage, threshold=0.5, num_perm=16) as lsh:
        async with lsh.insertion_session(batch_size=1000) as session:
            fs = (session.insert(key, minhash, check_duplication=False) for key, minhash in data)
        await asyncio.gather(*fs)

        async with lsh.delete_session(batch_size=3) as session:
            fs = (session.remove(key) for key in keys_to_remove)
            await asyncio.gather(*fs)
```

--------------------------------

### Initialize HNSW

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize an HNSW index with a distance function and construction parameters.

```python
from datasketch.hnsw import HNSW
from scipy.spatial.distance import euclidean

hnsw = HNSW(euclidean, m=16, ef_construction=200, m0=32, seed=100, reversed_edges=True)
```

--------------------------------

### Initialize MinHash with default parameters

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Creates a MinHash object with default settings for number of permutations and seed. The default hash function is SHA1.

```python
from datasketch import MinHash

minhash = MinHash()
```

--------------------------------

### Create LeanMinHash from Hash Values and Seed

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize a LeanMinHash using the seed and hash values from an existing MinHash object. This method is suitable when you have the raw components of a MinHash and want to create a lean version.

```python
lean_minhash = LeanMinHash(seed=minhash.seed, hashvalues=minhash.hashvalues)
```

--------------------------------

### datasketch.HyperLogLogPlusPlus

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes a HyperLogLog++ object for cardinality estimation. It supports custom precision and hash functions.

```APIDOC
## class datasketch.HyperLogLogPlusPlus

### Description
Initializes a HyperLogLog++ object for cardinality estimation. It supports custom precision and hash functions.

### Parameters
* **p** (*int*) – The precision parameter. It is ignored if the reg is given.
* **reg** (*Optional* *[**numpy.array* *]*) – The internal state. This argument is for initializing the HyperLogLog from an existing one.
* **hashfunc** (*Callable*) – The hash function used by this MinHash. It takes the input passed to the update method and returns an integer that can be encoded with 64 bits. The default hash function is based on SHA1 from [hashlib](https://docs.python.org/3.5/library/hashlib.html).
* **hashobj** (**deprecated**) – This argument is deprecated since version 1.4.0. It is a no-op and has been replaced by hashfunc.

### Methods
#### __init__(p: int = 8, reg: ndarray | None = None, hashfunc: Callable = <function sha1_hash64>, hashobj: object | None = None)
#### count() → float
Estimate the cardinality of the data values seen so far.
```

--------------------------------

### Commit and push changes

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Commit your changes with a descriptive message and push them to your forked repository.

```bash
git commit -m "Add feature: brief description of what was changed"
git push origin your-branch-name
```

--------------------------------

### __getstate__

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Helper method for pickling the HyperLogLog sketch.

```APIDOC
### __getstate__()

#### Description
Helper for pickle.

```

--------------------------------

### Batch Insertion with Async MinHash LSH

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Demonstrates efficient batch insertion of MinHash objects into an asynchronous LSH index using an insertion session.

```python
from datasketch.aio import AsyncMinHashLSH
from datasketch import MinHash

def chunk(it, size):
    it = iter(it)
    return iter(lambda: tuple(islice(it, size)), ()))

_chunked_str = chunk((random.choice(string.ascii_lowercase) for _ in range(10000)), 4)
seq = frozenset(chain((''.join(s) for s in _chunked_str), ('aahhb', 'aahh', 'aahhc', 'aac', 'kld', 'bhg', 'kkd', 'yow', 'ppi', 'eer')))
objs = [MinHash(16) for _ in range(len(seq))]
for e, obj in zip(seq, objs):
    for i in e:
        obj.update(i.encode('utf-8'))
data = [(e, m) for e, m in zip(seq, objs)]

_storage = {'type': 'aiomongo', 'mongo': {'host': 'localhost', 'port': 27017, 'db': 'lsh_test'}}
async def func():
    async with AsyncMinHashLSH(storage_config=_storage, threshold=0.5, num_perm=16) as lsh:
        async with lsh.insertion_session(batch_size=1000) as session:
            fs = (session.insert(key, minhash, check_duplication=False) for key, minhash in data)
        await asyncio.gather(*fs)
```

--------------------------------

### MinHashLSHBloom Constructor

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes the MinHashLSHBloom index. This index is optimized for Jaccard similarity threshold and uses Bloom filters to reduce space. It can be configured with parameters like threshold, number of permutations, estimated dataset size, false positive rate, and save directory.

```APIDOC
## MinHashLSHBloom Constructor

### Description
Initializes the MinHashLSHBloom index, optimized for Jaccard similarity queries with reduced space complexity using Bloom filters. It supports configuration for threshold, number of permutations, estimated dataset size, false positive rate, and save directory.

### Parameters
* **threshold** (*float*) - The Jaccard similarity threshold between 0.0 and 1.0. The initialized LSH index will be optimized for the threshold by minimizing the false positive and false negative.
* **num_perm** (*int*) - The number of permutation functions used by the MinHash to be indexed. For weighted MinHash, this is the sample size (sample_size).
* **n** (*int | None*) - The number of elements to be inserted (estimate of dataset size).
* **fp** (*float | None*) - The false positive rate for each Bloom filter. Must be in (0,1).
* **save_dir** (*str | None*) - The directory to save the Bloom filter index to. If Bloom filters already exist in this directory, the index will be loaded from here. If None, an in-memory index will be created - this index can not be persisted.
* **weights** (*tuple[float, float]*) - Used to adjust the relative importance of minimizing false positive and false negative when optimizing for the Jaccard similarity threshold. weights is a tuple in the format of (false_positive_weight, false_negative_weight).
* **params** (*tuple[int, int] | None*) - The LSH parameters (i.e., number of bands and size of each bands). This is used to bypass the parameter optimization step in the constructor. threshold and weights will be ignored if this is given.
```

--------------------------------

### Async MinHash LSH with Context Manager and MongoDB

Source: https://github.com/ekzhu/datasketch/blob/master/docs/lsh.md

Shows how to use AsyncMinHashLSH as a context manager for automatic resource management with MongoDB storage.

```python
from datasketch.aio import AsyncMinHashLSH
from datasketch import MinHash

_storage = {'type': 'aiomongo', 'mongo': {'host': 'localhost', 'port': 27017, 'db': 'lsh_test'}}

async def func():
    async with AsyncMinHashLSH(storage_config=_storage, threshold=0.5, num_perm=16) as lsh:
        m1 = MinHash(16)
        m1.update('a'.encode('utf8'))
        m2 = MinHash(16)
        m2.update('b'.encode('utf8'))
        await lsh.insert('a', m1)
        await lsh.insert('b', m2)
        print(await lsh.query(m1))
        print(await lsh.query(m2))
```

--------------------------------

### MinHashLSH Constructor

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes a MinHashLSH index. Parameters can be optimized automatically or set manually. Supports Redis storage.

```APIDOC
## MinHashLSH.__init__

### Description
Initializes a MinHashLSH index. It can automatically optimize LSH parameters (number of bands and band size) based on the desired Jaccard threshold and number of permutation functions, or accept manually specified parameters. It also supports configuring a Redis backend for storage.

### Parameters
* **threshold** (float) - Optional - The Jaccard similarity threshold for the index. Defaults to 0.9.
* **num_perm** (int) - Optional - The number of permutation functions for the MinHash. Defaults to 128.
* **weights** (tuple[float, float]) - Optional - Weights for false positive and false negative probabilities used in optimization. Defaults to (0.5, 0.5).
* **params** (tuple[int, int] | None) - Optional - A tuple specifying the number of bands (b) and the size of each band (r). If provided, optimization is skipped.
* **storage_config** (dict | None) - Optional - Configuration for the storage backend. If type is 'redis', requires 'redis' dictionary with host and port. Can include an optional 'basename' for key prefixes.
* **prepickle** (bool | None) - Optional - Whether to pickle MinHash objects before storing.
* **hashfunc** (Callable[[bytes], bytes] | None) - Optional - A custom hash function to use.
```

--------------------------------

### Create Optimized MinHashLSH Index

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize a MinHashLSH index with optimized parameters for Jaccard threshold and number of permutations. The built-in optimizer determines the optimal number of bands and their sizes.

```python
lsh = MinHashLSH(threshold=0.9, num_perm=128)
print(lsh.b, lsh.r)
# 5 25
```

--------------------------------

### AsyncMinHashLSH.insertion_session()

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Creates an asynchronous context manager for efficient batch insertion of keys into the LSH index. This is optimized for inserting large numbers of items.

```APIDOC
## insertion_session(batch_size=10000)

### Description
Create a asynchronous context manager for fast insertion in index.

### Parameters
* **batch_size** (*int*) – The size of chunks to use in insert_session mode (default=10000).

### Returns
datasketch.aio.lsh.AsyncMinHashLSHInsertionSession
```

--------------------------------

### copy

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Creates a copy of the current HyperLogLog sketch by exporting its state.

```APIDOC
### copy() → [HyperLogLog](#datasketch.HyperLogLog)

#### Returns
A copy of the current HyperLogLog.

#### Return type
[HyperLogLog](#datasketch.HyperLogLog)
```

--------------------------------

### Bulk MinHash Computation

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Demonstrates computing multiple MinHash objects efficiently in bulk using the `bulk` class method. Each inner list in the data is hashed into one MinHash.

```python
from datasketch import MinHash

data = [[b"token1", b"token2", b"token3"], [b"token4", b"token5", b"token6"]]
minhashes = MinHash.bulk(data, num_perm=64)
```

--------------------------------

### Create LeanMinHash from MinHash

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initialize a LeanMinHash object using an existing MinHash object. This is useful for saving memory when the MinHash no longer requires updates.

```python
lean_minhash = LeanMinHash(minhash)
```

--------------------------------

### MinHash Constructor

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes a MinHash object. It can be configured with the number of permutations, a random seed, GPU mode, a custom hash function, and optionally pre-computed hash values or permutations for faster initialization.

```APIDOC
## MinHash Constructor

### Description
Initializes a MinHash object. It can be configured with the number of permutations, a random seed, GPU mode, a custom hash function, and optionally pre-computed hash values or permutations for faster initialization.

### Parameters
* **num_perm** (*int*) – Number of random permutation functions. It will be ignored if hashvalues is not None.
* **seed** (*int*) – The random seed controls the set of random permutation functions generated for this MinHash.
* **gpu_mode** (*Literal['disable', 'detect', 'always']*) – Controls GPU use in `update_batch()`. Defaults to 'disable'.
    - 'disable' — always CPU.
    - 'detect' — use GPU if available, otherwise CPU.
    - 'always' — require GPU; raise `RuntimeError` if CuPy/CUDA is unavailable.
* **hashfunc** (*Callable*) – The hash function used by this MinHash. It takes the input passed to the `update()` method and returns an integer that can be encoded with 32 bits. The default hash function is based on SHA1 from `hashlib`. Users can use farmhash for better performance.
* **hashobj** (**deprecated**) – This argument is deprecated since version 1.4.0. It is a no-op and has been replaced by hashfunc.
* **hashvalues** (*Optional* *[**ArrayLike*]*)* – The hash values is the internal state of the MinHash. It can be specified for faster initialization using the existing `hashvalues` of another MinHash.
* **permutations** (*Optional* *[**Union* *[*Tuple* *[*ArrayLike*, *ArrayLike*]*, *ArrayLike*]*]*)* – The permutation function parameters as a tuple of two lists. This argument can be specified for faster initialization using the existing `permutations` from another MinHash.

### Notes
- Hashing and permutation generation always run on CPU to preserve existing semantics; only the permutation application and the columnwise min-reduction inside `update_batch()` may run on GPU.
- To save memory usage, consider using `datasketch.LeanMinHash`.
- Since version 1.1.1, MinHash will only support serialization using `pickle`. `serialize` and `deserialize` methods are removed, and are supported in `datasketch.LeanMinHash` instead.
- Since version 1.1.3, MinHash uses Numpy’s random number generator instead of Python’s built-in random package. This change makes the hash values consistent across different Python versions.
```

--------------------------------

### Check code quality with ruff

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Use ruff to check for code style issues and automatically format the code.

```bash
# Check for issues
uvx ruff check .

# Auto-fix formatting issues
uvx ruff format .
```

--------------------------------

### index

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Indexes all sets given their keys, MinHashes, and sizes. This method can only be called once after the index is created.

```APIDOC
## index

### Description
Indexes all sets given their keys, MinHashes, and sizes. This method can only be called once after the index is created.

### Parameters
* **entries** (*Iterable[tuple[Hashable, MinHash, int]]*) – An iterable of tuples, each must be in the form of `(key, minhash, size)`, where `key` is the unique identifier of a set, `minhash` is the MinHash of the set, and `size` is the size or number of unique items in the set.

### Raises
* **ValueError** – If the index is not empty or `entries` is empty.
```

--------------------------------

### Clone datasketch repository

Source: https://github.com/ekzhu/datasketch/blob/master/docs/index.md

Clone the datasketch project repository from GitHub.

```bash
git clone https://github.com/ekzhu/datasketch.git
cd datasketch
```

--------------------------------

### __eq__

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Checks for equivalence between two HyperLogLog sketches based on their internal state.

```APIDOC
### __eq__(other: [HyperLogLog](#datasketch.HyperLogLog)) → bool

#### Parameters
* **other** ([*HyperLogLog*](#datasketch.HyperLogLog))

#### Returns
True if both have the same internal state.

#### Return type
bool
```

--------------------------------

### LeanMinHash Initialization

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Initializes a LeanMinHash object. It can be created from an existing MinHash object, or from a seed and hash values.

```APIDOC
## LeanMinHash

### Description
Initializes a LeanMinHash object. It can be created from an existing MinHash object, or from a seed and hash values. Lean MinHash is a memory-efficient version of MinHash with a frozen internal state.

### Method
__init__

### Parameters
#### Parameters:
- **minhash** (datasketch.MinHash, optional) - The MinHash object used to initialize the LeanMinHash. If not set, then seed and hashvalues must be set.
- **seed** (int | None, optional) - The random seed that controls the set of random permutation functions. Must be used together with hashvalues.
- **hashvalues** (Iterable | None, optional) - The hash values used to initialize the state of the LeanMinHash. Must be used together with seed.

### Example
```python
# From an existing MinHash
lean_minhash = LeanMinHash(minhash)

# From seed and hashvalues
lean_minhash = LeanMinHash(seed=minhash.seed, hashvalues=minhash.hashvalues)
```
```

--------------------------------

### LeanMinHash.copy

Source: https://github.com/ekzhu/datasketch/blob/master/docs/documentation.md

Returns a copy of the LeanMinHash object.

```APIDOC
## copy

### Description
Returns a copy of the LeanMinHash object. This operation preserves the gpu_mode and lazily rehydrates caches.

### Method
copy

### Returns
- LeanMinHash - A copy of the current LeanMinHash object.
```